Polynomial regression

2.2.6 · AI-ML › Linear & Logistic Regression

Core Idea

YEH KAAM KYUN KARTA HAI? Kyunki hum abhi bhi linear regression hi kar rahe hain! "Linear" in linear regression ka matlab hai parameters ${\theta }_i$ mein linear, features mein nahi. Agar hum nayi features define karein $x_1=x$, $x_2=x^2$, $x_3=x^3$, toh humara model ban jaata hai: $h_{\theta }(\mathbf{x})={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+{\theta }_3{x}_3$

Yeh $\theta$ mein linear hai! Toh hum wohi normal equation ya gradient descent use kar sakte hain jo humein pehle se pata hai.

Model ko First Principles se Derive Karna

Step 1: Feature Transformation

Training data $(x^{(i)},y^{(i)})$ se shuru karo jahan $i=1,\dots ,m$.

Original feature vector: ${\mathbf{x}}^{(i)}=[x^{(i)}]$ (sirf ek feature)

Transformed feature vector: $\varphi ({\mathbf{x}}^{(i)})=[1,x^{(i)},(x^{(i)}{)}^2,(x^{(i)}{)}^3,\dots ,(x^{(i)}{)}^d]$

Kyun? Hum original $x$ se $d$ nayi features banate hain. Isse feature engineering kehte hain. Hypothesis ban jaata hai: $h_{\theta }(x)={\theta }^T\varphi (x)$

Step 2: Linear Regression Apply Karo

Design matrix $\Phi$ banao jahan har row $\varphi ({\mathbf{x}}^{(i)})$ hai:

1 & x^{(1)} & (x^{(1)})^2 & \cdots & (x^{(1)})^d \\ 1 & x^{(2)} & (x^{(2)})^2 & \cdots & (x^{(2)})^d \\ \vdots & \vdots & \ddots & \vdots \\ 1 & x^{(m)} & (x^{(m)})^2 & \cdots & (x^{(m)})^d \end{bmatrix}$$ **YEH SHAPE KYUN?** Har row ek training example ko represent karti hai apne saare polynomial features ke saath. Hamare paas $m$ rows (examples) aur $d+1$ columns (features including intercept) hain. Cost function linear regression jaisi hi hai: $$J(\theta) = \frac{1}{2m} \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})^2$$ > [!formula] Polynomial Regression ke liye Normal Equation > $$\theta = (\Phi^T \Phi)^{-1} \Phi^T \mathbf{y}$$ > **Derivation:** Linear regression se bilkul same. Hum $J(\theta)$ minimize karte hain $\nabla_\theta J = 0$ set karke: > $$\frac{\partial J}{\partial \theta} = \frac{1}{m}\Phi^T(\Phi\theta - \mathbf{y}) = 0$$ > $$\Phi^T\Phi\theta = \Phi^T\mathbf{y}$$ > $$\theta = (\Phi^T\Phi)^{-1}\Phi^T\mathbf{y}$$ > > **YEH KAAM KYUN KARTA HAI?** Hum transformed feature space mein least-squares problem solve kar rahe hain. Algebra bilkul same hai; sirf matrix $\Phi$ badli hai. ## Worked Examples > [!example] Example 1: Quadratic Fit Karna > **Data:** $(1, 3), (2, 7), (3, 13), (4, 21)$ > **Goal:** $h_\theta(x) = \theta_0 + \theta_1 x + \theta_2 x^2$ fit karo > > **Step 1: $\Phi$ banao** > $$\Phi = \begin{bmatrix} > 1 & 1 & 1 \\ > 1 & 2 & 4 \\ > 1 & 3 & 9 \\ > 1 & 4 & 16 > \end{bmatrix}, \quad \mathbf{y} = \begin{bmatrix} 3 \\ 7 \\ 13 \\ 21 \end{bmatrix}$$ > > **YEH NUMBERS KYUN?** Har row $[1, x^{(i)}, (x^{(i)})^2]$ hai. For example, row 1: $[1, 1, 1^2] = [1, 1, 1]$; row 2: $[1, 2, 2^2] = [1, 2, 4]$. > > **Step 2: $\Phi^T \Phi$ compute karo** > $$\Phi^T\Phi = \begin{bmatrix} > 4 & 10 & 30 \\ > 10 & 30 & 100 \\ > 30 & 100 & 354 > \end{bmatrix}$$ > > **KYUN?** Matrix multiplication: element $(i,j)$ column $i$ of $\Phi^T$ aur column $j$ of $\Phi$ ka dot product hai. For example, top-left entry hai $\sum_i 1 \cdot 1 = 4$ (chaar rows). > > **Step 3: $\Phi^T \mathbf{y}$ compute karo** > $$\Phi^T\mathbf{y} = \begin{bmatrix} 44 \\ 140 \\ 484 \end{bmatrix}$$ > > **YEH NUMBERS KYUN?** Pehli entry: $\sum_i 1 \cdot y^{(i)} = 3+7+13+21 = 44$. Doosri: $\sum_i x^{(i)} y^{(i)} = 1\cdot3 + 2\cdot7 + 3\cdot13 + 4\cdot21 = 3+14+39+84 = 140$. Teesri: $\sum_i (x^{(i)})^2 y^{(i)} = 1\cdot3 + 4\cdot7 + 9\cdot13 + 16\cdot21 = 3+28+117+336 = 484$. > > **Step 4: $\theta$ ke liye solve karo** > $$\theta = (\Phi^T\Phi)^{-1}\Phi^T\mathbf{y} = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}$$ > > **Result:** $h_\theta(x) = 1 + x + x^2$ > > **Verification:** $h(1) = 1+1+1 = 3$ ✓, $h(2) = 1+2+4 = 7$ ✓, $h(3) = 1+3+9 = 13$ ✓, $h(4) = 1+4+16 = 21$ ✓. > > Saare chaar points **exactly** $y = x^2 + x + 1$ pe hain! 3 parameters aur 4 collinear-in-pattern points ke saath, least-squares fit unse perfectly guzarta hai (zero training error). Yeh ek rare exact fit hai; aam taur pe real data mein noise hota hai aur fit sirf approximate hoti hai. > [!example] Example 2: Polynomial Degree Chunna > **Data:** Temperature (°C) vs Ice Cream Sales (units): $(10, 20), (15, 40), (20, 80), (25, 120), (30, 150)$ > > **Degree 1 try karo (linear):** $h(x) = \theta_0 + \theta_1 x$ > - Underfit karta hai: Accelerating growth capture nahi kar sakta > - High training error > > **Degree 2 try karo (quadratic):** $h(x) = \theta_0 + \theta_1 x + \theta_2 x^2$ > - Accha fit: Curve capture karta hai > - Low training error, achhi tarah generalize karta hai > > **Degree 4 try karo:** $h(x) = \sum_{i=0}^{4} \theta_i x^i$ > - Overfit karta hai: 5 parameters aur 5 data points ke saath, har point se hokar wiggle karta hai > - Zero training error, lekin naye data pe bahut kharab > > **HIGH DEGREE OVERFIT KYUN KARTA HAI?** Degree 4 ke saath hamare paas 5 parameters ($\theta_0, \ldots, \theta_4$) hain aur sirf 5 data points. Model ke paas har point se exactly guzarne ki itni freedom hai—toh woh pattern seekhne ki jagah noise memorize karta hai. Yahi ==bias-variance tradeoff== hai. ## Feature Scaling Bahut Zaroori Hai > [!formula] Polynomial Features ko Scale Kyun Karo? > Agar $x \in [1, 100]$ ho, toh $x^2 \in [1, 10000]$ aur $x^3 \in [1, 1000000]$. > > **Problem:** Features ke scales bahut alag hain → gradient descent dheere converge karta hai, normal equation numerically unstable ho jaata hai. > **Solution:** Transformation ke baad ==feature scaling== apply karo: > $$x_j^{(i)} \leftarrow \frac{x_j^{(i)} - \mu_j}{\sigma_j}$$ > jahan $\mu_j$ mean hai aur $\sigma_j$ feature $j$ ka standard deviation hai. > > **KYUN?** Isse saare features ka mean 0 aur standard deviation 1 ho jaata hai, sab barabar footing pe aa jaate hain. ## Multiple Features Multiple input features $x_1, x_2, \ldots, x_n$ ke liye, hum include kar sakte hain: - **Pure powers:** $x_1^2, x_1^3, x_2^2, x_2^3, \ldots$ - **Interaction terms:** $x_1 x_2, x_1 x_2^2, x_1^2 x_2, \ldots$ Do features, degree 2 ka example: $$h_\theta(x_1, x_2) = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \theta_3 x_1^2 + \theta_4 x_2^2 + \theta_5 x_1 x_2$$ **INTERACTION TERMS KYUN?** Yeh capture karte hain ki features mil ke kaise kaam karte hain. For example, crop yield predict karne mein rainfall AUR temperature dono ka saath mein interaction depend kar sakta hai (zyada rain + zyada heat = kharab). > [!mistake] Common Error: Scale Karna Bhool Jaana > **Galat soch:** "Feature scaling optional hai, bas convergence speed up karta hai." > > **Yeh sahi kyun lagta hai:** Similar-scale features wale linear regression mein scaling itni critical nahi hoti. > > **Yeh galat kyun hai:** Polynomial features ke saath, $x^2$ aur $x^3$ magnitude mein orders se alag ho sakte hain. Scaling ke bina: > - Normal equation: $(\Phi^T\Phi)^{-1}$ ill-conditioned ho jaata hai (near-singular) > - Gradient descent: Bahut time lagta hai ya diverge karta hai > > **Fix:** Polynomial transformation ke baad, training se pehle hamesha features scale karo. > [!mistake] Common Error: Bahut Zyada Degree Use Karna > **Galat soch:** "Zyada degree = zyada flexible = better model." > > **Yeh sahi kyun lagta hai:** Higher-degree polynomials training data better fit karte hain (lower $J_{\text{train}}$). > > **Yeh galat kyun hai:** Woh overfit karte hain! Degree $d$ ke saath $d+1$ parameters hote hain. Jab $d+1$ training examples $m$ ki number ke kareeb ya zyada ho jaata hai, model ke paas har point se exactly guzarne ki—noise including—itni freedom ho jaati hai. Hamare 5-point example mein, degree 4 se 5 points ke liye 5 parameters milte hain, toh training error zero ho jaata hai lekin fit points ke beech bahut wildly wiggle karta hai. > > **Steel-man:** Haan, zyada degree bias kam karta hai (model complexity). Lekin variance (training data ke liye sensitivity) explode ho jaata hai. Test error badh jaata hai. > > **Fix:** Degree choose karne ke liye ==cross-validation== use karo. Training aur validation error ko degree ke against plot karo; woh degree chuno jahan validation error sabse kam ho. > [!recall]- 12 Saal ke Bacche ko Samjhao > Socho tum ek graph pe kuch dots ke through line kheenchne ki koshish kar rahe ho. Agar dots ek curve banate hain (jaise ek smile 😊), toh seedhi line kaam nahi karegi—woh zyaatar dots se miss ho jaayegi! > > Polynomial regression aise kehna hai: "Sirf seedhi lines kheenchne ki jagah, main curves bhi try karta hoon—jaise parabolas (U-shapes) ya aur bhi wigglier curves." > > Trick yeh hai: Hum apna data (jaise dots ki x-position) lete hain aur naye "super-features" banate hain—ise square karte hain ($x^2$), cube karte hain ($x^3$), aur aage bhi. Phir hum in nayi features pe apna purana straight-line method (linear regression) use karte hain! > > Yeh computer se kehne jaisa hai: "Yaar, main jaanta hoon tum sirf seedhi lines kheeench sakte ho, lekin kya ho agar main tumhe $x$, $x^2$, aur $x^3$ alag-alag features de doon? Phir is naye space mein tumhari 'seedhi line' original space mein curve jaisi dikhegi!" > > Lekin savdhaan raho! Agar tum curve ko BAHUT zyada wiggly bana do (bahut zyada degree), woh har ek dot se perfectly snake karegi—lekin isse "memorizing" kehte hain, "learning" nahi. Yeh bilkul practice problems ke answers memorize karne jaisa hai, concept samjhne ki jagah. Jab koi naya problem aayega (naya data), tum fail ho jaoge! > [!mnemonic] Polynomial Regression Kab Use Karna Yaad Rakhne Ke Liye > **"Curves Need Curves"** > - Agar aapka data **Curves** kare, tumhe polynomial features **Need** hain (seedhi line se kaam nahi chalega) > - Lekin yaad raho: **C**ross-validation se degree **C**huno > > **Degree selection:** 1, 2, 3 se shuru karo... "Low, Squared, Cubed" > - **1** = Low (linear) > - **2** = Squared (quadratic, parabola) > - **3** = Cubed (cubic, S-curves) > > **"Scale Before You Sail"** — Training se pehle hamesha features scale karo! ## Connections - [[2.2.01-Linear-regression-fundamentals]] — Polynomial regression same cost function aur normal equation use karta hai - [[2.2.05-Feature-scaling]] — Polynomial features ke liye critical hai kyunki magnitudes bahut alag hote hain - [[2.3.02-Overfitting-and-regularization]] — High-degree polynomials mein overfitting ka darr rehta hai; regularization help karta hai - [[2.4.01-Train-validation-test-split]] — Optimal polynomial degree select karne ke liye validation set use karo - [[3.1.03-Bias-variance-tradeoff]] — Degree choice directly bias (underfitting) vs variance (overfitting) ko affect karti hai - [[2.2.03-Gradient-descent-variants]] — Large feature sets ke liye normal equation ki jagah gradient descent use kar sakte hain #flashcards/ai-ml Polynomial regression kya hai? ::: Ek regression technique jo curved relationships fit karti hai input features ko polynomial terms (x, x², x³, ...) mein transform karke aur in transformed features pe linear regression apply karke. Polynomial regression abhi bhi "linear" regression kyun kehlaata hai? ::: Kyunki yeh parameters θ mein linear hai, features mein nahi. h(x) = θ₀ + θ₁x + θ₂x² θ mein linear hai, chahe x mein nonlinear ho. Degree d ke polynomial regression ke liye design matrix Φ kaise banate hain? ::: Har row i mein [1, x⁽ⁱ⁾, (x⁽ⁱ⁾)², ..., (x⁽ⁱ⁾)ᵈ] hota hai. Matrix mein m rows (examples) aur d+1 columns (intercept including features) hote hain. Polynomial regression ke liye normal equation kya hai? ::: θ = (ΦᵀΦ)⁻¹Φᵀy, jahan Φ polynomial features wala design matrix hai. Yeh linear regression se bilkul same hai lekin transformed features use karta hai. Polynomial regression ke liye feature scaling itni critical kyun hai? ::: Kyunki polynomial features ke magnitudes bahut alag hote hain (x² aur x³ orders of magnitude se alag ho sakte hain), jo normal equation mein numerical instability aur gradient descent mein dheere convergence karti hai. Polynomial regression mein interaction terms kya hote hain? ::: Alag features ke products jaise x₁x₂ ya x₁²x₂, jo yeh capture karte hain ki features mil ke kaise kaam karte hain, sirf inke alag-alag effects nahi. Bahut zyada polynomial degree use karne se kya hota hai? ::: Overfitting—model ke paas training examples ke relative bahut zyada parameters ho jaate hain, toh woh patterns seekhne ki jagah noise memorize karta hai. Training error drop hota hai lekin test error badh jaata hai. Optimal polynomial degree kaise chooste hain? ::: Cross-validation use karo: alag-alag degrees ke models train karo, training aur validation error ko degree ke against plot karo, aur woh degree chuno jahan validation error sabse kam ho. Degree-2 polynomial ke liye transformation function φ(x) kya hai? ::: φ(x) = [1, x, x²], ek single feature x ko teen features mein transform karta hai (intercept term including). Degree 1 ke saath polynomial regression underfit kyun karta hai? ::: Degree-1 polynomial sirf ek seedhi line hai, jo data mein curved relationships capture nahi kar sakti, jisse high bias aur high training error hota hai. ## 🖼️ Concept Map ```mermaid flowchart TD LR[Linear regression] CURVE[Curved data] PR[Polynomial regression] LINP[Linear in parameters] FT[Feature transformation phi x] DM[Design matrix Phi] COST[Cost function J theta] NE[Normal equation] THETA[Parameters theta] GD[Gradient descent] CURVE -->|motivates| PR LR -->|extended by| PR PR -->|relies on| LINP LINP -->|allows reuse of| LR PR -->|starts with| FT FT -->|builds| DM DM -->|minimizes| COST COST -->|solved by| NE COST -->|solved by| GD NE -->|yields| THETA GD -->|yields| THETA ```