4.9.17
Hamare paas data x 1 , … , x n vishwas hai kisi distribution se aaya hai jisme ek unknown parameter θ μ λ Estimation poochta hai: data dekhke hum kaunsa ek best guess θ ^
MLE : woh θ observed data ko sabse zyada probable banata ho. "Main sabse zyada kis duniya mein rehta hoon?"Method of Moments (MoM) : theoretical moments (mean, variance...) ko sample moments ke barabar force karo, phir solve karo. "Averages match karo."
Definition Estimator vs estimateEk statistic data ka koi bhi function hota hai. Ek estimator θ ^ = T ( X 1 , … , X n ) θ random variable hai (random sample par depend karta hai). Actual data se jo ek number milta hai woh estimate hai.
Intuition YEH kyon kaam karta haiPopulation ka k μ k = E [ X k ] θ known formula hai. Sample ka k m k = n 1 ∑ x i k m k → μ k inhe equal set karein aur θ
Worked example MoM for Exponential,f ( x ) = λ e − λ x λ k = 1
Step 1 : theoretical mean E [ X ] = ∫ 0 ∞ x λ e − λ x d x = 1/ λ Yeh step kyon? Ek parameter ke saath yahi ek cheez match kar sakte hain.Step 2 : 1/ λ = x ˉ Kyon? Population moment ko sample moment ke barabar karo.Step 3 : solve karo λ ^ = 1/ x ˉ
N ( μ , σ 2 )
μ 1 = μ μ ^ = x ˉ μ 2 = E [ X 2 ] = σ 2 + μ 2 σ 2 + μ 2 = n 1 ∑ x i 2 Solve karo: σ ^ 2 = n 1 ∑ x i 2 − x ˉ 2 = n 1 ∑ ( x i − x ˉ ) 2 Yeh simplification kyon? Algebraic identity x 2 − x ˉ 2
Definition Likelihood functionJoint density (ya pmf) ko θ fixed rakhte hue:
L ( θ ) = ∏ i = 1 n f ( x i ; θ ) . MLE θ ^ L ( θ )
Intuition Product kyon maximize karein, aur log kyon leinIndependent data → joint probability ek product hai. Hum woh θ log strictly increasing hai, isliye arg max L = arg max ℓ log-likelihood
ℓ ( θ ) = ∑ i = 1 n log f ( x i ; θ )
Worked example MLE for Exponential — full derivationf ( x ; λ ) = λ e − λ x
Step 1 Likelihood: L = ∏ λ e − λ x i = λ n e − λ ∑ x i Kyon? Densities ka product (independence).Step 2 Log: ℓ = n ln λ − λ ∑ x i Log kyon? Power aur exp ko sum mein convert karta hai.Step 3 Differentiate karke 0 ℓ ′ = λ n − ∑ x i = 0 Kyon? Maximum ka slope zero hota hai.Step 4 Solve karo: λ ^ = ∑ x i n = x ˉ 1 Step 5 Max confirm karo: ℓ ′′ = − n / λ 2 < 0
Worked example MLE for Bernoulli(p 0/1 p x ( 1 − p ) 1 − x ∑ x i = s
L = p s ( 1 − p ) n − s Kyon? Saare trials par pmf multiply karo.ℓ = s ln p + ( n − s ) ln ( 1 − p ) ℓ ′ = p s − 1 − p n − s = 0 s ( 1 − p ) = ( n − s ) p p ^ = n s = x ˉ Yeh important kyon hai? "Obvious" sample proportion provably optimal hai.
N ( μ , σ 2 ) ℓ = − 2 n ln ( 2 π σ 2 ) − 2 σ 2 1 ∑ ( x i − μ ) 2
∂ ℓ / ∂ μ = σ 2 1 ∑ ( x i − μ ) = 0 ⇒ μ ^ = x ˉ ∂ ℓ / ∂ σ 2 = − 2 σ 2 n + 2 σ 4 1 ∑ ( x i − μ ) 2 = 0 ⇒ σ ^ 2 = n 1 ∑ ( x i − x ˉ ) 2 lekin note karo ki σ ^ 2 (n n − 1
Definition Bias, consistency, efficiency
Bias : b ( θ ^ ) = E [ θ ^ ] − θ Unbiased ⟺ b = 0 Consistent : θ ^ P θ n → ∞ MSE = E [( θ ^ − θ ) 2 ] = Var ( θ ^ ) + b 2
Intuition Forecast-then-verifyForecast : kaunsa divisor σ ^ 2 n Verify : E [ n 1 ∑ ( x i − x ˉ ) 2 ] = n n − 1 σ 2 = σ 2 n kam estimate karta hai; n − 1
MoM : trivially easy, starting guess ke liye badhiya; kabhi kabhi silly out-of-range values de sakta hai aur usually kam efficient hota hai.MLE : algebra mushkil hai lekin asymptotically efficient hai (bade n g ( θ ) = g ( θ ^ ) ℓ
Common mistake Steel-manned errors
"Likelihood θ Sahi lagta hai kyunki L Fix : L θ fixed data ke liye; yeh θ "L Sahi lagta hai , yeh actual target hai. Fix : products underflow karte hain aur differentiate karne mein mushkil hote hain; ℓ = log L "MLE hamesha unbiased hota hai." Sahi lagta hai kyunki yeh "optimal" hai. Fix : σ ^ M L E 2 n unbiasedness nahi ."Second-derivative / boundary check bhool jaana." Score-equation ka root minimum ho sakta hai ya boundary par ho sakta hai (jaise uniform U ( 0 , θ ) θ ^ = max x i differentiation se nahi ).
MLE kya maximize karta hai? Likelihood L ( θ ) = ∏ f ( x i ; θ ) θ
Likelihood ka log kyon lein? log strictly increasing hai isliye maximizer preserve hota hai, aur product ko ek aisi sum mein badal deta hai jise differentiate karna aasaan hai.
Method of moments recipe? Population moments μ k = E [ X k ] n 1 ∑ x i k k = 1.. p p
Exponential rate λ λ ^ = 1/ x ˉ
Bernoulli p p ^ = x ˉ =
Kya normal ke liye σ 2 Nahi; yeh n E [ σ ^ 2 ] = n n − 1 σ 2 n − 1
MSE decomposition? MSE = Var ( θ ^ ) + bias 2
Differentiate karna MLE dhundhne mein kab fail karta hai? Boundary/support-dependent parameters par, jaise U ( 0 , θ ) θ ^ = max x i
MLE ki invariance property? Agar θ ^ θ g ( θ ^ ) g ( θ )
Consistent estimator define karo.
Recall Feynman: 12-saal ke bacche ko explain karoSocho tumhe keeche mein panje mile. Tumhe nahi pata kaunse jaanwar ne banaye, lekin tum poochte ho: kaunsa jaanwar exactly yeh panje chodne ki sabse zyada probability rakhta hai? Woh jaanwar tumhara best guess hai. Yahi MLE hai — woh cause chuno jo tumne jo dekha usse sabse achha explain kare. Method of moments aur bhi simple hai: agar tum jaante ho ki average mein kutte 20kg ke hote hain, aur tumhare mystery kuton ka average bhi 20kg hai, toh tum maan lete ho ki yeh normal kutte hain — averages match karo aur solve karo. Dono sirf clever tarike hain "jo data dekha" ko "hidden number ka best guess" mein badalne ke liye.
Law of Large Numbers — justify karta hai kyon sample moments converge hote hain (MoM ka basis).Probability Distributions — exponential, Bernoulli, normal densities yahan use hue.Bias and Variance — MSE decomposition.Cramér-Rao Lower Bound — "efficient" MLE asymptotically kya achieve karta hai.Logarithms — kyon log-likelihood legal hai.Bayesian Estimation — MLE, MAP estimate ka flat-prior limit hai.
strictly increasing keeps
Data x1..xn from unknown theta
Random variable / statistic
Sample moment mk = population moment muk
Likelihood L = product f xi theta
Log-likelihood ell = sum log f
Same argmax easier algebra
Exp lambda-hat = 1 / xbar
Normal mu-hat=xbar sigma2-hat=var
