Chi-squared, t, F distributions — definition, degrees of freedom

---

1. The Chi-squared distribution $\chi^2_k$

Deriving the mean and variance (from scratch)

WHY this works: expectation is linear, and we only need facts about a single $Z^2$.

For one $Z\sim N(0,1)$: $E[Z^2] = \operatorname{Var}(Z) + (E[Z])^2 = 1 + 0 = 1.$ Why this step? Because $\operatorname{Var}(Z)=E[Z^2]-(E[Z])^2$, rearranged.

The fourth moment of a standard normal is $E[Z^4]=3$, so $\operatorname{Var}(Z^2) = E[Z^4] - (E[Z^2])^2 = 3 - 1 = 2.$

Now sum $k$ independent copies:

\operatorname{Var}(\chi^2_k) = \sum_{i=1}^k \operatorname{Var}(Z_i^2) = 2k.$$ *Why this step?* Variance adds over independent terms; that independence is the whole point of the df count. > [!formula] Chi-squared facts > $$\chi^2_k = \sum_{i=1}^k Z_i^2,\qquad E[\chi^2_k]=k,\qquad \operatorname{Var}(\chi^2_k)=2k.$$ > **Additive:** $\chi^2_a + \chi^2_b = \chi^2_{a+b}$ if independent (just append more squared normals). > **Key statistics result:** for a sample of size $n$ from $N(\mu,\sigma^2)$, > $$\frac{(n-1)s^2}{\sigma^2}\sim \chi^2_{n-1},\quad s^2=\tfrac{1}{n-1}\sum(x_i-\bar x)^2.$$ > Df is $n-1$, not $n$, because estimating $\bar x$ imposes $\sum(x_i-\bar x)=0$ — ==one constraint, lose one df==. --- ## 2. The Student's t distribution $t_k$ > [!definition] Definition > If $Z\sim N(0,1)$ and $V\sim\chi^2_k$ are **independent**, then > $$ t_k = \frac{Z}{\sqrt{V/k}} $$ > follows a $t$ distribution with $k$ degrees of freedom. The denominator is a $\chi^2$ ==divided by its df, then square-rooted== — i.e. an estimated standard deviation. > [!intuition] WHY it has fatter tails than the normal > In $t = Z/\sqrt{V/k}$, the denominator is **random** (it's an estimate of $\sigma$, not the true $\sigma$). Sometimes the denominator is small by chance → $t$ blows up → more extreme values → **heavier tails**. As $k\to\infty$, $V/k\to 1$ (law of large numbers) so the denominator stabilises to 1 and $t_k \to Z$. ### Where it comes from in practice (derivation sketch) We want to test a mean with **unknown variance**. Start from the standardised sample mean: $$\frac{\bar x - \mu}{\sigma/\sqrt n}\sim N(0,1)=Z.$$ *Why?* $\bar x\sim N(\mu,\sigma^2/n)$, then standardise. But $\sigma$ is unknown — replace it by $s$: $$\frac{\bar x-\mu}{s/\sqrt n} = \frac{(\bar x-\mu)/(\sigma/\sqrt n)}{s/\sigma} = \frac{Z}{\sqrt{\dfrac{(n-1)s^2/\sigma^2}{n-1}}} = \frac{Z}{\sqrt{V/k}},\quad V=\tfrac{(n-1)s^2}{\sigma^2}\sim\chi^2_{n-1}.$$ *Why this step?* We divide numerator and denominator by $\sigma$ so the top becomes exactly $Z$ and the bottom becomes a $\chi^2_{n-1}$ scaled by its df — matching the definition with $k=n-1$. > [!formula] t-distribution facts > $$t_k=\frac{Z}{\sqrt{V/k}},\quad E[t_k]=0\ (k>1),\quad \operatorname{Var}(t_k)=\frac{k}{k-2}\ (k>2).$$ > Symmetric about 0, bell-shaped, fatter tails than $N(0,1)$; $t_k\to N(0,1)$ as $k\to\infty$. --- ## 3. The F distribution $F_{d_1,d_2}$ > [!definition] Definition > If $U\sim\chi^2_{d_1}$ and $V\sim\chi^2_{d_2}$ are **independent**, then > $$ F_{d_1,d_2} = \frac{U/d_1}{V/d_2} $$ > is the $F$ distribution with **numerator df $d_1$** and **denominator df $d_2$**. It is a ==ratio of two chi-squareds, each divided by its own df==. > [!intuition] WHY divide each by its df > A raw $\chi^2_k$ grows with $k$ (mean $=k$). To compare two of them *fairly* you normalise each to "per degree of freedom", so each part has mean $\approx 1$. Then $F$ measures **how many times bigger** one normalised variance estimate is than the other. If two populations have equal variance, $F\approx 1$. ### Connections between the three (derive, don't memorise) **$t^2$ is an $F$:** $$t_k^2 = \frac{Z^2}{V/k} = \frac{Z^2/1}{V/k} = F_{1,k}.$$ *Why?* $Z^2\sim\chi^2_1$, so the squared $t$ is a $\chi^2_1$ (df=1) over a $\chi^2_k/k$ — exactly $F_{1,k}$. **Reciprocal symmetry:** swapping numerator/denominator inverts $F$: $$\frac{1}{F_{d_1,d_2}} = \frac{V/d_2}{U/d_1}\sim F_{d_2,d_1}.$$ > [!formula] F-distribution facts > $$F_{d_1,d_2}=\frac{U/d_1}{V/d_2},\qquad E[F_{d_1,d_2}]=\frac{d_2}{d_2-2}\ (d_2>2).$$ > Used in ANOVA and to compare two sample variances $s_1^2/s_2^2$. ![[4.9.09-Chi-squared,-t,-F-distributions-—-definition,-degrees-of-freedom.png]] --- ## Worked examples > [!example] Example 1 — mean & variance of $\chi^2_5$ > Find $E$ and $\operatorname{Var}$ of $\chi^2_5$. > **Step 1:** $E=k=5$. *Why?* Each $Z_i^2$ contributes mean 1, summed 5 times. > **Step 2:** $\operatorname{Var}=2k=10$. *Why?* Each $Z_i^2$ has variance 2, summed over 5 independent terms. > [!example] Example 2 — sample variance to chi-squared > A sample of $n=10$ from $N(\mu,\sigma^2)$ gives $s^2 = 4$ with true $\sigma^2=3$. What is the distribution and value of the test statistic? > **Step 1:** Statistic $\frac{(n-1)s^2}{\sigma^2}=\frac{9\cdot 4}{3}=12$. > **Step 2:** Distribution is $\chi^2_{n-1}=\chi^2_9$. *Why df=9?* Estimating $\bar x$ costs one df. > [!example] Example 3 — building a $t$ statistic > $n=16$, $\bar x = 52$, $s=8$, test $\mu=50$. > **Step 1:** $t = \dfrac{\bar x-\mu}{s/\sqrt n}=\dfrac{52-50}{8/4}=1$. > **Step 2:** Reference distribution $t_{15}$. *Why 15?* $k=n-1=15$. > **Step 3:** Note we use $t$ not $Z$ because $\sigma$ is **estimated** by $s$. > [!example] Example 4 — $t^2$ as $F$ > If $t\sim t_{12}$, what is $t^2$? > **Step 1:** $t^2 = F_{1,12}$. *Why?* Numerator $Z^2$ is $\chi^2_1$ (df 1) over $\chi^2_{12}/12$. > [!example] Example 5 — F for two variances > Two samples: $s_1^2=10$ ($n_1=6$), $s_2^2=4$ ($n_2=9$), equal true variances. > **Step 1:** $F = s_1^2/s_2^2 = 10/4 = 2.5$. > **Step 2:** Distribution $F_{d_1,d_2}=F_{5,8}$. *Why?* $d_1=n_1-1=5$, $d_2=n_2-1=8$. --- > [!mistake] Common mistakes (steel-manned) > **1. Using $n$ df for sample variance.** > *Why it feels right:* there are $n$ data points, so "$n$ pieces of information." > *The fix:* one piece is used up forcing $\sum(x_i-\bar x)=0$. Free pieces $= n-1$. **Df = data − estimated parameters.** > > **2. Forgetting to divide each $\chi^2$ by its df in $F$.** > *Why it feels right:* "$F$ is just a ratio of two chi-squareds." > *The fix:* without dividing by df, the ratio's scale depends on $k$. Dividing makes each part mean $\approx 1$ so $F\approx 1$ under equal variances. > > **3. Using $Z$ instead of $t$ for small samples with unknown $\sigma$.** > *Why it feels right:* "$\bar x$ is normal, so standardise it." > *The fix:* you replaced $\sigma$ by random $s$, adding uncertainty → fatter tails → must use $t_{n-1}$. > > **4. Thinking $t$ has heavier tails forever.** > *Why it feels right:* "$t$ is always different from normal." > *The fix:* As $k\to\infty$, $s\to\sigma$, denominator → 1, and $t_k\to N(0,1)$. --- > [!recall]- Feynman: explain to a 12-year-old > Imagine you measure how far darts land from the bullseye. **Chi-squared** adds up the *squared* misses — a "total wobble" score; more darts ($k$) means a bigger expected score. **t** is like guessing your aim's accuracy when you *don't know* your own steadiness — you estimate it from a few throws, so you're less sure, and crazy outcomes happen more often (fat tails). **F** compares two players' wobbliness as a ratio: if it's near 1, they're equally shaky. "Degrees of freedom" = how many throws were *truly free* to vary after you used some up to compute your average. > [!mnemonic] Remember the builders > **"Square, Scale, Split."** > - $\chi^2$ = **Square** and add normals. > - $t$ = a normal **Scaled** by an estimated SD ($\sqrt{\chi^2/k}$). > - $F$ = **Split** one chi-squared-per-df by another. > And **"df = data − constraints."** --- ## Connections - [[Standard Normal Distribution]] — the atom all three are built from. - [[Sample Variance and Bessel's Correction]] — source of the $n-1$ df. - [[Hypothesis Testing]] — t-tests, chi-squared goodness-of-fit, F-tests. - [[ANOVA]] — F arises as ratio of between/within mean squares. - [[Central Limit Theorem]] — why $\bar x$ is normal, enabling $t$. - [[Gamma Distribution]] — $\chi^2_k$ is $\text{Gamma}(k/2,\,1/2)$. --- ## #flashcards/maths How is $\chi^2_k$ defined from normals? ::: As $\sum_{i=1}^k Z_i^2$ where $Z_i$ are independent $N(0,1)$. Mean and variance of $\chi^2_k$? ::: Mean $=k$, variance $=2k$. Why is $E[\chi^2_k]=k$? ::: Because $E[Z^2]=1$ for each of the $k$ independent terms. What distribution does $(n-1)s^2/\sigma^2$ follow? ::: $\chi^2_{n-1}$. Why $n-1$ df for sample variance? ::: Estimating $\bar x$ imposes $\sum(x_i-\bar x)=0$, costing one df. Definition of $t_k$? ::: $Z/\sqrt{V/k}$ with $Z\sim N(0,1)$, $V\sim\chi^2_k$, independent. Why does $t$ have fatter tails than $Z$? ::: Its denominator is a random estimate of $\sigma$; small estimates inflate $t$. What does $t_k$ converge to as $k\to\infty$? ::: The standard normal $N(0,1)$. Variance of $t_k$? ::: $k/(k-2)$ for $k>2$. Definition of $F_{d_1,d_2}$? ::: $(U/d_1)/(V/d_2)$ with $U\sim\chi^2_{d_1}$, $V\sim\chi^2_{d_2}$, independent. What is $t_k^2$ distributed as? ::: $F_{1,k}$. Reciprocal property of $F$? ::: $1/F_{d_1,d_2}\sim F_{d_2,d_1}$. Why divide each $\chi^2$ by its df in $F$? ::: To normalise each to mean $\approx 1$ so $F\approx 1$ under equal variances. Mean of $F_{d_1,d_2}$? ::: $d_2/(d_2-2)$ for $d_2>2$. General rule for degrees of freedom? ::: df = (number of data points) − (number of estimated parameters/constraints). ## 🖼️ Concept Map ```mermaid flowchart TD Z[Standard normal Z ~ N 0,1] CHI[Chi-squared with k df] T[Student t with k df] F[F distribution] DF[Degrees of freedom] CON[Constraint sum of deviations = 0] S2[Sample variance s squared] MV[Mean k, Variance 2k] Z -->|sum of k squared| CHI CHI -->|counts free directions| DF CON -->|removes one df| DF Z -->|numerator| T CHI -->|scaled denominator| T T -->|estimated variance| FT[Fatter tails than normal] CHI -->|ratio of two per df| F CHI -->|gives| MV S2 -->|n-1 s2 over sigma2| CHI CON -->|from estimating mean| S2 ``` ## 🔊 Hinglish (regional understanding) > [!intuition]- Hinglish mein samjho > Dekho, ye teen distributions — chi-squared, t, aur F — sab ek hi cheez se bante hain: **standard normal $Z$**. Inko "sampling distributions" kehte hain kyunki ye batate hain ki sample se nikale gaye *statistics* (jaise sample variance ya sample mean) kaise behave karte hain. Chi-squared $\chi^2_k$ ka matlab hai: $k$ independent normals ko square karke jod do — ye ek "total wobble" ya total squared deviation measure karta hai. Iska mean $k$ hota hai aur variance $2k$, kyunki har $Z^2$ ka mean 1 aur variance 2 hai. > > **Degrees of freedom (df)** ka concept simple hai: kitne pieces of information *azaad* hain vary karne ke liye. Jab tum $\bar x$ estimate karte ho, to ek constraint lag jata hai ($\sum(x_i-\bar x)=0$), isliye ek df chala jaata hai — tabhi sample variance ka df $n-1$ hota hai, $n$ nahi. Yaad rakho: **df = data points − estimated parameters.** > > **t distribution** tab aata hai jab population ka $\sigma$ unknown ho aur usko $s$ se estimate karna pade. $t = Z/\sqrt{V/k}$ — denominator random hai (estimated SD), isliye kabhi-kabhi chhota ho jata hai aur $t$ bada ho jata hai → **fatter tails**. Jaise-jaise $k$ bada hota hai, $s\to\sigma$ ho jata hai aur $t$ wapas normal jaisa ban jaata hai. **F distribution** do variances ko compare karta hai: $F=(U/d_1)/(V/d_2)$, dono $\chi^2$ ko apne df se divide karte hain taaki fair comparison ho. Agar dono variances equal hain to $F\approx 1$. Yeh sab ANOVA aur hypothesis testing mein kaam aata hai — isliye inhe achhe se samajhna zaroori hai. ![[audio/4.9.09-Chi-squared,-t,-F-distributions-—-definition,-degrees-of-freedom.mp3]]