Intuition The one-line ideaConditional expectation answers: "Given that I already know some information, what is my best guess for the average of X X X
Plain expectation E [ X ] E[X] E [ X ] before you know anything. Conditional expectation E [ X ∣ Y ] E[X \mid Y] E [ X ∣ Y ] updated guess after learning Y Y Y — and crucially, since you don't know Y Y Y random variable .
Definition Conditional expectation given an eventFor a discrete random variable X X X B B B P ( B ) > 0 P(B)>0 P ( B ) > 0 E [ X ∣ B ] = ∑ x x P ( X = x ∣ B ) = ∑ x x P ( X = x , B ) P ( B ) E[X \mid B] = \sum_x x\, P(X = x \mid B) = \sum_x x\,\frac{P(X=x,\,B)}{P(B)} E [ X ∣ B ] = ∑ x x P ( X = x ∣ B ) = ∑ x x P ( B ) P ( X = x , B ) conditional probability distribution of X X X B B B
Definition Conditional expectation given a variableE [ X ∣ Y = y ] E[X \mid Y = y] E [ X ∣ Y = y ] number (one for each value y y y E [ X ∣ Y = y ] = ∑ x x P ( X = x ∣ Y = y ) ( discrete ) E[X \mid Y=y] = \sum_x x\, P(X=x \mid Y=y) \quad (\text{discrete}) E [ X ∣ Y = y ] = ∑ x x P ( X = x ∣ Y = y ) ( discrete ) E [ X ∣ Y = y ] = ∫ − ∞ ∞ x f X ∣ Y ( x ∣ y ) d x ( continuous ) E[X \mid Y=y] = \int_{-\infty}^{\infty} x\, f_{X\mid Y}(x\mid y)\,dx \quad (\text{continuous}) E [ X ∣ Y = y ] = ∫ − ∞ ∞ x f X ∣ Y ( x ∣ y ) d x ( continuous ) g ( y ) = E [ X ∣ Y = y ] g(y) = E[X \mid Y=y] g ( y ) = E [ X ∣ Y = y ] random input Y Y Y E [ X ∣ Y ] = g ( Y ) E[X \mid Y] = g(Y) E [ X ∣ Y ] = g ( Y )
Why two flavours? E [ X ∣ B ] E[X\mid B] E [ X ∣ B ] fixed event → a number. E [ X ∣ Y ] E[X\mid Y] E [ X ∣ Y ] all the answers "for every possible y y y Y Y Y
This is THE key theorem. Let's build it from scratch.
Derivation from first principles (discrete):
Start with the definition of E [ X ∣ Y = y ] E[X\mid Y=y] E [ X ∣ Y = y ] Y Y Y E [ E [ X ∣ Y ] ] = ∑ y E [ X ∣ Y = y ] P ( Y = y ) E\big[E[X\mid Y]\big] = \sum_y E[X\mid Y=y]\,P(Y=y) E [ E [ X ∣ Y ] ] = ∑ y E [ X ∣ Y = y ] P ( Y = y )
Why this step? E [ X ∣ Y ] E[X\mid Y] E [ X ∣ Y ] Y Y Y g ( Y ) g(Y) g ( Y ) ∑ y g ( y ) P ( Y = y ) \sum_y g(y)P(Y=y) ∑ y g ( y ) P ( Y = y )
Substitute the definition E [ X ∣ Y = y ] = ∑ x x P ( X = x ∣ Y = y ) E[X\mid Y=y]=\sum_x x\,P(X=x\mid Y=y) E [ X ∣ Y = y ] = ∑ x x P ( X = x ∣ Y = y ) = ∑ y ( ∑ x x P ( X = x ∣ Y = y ) ) P ( Y = y ) = \sum_y \Big(\sum_x x\,P(X=x\mid Y=y)\Big)P(Y=y) = ∑ y ( ∑ x x P ( X = x ∣ Y = y ) ) P ( Y = y )
Why? Just plugging in. Now use P ( X = x ∣ Y = y ) P ( Y = y ) = P ( X = x , Y = y ) P(X=x\mid Y=y)P(Y=y) = P(X=x, Y=y) P ( X = x ∣ Y = y ) P ( Y = y ) = P ( X = x , Y = y ) = ∑ y ∑ x x P ( X = x , Y = y ) = ∑ x x ∑ y P ( X = x , Y = y ) ⏟ P ( X = x ) = \sum_y\sum_x x\, P(X=x, Y=y) = \sum_x x \underbrace{\sum_y P(X=x,Y=y)}_{P(X=x)} = ∑ y ∑ x x P ( X = x , Y = y ) = ∑ x x P ( X = x ) y ∑ P ( X = x , Y = y )
Why the inner sum collapses? Summing the joint over all y y y marginalises y y y P ( X = x ) P(X=x) P ( X = x ) = ∑ x x P ( X = x ) = E [ X ] ■ = \sum_x x\, P(X=x) = E[X] \qquad \blacksquare = ∑ x x P ( X = x ) = E [ X ] ■
Intuition Why the Tower Rule must be trueImagine averaging exam scores. First average within each class (E [ X ∣ Y ] E[X\mid Y] E [ X ∣ Y ] Y Y Y weighted by class size (P ( Y = y ) P(Y=y) P ( Y = y ) every student at once . Splitting then recombining can't change the overall mean.
Worked example 1. Roll two dice; expected total given the first is evenLet X X X Y Y Y E [ X ∣ Y even ] E[X\mid Y\text{ even}] E [ X ∣ Y even ] E [ X ∣ Y = y ] = y + E [ second die ] = y + 3.5 E[X\mid Y=y] = y + E[\text{second die}] = y + 3.5 E [ X ∣ Y = y ] = y + E [ second die ] = y + 3.5 Why? Second die is independent, so conditioning on Y Y Y Y Y Y Y ∈ { 2 , 4 , 6 } Y\in\{2,4,6\} Y ∈ { 2 , 4 , 6 } 1 / 3 1/3 1/3 y = 4 y = 4 y = 4 E [ X ∣ Y even ] = 4 + 3.5 = 7.5 E[X\mid Y\text{ even}] = 4 + 3.5 = 7.5 E [ X ∣ Y even ] = 4 + 3.5 = 7.5
Worked example 2. Random number of summands (Wald-style)Flip a fair coin; if heads, X = X= X = X = X= X = Y Y Y E [ X ∣ Y = H ] = 3.5 E[X\mid Y=H] = 3.5 E [ X ∣ Y = H ] = 3.5 E [ X ∣ Y = T ] = 7 E[X\mid Y=T] = 7 E [ X ∣ Y = T ] = 7 Tower: E [ X ] = 1 2 ( 3.5 ) + 1 2 ( 7 ) = 5.25 E[X] = \tfrac12(3.5) + \tfrac12(7) = 5.25 E [ X ] = 2 1 ( 3.5 ) + 2 1 ( 7 ) = 5.25 Why this step? We never compute X X X
Worked example 3. Continuous:Y ∼ U ( 0 , 1 ) Y\sim U(0,1) Y ∼ U ( 0 , 1 ) X ∣ Y = y ∼ U ( 0 , y ) X\mid Y=y \sim U(0,y) X ∣ Y = y ∼ U ( 0 , y ) E [ X ∣ Y = y ] = 0 + y 2 = y 2 E[X\mid Y=y] = \dfrac{0+y}{2} = \dfrac{y}{2} E [ X ∣ Y = y ] = 2 0 + y = 2 y [ 0 , y ] [0,y] [ 0 , y ] E [ X ∣ Y ] = Y / 2 E[X\mid Y] = Y/2 E [ X ∣ Y ] = Y /2 Tower: E [ X ] = E [ Y / 2 ] = 1 2 E [ Y ] = 1 2 ⋅ 1 2 = 1 4 E[X] = E[Y/2] = \tfrac12 E[Y] = \tfrac12\cdot\tfrac12 = \tfrac14 E [ X ] = E [ Y /2 ] = 2 1 E [ Y ] = 2 1 ⋅ 2 1 = 4 1 Why? We replace the hard direct integral over the joint density with two easy steps.
E [ X ∣ Y ] E[X\mid Y] E [ X ∣ Y ] Why it feels right: E [ X ∣ Y = y ] E[X\mid Y=y] E [ X ∣ Y = y ] is a number, so people drop the distinction.
Fix: E [ X ∣ Y = y ] E[X\mid Y=y] E [ X ∣ Y = y ] y y y E [ X ∣ Y ] E[X\mid Y] E [ X ∣ Y ] Y Y Y random variable — that's exactly why E [ E [ X ∣ Y ] ] E[\,E[X\mid Y]\,] E [ E [ X ∣ Y ] ]
E [ X Y ∣ Y ] = Y E [ X ] E[XY\mid Y] = Y\,E[X] E [ X Y ∣ Y ] = Y E [ X ] Why it feels right: you remember "pull out known stuff" and "independence gives E [ X ] E[X] E [ X ] Fix: Pulling out Y Y Y E [ X Y ∣ Y ] = Y E [ X ∣ Y ] E[XY\mid Y]=Y\,E[X\mid Y] E [ X Y ∣ Y ] = Y E [ X ∣ Y ] E [ X ∣ Y ] E[X\mid Y] E [ X ∣ Y ] E [ X ] E[X] E [ X ] independence . Don't fuse the two rules.
Common mistake Forgetting to weight byP ( Y = y ) P(Y=y) P ( Y = y ) Why it feels right: averaging the conditional means looks symmetric.
Fix: E [ X ] = ∑ y E [ X ∣ Y = y ] P ( Y = y ) E[X]=\sum_y E[X\mid Y=y]P(Y=y) E [ X ] = ∑ y E [ X ∣ Y = y ] P ( Y = y )
Recall Feynman: explain to a 12-year-oldYou want to guess how tall a random kid in school is. Your first guess is the school average — that's E [ X ] E[X] E [ X ] Y Y Y 8th-grade average — that's E [ X ∣ Y ] E[X\mid Y] E [ X ∣ Y ]
"Average the averages, weighted by the crowds." — the Tower Rule.
And: E inside, E outside, the middle disappears → E [ E [ X ∣ Y ] ] = E [ X ] E[E[X\mid Y]] = E[X] E [ E [ X ∣ Y ]] = E [ X ]
What kind of object is E [ X ∣ Y ] E[X\mid Y] E [ X ∣ Y ] Y Y Y A random variable — a function
g ( Y ) g(Y) g ( Y ) of the random
Y Y Y .
What kind of object is E [ X ∣ Y = y ] E[X\mid Y=y] E [ X ∣ Y = y ] A number, one for each fixed value
y y y .
State the Tower / Law of Iterated Expectation. E [ E [ X ∣ Y ] ] = E [ X ] E[\,E[X\mid Y]\,]=E[X] E [ E [ X ∣ Y ] ] = E [ X ] .
First step in deriving the tower (discrete)? E [ E [ X ∣ Y ] ] = ∑ y E [ X ∣ Y = y ] P ( Y = y ) E[E[X\mid Y]]=\sum_y E[X\mid Y=y]\,P(Y=y) E [ E [ X ∣ Y ]] = ∑ y E [ X ∣ Y = y ] P ( Y = y ) .
Why does the inner sum ∑ y P ( X = x , Y = y ) \sum_y P(X=x,Y=y) ∑ y P ( X = x , Y = y ) Marginalising over
y y y gives
P ( X = x ) P(X=x) P ( X = x ) .
"Take out what is known" rule? E [ g ( Y ) X ∣ Y ] = g ( Y ) E [ X ∣ Y ] E[g(Y)X\mid Y]=g(Y)E[X\mid Y] E [ g ( Y ) X ∣ Y ] = g ( Y ) E [ X ∣ Y ] .
If X ⊥ Y X\perp Y X ⊥ Y E [ X ∣ Y ] E[X\mid Y] E [ X ∣ Y ] The constant
E [ X ] E[X] E [ X ] .
Y ∼ U ( 0 , 1 ) Y\sim U(0,1) Y ∼ U ( 0 , 1 ) X ∣ Y = y ∼ U ( 0 , y ) X\mid Y=y\sim U(0,y) X ∣ Y = y ∼ U ( 0 , y ) E [ X ] E[X] E [ X ] E [ X ∣ Y ] = Y / 2 E[X\mid Y]=Y/2 E [ X ∣ Y ] = Y /2 , so
E [ X ] = E [ Y / 2 ] = 1 / 4 E[X]=E[Y/2]=1/4 E [ X ] = E [ Y /2 ] = 1/4 .
Coin: H→one die, T→sum of two dice. Find E [ X ] E[X] E [ X ] 1 2 ( 3.5 ) + 1 2 ( 7 ) = 5.25 \tfrac12(3.5)+\tfrac12(7)=5.25 2 1 ( 3.5 ) + 2 1 ( 7 ) = 5.25 .
Common slip when applying tower across groups? Forgetting to weight conditional means by
P ( Y = y ) P(Y=y) P ( Y = y ) .
feed random Y into g of Y
E X given Y is a random variable
Tower Rule E of E X given Y equals E X
Intuition Hinglish mein samjhoConditional expectation ka matlab simple hai: pehle aapke paas X X X E [ X ] E[X] E [ X ] Y Y Y E [ X ∣ Y = y ] E[X\mid Y=y] E [ X ∣ Y = y ] y y y Y Y Y E [ X ∣ Y ) E[X\mid Y) E [ X ∣ Y ) random variable ban jaata hai. Yahi sabse important point hai jo students miss karte hain.
Sabse powerful tool hai Tower Rule : E [ E [ X ∣ Y ] ] = E [ X ] E[\,E[X\mid Y]\,]=E[X] E [ E [ X ∣ Y ] ] = E [ X ]
Practical fायda yeh hai ki kai baar X X X E [ X ] E[X] E [ X ] Y Y Y 1 2 ( 3.5 ) + 1 2 ( 7 ) = 5.25 \tfrac12(3.5)+\tfrac12(7)=5.25 2 1 ( 3.5 ) + 2 1 ( 7 ) = 5.25
Do galtiyaan avoid karo: (1) "known cheez bahar nikaalna" hamesha valid hai — E [ g ( Y ) X ∣ Y ] = g ( Y ) E [ X ∣ Y ] E[g(Y)X\mid Y]=g(Y)E[X\mid Y] E [ g ( Y ) X ∣ Y ] = g ( Y ) E [ X ∣ Y ] E [ X ] E[X] E [ X ] X X X Y Y Y P ( Y = y ) P(Y=y) P ( Y = y )
