Intuition The One Core IdeaA machine-learning model is a guessing machine: you feed it examples, it learns a rule, and you hope that rule works on new examples it has never seen. Underfitting and overfitting are the two ways that hope can fail — the rule is either too dumb to fit anything, or so obsessed with the practice examples that it flops on new ones.
Before you can diagnose those two failures, you need a small toolbox of ideas and symbols. The parent note (2.6.02 ) throws around things like J t r ain ( θ ) E [ ⋅ ] every one of them from nothing , in an order where each idea leans on the one before it.
Everything starts with a table of examples.
Definition Example, feature, labelAn example is one row of your data — one dog photo, one audio clip, one house.
The features are the numbers describing it (pixel brightness, house size). We call the bundle of features x
The label is the correct answer we want (dog/cat, the house price). We call it y
A single example is the pair ( x , y )
We write the i x ( i ) y ( i ) ( i ) name tag — "example number i x ( 3 ) x
x y x y state what the model is trying to do. Look at figure s01: each dot is one ( x , y )
h θ ( x ) h hypothesis — a fancy word for "the model's guess." You feed it features x
The subscript θ knobs the model can turn — its adjustable numbers, called parameters .
So h θ ( x ) x θ
Picture a machine with dials on the front. Turning the dials (θ
Intuition Why a subscript?We separate x θ h θ inputs go in the parentheses, learned settings go underneath.
Worked example A concreteh θ h θ ( x ) = θ 0 + θ 1 x θ 0 θ 1
L ( y ^ , y ) Loss is a single number saying "how wrong was this one guess?" Small loss = good guess; big loss = bad guess. We write y ^ = h θ ( x ) y
A common loss for numbers is the squared error :
L ( y ^ , y ) = ( y ^ − y ) 2
squared , why not just the difference?
If we used y ^ − y + 5 − 5 look perfect. Squaring makes every error positive, so mistakes never cancel. It also punishes big misses far more than small ones (a miss of 10 100 2 4
The vertical red gap in figure s03 between a data point and the model's line is the error y ^ − y area of the square built on that gap.
One loss measures one example. But we train on many. So we average.
J J average loss over a whole set of examples . The letter J
J ( θ ) = m 1 i = 1 ∑ m L ( h θ ( x ( i ) ) , y ( i ) ) Let's decode every piece:
Definition Reading the sum
m ∑ i = 1 m i 1 m m 1 average , so 10 examples and 10,000 examples give comparable numbers.
Intuition Why average and not just total?If you only summed, more data would always give a bigger, scarier number even if each guess got better. Dividing by m
Now the parent's mysterious symbols become plain:
The entire diagnosis is just comparing these two numbers : is J t r ain J c v
Definition Squiggles that compare sizes
≈ roughly equal ("about the same").≫ much bigger than (the two little arrows point from the tiny side to the huge side).The gap is literally the subtraction J c v − J t r ain
Mnemonic Reading the signatures
Underfit: J t r ain big , gap small → "bad everywhere, evenly bad."
Overfit: J t r ain tiny , gap big → "ace at home, fails the exam."
The parent writes E [ Test Error ] = Bias 2 + Variance + Irreducible Error E
E [ ⋅ ] E [ X ] long-run average value of X E [ Test Error ] typically get, averaging over all the training sets you might have been handed.
Intuition Why bother imagining many training sets?Your one dataset is a lucky (or unlucky) draw. The concepts bias and variance only make sense across many possible draws: bias is how far off you are on average; variance is how much your answer jumps around from one draw to the next. Without E
Definition Bias and Variance in one breath
Bias = how far the model's average guess sits from the truth. High bias = model too simple = underfitting .Variance = how wildly the model's guess changes when the training data changes slightly. High variance = model too twitchy = overfitting .Irreducible error = pure noise in the data that no model can beat.
Full detail lives in 2.6.01-Bias-variance-tradeoff ; here you only need to know they are the two dials behind the two failure modes.
E weighted average , nothing more. Every time you see E [ something ]
Definition Model capacityCapacity = how flexible the model is, i.e. how wiggly a curve it can draw. A straight line has low capacity; a degree-15 polynomial or a 20-layer network has huge capacity.
Capacity is the cause of both diseases:
Too little capacity → can't bend to the pattern → underfit (high bias).Too much capacity → bends to fit every noisy dot → overfit (high variance).
More on where capacity comes from in neural nets: 3.2.01-Neural-network-capacity . Ways to tame excess capacity: 2.6.05-Regularization-methods and 4.1.02-Data-augmentation-strategies .
expectation E over many draws
compare sizes gap J-cv minus J-train
Underfit vs Overfit diagnosis
Read top to bottom: raw examples become guesses, guesses become a measured badness, badness measured on two sets gives you a gap, and the gap plus the bias/variance idea lets you diagnose. Capacity feeds in from the side as the root cause.
Test yourself — cover the right side and answer before revealing.
What does the superscript ( i ) x ( i ) "Example number i
What is h θ ( x ) The model's guess for input x θ
Why do we square the error in the loss L = ( y ^ − y ) 2 So over- and under-guesses don't cancel, and big misses are punished more.
What does the m 1 J Turns a total into a per-example average, fair across dataset sizes.
What is the difference between J t r ain J c v J t r ain J c v
Translate J t es t ≫ J t r ain Test error is much bigger than training error — the overfitting signature.
What does E [ ⋅ ] The long-run average over many repeated random training sets.
Bias vs variance in one line each? Bias = average distance from truth (underfit); variance = how much the guess jumps when data changes (overfit).
What is model capacity, and how does it cause both failures? Flexibility of the curve; too little → underfit, too much → overfit.
What is ϵ ∗ The best possible (irreducible / human-level) error — the floor no model beats.
