Cost function (MSE) and gradient descent fitting: Question bank

This is a question bank, not a lecture. Each line below hides an answer behind :::. Read the prompt, commit to an answer out loud, then reveal. If your reason doesn't match the reason given, that gap is exactly the misconception this bank is hunting.

Before we start, three symbols we use everywhere — pinned here so nothing is unearned:

Everything else ( $h_{θ} (x_{i})$ the prediction, $e_{i} = h_{θ} (x_{i}) - y_{i}$ the residual) is built from the parent note. If a term appears you haven't met, it is in the parent topic.

True or false — justify

MSE and gradient descent always find the single global best line.

True for linear regression — the MSE surface is a bowl (convex), so the one valley floor is the global minimum; this is not true in general (see 3.1.02-Convex-optimization).

Squaring the errors is only about making them positive.

False — absolute value also removes signs, but squaring additionally makes the surface smooth/differentiable and punishes big errors far more (quadratic penalty), which is why we prefer it.

If the gradient is the zero vector, we have reached the minimum of the cost.

True for a convex bowl like linear-regression MSE, because the only flat spot is the bottom; in non-convex costs a zero gradient could be a saddle or local minimum.

Gradient descent moves in the direction of the gradient.

False — the gradient points toward steepest increase; we subtract it, so we move in the opposite (downhill) direction.

A larger learning rate always means faster convergence.

False — too large and each step overshoots the valley and the cost can grow every iteration (divergence); speed only improves up to a point.

The

\frac{1}{2}

\frac{1}{2 n}

changes which line is optimal.

False — a positive constant scaling of

J

leaves the location of the minimum unchanged; the

\frac{1}{2}

only tidies the derivative by cancelling the 2 from the power rule.

If two parameters share the same gradient value, they get the same update.

True only if

α

is the same for both (it is) and they start equal — the update is

θ_{j} - α \cdot g_{j}

, so equal

g_{j}

gives equal step, but different starting values still differ afterwards.

Batch gradient descent uses one training example per step.

False — batch uses all

n

examples to compute one averaged gradient per step; using one example per step is 4.2.01-Stochastic-gradient-descent.

Dividing the cost by

n

is essential to getting the right answer.

False for the location of the minimum —

n

is a constant, so it rescales

J

without moving the valley; it matters for making

α

behave consistently across dataset sizes.

Spot the error

"I updated

θ_{1}

using the new

θ_{0}

I just computed this step."

Wrong — updates must be simultaneous: both new values are computed from the old

θ

pair, then applied together. Using a freshly-updated

θ_{0}

corrupts the gradient for

θ_{1}

"My cost went up this iteration, so gradient descent is broken."

Not broken — a rising cost almost always means

α

is too large and you overshot; shrink

α

(or use 4.2.03-Learning-rate-schedules), the algorithm is fine.

"The gradient for

θ_{0}

\frac{1}{n} \sum (h_{θ} (x_{i}) - y_{i}) x_{i}

Wrong feature —

θ_{0}

multiplies the bias feature

x_{i}^{(0)} = 1

, so its gradient is

\frac{1}{n} \sum (h_{θ} (x_{i}) - y_{i}) \cdot 1

; the

x_{i}

belongs to the

θ_{1}

gradient.

"I'll compute residuals as

y_{i} - h_{θ} (x_{i})

; the sign won't matter after squaring."

The square is fine, but the gradient uses the raw residual, so flipping the sign flips the gradient direction — you'd then climb the hill. Keep

e_{i} = h_{θ} (x_{i}) - y_{i}

consistently.

"Features range 1–100000, so I'll just pick a tiny

α

and it's equivalent to scaling them."

Not equivalent — wildly different feature scales stretch the bowl into a long thin ravine that zig-zags slowly regardless of

α

; the fix is 2.06-Feature-scaling-and-normalization.

"MSE isn't differentiable at zero error, like absolute value is."

Backwards — the absolute-value cost has the kink at zero; the squared cost is perfectly smooth everywhere, which is the whole reason we square.

"Since the derivative of

u^{2}

2 u

, my final gradient carries a factor of 2."

The 2 is cancelled by the

\frac{1}{2}

we deliberately put in front — that cancellation is why

\frac{1}{2}

exists, so the clean gradient has no 2.

Why questions

Why average the squared errors instead of just summing them?

So the cost doesn't balloon with dataset size — dividing by

n

makes "typical error per example" the quantity, letting the same

α

work on 100 or 10000 points.

Why do we need a cost function at all instead of eyeballing the line?

To turn "best fit" into one measurable number we can systematically minimize with calculus; without it there is no gradient to follow and no objective notion of "better".

Why does subtracting the gradient guarantee we go downhill (for small enough

α

Because the gradient is the direction of steepest ascent; stepping the opposite way decreases

J

to first order, and a small step keeps the linear approximation valid.

Why is the MSE surface for linear regression a bowl and not something bumpy?

Because

J

is a sum of squares of linear functions of

θ

, which is quadratic and convex in

θ

— one valley, no false bottoms. Contrast with the 2.3.02-Logistic-cost-function where squared error is not convex, forcing a different cost.

Why can't we just set the gradient to zero and solve directly?

We can — that's the 2.2.05-Normal-equation; gradient descent is preferred when the number of features is huge and the matrix inversion becomes too expensive.

Why does the same update rule generalize to networks with millions of parameters?

Because "subtract

α

times the slope" doesn't care how many dials there are — the gradient is computed per parameter, so the recipe extends to 2.4.01-Gradient-descent-for-neural-networks.

Edge cases

What happens if you initialize

θ_{0} = θ_{1} = 0

on data that already passes through the origin perfectly?

The residuals are already zero, so the gradient is zero and no update occurs — you started at the minimum, which is a legitimate (not broken) outcome.

What if all your

y_{i}

values are identical (a flat dataset)?

The best line is horizontal:

θ_{1} \to 0

and

θ_{0} \to

that constant; the slope's gradient drives

θ_{1}

toward zero because

x_{i}

has no relationship to

y

What if

α = 0

Nothing moves — the update becomes

θ_{j} := θ_{j} - 0

, so you stay put forever regardless of how wrong the line is.

What if two data points share the same

x

but different

y

Perfectly fine — no line can hit both, so the minimum cost is nonzero; gradient descent settles at the line minimizing the average squared miss, not at zero cost.

What if the gradient is enormous because errors are huge early on?

The step

α \cdot g

can be huge too and overshoot; this is the classic reason cost explodes on iteration 1 with an unscaled dataset — either scale features or lower

α

As iterations

\to \infty

with a good

α

, what does each gradient component approach?

Zero — at the valley floor the slope in every direction flattens, so updates shrink toward nothing and the parameters stop changing (convergence).

Recall Fastest self-check

The step size formula is ::: $θ_{j} := θ_{j} - α \frac{\partial J}{\partial θ _{j}}$ , all $θ_{j}$ updated simultaneously from old values. A cost that rises every iteration means ::: $α$ is too large; you are overshooting the minimum. Zero gradient in a convex bowl means ::: you have reached the global minimum.