ResNet and skip connections: Exercises — AI-ML deep dive

These problems climb a ladder: first you just recognize the pieces of a residual block, then you apply the forward and backward rules, then you analyze why they work, then you synthesize whole architectures, and finally you reach mastery — the subtle edge cases most people trip over. Every problem has a full solution hidden inside a collapsible callout, so cover the answer, try it yourself, then reveal.

New to the whole idea? Start from the parent note: ResNet and skip connections.

Level 1 — Recognition

Goal: name the parts. No arithmetic yet.

Recall Solution 1.1

$x$ — the input to the block, and also the thing carried untouched along the skip connection (the highway).
$F (x)$ — the residual mapping: the small correction learned by the stacked conv layers (the detour).
$+ x$ — the identity shortcut addition: it adds the untouched input back so the layers only have to learn the difference $F (x) = H (x) - x$ , not the whole target $H (x)$ .

Recall Solution 1.2

The addition happens before the final ReLU: ... → BN → Add → ReLU. Why: we want the nonlinearity to act on the combined signal $F (x) + x$ , not on $F (x)$ alone. If ReLU came first it would clip the residual to be $\geq 0$ , so the block could never produce a negative correction — it could only ever push values up, never down. Adding first, then ReLU, keeps the correction free to go either way.

Recall Solution 1.3

Basic Block (two $3 \times 3$ convs): ResNet-18, ResNet-34.
Bottleneck Block ( $1 \times 1 \to 3 \times 3 \to 1 \times 1$ ): ResNet-50, ResNet-101, ResNet-152. Rule of thumb: 50 and deeper use bottlenecks, because the $1 \times 1$ "squeeze then expand" trick keeps the $3 \times 3$ conv cheap at high depth.

Level 2 — Application

Goal: push numbers and tensors through the rules.

Recall Solution 2.1

A $3 \times 3$ conv with stride 1 and padding 1 preserves spatial size, and "64 filters" fixes 64 output channels. So after each conv the shape stays $64 \times 56 \times 56$ . Thus $F (x)$ is $64 \times 56 \times 56$ , identical to $x$ , so they add element-wise and $y$ is $64 \times 56 \times 56$ .

Recall Solution 2.2

Inner pre-activation: $W_{1} x + b_{1} = 2 (3) - 4 = 2$ . ReLU: $ReLU (2) = 2$ (positive, passes through). Detour: $F (x) = 5 (2) + 1 = 11$ . Output: $y = 11 + 3 = 14$ .

Recall Solution 2.3

Inner: $0 \cdot 3 + 0 = 0$ ; ReLU $(0) = 0$ ; detour $F (x) = 0 \cdot 0 + 0 = 0$ . So $y = 0 + 3 = 3 = x$ . The block collapsed to the identity map just by driving weights to zero — this is exactly why residual learning is easy: "do nothing" costs zero weights, whereas a plain layer must learn the identity from scratch.

Recall Solution 2.4

The addition needs both tensors to match exactly. So $W_{s}$ is a $1 \times 1$ conv with 128 filters (to reach 128 channels) and stride 2 (to halve $56 \to 28$ spatially). Result of $W_{s} x$ : $128 \times 28 \times 28$ , matching $F (x)$ . ✓

Level 3 — Analysis

Goal: explain the mechanism, especially the backward pass.

Recall Solution 3.1

Chain rule: $\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} \cdot \frac{\partial y}{\partial x}$ . Since $y = F (x) + x$ : $\frac{\partial y}{\partial x} = \frac{\partial F}{\partial x} + I$ . So $\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} highway + \frac{\partial L}{\partial y} \cdot \frac{\partial F}{\partial x} .$ The bare $\frac{\partial L}{\partial y}$ term (from the " $+ 1$ " / $+ I$ ) is the highway: even if the detour's Jacobian $\partial F / \partial x \to 0$ , this term survives, so early layers still receive a gradient of size $\geq \partial L / \partial y$ . That is what kills vanishing gradients.

Recall Solution 3.2

Plain: $0. 5^{10} = 1/1024 \approx 9.77 \times 1 0^{- 4}$ — the signal has nearly vanished. Residual: $1. 5^{10} \approx 57.67$ — the signal is amplified, easily flowing back. Ratio residual/plain $= 3^{10} = 59049$ . The highway makes the deep gradient ~ $59, 000 \times$ larger here.

Recall Solution 3.3

Overfitting means the model fits the training data too well and only hurts test accuracy — but here the deeper net is worse on the training set itself, so it isn't even memorizing well. That means the extra layers can't be optimized to at least copy the shallower net (which they could do by learning identity mappings), so the failure is in optimization / trainability, not generalization — precisely the degradation problem residuals fix.

Level 4 — Synthesis

Goal: assemble and cost whole structures.

Recall Solution 4.1

One conv: $3 \times 3 \times 64 \times 64 = 9 \times 4096 = 36864$ . Two such convs: $2 \times 36864 = 73728$ weights (plus BN scale/shift, ignored here).

Recall Solution 4.2

$1 \times 1$ reduce: $1 \cdot 256 \cdot 64 = 16384$ .
$3 \times 3$ middle: $9 \cdot 64 \cdot 64 = 36864$ .
$1 \times 1$ expand: $1 \cdot 64 \cdot 256 = 16384$ .
Bottleneck total: $16384 + 36864 + 16384 = 69632$ .
Naive $3 \times 3 (256 \to 256)$ : $9 \cdot 256 \cdot 256 = 589824$ . The bottleneck is $589824/69632 \approx 8.47 \times$ cheaper for the same in/out channels. That is why deep ResNets use it.

Recall Solution 4.3

Conv layers inside blocks: $2 \times (3 + 4 + 6 + 3) = 2 \times 16 = 32$ . Add the stem conv ( $+ 1$ ) and the final FC layer ( $+ 1$ ): $32 + 1 + 1 = 34$ . That is where "ResNet-34" gets its number — counted layers with weights. (Poolings and additions carry no weights, so they aren't counted.)

Level 5 — Mastery

Goal: the subtle edge cases and limits.

Recall Solution 5.1

With $F (x) = 0$ : $y = 0 + x = x$ exactly — identity, with no dependence on getting weights precisely right, since zero weights already give $F = 0$ . A plain block computes $y = F (x)$ with no $+ x$ ; to be the identity it must satisfy $F (x) = x$ for every $x$ , which for nonlinear (ReLU) layers is a specific, hard function to learn — not achievable by simply zeroing weights (zeroing gives $y = 0$ , not $y = x$ ). Hence extra plain layers can hurt, extra residual layers can at worst do nothing.

Recall Solution 5.2

ReLU $(- 3) = 0$ , so the detour is dead: $F (x) = W_{2} \cdot 0 + b_{2} = 0$ . Then $y = 0 + 5 = 5 = x$ . Gradient: $\partial y / \partial x = \partial F / \partial x + 1$ . With ReLU input negative, $\partial F / \partial x = 0$ , so $\partial y / \partial x = 0 + 1 = 1$ . Punchline: even when the entire detour is dead (a scenario that would zero-out a plain block's gradient), the highway still delivers gradient $= 1$ . This is the vanishing-gradient cure in a single number.

Recall Solution 5.3

With every $F_{k} \approx 0$ , each block is $\approx$ identity, so the composition of $N$ blocks is $\approx$ identity: the network passes the input straight through at init, regardless of $N$ . This is ideal because:

A very deep net begins in a well-behaved state (no exploding/vanishing forward signal),
Optimization then only has to carve small nonzero $F_{k}$ where useful — a gentle departure from identity — instead of taming a wildly transformed signal. So depth becomes "safe": adding blocks starts as a no-op and can only help.

Recall Solution 5.4

$\frac{\partial y}{\partial x} = \frac{\partial F}{\partial x} + W_{s}^{⊤}$ (the $1 \times 1$ conv's Jacobian replaces the identity $I$ ). So the highway is no longer the pristine " $+ 1$ "; it becomes " $+ W_{s}$ ". The gradient floor is now set by $W_{s}$ instead of exactly $1$ — still a non-nonlinear, single-layer shortcut (no ReLU in the way), so it's far healthier than a plain deep chain, but it is not a perfect identity highway. That's why identity shortcuts (no projection) are preferred whenever shapes already match, and projections are used only where they must be.

Recall Self-test cloze recap

The block computes $y = F (x) + x$ , where the residual $F (x) = H (x) - x$ is only the correction ::: correct, because pushing $F$ to zero yields the identity for free. The backward highway term that survives even when $\partial F / \partial x \to 0$ is ::: the $+ 1$ (identity) term, giving $\partial L / \partial x \supseteq \partial L / \partial y$ . Bottleneck order is ::: $1 \times 1$ reduce → $3 \times 3$ → $1 \times 1$ expand. Projection shortcut is used when ::: spatial size or channel count of $F (x)$ differs from $x$ , via a $1 \times 1$ conv $W_{s}$ .