ResNet and skip connections: Worked examples — AI-ML deep dive

This page is the "throw everything at it" companion to the ResNet topic note. We will not introduce a single new idea without a picture and a plain-word reason. If you have never seen a "channel" or a "gradient" before, start here at line one.

What a residual block does, in one picture

Before any example, let us fix the vocabulary so no symbol is ever used cold.

Look at the figure. The orange wire is the residual path (it does work). The blue wire is the skip — it copies $x$ across untouched. The little green circle is element-wise addition: sheet-by-sheet, pixel-by-pixel, the two stacks are summed. For that green $+$ to even be legal, the two stacks arriving must be the same shape. That single requirement — "shapes must match at the $+$ " — generates almost every case below.

The scenario matrix

Every situation a residual block can face reduces to one of these cells. Our examples cover all of them.

Cell	Scenario	What is special	Example
A	Shapes already match	Identity skip, plain add	Ex 1
B	Channels grow + spatial shrinks	Need projection $W_{s}$	Ex 2
C	$F (x) = 0$ (degenerate)	Block passes input through	Ex 3
D	Gradient flow, deep stack	The "+1" highway	Ex 4
E	Vanishing-gradient limit	$\frac{\partial F}{\partial x} \to 0$	Ex 5
F	Bottleneck FLOP count	Word problem, cost	Ex 6
G	Exam twist: does depth ever hurt?	Concept trap	Ex 7
H	Numeric add on tiny tensor	Sanity of "element-wise"	Ex 8

Example 1 — Cell A: shapes already match

Worked example Basic block, identity skip

Input $x$ has shape $64 \times 56 \times 56$ . Two $3 \times 3$ convs, each with 64 filters and stride 1, produce $F (x)$ . What is the output shape, and is the skip an identity or a projection?

Forecast: guess the output shape and whether we need $W_{s}$ before reading on.

First conv keeps shape at $64 \times 56 \times 56$ . Why this step? A $3 \times 3$ conv with stride 1 and "same" padding does not change height/width; 64 filters means 64 output channels.
Second conv (64 filters, stride 1) gives $F (x)$ at $64 \times 56 \times 56$ . Why this step? Same reasoning — shape is preserved.
At the green $+$ : $x$ is $64 \times 56 \times 56$ , $F (x)$ is $64 \times 56 \times 56$ . They match, so we add directly: $y = F (x) + x$ . Why this step? Element-wise addition is only defined when every dimension agrees.
Output shape $= 64 \times 56 \times 56$ .

Verify: input shape equals output shape, so the skip can be a bare identity (no learned $W_{s}$ ). Correct — this is exactly the "identity shortcut" case.

Example 2 — Cell B: channels grow, space shrinks

Worked example Projection shortcut needed

Input $x$ is $64 \times 56 \times 56$ . The residual path downsamples with stride 2 and expands to 128 channels, so $F (x)$ is $128 \times 28 \times 28$ . How do we make the $+$ legal?

Forecast: the two arriving stacks disagree in both channels (64 vs 128) and space (56 vs 28). Can a plain identity skip work? Guess.

Compute $F (x)$ target shape: $128 \times 28 \times 28$ . Why this step? Stride 2 halves $56 \to 28$ ; the last $1 \times 1$ conv sets channels to 128.
The raw $x$ is still $64 \times 56 \times 56$ — it does not match. Why this step? The identity skip copies $x$ unchanged, so it cannot magically become $128 \times 28 \times 28$ .
Insert a projection $W_{s}$ : a $1 \times 1$ conv with 128 filters and stride 2 applied to $x$ . Why this step? Stride 2 fixes the spatial mismatch ( $56 \to 28$ ); 128 filters fix the channel mismatch ( $64 \to 128$ ).
Now $W_{s} x$ is $128 \times 28 \times 28$ , matching $F (x)$ . Add: $y = F (x) + W_{s} x$ .

Verify: $28 = ⌊(56 - 1) /2 ⌋ + 1 = 28$ ✓ and channel count $128 = 128$ ✓. Both dimensions agree, so the $+$ is legal.

Example 3 — Cell C: the degenerate block $F (x) = 0$

Worked example What if the layers learn nothing?

Suppose training pushes all residual weights so that $F (x) = 0$ for every input. What does the block output?

Forecast: does the block break, or do something useful?

Plug $F (x) = 0$ into $y = F (x) + x$ . Why this step? This is the whole point of residual design — test the "do nothing" corner.
$y = 0 + x = x$ . The block becomes the identity. Why this step? Adding zero returns the input untouched.
Contrast a plain block $y = F (x)$ : with $F (x) = 0$ it outputs all zeros — information destroyed. Why this step? Shows why residual form is safer: its "default" is harmless, a plain block's default is fatal.

Verify: the easiest map to learn (all weights $\to 0$ ) yields identity, so extra depth can never do worse than shallower depth — deeper ResNets are guaranteed at least as expressive. This is precisely why they cure the degradation problem.

Example 4 — Cell D: the gradient highway

Worked example Gradient through one block

During backprop we know $\frac{\partial L}{\partial y} = g$ (a known number/vector, the "gradient arriving from above"). The residual path has local slope $\frac{\partial F}{\partial x} = 0.3$ . Find $\frac{\partial L}{\partial x}$ .

Forecast: a plain block would give $g \cdot 0.3$ . Guess whether the skip makes it bigger or smaller.

Write $y = F (x) + x$ , so $\frac{\partial y}{\partial x} = \frac{\partial F}{\partial x} + 1$ . Why this step? The "+1" comes from $\frac{\partial x}{\partial x} = 1$ — the skip's own contribution.
Chain rule: $\frac{\partial L}{\partial x} = g \cdot (\frac{\partial F}{\partial x} + 1) = g (0.3 + 1) = 1.3 g$ . Why this step? Multiply arriving gradient by local slope.
Plain block would give $0.3 g$ — the signal shrinks each layer. Why this step? With no "+1", repeated multiplication of numbers $< 1$ vanishes.

Verify: $1.3 g > g$ , meaning even a weak residual path lets at least $g$ pass straight through. Over 100 layers, $1. 3^{?}$ never collapses to zero because the "+1" guarantees a floor of the raw $g$ .

Example 5 — Cell E: the vanishing limit

Example 6 — Cell F: word problem, FLOP cost of a bottleneck

Worked example Why bottlenecks save computation

A bottleneck block on a $256 \times 56 \times 56$ input does: $1 \times 1$ conv to 64 ch, then $3 \times 3$ conv (64 ch), then $1 \times 1$ conv back to 256 ch. Compare the $3 \times 3$ conv's multiply count against a naive $3 \times 3$ conv done directly on all 256 channels (256 out). Use FLOPs $= H \cdot W \cdot C_{in} \cdot C_{o u t} \cdot k^{2}$ .

Forecast: guess the rough ratio (2×? 10×? 16×?).

Bottleneck $3 \times 3$ : $C_{in} = 64, C_{o u t} = 64$ . FLOPs $= 56 \cdot 56 \cdot 64 \cdot 64 \cdot 9$ . Why this step? The $1 \times 1$ layers first shrank channels to 64, so the pricey $3 \times 3$ runs on far fewer channels.
Compute: $56 \cdot 56 = 3136$ ; $64 \cdot 64 = 4096$ ; $3136 \cdot 4096 \cdot 9 = 115, 605, 504$ . Why this step? Direct arithmetic of the FLOP formula.
Naive $3 \times 3$ on 256→256: $3136 \cdot (256 \cdot 256) \cdot 9 = 3136 \cdot 65536 \cdot 9 = 1, 849, 688, 064$ . Why this step? Same formula with $C_{in} = C_{o u t} = 256$ .
Ratio $= 1, 849, 688, 064/115, 605, 504 = 16$ . Why this step? $(256/64)^{2} = 4^{2} = 16$ — shrinking channels by 4 saves $4^{2}$ on the square-kernel conv.

Verify: the $3 \times 3$ conv is exactly 16× cheaper in the bottleneck design. That is why ResNet-50+ use bottlenecks.

Example 7 — Cell G: exam twist, "does depth ever hurt?"

Worked example Conceptual trap

True or false: "A 56-layer plain net trains worse than a 20-layer plain net, therefore neural nets are fundamentally limited past ~20 layers."

Forecast: decide true/false before reading.

Identify the observed fact: deeper plain net had higher training error. Why this step? Separate the empirical observation from the false conclusion attached to it.
Ask: is this overfitting? No — overfitting hurts test error while training error keeps dropping. Here training error rose. Why this step? Rules out generalization as the cause; it is an optimization failure.
ResNet-152 trains fine — 152 layers, lower error than 20. So depth itself is not the limit. Why this step? A counterexample destroys the "fundamentally limited" claim.
Conclusion: False. The problem was optimizability of plain stacks; skip connections fix it.

Verify: ResNet-152 (152 layers) beats ResNet-18 in Top-5 error (6.16% vs 10.76%). More layers helped once skips existed — the claim is false.

Example 8 — Cell H: numeric element-wise add

Recall Quick self-test

When can the skip be a bare identity? ::: When $x$ and $F (x)$ share the exact same shape (channels, height, width). What survives backprop even if the residual path's slope is 0? ::: The full arriving gradient $g$ , thanks to the "+1" from the skip. Why is a bottleneck's $3 \times 3$ conv 16× cheaper here? ::: Channels are cut 4×, and cost scales with $C_{in} \cdot C_{o u t}$ , so $4^{2} = 16$ . What does $F (x) = 0$ turn a residual block into? ::: The identity map $y = x$ .