ResNet and skip connections: Visual walkthrough

We are going to follow one number as it travels forward through a stack of layers and then travels backward as a correction signal. Everything else — identity, residual, gradient highway — falls out of watching that one number.

Step 1 — What is a "layer", really?

WHAT. We draw the box as an arrow-fed cylinder. A value goes in on the left, comes out changed on the right.

WHY. Before we can talk about "adding $x$ back", we must be crystal clear that a plain network is only the right-hand output $F (x)$ — the original $x$ is thrown away the instant the box finishes. That thrown-away $x$ is the whole story.

PICTURE. In the figure the orange arrow is the input $x$ ; the magenta arrow leaving the box is $F (x)$ . Notice there is no copy of the orange arrow surviving past the box.

Step 2 — What we secretly want the box to compute

WHAT. We split the target into two possibilities and draw them side by side:

Sometimes the perfect answer is very different from the input.
Very often — especially deep in a network where features are already good — the perfect answer is almost the input itself: $H (x) \approx x$ .

WHY. Ask the real question: which is easier to learn, the whole answer or just the change? If the best thing a layer can do is "leave the numbers nearly alone", then forcing it to rebuild $x$ exactly through multiplications and ReLU bends is wasteful and fragile. This is the seed of the whole idea.

PICTURE. The left panel shows $H (x)$ far from $x$ (a big magenta jump). The right panel shows $H (x)$ sitting almost on top of $x$ (a tiny nudge). The tiny nudge is what we want to make easy.

Step 3 — Give $F$ its job: learn only the change

WHAT. Rearrange that one equation by adding $x$ to both sides: $what we want H (x) = the change the box learns F (x) + the input, kept alive x$

WHY. Now the box no longer has to invent the whole answer. It only has to describe how much to change $x$ . If no change is needed, the box can output $F (x) = 0$ — and $0$ is the easiest thing on earth for weights to produce (just push them toward zero). Compare: making a stack of nonlinear layers reproduce $x$ exactly is genuinely hard.

PICTURE. The orange input arrow now survives around the box (the curved shortcut). The box only produces the short magenta "change" arrow. The two arrows meet at a ⊕ and sum to the tall answer arrow.

Step 4 — The residual block, fully assembled

Inside the box there are, in order, two Conv layers (the weight multiplications), each followed by a BatchNorm step, with a ReLU bend between them.

WHAT. So the full inner function $F$ , written as one formula, is a Conv (weights $W_{1}$ , bias $b_{1}$ ), then a ReLU bend, then a second Conv (weights $W_{2}$ , bias $b_{2}$ ): $F (x, {W_{i}}) = W_{2} \cdot ReLU (W_{1} x + b_{1}) + b_{2}$

$W_{1}, W_{2}$ ::: weight matrices of the two Conv layers (the tunable numbers, collectively ${W_{i}}$ )
$b_{1}, b_{2}$ ::: biases — constant offsets added inside each Conv layer
$ReLU$ ::: the bend that keeps positives, zeroes negatives — the source of nonlinearity

The BN steps are folded silently into the two Convs here (BN is a fixed rescale, so it merges into the surrounding $W$ and $b$ ); the diagram spells them out as separate boxes so the picture matches a real ResNet, while the formula keeps only the parts that carry weights.

Then we add the skip and only afterwards apply the final ReLU: $y = ReLU (F (x, {W_{i}}) + x)$

WHY. The final ReLU is applied after the addition so the identity path $x$ passes through the ⊕ completely unmodified — if we bent it first, the "keep the input alive" guarantee would break.

PICTURE. Follow the two roads: the long residual road (Conv → BN → ReLU → Conv → BN, matching the formula above) and the short skip road (a straight wire). They rejoin at ⊕, then one last ReLU.

Step 5 — Edge case: what if the two roads don't line up?

WHAT. We put a $1 \times 1$ conv (with stride 2 and 128 filters, say) on the shortcut so it produces exactly the shape $F (x)$ produces.

WHY. Without matching shapes the ⊕ is undefined — you cannot add a $64 \times 56 \times 56$ block to a $128 \times 28 \times 28$ block. The projection is the minimum fix: a $1 \times 1$ conv is the cheapest possible reshaper, so the skip road stays almost free.

PICTURE. The clash is drawn as two mismatched grids (orange 56×56 vs magenta 28×28). The $W_{s}$ box on the skip road resizes the orange grid down so both grids match before ⊕.

Step 6 — Now run it backward: why the "+x" is a gradient highway

WHAT — first look at the sum before the final ReLU. Call the pre-activation sum $z = F (x) + x$ (the value that enters the last ReLU). Differentiate $z$ with respect to the input $x$ using the chain rule: $\frac{\partial z}{\partial x} = slope of the residual road \frac{\partial F ( x )}{\partial x} + = 1, the skip road \frac{\partial x}{\partial x} = \frac{\partial F ( x )}{\partial x} + 1$

$z$ ::: the sum $F (x) + x$ , the number fed into the final ReLU
$\frac{\partial z}{\partial x}$ ::: how that sum moves when the input moves
the $+ 1$ ::: comes from the skip road and can never disappear

WHY this tool. We use the derivative because a derivative answers exactly one question: "if I wiggle the input a little, how much does the output wiggle?" That wiggle-ratio is precisely what training multiplies down the chain. No other tool measures sensitivity.

WHAT — now include the final ReLU honestly. The true block output is $y = ReLU (z)$ . So the full slope carries one extra factor, the ReLU's own slope $ReLU^{'} (z)$ , which is $1$ where $z > 0$ and $0$ where $z < 0$ : $\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} \cdot ReLU^{'} (z) \cdot [\frac{\partial F ( x )}{\partial x} + 1]$

$ReLU^{'} (z)$ ::: the gate — $1$ if this unit is active ( $z > 0$ ), $0$ if it is switched off ( $z < 0$ )
the bracket ::: still carries the protective $+ 1$ whenever the gate is open

WHY this matters (the honest highway). The $+ 1$ highway is real, but it is gated by the final ReLU: on units that are switched off ( $z < 0$ ) the whole thing is multiplied by $0$ and nothing flows through that unit. So why does ResNet still work? Because ReLU passes roughly half its units on average, and — crucially — the gate is a per-unit on/off switch, not a shrinking factor. An open gate multiplies by exactly $1$ (no shrinkage), whereas a plain deep network multiplies every unit by a fractional slope at every layer. Across the units that stay active, the bracket delivers its full $+ 1$ with no attenuation, so the signal reaches early layers intact. (In the popular pre-activation ResNet the ReLU is moved off the main sum entirely, making the highway perfectly ungated — which is exactly why it trains even deeper networks.)

PICTURE. Two backward-pointing arrows. The magenta one crawls back through $F$ , getting shorter (multiplied by a small slope). The orange one shoots straight back along the skip with a fixed strength of $1$ — untouched, as long as the final ReLU gate is open.

Step 7 — Stacking many blocks: the highway spans the whole network

WHAT. Chain the brackets from Step 6 across all blocks. The signal from the deepest output $x_{L}$ back to the earliest output $x_{1}$ is the product of every block's bracket: $\frac{\partial L}{\partial x _{1}} = \frac{\partial L}{\partial x _{L}} \prod_{i = 2}^{L} [\frac{\partial F _{i}}{\partial x _{i - 1}} + 1]$

$\frac{\partial L}{\partial x _{1}}$ ::: signal reaching the earliest block's output
$\frac{\partial L}{\partial x _{L}}$ ::: signal at the deepest block's output
$\prod_{i = 2}^{L}$ ::: multiply the brackets of blocks $2$ through $L$ together
each bracket ::: still carries its own protective $+ 1$

WHY. Expand the product and one term is $1 \times 1 \times \dots \times 1 = 1$ : a direct path from the last block to the first that skips every nonlinearity. Even across 100 blocks, that path never shrinks. Contrast the plain network, whose product has no $+ 1$ terms — every factor can be $< 1$ , so the whole product collapses.

PICTURE. Left: the plain-network product of small slopes fading to black by the bottom. Right: the ResNet product with a bright orange $1 \cdot 1 \dots 1$ spine that reaches all the way down at full brightness.

The one-picture summary

Everything above — forward split, residual, skip, backward $+ 1$ , stacked highway — compressed into a single diagram.

Recall Feynman retelling — say it back in plain words

A plain layer throws your input away and must rebuild the whole answer from scratch. That is hard, and worse, the correction signal that teaches it has to squeeze back through every layer, shrinking each time until the early layers learn nothing.

ResNet does one small, brilliant thing: it keeps a copy of the input and carries it around the layers on a straight wire, then adds it back. Now the layers only have to learn the change — and if no change is best, they just output zero, which is easy.

The magic shows up when we teach the network backward. Because the sum is "change + input", its slope is "(slope of the change) + 1". That $+ 1$ is a wire the correction signal rides with full strength. There is one honest caveat: the final ReLU gates each unit — where a unit is switched off, its signal is blocked — but where it is open, the gate multiplies by exactly $1$ , never a shrinking fraction. So across the active units the highway carries the signal to layer one at full brightness. That single $+ 1$ is the whole trick.

Recall Quick self-check

Why is learning $F (x) = 0$ easier than learning the identity $H (x) = x$ ? ::: Producing zero just means pushing weights toward zero; reproducing $x$ exactly through nonlinear ReLU layers is a precise, fragile target. In the backward pass, what is $\frac{\partial z}{\partial x}$ for the sum $z = F (x) + x$ ? ::: $\frac{\partial F ( x )}{\partial x} + 1$ — the $+ 1$ is the skip road. What extra factor does the final ReLU add to the block's gradient? ::: $ReLU^{'} (z)$ , which is $1$ where $z > 0$ and $0$ where $z < 0$ — a per-unit open/shut gate. When do you need the projection $W_{s}$ ? ::: Only when $F (x)$ and $x$ have different shapes (channel count or spatial size changed). In Step 7, what does $x_{i}$ denote? ::: The output of block $i$ (the input handed to block $i + 1$ ).

Parent: this page expands the derivation in the topic note.