Foundations — ResNet and skip connections

This page assumes nothing. Every letter, arrow, and word the parent note (3.4.09 ResNet and skip connections (Hinglish) is the Hinglish twin) throws at you is built here from the ground up, in an order where each idea rests on the one before it.

1. What is $x$? — a picture of a "tensor"

Look at the figure: each pastel sheet is one channel, a $56\times 56$ grid of numbers. The whole stack is one $x$. When the parent note writes "$x$ is $64\times 56\times 56$", it means exactly this book of 64 pages.

Why the topic needs it: everything ResNet does — convolutions, the skip wire, the addition — operates on these blocks. If you can't picture $x$ as a stack of grids, the shape-matching later ($128\times 28\times 28$ vs $64\times 56\times 56$) will feel like magic. It isn't.

2. What is a function $H(x)$, and what is $F(x)$?

In the figure, $x$ sits at a starting dot. The desired answer $H(x)$ sits at another dot. The straight arrow $x\to H(x)$ is the whole journey. The short red arrow is $F(x)$ — only the gap between where you already are and where you want to be.

Why the topic needs it: the entire ResNet idea is the equation $H(x)=F(x)+x$. Rearranged, that is exactly $F(x)=H(x)-x$. Without the word "residual" meaning "the gap", the name Residual Network is empty.

3. The skip connection — the "$+x$" wire

Trace the figure: $x$ enters and splits. One copy goes down through the processing layers (that path produces $F(x)$). The other copy takes the curved bypass over the top, untouched. At the merge circle they are summed to give $y$. The bypass is the skip connection.

Element-wise addition only works if the two blocks are the same shape. That single requirement is the reason the parent note spends a whole example on "dimension matching" — hold that thought.

4. Symbols inside $F(x)$ — weights, ReLU, and the notation $\{W_i\}$

The parent writes $F(x)=W_2\cdot \text{ReLU}(W_1\cdot x+b_1)+b_2$. Let's earn every piece.

The figure shows ReLU's shape: flat at zero for negative inputs, then a straight $45^{\circ }$ line for positives — a bent hinge.

5. Loss $\mathcal{L}$, the derivative $\frac{\partial }{\partial }$, and the chain rule

The gradient section of the parent is scary only because of unfamiliar symbols. Here they are, one at a time.

Now the "$+1$" miracle reads plainly. With $y=F(x)+x$: $\frac{\partial y}{\partial x}=\frac{\partial F(x)}{\partial x}+\underset{=1}{\underbrace{\frac{\partial x}{\partial x}}}$ The skip wire contributes a clean $+1$ that the shrinking product can never kill. Even if $\frac{\partial F}{\partial x}\to 0$, the sensitivity keeps its $+1$ and the gradient still reaches the early layers.

6. Two shape-changing symbols: stride and the $1\times 1$ convolution

Prerequisite map

Equipment checklist

Cover the right side and answer aloud. If any stumps you, re-read its section above.