ResNet and skip connections: Question bank — AI-ML deep dive

This is a rapid-fire self-test for ResNet and skip connections. Every line below is a Question ::: Answer reveal — cover the right side, commit to an answer, then check. The goal is to hunt down the misconceptions the topic invites, not to grind numbers.

True or false — justify

Deeper plain networks fail because they overfit the training data.

False. The failure shows up on the training set itself — the deep net can't even fit the data it sees. This is the degradation problem (an optimization failure), not overfitting (a generalization failure).

A ResNet has strictly more representational capacity than the equivalent plain network.

False — capacity is essentially the same. The skip connection adds no new weights; it only re-parameterizes what the layers must learn, making the identity mapping easy to reach.

If every residual block learned

F (x) = 0

, the whole ResNet would collapse and output garbage.

False. If all

F (x) = 0

then

y = x

everywhere, so the network becomes one big identity function — it passes the input straight through unchanged, which is exactly the safe fallback that makes deep ResNets trainable.

The skip connection introduces extra parameters that must be trained.

False for the identity shortcut — it is a plain copy of

x

with zero parameters. Only the projection shortcut

W_{s} x

(used when dimensions mismatch) adds a small

1 \times 1

conv worth of weights.

Skip connections eliminate the vanishing-gradient problem completely.

False — "reduce," not "eliminate." The

+ 1

term in

\frac{\partial y}{\partial x} = \frac{\partial F}{\partial x} + 1

gives gradients a direct path, so they can't vanish to zero, but the

F

-path can still shrink; the point is the sum never collapses.

ResNet-50 has more parameters than ResNet-34.

True — 25.6M vs 21.8M. Despite using cheaper bottleneck blocks per layer, ResNet-50 is deeper and wider overall, so total parameters go up.

The final ReLU in a basic block is applied before adding the skip connection.

False. The order is Conv → BN → ReLU → Conv → BN → Add → ReLU. ReLU comes after the addition, so the identity

x

can carry negative values through the sum before being clipped.

Batch Normalization sits inside the residual path

F (x)

, not on the skip connection.

True. BN normalizes the learned transformation; the skip path stays a clean, unmodified copy of

x

so it can act as a gradient highway.

Spot the error

"The residual block learns

H (x)

directly, which is why it's easier to train."

Error: it learns the residual

F (x) = H (x) - x

, then adds

x

back. Learning "how much to change

x

" is easier than re-deriving the whole target

H (x)

from scratch.

"When channels change from 64 to 128, we can still add

x

F (x)

element-wise directly."

Error: shapes must match for element-wise addition. You need a projection shortcut

W_{s} x

1 \times 1

conv) to reshape

x

128

channels first.

"The gradient formula is

\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} \cdot \frac{\partial F}{\partial x}

Error: it drops the identity term. The correct result is

\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} (\frac{\partial F}{\partial x} + 1)

— that missing

+ 1

is the entire reason gradients don't vanish.

"A bottleneck block uses three

3 \times 3

convolutions to save computation."

Error: it uses

1 \times 1 \to 3 \times 3 \to 1 \times 1

. The

1 \times 1

convs squeeze channels down before and expand after the expensive

3 \times 3

, which is what saves computation.

"With a stride-2 downsampling block, only

F (x)

needs stride 2; the shortcut can stay stride 1."

Error: the shortcut also needs stride 2 (and the channel change), or its spatial size won't match

F (x)

and the addition fails.

"Since ReLU already fixes gradients, ResNet's skip connections are unnecessary in modern nets."

Error: ReLU helps per-layer gradient magnitude, but very deep plain nets still suffer optimization difficulty and degradation. Skip connections address the depth-composition problem ReLU alone cannot.

H (x) = F (x) + x

means the network can only represent functions close to the identity."

Error:

F (x)

is a full nonlinear stack and can be arbitrarily large, so

H (x)

can be anything. The formulation merely makes the identity easy to reach, it doesn't restrict the reachable set.

Why questions

Why is pushing

F (x) \to 0

easier than learning an exact identity map through nonlinear layers?

Driving weights toward zero is a simple, well-behaved optimization target, whereas forcing a stack of nonlinear convs to reproduce their input exactly requires them to precisely cancel their own transformations.

Why does the "

+ 1

" in the gradient act as a highway across 100 layers?

At each block the gradient is multiplied by

(\frac{\partial F}{\partial x} + 1)

; the

+ 1

guarantees a term that survives even if all the

\frac{\partial F}{\partial x}

factors shrink, so the product doesn't decay exponentially to zero.

Why place the addition before the final ReLU rather than after?

Adding first lets the skip connection

x

contribute negative and positive values to the sum; a ReLU before the add would clip

F (x)

and break the clean identity gradient path.

Why do deeper models like ResNet-152 still use bottleneck blocks instead of basic blocks?

Bottlenecks make each block far cheaper (channel squeeze via

1 \times 1

), so you can afford far more blocks within a similar compute/parameter budget — depth becomes affordable.

Why does the degradation problem count as evidence against simply stacking more layers?

A deeper plain net should at worst copy the shallower net and set extra layers to identity, yet it does worse — proving the optimizer can't even find that trivial solution, which is what skip connections hand it for free.

Edge cases

What happens in a residual block when the input is all zeros,

x = 0

Then

y = F (0) + 0 = F (0)

, so the block reduces to a plain layer stack with no skip contribution — the identity term simply vanishes because there's nothing to carry forward.

What is the output if

F (x) = - x

exactly?

y = F (x) + x = - x + x = 0

before the final ReLU, so the block can also cancel its input entirely — the residual formulation doesn't force the output toward

x

, it just makes

x

a cheap default.

In a downsampling block, what if you use identity (not projection) shortcut despite a shape mismatch?

The element-wise add is undefined — spatial sizes and channel counts differ — so the forward pass errors out. A projection

W_{s} x

(or zero-padding + subsampling) is mandatory here.

For a block where the optimal mapping is genuinely complex and far from identity, is ResNet still fine?

Yes —

F (x)

is a full nonlinear function, so it can learn a large, complex residual. The skip connection never restricts expressiveness; it only changes the starting point of optimization.

If we removed every nonlinearity so

F

were purely linear, would the block still help gradients?

The

+ 1

gradient term would still exist, so gradient flow is preserved, but the block would then represent only an affine map of

x

— you'd lose the representational power, not the gradient highway.

Recall One-line summary to lock in

The single equation to remember ::: $y = F (x) + x$ makes identity the cheap default and creates the $+ 1$ gradient path that lets very deep networks train.