Feed-forward network sublayers: Question bank

Before we start, one shared picture in words. The FFN is a two-step machine applied to one token vector at a time: first stretch the vector from $d_{model}$ numbers up to $d_{f f}$ numbers (the "expansion"), bend it with a non-linear function $σ$ , then squeeze it back down to $d_{model}$ numbers (the "projection"). "Position-wise" means the same machine runs on every token separately, like the same stamp pressed onto every page.

True or false — justify

Every "answer" below gives the reason, never a bare verdict.

Removing the FFN and stacking only attention layers still gives a deep non-linear model.

False — attention combines value vectors with weights that sum to 1, i.e. it is a (data-dependent) linear average, so stacking such layers collapses toward one linear map; the FFN's

σ

is the only non-linearity, see Activation Functions.

The FFN mixes information between different tokens in the sequence.

False — it is position-wise: token

i

's output

FFN (x_{i})

depends only on

x_{i}

. Cross-token mixing already happened in attention; the FFN just refines each routed vector.

Making

d_{f f}

larger than

d_{model}

is wasteful because the output is squeezed back down anyway.

False — the wide hidden layer is where many feature detectors live; the squeeze forces the network to keep only useful combinations. A bottleneck (

d_{f f} \leq d_{model}

) would starve capacity.

Because

FFN (X) = [FFN (x_{1}), \dots, FFN (x_{n})]

, each token gets its own private weights.

False — the same

W_{1}, b_{1}, W_{2}, b_{2}

are shared across all positions; only the input data

x_{i}

differs. That sharing is what makes it "one stamp, many pages".

The FFN holds fewer parameters than the attention sublayer, so it matters less.

False — with

d_{f f} = 4 d_{model}

the FFN carries

\approx 8 d_{model}^{2}

params vs attention's

\approx 4 d_{model}^{2}

; the FFN is where most parameters (and much learning) live.

Two stacked linear layers with no activation between them are as expressive as the real FFN.

False —

x W_{1} W_{2} = x W_{eff}

is a single linear map; without

σ

the whole thing reduces to one matrix, and the Universal Approximation Theorem guarantee vanishes.

The FFN's bias vectors

b_{1}, b_{2}

can always be dropped with no effect.

False — the biases shift the pre-activation, moving where

σ

bends; dropping

b_{1}

especially can push many ReLU units permanently off (dead) or on, changing which patterns fire.

Applying LayerNorm after the residual add (

LayerNorm (x + FFN (x))

) is just cosmetic.

False — it re-centres and re-scales the summed stream so the next block sees a stable distribution; see Layer Normalization and Residual Connections. Removing it typically breaks training at depth.

Spot the error

Each line states a flawed claim; the reveal names the flaw.

"The FFN is a standard MLP over the flattened sequence, mlp(x.view(-1))."

Error: flattening mixes all positions into one giant vector. The correct view is a shared network broadcast over the sequence axis — equivalently a

1 \times 1

convolution, so mlp(x) acting per-position.

"ReLU is smooth, so its gradient is never zero."

Error: ReLU

(x) = max (0, x)

has gradient

0

for all

x < 0

(dead-neuron zone) and is not differentiable at

0

. That hard cutoff is exactly why GELU/SiLU were introduced.

"GELU

(x) = x \cdot Φ (x)

means we multiply

x

by a number bigger than 1."

Error:

Φ (x) = P (X \leq x)

is a probability in

[0, 1]

, so GELU scales

x

down, smoothly gating small negatives toward zero instead of chopping them.

"Since

d_{f f} = 4 d_{model}

, exactly

4 d_{model}

hidden units fire on every input."

Error: ReLU gives sparsity — only roughly half the

d_{f f}

units are active per input, so effective width is

\sim 2 d_{model}

, not

4 d_{model}

"FFN output can replace attention because it also transforms token vectors."

Error: the FFN never looks across positions, so it cannot route information between tokens. Attention routes; FFN extracts — different jobs, not interchangeable.

"SiLU and Swish are unrelated functions."

Error: SiLU

(x) = x σ (x) = \frac{x}{1 + e ^{- x}}

is Swish (with gate coefficient 1); they name the same self-gated activation.

"Parameter count of the FFN is

d_{model} \cdot d_{f f}

Error: there are two weight matrices (

W_{1}, W_{2}

) plus biases, giving

2 d_{model} d_{f f} + d_{f f} + d_{model}

; the single-matrix estimate is roughly half the true count.

Why questions

Why expand to

d_{f f}

first and only then apply the non-linearity, rather than bending in the original

d_{model}

space?

A wider space lets

σ

carve many independent linear pattern-detectors (

x W_{1} [:, k] + b_{1} [k]

) before compression; bending in the cramped input space would limit how many distinct features the approximation can express.

Why is the FFN placed after attention in the block, not before?

Attention first aggregates context (linear mixing) so each token vector already carries neighbour information; the FFN then does non-linear feature extraction on that enriched vector — extracting before routing would work on raw, context-poor inputs.

Why does modern practice prefer GELU or SiLU over ReLU inside the FFN?

They are smooth everywhere and self-gated, giving small non-zero gradients for negative inputs — this avoids dead neurons and empirically trains large models more stably than ReLU's hard cutoff.

Why can the FFN be run for every position fully in parallel?

Because there is no cross-position dependency;

FFN (x_{i})

needs only

x_{i}

, so all

n

tokens pass through the shared weights simultaneously with no sequential ordering required.

Why is

σ

applied only between the two linear layers, not at the very end?

The final projection must land back in the residual-stream dimension

d_{model}

as an unclamped value so it can be added cleanly to

x

; a trailing ReLU would forbid negative contributions and cripple the residual sum.

Why is the "

4 \times

" expansion an empirical rule rather than a proven optimum?

Universal approximation guarantees some sufficient width exists but not the number;

4 \times

is the width that balances capacity, ReLU sparsity (

\sim 2 \times

effective), and compute cost well in practice.

Edge cases

d_{f f} = d_{model}

(no expansion), is the FFN still non-linear?

Yes —

σ

is still present so it remains non-linear, but it becomes a capacity bottleneck: no room to spread out rich feature combinations, so representational power drops sharply.

What happens to an FFN whose pre-activation is negative for a given unit across all training inputs with ReLU?

That unit's output and gradient are permanently

0

— a dead neuron; it never updates. GELU/SiLU avoid this by leaking a small gradient for negative pre-activations.

For an all-zero input token

x = 0

, is the FFN output necessarily zero?

No — the biases survive:

h = σ (b_{1})

and output

= σ (b_{1}) W_{2} + b_{2}

, generally non-zero. Only the weight-driven term vanishes, not the bias-driven one.

If the sequence has length

1

(a single token), does the FFN behave differently from length

n

No — since processing is position-wise with shared weights, length is irrelevant to what each token computes; only the number of independent applications changes, not the function.

x \to + \infty

, how do ReLU, GELU, and SiLU compare?

All three approach the identity

σ (x) \approx x

for large positive

x

(GELU's

Φ (x) \to 1

, SiLU's

σ (x) \to 1

), so they differ mainly near and below zero, not in the far-positive tail.

Does batch size or the choice between batch- vs layer-norm change what the FFN itself computes per token?

No — the FFN is independent of batch statistics; normalization choice (Batch Normalization vs Layer Normalization) affects the surrounding stream, but the per-token map

FFN (x_{i})

is unchanged by neighbouring examples.

Is the FFN affected by the Positional Encoding scheme used earlier?

Only indirectly — the FFN sees whatever vector arrives, which already carries position info baked in upstream; the FFN has no separate notion of position and applies identical weights regardless of where the token sits.

Recall One-line self-test

"Attention routes, FFN ______, and only ______ makes the FFN non-linear." ::: FFN extracts (features); only the activation $σ$ makes it non-linear.