DenseNet and EfficientNet: Question bank — AI-ML deep dive

These are the ideas people get wrong, not the arithmetic. If a claim mentions channel counts or FLOPs, we care about why the number behaves that way, not the multiplication itself. Each item is a one-line reveal: read the prompt, answer in your head, then check.

Before you start, one vocabulary reminder so no symbol appears unexplained:

A feature map is one 2D grid of numbers a filter produces — one "channel" of the layer's output.
Concatenation stacks feature maps side by side along the channel axis (more channels, same values kept). Addition overlays them numerically (same channel count, values summed).
$k$ (growth rate) = how many new feature maps each DenseNet layer adds.
$ϕ$ (compound coefficient) = the single knob EfficientNet turns to grow depth, width, and resolution together.

Related material: Hinglish version · 3.4.7-ResNet-and-Skip-Connections · 3.4.9-MobileNet-and-Depthwise-Separable-Convolutions · 3.4.11-Neural-Architecture-Search.

True or false — justify

DenseNet layers add their output to previous features like ResNet does

False — DenseNet concatenates, so every prior feature map stays intact and separately addressable; ResNet's addition merges identities into one tensor.

A DenseNet with growth rate

k = 32

means every layer outputs exactly 32 channels

True — each layer's output is always

k = 32

new maps; what grows is the input it sees, since inputs are all previous outputs concatenated.

Because every layer connects to every other, DenseNet has more parameters than a plain CNN of the same depth

False — the tiny growth rate means each layer adds very few filters, so DenseNet is usually more parameter-efficient despite the dense connectivity.

The transition layer exists mainly to add non-linearity between blocks

False — its job is spatial downsampling (pooling) and channel compression (the

1 \times 1

conv with factor

θ

) to stop the channel count from exploding block after block.

In EfficientNet, doubling the compound coefficient

ϕ

doubles the FLOPs

False — each unit increase of

ϕ

roughly doubles FLOPs, so FLOPs grow as

2^{ϕ}

; going from

ϕ

2 ϕ

multiplies FLOPs by about

2^{ϕ}

, not by 2.

Width scaling costs more FLOPs per unit than depth scaling

True — depth scales FLOPs linearly (

\propto d

) but width scales them quadratically (

\propto w^{2}

) because channels multiply on both input and output sides of a convolution.

Compound scaling requires you to hand-tune

α, β, γ

for every model B0–B7

False — you search for

α, β, γ

once on the small B0 baseline, then reuse them; only

ϕ

changes to produce B1 through B7.

The MBConv block first shrinks channels, then does the depthwise convolution

False — MBConv is an inverted bottleneck: it first expands channels (by factor

t \approx 6

), does the cheap depthwise conv in that wide space, then projects back down.

Spot the error

"DenseNet block output has

k_{0} \cdot L

channels for

L

layers."

The error is multiplication — it should be

k_{0} + L \cdot k

. You add

k

new maps per layer to the starting

k_{0}

; you never scale the input by the layer count.

"We use a

1 \times 1

convolution in the bottleneck to increase spatial resolution before the

3 \times 3

conv."

1 \times 1

conv touches only the channel axis — it changes channel count, never spatial size. It's there to reduce channels so the

3 \times 3

conv is cheaper.

"EfficientNet scales resolution because bigger images always improve accuracy."

Resolution helps only when depth and width grow with it — a high-res image needs deeper/wider layers to actually use the extra detail. That balance is the whole point of the constraint

α β^{2} γ^{2} \approx 2

"DenseNet's dense connections make it immune to vanishing gradients entirely."

They alleviate the problem by giving gradients short direct paths to early layers, but no architecture is fully immune — very deep or poorly initialized nets can still struggle.

"Compression factor

θ = 0.5

means half the layers are removed at each transition."

θ

compresses channels, not layers. With

θ = 0.5

a block's

m

output channels become

⌊ 0.5 m ⌋

; the layer structure is untouched.

"Since width scales as

w^{2}

in FLOPs, the constraint should be

α^{2} \cdot β \cdot γ \approx 2

The exponents are on the wrong symbols. Depth is linear, width and resolution are both quadratic, giving

α^{1} \cdot β^{2} \cdot γ^{2} \approx 2

"Bottleneck layers always save parameters, no matter the input channel count."

They save parameters only when the input channel count

m

is large. Early in a block (small

m

), the extra

1 \times 1

conv can cost more than it saves; the payoff comes as

m

grows deep in the block.

Why questions

Why does DenseNet use such a small growth rate like

k = 12

32

Because each layer already sees the rich concatenated features of all predecessors, it only needs to contribute a little new information — keeping

k

small keeps the channel growth (and parameters) manageable.

Why does concatenation cost more memory than ResNet's addition?

Concatenation keeps every prior feature map stored to be reused, so memory grows with depth; addition collapses them into one fixed-size tensor, discarding individual copies.

Why does EfficientNet balance three dimensions instead of just going deeper?

Scaling one dimension alone hits diminishing returns fast — a very deep but narrow, low-res net wastes capacity. Balanced scaling matches receptive field, channel capacity, and input detail so each supports the others.

Why is the depthwise convolution in MBConv done after the channel expansion?

Depthwise conv processes each channel independently and cheaply; doing it in the expanded (wide) space lets the block extract richer per-channel features before projecting back down, at low extra cost.

Why does DenseNet put BN and ReLU before the convolution (pre-activation order)?

Pre-activation (BN then ReLU then Conv) keeps the concatenated feature maps as clean, un-activated signals that each new layer can normalize consistently, improving gradient flow through the dense connections.

Why can't you just double every dimension when doubling your compute budget?

Doubling all three would multiply FLOPs by

2 \cdot 2^{2} \cdot 2^{2} = 32

, far past your budget. The constraint

α β^{2} γ^{2} \approx 2

spreads a single

2 \times

FLOP increase optimally across the three.

Edge cases

What is a DenseNet block's output channel count when it has zero layers (

L = 0

Just

k_{0}

— the input passes through unchanged, since

k_{0} + 0 \cdot k = k_{0}

. The formula degrades gracefully.

What happens at

ϕ = 0

in EfficientNet?

Every scaling factor becomes

α^{0} = β^{0} = γ^{0} = 1

, so you recover the untouched B0 baseline —

ϕ = 0

is the "no scaling" identity point.

What if the compression factor is

θ = 1

No channel compression occurs; the transition layer still downsamples spatially via pooling but passes all

m

channels through, giving the largest (most memory-hungry) DenseNet variant.

Does a single-layer dense block behave any differently from a normal conv layer?

No — with one layer there are no preceding layers to concatenate, so it's just

H_{1} (x_{0})

, an ordinary BN-ReLU-Conv. Dense connectivity only shows its effect from the second layer onward.

If you set the MBConv expansion factor

t = 1

, what block do you get?

No expansion happens, so the "inverted bottleneck" collapses into a plain depthwise-separable block — essentially the MobileNet building block without the widening step.

Recall Quick self-check

Concatenation vs addition — which is DenseNet? ::: Concatenation — it keeps every prior feature map. Which single symbol controls all three EfficientNet dimensions? ::: The compound coefficient $ϕ$ . One unit increase in $ϕ$ multiplies FLOPs by roughly? ::: 2.