Exercises — DenseNet and EfficientNet

DenseNet concatenates; ResNet adds. $\text{DenseNet: }{\mathbf{x}}_{\ell }=H_{\ell }([{\mathbf{x}}_0,{\mathbf{x}}_1,\dots ,{\mathbf{x}}_{\ell -1}])$ $\text{ResNet: }{\mathbf{x}}_{\ell }=H_{\ell }({\mathbf{x}}_{\ell -1})+{\mathbf{x}}_{\ell -1}$ Concatenation stacks feature maps side-by-side along the channel axis, so every earlier feature survives untouched. Addition merges them into one tensor of the same channel count, so individual identities are lost but memory stays flat. See 3.4.7-ResNet-and-Skip-Connections.

• Depth (number of layers): base $\alpha$
• Width (channels per layer): base $\beta$
• Resolution (input image side length): base $\gamma$

All three are raised to the same compound coefficient $\varphi$: $d={\alpha }^{\varphi },w={\beta }^{\varphi },r={\gamma }^{\varphi }$.

What we do: apply the running rule "layer $\ell$ receives $k_0+(\ell -1)k$, outputs $k$ new." Why: each layer's output is stacked onto the pile, so the pile grows by exactly $k$ each step.

Output channels $=k_0+Lk=24+5\cdot 12=84$.

$\lfloor \theta \cdot m\rfloor =\lfloor 0.5\times 84\rfloor =\lfloor 42\rfloor =42\text{ channels.}$ The floor $\lfloor \cdot \rfloor$ matters only when $\theta m$ is not an integer; here it is exact. Transition also halves spatial size via $2\times 2$ average pooling.

$1.2\times 1.1^2\times 1.15^2=1.2\times 1.21\times 1.3225\approx \mathrm{1.920.}$ This is close to $2$ — the small shortfall is why the paper writes "$\approx 2$". Each unit of $\varphi$ therefore roughly (not exactly) doubles FLOPs.

With bottleneck ($1\times 1$ to $4k$, then $3\times 3$ to $k$): $\underset{1\times 1}{\underbrace{m\cdot 4k}}+\underset{3\times 3}{\underbrace{4k\cdot k\cdot 9}}=4mk+36k^2.$ Without bottleneck (direct $3\times 3$ from $m$ to $k$): $m\cdot k\cdot 9=9mk.$ Bottleneck saves parameters when $4mk+36k^2<9mk⟹36k^2<5mk⟹m>\frac{36}{5}k=7.2k.$ Interpretation: early in a block $m$ is small, so the bottleneck's fixed $36k^2$ cost dominates and it does not help. Late in a block $m$ has grown well past $7.2k$, so $4mk\ll 9mk$ and the bottleneck wins. That is exactly where channel counts explode — precisely where you want savings.

$d=1.2^6\approx 2.986,w=1.1^6\approx 1.772,{\gamma }^6=1.15^6\approx \mathrm{2.313.}$ Resolution: $224\times 2.313\approx 518$ px — the paper rounds the architecture to a network-friendly $600$ px in practice, but the scaling law predicts $\approx 518$. (Depth and width multiplier magnitudes match the parent note's $\approx 3.0$ and $\approx 1.77$.)

Depth-only: FLOPs $\propto d$, so $\text{FLOPs}\times 2$ needs $d\times 2$. Compound: FLOPs $\propto (\alpha {\beta }^2{\gamma }^2{)}^{\varphi }=2^{\varphi }$. One extra unit of $\varphi$ doubles FLOPs while multiplying depth by only $\alpha =1.2$, width by $\beta =1.1$, resolution by $\gamma =1.15$. Why spreading wins: each single dimension has diminishing returns — a very deep but narrow, low-resolution net saturates. Balancing keeps all three below their saturation points, so the same FLOP budget buys more accuracy. This is the empirical finding behind EfficientNet; see 3.4.11-Neural-Architecture-Search for how $\alpha ,\beta ,\gamma$ were discovered.

Block 1 output: $32+4\cdot 16=96$. Transition 1: $\lfloor 0.5\times 96\rfloor =48$. Block 2 output: $48+4\cdot 16=112$. Transition 2: $\lfloor 0.5\times 112\rfloor =56$.

Sequence: $32\to 96\to 48\to 112\to 56$ channels. Notice the transition resets the "pile" so the next block starts from a controlled base — this is what keeps DenseNet's channel growth from exploding, unlike a single giant block.

FLOPs factor $=(\alpha {\beta }^2{\gamma }^2{)}^{\varphi }=1.920^2\approx 3.686$. Resolution $=224\times 1.15^2=224\times 1.3225\approx 296$ px. So $\approx 3.7\times$ the compute buys a jump to $\approx 296$ px input alongside proportional depth ($1.2^2\approx 1.44\times$) and width ($1.1^2\approx 1.21\times$) growth.

The claim is false on feature reuse.

• Addition requires all summands to share the same channel count and merges them: $\mathbf{y}=\mathbf{a}+\mathbf{b}$. A downstream layer sees one blended tensor and cannot recover $\mathbf{a}$ or $\mathbf{b}$ separately. Information is superimposed, not preserved.
• Concatenation keeps $\mathbf{a}$ and $\mathbf{b}$ as distinct slices: any later layer's $1\times 1$ conv can weight them independently. This is true feature reuse — the raw low-level maps from layer 0 remain literally addressable at layer 100.

So memory is the genuine cost of concatenation (channel pile grows as $k_0+\ell k$). The correct fixes are the ones DenseNet already ships and that MobileNet inspires:

1. Bottleneck $1\times 1$ convs cap the expensive $3\times 3$ input at $4k$ (Ex 3.1).
2. Transition compression $\theta =0.5$ halves the pile between blocks (Ex 2.2).
3. Depthwise-separable convs (MobileNet) cut the $3\times 3$ cost per channel, orthogonal to the concat memory. None of these sacrifice the concatenation semantics, so they save resources without destroying reuse — unlike swapping to addition.

The growth rate $k$ is DenseNet's width knob: it sets how many channels each layer emits, exactly the role $\beta$ scales in EfficientNet. The catch: in a plain CNN, scaling width multiplies FLOPs by ${\beta }^2$ and channel count grows linearly with depth. In DenseNet, the input to layer $\ell$ is $k_0+(\ell -1)k$, so channel count grows linearly in $k$ and in depth simultaneously — the two scaling axes are coupled. Doubling $k$ in a deep block inflates every subsequent layer's input, so FLOPs grow faster than the naive ${\beta }^2$ prediction. A compound-scaled DenseNet must therefore re-tighten the constraint $\alpha {\beta }^2{\gamma }^2\approx 2$ empirically (a job for 3.4.11-Neural-Architecture-Search), and lean harder on compression $\theta$ and bottlenecks to stay in budget.

DenseNet layer $\ell$ receives $k_0+(\ell -1)k$ channels ::: and outputs $k$ new ones by concatenation. Bottleneck saves parameters once $m$ exceeds ::: $7.2k$. Compound scaling constraint ::: $\alpha {\beta }^2{\gamma }^2\approx 2$. Growth rate $k$ is DenseNet's analogue of EfficientNet's ::: width $\beta$.