DenseNet and EfficientNet

Overview

Two landmark architectures that revolutionized CNN design: DenseNet maximizes feature reuse through dense connections, while EfficientNet systematically scales networks for optimal accuracy-efficiency tradeoffs.

DenseNet: Dense Convolutional Networks

For layer $\ell$ in a dense block: $\mathbf{x}_\ell = H_\ell([\mathbf{x}_0, \mathbf{x}_1, \ldots, \mathbf{x}_{\ell-1}])$

where $[\mathbf{x}_0, \mathbf{x}_1, \ldots, \mathbf{x}_{\ell-1}]$ denotes concatenation along the channel dimension, and $H_\ell$ is a composite function: BN → ReLU → Conv3×3.

Why this matters:

• Gradient highway: Gradients can flow directly to early layers (alleviates vanishing gradients)
• Feature reuse: Late layers access raw low-level features from early layers
• Parameter efficiency: No need to re-learn redundant features

Deriving Growth Rate and Parameters

Derivation from first principles:

• Layer0: input has $k_0$ channels
• Layer 1: receives $k_0$ channels, outputs $k$ new channels → total = $k_0 + k$
• Layer 2: receives $k_0 + k$ channels, outputs $k$ new channels → total = $k_0 + 2k$
• Layer $\ell$: receives $k_0 + (\ell-1)k$ channels, outputs $k$ → total = $k_0 + \ell k$
• After $L$ layers: $k_0 + Lk$ channels

Why small $k$ works: Typical $k = 12$ or $32$. Each layer contributes a small set of feature maps to the "collective knowledge," but accesses the rich combined feature set from all predecessors.

$H_\ell: \text{BN-ReLU-Conv1×1} \rightarrow 4k \text{ channels} \rightarrow \text{BN-ReLU-Conv3×3} \rightarrow k \text{ channels}$

Why 1×1 convolution? Reduces dimensionality before expensive3×3 convolution.

Parameter calculation for one bottleneck layer:

• If input has $m$ channels:
  • 1×1 conv: $m \times 4k \times 1 \times 1 = 4mk$ parameters
  • 3×3 conv: $4k \times k \times 3 \times 3 = 36k^2$ parameters
• Without bottleneck: $m \times k \times 3 = 9mk$ parameters

As $m$ grows (late in the block), $4mk \ll 9mk$ for reasonable $k$, so bottlenecks save parameters.

If a dense block outputs $m$ channels, the transition layer compresses to $\lfloor \theta m \rfloor$ channels where $\theta \in (0, 1]$ is the compression factor (typically $\theta = 0.5$).

Why compression? Controls model size and computational cost. Halving channels after each dense block keeps growth manageable.

Total layers: $1 + (6+12+24+16) = 59$ conv layers inside dense blocks, plus transitions≈ 121 layers with bottlenecks.

Why this configuration works: Early blocks (few layers) learn low/mid-level features. Deep blocks (many layers) learn high-level abstractions. Each has access to all prior features.

Why this step? Each layer concatenates its output ($k=8$) to the running total.

After transition with $\theta=0.5$: $48 \times 0.5 = 24$ channels.

Why it feels right: Both are "skip" connections that help gradients.

The fix: DenseNet concatenates feature maps: $\mathbf{x}_\ell = H_\ell([\mathbf{x}_0, \ldots, \mathbf{x}_{\ell-1}])$. This preserves all prior information explicitly, while ResNet's addition merges features (loses individual identity). Concatenation enables true feature reuse but increases memory; addition is memory-efficient but reuses less.

EfficientNet: Compound Scaling

$\text{depth: } d = \alpha^\phi$ $\text{width: } w = \beta^\phi$ $\text{resolution: } r = \gamma^\phi$

subject to the constraint: $\alpha \cdot \beta^2 \cdot \gamma^2approx 2$, with $\alpha \geq 1, \beta \geq 1, \gamma \geq 1$.

Why this constraint? Approximates FLOPs doubling per unit increase in $\phi$.

Deriving the Scaling Law

Total FLOPs for a layer: $\text{FLOPs} \propto d \cdot w^2 \cdot r^2$

If we scale each dimension: $\text{FLOPs}_{\text{new}} \propto (d \cdot \alpha^\phi) \cdot (w \cdot \beta^\phi)^2 \cdot (r \cdot \gamma^\phi)^2$ $= d \cdot w^2 \cdot r^2 \cdot \alpha^\phi \cdot \beta^{2\phi} \cdot \gamma^{2\phi}$ $= \text{FLOPs}_{\text{base}} \cdot (\alpha \cdot \beta^2 \cdot \gamma^2)^\phi$

To double FLOPs per increment of $\phi$: $\alpha \cdot \beta^2 \cdot \gamma^2 = 2$

Why this matters: For a given computational budget (FLOPs), this formula tells you how to distribute scaling across depth, width, and resolution for maximum accuracy.

Check: $1.2 \times 1.1^2 \times 1.15^2 = 1.2 \times 1.21 \times 1.3225 \approx 1.92 \approx 2$ ✓

To create EfficientNet-B1 with $\phi = 1$:

• Depth: $d_{B1} = 1.2^1 \times d_{B0} = 1.2 \times d_{B0}$
• Width: $w_{B1} = 1.1^1 \times w_{B0} = 1.1 \times w_{B0}$
• Resolution: $r_{B1} = 1.15^1 \times 224 \approx 240$ pixels

For EfficientNet-B7 with $\phi = 6$:

• Depth: $1.2^6 \approx 3.0 \times d_{B0}$
• Width: $1.1^6 \approx 1.77 \times w_{B0}$
• Resolution: $1.15^6 \times 224 \approx 456$ pixels

Why this step? Each dimension grows exponentially with $\phi$, but constrained so total FLOPs grow exponentially too (each increment of $\phi$ ≈ 2× FLOPs).

1. Expand: 1×1 conv to expand channels by factor $t$ (typically 6)
2. Depthwise: 3×3 or 5×5 depthwise conv (one filter per channel)
3. Squeeze-and-Excitation (SE): channel attention
4. Project: 1×1 conv to project back to output channels
5. Skip connection if input/output dimensions match

Why MBConv? Extremely parameter-efficient. Depthwise separable convolutions reduce FLOPs dramatically compared to standard convolutions.

Strategy 1: Depth only ($\alpha = 2, \beta = 1, \gamma = 1$)

• FLOPs: $2 \times 1^2 \times 1^2 = 2$ ✓
• Result: Very deep, narrow network. Risk: vanishing gradients, harder to train.

Strategy 2: Width only ($\alpha = 1, \beta = \sqrt{2}, \gamma = 1$)

• FLOPs: $1 \times 2 \times 1 = 2$ ✓
• Result: Wide, shallow network. Risk: limited representational depth.

Strategy 3: Resolution only ($\alpha = 1, \beta = 1, \gamma = \sqrt{2}$)

• FLOPs: $1 \times 1 \times 2 = 2$ ✓
• Result: Higher resolution, same depth/width. Risk: fine spatial details but no added abstraction layers.

Strategy 4: Compound scaling ($\alpha = 1.2, \beta = 1.1, \gamma = 1.15 \phi = 1$)

• FLOPs: $1.2 \times 1.21 \times 1.3225 \approx 2$ ✓
• Result: Balanced growth in all dimensions. Empirically achieves higher accuracy.

Why compound scaling wins: Depth adds abstraction, width adds capacity per layer, resolution captures finer patterns. Scaling all three avoids bottlenecks in any single dimension.

Why it feels right: More is better, right?

The fix: Unconstrained scaling explodes FLOPs: $2 \times 4 = 32\times$ increase FLOPs (not 2×). You blow your compute budget. The $\alpha \beta^2 \gamma^2 \approx 2$ constraint ensures you scale within a target computational envelope. Empirically, the exponents 1, 2, 2 reflect how depth, width, and resolution impact FLOPs in convolutional layers.

Comparing DenseNet and EfficientNet

| Aspect | DenseNet | EfficientNet | |--------|--------------| | Core Idea | Dense connectivity for feature reuse | Compound scaling for efficiency | | Parameter Efficiency | Fewer params via feature reuse | Fewer params via MBConv + AutoML | | Memory | High (concatenation stores all intermediate features) | Low (inverted bottlenecks) | | Training | Can be memory-intensive | Efficient, but needs careful scaling | | Strength | Excellent gradient flow, feature reuse | State-of-the-art accuracy/FLOP tradeoff | | Use Case | Tasks where feature reuse matters (medical imaging, fine-grained classification) | Resource-constrained or production deployment |

DenseNet: Every time you add a new layer of LEGO bricks, you attach it not just to the layer right below, but you run strings (connections) to every layer below it. So the top brick is connected to the bottom, the middle, everything. This means every brick can "see" what all the earlier bricks learned. It's like everyone in a class sharing their notes with everyone else—nobody mises out on important info. The downside? You need a lot of string (memory).

EfficientNet: Now imagine you want to make your tower taller. You could just stack more layers (make it deeper). Or make each layer wider (more bricks per layer). Or use bigger bricks (higher resolution images). EfficientNet says: do all three, but in a balanced way. If you make it20% deeper, make it 10% wider and use 15% bigger bricks, all at once. This way, the tower stays strong and balanced without topling over or using too much material.

Both are about being smart: DenseNet shares everything, EfficientNet scales everything proportionally.

Connections

• ResNet - DenseNet extends skip connections to full dense connectivity
• Inception - Multi-scale processing, both use1×1 bottlenecks
• MobileNet - EfficientNet builds on MBConv blocks from MobileNetV2
• Neural Architecture Search - EfficientNet uses NAS to find baseline architecture
• Batch Normalization - Critical component in both architectures
• Model Compression - Both architectures motivate efficiency research

#flashcards/ai-ml

What is the dense connectivity pattern in DenseNet? :: Each layer receives feature maps from all preceding layers via concatenation along the channel dimension, and passes its output to all subsequent layers.

What is compound scaling in EfficientNet? :: Uniformly scaling network depth ($d = \alpha^\phi$), width ($w = \beta^\phi$), and resolution ($r = \gamma^\phi$) together using a compound coefficient $\phi$, subject to $\alpha \beta^2 \gamma^2 \approx 2$ to control FLOPs.

What are MBConv blocks and why does EfficientNet use them? :: Mobile Inverted Bottleneck Convolution blocks: expand channels (1×1 conv), apply depthwise conv (3×3 or 5×5), add SE attention, project down (1×1 conv), skip connection. Extremely parameter-efficient due to depthwise separable convolutions.

Concept Map

Hinglish (regional understanding)