Visual walkthrough — DenseNet and EfficientNet

This page rebuilds one result from the parent note, DenseNet and EfficientNet, from absolute zero: the EfficientNet rule

$\alpha \cdot {\beta }^2\cdot {\gamma }^2\approx 2.$

The parent note stated this. Here we earn it — every symbol, every square, every "2" — with a picture per step. If you have not yet met the ideas of layers, channels, and image size, we build them from scratch below. Related reading: 3.4.9-MobileNet-and-Depthwise-Separable-Convolutions and 3.4.11-Neural-Architecture-Search.

---

Step 1 — What is a convolutional layer counting?

WHAT. A CNN has three independent sizes:

• Depth — how many layers are stacked (a taller tower).
• Width — how many channels each layer holds (a fatter tower).
• Resolution — how many pixels tall and wide the image is (a bigger picture).

WHY. Historically people pulled one arrow at a time — "let's just add layers", or "just add channels". EfficientNet's whole point is that these are three different arrows, so we must first name each one cleanly and separately before we can talk about pulling them together.

PICTURE. Figure s01 — a single network drawn as a point, with three coloured arrows leaving it: a lavender arrow going up labelled "Depth $d_0$ (more layers)", a coral arrow going right labelled "Width $w_0$ (more channels)", and a mint arrow going up-right labelled "Resolution $r_0$ (bigger image)". The figure's job is to establish that these are three separate, independent directions of growth before any algebra appears.

Each of $d_0,w_0,r_0$ is just a plain positive number. Nothing hidden yet.

---

Step 2 — Counting the work of ONE layer

WHAT. Take a single convolution. A filter of size $F\times F$ slides over an $r\times r$ grid (using the scaled resolution $r$). At each of the $r^2$ positions it touches:

• $w$ input channels,
• to produce $w$ output channels,
• doing $F^2$ multiply-adds per channel-pair.

So the multiply-adds for one layer are

$$\underset{\text{how many grid positions}}{\underbrace{r^2}}\times \underset{\text{input channels}}{\underbrace{w}}\times \underset{\text{output channels}}{\underbrace{w}}\times \underset{\text{filter taps}}{\underbrace{F^2}}.$$

WHY these factors, term by term.

• $r^2$ — the filter must stop at every pixel of an $r\times r$ image, and there are $r^2$ pixels. Two spatial directions ⇒ the square.
• $w\times w$ — every one of the $w$ output channels is built by looking at all $w$ input channels. Two channel counts multiplied ⇒ $w^2$. (This "dense" all-to-all channel mixing is exactly what depthwise-separable convolutions in real EfficientNet avoid — another place the toy model is simpler than reality.)
• $F^2$ — the fixed filter size (say $3\times 3=9$); by the assumption above it is the same constant in every layer, so we factor it into the proportionality constant and ignore it from here.

PICTURE. Figure s02 — the three counted things laid side by side: on the left a mint $r\times r$ grid labelled "$r^2$ positions"; in the middle a coral column of $w$ input channels feeding through an "$F\times F$ filter" arrow into a lavender column of $w$ output channels; a caption "each out reads each in ⇒ $w\times w=w^2$". The figure makes visible why resolution and width each contribute a square.

---

Step 3 — Counting the whole network

WHAT. In the toy model a network of depth $d$ is just $d$ identical layers stacked. Total work is the per-layer work times the number of layers:

$$\text{FLOPs}\propto \underset{\text{how many layers}}{\underbrace{d}}\times \underset{\text{per-layer channel work}}{\underbrace{w^2}}\times \underset{\text{per-layer spatial work}}{\underbrace{r^2}}.$$

WHY. Depth enters linearly (one copy of the work per layer), while width and resolution each enter squared. This asymmetry — one linear, two squared — is the whole reason the final rule has a ${\beta }^2$ and a ${\gamma }^2$ but only ${\alpha }^1$. Keep your eye on it.

PICTURE. Figure s03 — a vertical stack of five identical butter-coloured bricks, each labelled "layer: $w^2{r}^2$", with a lavender arrow up the side labelled "depth $d$ copies" and a coral caption "total FLOPs ~ $d{w}^2{r}^2$". The figure shows depth as repetition (linear) versus the squared per-brick cost.

---

Step 4 — Scaling each knob by its own factor

WHAT. Introduce three growth factors, all $\ge 1$, that stretch the baseline sizes:

• $\alpha$ multiplies depth: $d=\alpha d_0$,
• $\beta$ multiplies width: $w=\beta w_0$,
• $\gamma$ multiplies resolution: $r=\gamma r_0$.

Substitute into the total-work formula from Step 3:

$${\text{FLOPs}}_{\text{new}}\propto (\alpha d_0)(\beta w_0{)}^2(\gamma r_0{)}^2.$$

Now expand each squared bracket, term by term:

$$(\beta w_0{)}^2={\beta }^2{w}_0^2,(\gamma r_0{)}^2={\gamma }^2{r}_0^2.$$

So

$${\text{FLOPs}}_{\text{new}}\propto \underset{\text{baseline work}}{\underbrace{d_0{w}_0^2{r}_0^2}}\cdot \underset{\text{the growth multiplier}}{\underbrace{\alpha {\beta }^2{\gamma }^2}}.$$

WHY. Look what happened: the baseline work $d_0{w}_0^2{r}_0^2$ factored straight back out, leaving a single tidy multiplier $\alpha {\beta }^2{\gamma }^2$. The squares on $\beta$ and $\gamma$ come directly from the two squared factors in Step 2 — width and resolution each cost a square because each spans two directions.

PICTURE. Figure s04 — the three growth factors as small boxes ("$d=\alpha d_0$" lavender, "$w=\beta w_0$" coral, "$r=\gamma r_0$" mint) with arrows converging into one butter box labelled "multiplier $\alpha \cdot {\beta }^2\cdot {\gamma }^2$" and a note "(width & res squared)". The figure shows all scaling collapsing into one factor.

---

Step 5 — The compound coefficient $\varphi$: one dial for all three

WHAT. Define the depth, width and resolution growth as powers of $\varphi$, always relative to the baseline:

$$d={\alpha }^{\varphi }{d}_0,w={\beta }^{\varphi }{w}_0,r={\gamma }^{\varphi }{r}_0.$$

Here $\varphi$ is the compound coefficient. It is a real number $\varphi \ge 0$, not restricted to whole numbers — fractional values like $\varphi =0.5$ are perfectly meaningful (they give a network between two named models). In practice EfficientNet publishes models at the integer steps $\varphi =0,1,2,\dots ,7$ (named B0…B7) purely for convenience; the maths itself is continuous. Turning $\varphi$ up grows all three sizes at once. $\varphi =0$ gives ${\alpha }^0={\beta }^0={\gamma }^0=1$: no growth, so $d=d_0,w=w_0,r=r_0$ — the untouched baseline.

Plug these into the multiplier from Step 4. Using the exponent rule $x^{\varphi }\cdot y^{\varphi }=(xy{)}^{\varphi }$:

$${\alpha }^{\varphi }({\beta }^{\varphi }{)}^2({\gamma }^{\varphi }{)}^2={\alpha }^{\varphi }{\beta }^{2\varphi }{\gamma }^{2\varphi }=(\alpha {\beta }^2{\gamma }^2{)}^{\varphi }.$$

WHY, term by term.

• $({\beta }^{\varphi }{)}^2={\beta }^{2\varphi }$ — power of a power multiplies exponents.
• Factoring the common exponent $\varphi$ out gives $(\alpha {\beta }^2{\gamma }^2{)}^{\varphi }$: the same base $\alpha {\beta }^2{\gamma }^2$ from Step 4, now raised to the dial $\varphi$.

So the total work grows as

$$\frac{{\text{FLOPs}}_{\text{new}}}{{\text{FLOPs}}_{\text{base}}}=(\alpha {\beta }^2{\gamma }^2{)}^{\varphi }.$$

PICTURE. Figure s05 — a dial (labelled "phi ≥ 0 (real)") on the left whose output fans out to the three factor boxes "$d={\alpha }^{\varphi }{d}_0$", "$w={\beta }^{\varphi }{w}_0$", "$r={\gamma }^{\varphi }{r}_0$", which then feed a single box "$(\alpha {\beta }^2{\gamma }^2{)}^{\varphi }$". The figure shows one dial simultaneously driving all three scalings.

---

Step 6 — Why the base is chosen to be 2

WHAT. We want: increasing $\varphi$ by $1$ multiplies FLOPs by $2$. From Step 5, going from $\varphi$ to $\varphi +1$ multiplies FLOPs by

$$\frac{(\alpha {\beta }^2{\gamma }^2{)}^{\varphi +1}}{(\alpha {\beta }^2{\gamma }^2{)}^{\varphi }}=(\alpha {\beta }^2{\gamma }^2{)}^1=\alpha {\beta }^2{\gamma }^2.$$

Set that target equal to the desired factor $2$:

$$\boxed{\alpha \cdot {\beta }^2\cdot {\gamma }^2=2}$$

WHY. This is not a law of nature — it is a design choice. Fixing the target base to $2$ turns $\varphi$ into a "compute-doubling counter": $\varphi =1$ ⇒ $2\times$ FLOPs of the baseline, $\varphi =2$ ⇒ $4\times$, $\varphi =3$ ⇒ $8\times$, and so on. The constraint $\alpha ,\beta ,\gamma \ge 1$ guarantees we only ever grow, never shrink.

PICTURE. Figure s06 — a bar chart of FLOPs (in multiples of the baseline) against $\varphi =0,1,2,3,4,5$, each bar exactly double the previous ($1,2,4,8,16,32$), with a caption "ratio between steps = $\alpha {\beta }^2{\gamma }^2=2$". The figure makes the doubling-per-step meaning of the target concrete.

---

Step 7 — Checking the real EfficientNet numbers

WHAT. Neural Architecture Search found, for EfficientNet-B0:

$\alpha =1.2,\beta =1.1,\gamma =\mathrm{1.15.}$

Test the target, term by term:

$$\alpha \cdot {\beta }^2\cdot {\gamma }^2=1.2\times \underset{1.21}{\underbrace{1.1^2}}\times \underset{1.3225}{\underbrace{1.15^2}}\approx 1.2\times 1.21\times 1.3225\approx 1.92\approx 2.✓$$

The $\approx$ here is exactly the point of Step 6's box: the target is $2$, the realized value is $1.92$.

Now drive the dial. For B7, $\varphi =6$:

$$d=1.2^6{d}_0\approx 2.99d_0,w=1.1^6{w}_0\approx 1.77w_0,r=1.15^6{r}_0\approx 2.31r_0.$$

The raw formula on the baseline resolution $r_0=224$ gives $1.15^6\times 224\approx 517$ px.

WHY. Notice depth (base $1.2$) grows fastest, width (base $1.1$) slowest — but because width and resolution are squared in the cost, small $\beta ,\gamma$ still contribute plenty to the budget. That is the balance the squared exponents from Step 2 buy us.

PICTURE. Figure s07 — three curves of growth factor versus $\varphi \in [0,6]$: lavender "$1.2^{\varphi }$" (depth, steepest), coral "$1.1^{\varphi }$" (width, flattest), mint "$1.15^{\varphi }$" (resolution, middle), with dotted verticals marking B0 ($\varphi =0$), B1 ($\varphi =1$) and B7 ($\varphi =6$). The figure shows each knob growing exponentially at its own rate.

---

Step 8 — The degenerate case $\varphi =0$

WHAT. Set the dial to zero:

$$d={\alpha }^0{d}_0=d_0,w={\beta }^0{w}_0=w_0,r={\gamma }^0{r}_0=r_0.$$

Every growth factor becomes $1$, so nothing scales — this is exactly EfficientNet-B0, the baseline.

WHY. A good scaling law must contain "do nothing" as a special case. $\varphi =0$ giving multiplier $(\alpha {\beta }^2{\gamma }^2{)}^0=1$ (any positive number to the power $0$ is $1$) confirms the formula degrades gracefully: zero dial ⇒ $1\times$ FLOPs ⇒ the original network untouched.

PICTURE. Figure s08 — the same dial as s05 but pointed straight up at "phi = 0", feeding three boxes each reading "${\alpha }^0=1$", "${\beta }^0=1$", "${\gamma }^0=1$", with a caption "$d=d_0,w=w_0,r=r_0$ → nothing scales". The figure confirms the law contains the do-nothing case.

---

The one-picture summary

PICTURE. Figure s09 — a left-to-right flow that compresses the whole page: "3 knobs $d,w,r$" → "layer cost $w^2{r}^2$" → "net cost $d{w}^2{r}^2$" → "scaled × $\alpha {\beta }^2{\gamma }^2$" → "one dial: $(\alpha {\beta }^2{\gamma }^2{)}^{\varphi }$" → "double per step ⇒ $\alpha {\beta }^2{\gamma }^2=2$". Reading the chain top-to-bottom replays every step in one glance.

The whole derivation on one canvas: three knobs → per-layer cost $w^2{r}^2$ → network cost $d{w}^2{r}^2$ → scaled by $\alpha {\beta }^2{\gamma }^2$ → tied to one dial as $(\alpha {\beta }^2{\gamma }^2{)}^{\varphi }$ → doubling wish targets the base $2$.

Recall Feynman retelling — say it back in plain words

A CNN has three ways to get bigger: taller (more layers, $d_0$), fatter (more channels, $w_0$), and higher-detail (bigger image, $r_0$). We use a toy picture: one long stack of identical layers — real EfficientNet has varied stages, down-sampling and residuals, but they all scale the same way, so the toy gives the right exponents. One layer's work is "how many pixels" times "channels in" times "channels out" — pixels are two-dimensional so that's $r^2$, and channels-in times channels-out is $w^2$; the filter size $F^2$ is the same constant in every layer, so it factors out and we drop it. Stack $d$ layers and the total work is $d{w}^2{r}^2$. Now grow each baseline size by its own factor $\alpha ,\beta ,\gamma$; the extra work is exactly the multiplier $\alpha {\beta }^2{\gamma }^2$ — width and resolution get squared because they each stretch in two directions, depth doesn't. Tie all three factors to one dial $\varphi$ (any real $\varphi \ge 0$) by raising them to $\varphi$, and the multiplier becomes $(\alpha {\beta }^2{\gamma }^2{)}^{\varphi }$. Finally we decide that each click of the dial should double the compute; that decision targets $\alpha {\beta }^2{\gamma }^2=2$ exactly, though the searched constants $1.2,1.1,1.15$ only reach $1.92$ (hence $\approx$). Resolutions from the formula get rounded to clean, stride-friendly whole numbers — that's why B7 ships at $600$ px, not the raw $517$. Dial at zero ⇒ the plain baseline network.

Recall

Why is width squared in the FLOP formula but depth is not? ::: Width sets both the number of input channels and the number of output channels, and each output channel reads every input channel — two counts multiplied ⇒ $w^2$. Depth just stacks copies of a layer, so it enters linearly. What single design wish forces the target $\alpha {\beta }^2{\gamma }^2=2$? ::: The wish that increasing the compound coefficient $\varphi$ by 1 should exactly double the network's FLOPs. Is the constraint an exact equality or an approximation? ::: The target is exact ($=2$); the searched values $1.2,1.1,1.15$ only approximate it (product $\approx 1.92$), so we write $\approx 2$ when substituting real numbers. What network do you get when $\varphi =0$? ::: The unscaled baseline (EfficientNet-B0), because every growth factor raised to the power 0 equals 1. Is $\varphi$ restricted to whole numbers? ::: No — $\varphi$ is any real number $\ge 0$; the integer values 0…7 (B0…B7) are just the officially published models. Fractional $\varphi$ is perfectly valid and lands between named models. What does $F$ stand for and why does it drop out? ::: $F$ is the filter's side-length (e.g. $F=3$ for a $3\times 3$ window); held constant across all layers, its $F^2$ is the same factor everywhere and cancels from every ratio. Why is the derivation a "toy" model? ::: It treats the net as a stack of identical layers with one shared $w$ and $r$; real EfficientNet has multiple stages, down-sampling, expansion factors and residuals — but they all scale the same way, so the toy gives the correct scaling exponents. Why doesn't the raw formula give the published resolution (e.g. 517 vs 600 px)? ::: A pixel grid needs an integer side that is also divisible by powers of two (for the stride-2 down-sampling); the raw ${\gamma }^{\varphi }{r}_0$ is rounded to the nearest clean, stride-friendly whole number.