Worked examples — Autoencoders fundamentals

This is the practice companion to Autoencoders fundamentals. The parent note built the machinery — encoder, bottleneck, decoder, MSE vs BCE loss. Here we do the opposite of hand-waving: we enumerate every kind of situation an autoencoder problem can put in front of you, then solve one concrete example for each. If a scenario is not below, tell us — the goal is total coverage.

Every symbol we use is first re-anchored so you can read this cold.

Recall Quick symbol refresher (click to expand)

• $\mathbf{x}$ ::: the input — a list of numbers (a vector). For an image it is the pixel values laid in a row.
• $\hat{\mathbf{x}}$ ::: the reconstruction — what the decoder spits out, our attempt to rebuild $\mathbf{x}$.
• $\mathbf{z}$ ::: the latent code — the small squeezed representation in the middle (the bottleneck).
• $d_x$ ::: how many numbers are in $\mathbf{x}$ (input dimension). $d_z$ ::: how many are in $\mathbf{z}$.
• $f_{\varphi }$ ::: the encoder function ($\varphi$ is just the name-tag for its weights) — it is the machine, $\mathbf{z}=f_{\varphi }(\mathbf{x})$. $g_{\theta }$ ::: the decoder, $\hat{\mathbf{x}}=g_{\theta }(\mathbf{z})$.
• $\Vert \mathbf{a}{\Vert }^2$ ::: "squared length" — square each entry of $\mathbf{a}$ and add them up.

The scenario matrix

Think of a topic as a machine that can be fed many kinds of situation. Below is the full menu, followed by a schematic that groups the nine cases so you can see how they relate. Every worked example is tagged with the cell it covers, so you can check nothing is missing.

#	Scenario class	What is special about it	Example
A	Perfect compressible data — points sit on a line/plane	intrinsic dimension < input dimension, exact reconstruction possible	Ex 1
B	Bottleneck too wide ($d_z\ge d_x$) — degenerate	network cheats with the identity map	Ex 2
C	Continuous data, Gaussian noise — pick the loss	MSE is the right answer, compute it by hand	Ex 3
D	Binary data — pick the loss	BCE is right; MSE gives a wrong-feeling penalty	Ex 4
E	Boundary / degenerate loss input — $\hat{x}=0$ or $1$	BCE blows up to $\infty$ (the log-of-zero trap)	Ex 5
F	Gradient flow through the bottleneck	backprop by the chain rule on a tiny net	Ex 6
G	Real-world word problem — sensor compression	translate a story into $d_x,d_z$ and a compression ratio	Ex 7
H	Capacity / information limit — how small can $d_z$ go?	Shannon source-coding bound, a counting argument	Ex 8
I	Exam twist — data NOT on a line, undercomplete forced error	irreducible reconstruction error, the "elbow"	Ex 9

The schematic below shows the same nine cases grouped into three families — geometry of the data, choosing the loss, and the mechanics of learning — so the matrix is not just a flat list.

We will hit A through I. Cases A, B, I are geometric, so they get figures.

---

Case A — data that compresses perfectly

Look at Figure s01 below: all data sits on one ray through the origin (lavender line, coral dots). The mint arrow shows the single number — how far along the line — that fully describes each point. That is why one latent dimension suffices. Here $w_1,w_2$ are the two entries of the encoder $f_{\varphi }$ and $u_1,u_2$ the two entries of the decoder $g_{\theta }$.

Figure s01 — the three data points on $x_2=2x_1$; the mint arrow is the latent coordinate $z$ = signed distance along the line.

1. Pick the encoder as "distance along the line". Choose $w_1=1,w_2=0$, so $z=x_1$. Why this step? We only need $z$ to be some faithful label of position. The simplest label is the first coordinate — since $x_2$ is forced to $2x_1$, knowing $x_1$ knows everything.
2. Pick the decoder to rebuild both coordinates from $z=x_1$. We need ${\hat{x}}_1=x_1$ and ${\hat{x}}_2=2x_1$, so set $u_1=1,u_2=2$. Why this step? The decoder must invert step 1 and re-inject the line's constraint $x_2=2x_1$. That constraint is literally the 2:1 ratio.
3. Read off the pattern. Decoder ratio $u_2/u_1=2$ = the line's slope. The autoencoder discovered the line. Why this step? This is the parent note's claim "linear AE = PCA": the learned direction $(u_1,u_2)=(1,2)$ is the principal axis of the data.

Verify. Point $(2,4)$: $z=1\cdot 2=2$; ${\hat{x}}_1=1\cdot 2=2$; ${\hat{x}}_2=2\cdot 2=4$. Reconstruction $(2,4)$ ✓. Loss $=(2-2{)}^2+(4-4{)}^2=0$. Zero error because the data's intrinsic dimension (1) equals $d_z$ (1).

---

Case B — the bottleneck that cheats

1. Set $W=I$ (identity matrix) and $U=I$. Then $\mathbf{z}=f_{\varphi }(\mathbf{x})=I\mathbf{x}=\mathbf{x}$ and $\hat{\mathbf{x}}=g_{\theta }(\mathbf{z})=I\mathbf{z}=\mathbf{x}$. Why this step? With as many latent slots as inputs, the net can literally copy the input through. No squeezing happens. ($I$ = the matrix with 1's on the diagonal and 0's elsewhere; multiplying by it changes nothing.)
2. Compute the loss. $\mathcal{L}=\Vert \mathbf{x}-\hat{\mathbf{x}}{\Vert }^2=\Vert \mathbf{x}-\mathbf{x}{\Vert }^2=0$ for every $\mathbf{x}$. Why this step? Perfect score — and that is exactly the trap. Zero loss but the latent code learned no structure; it's just the input again.
3. Contrast with the picture. In Figure s02, the wide bottleneck (right) passes the cloud through unchanged; the narrow bottleneck (left) is forced to flatten the cloud onto a line, discovering its main axis. Why this step? This is the parent's "Common Mistake: bottleneck too wide". Compression requires $d_z<d_x$.

Figure s02 — left: narrow $d_z=1$ projects the cloud onto its principal axis (coral = reconstruction, mint = the axis). Right: wide $d_z=2$ leaves the cloud untouched — zero loss, zero learning.

Verify. For $\mathbf{x}=(3,-1)$: $\mathbf{z}=(3,-1)$, $\hat{\mathbf{x}}=(3,-1)$, loss $=0$. The identity map achieves zero loss ⇒ the metric alone cannot tell you the model is useless. You must constrain $d_z$.

---

Case C — continuous data → MSE

Recall MSE: average of the squared per-entry errors. Let $n$ stand for the number of entries in the vector — for a single input vector this is exactly $n=d_x$ (the input dimension from the refresher). We keep the letter $n$ only to match the parent note's formula: ${\mathcal{L}}_{\text{MSE}}=\frac{1}{n}{\sum }_{i=1}^n(x_i-{\hat{x}}_i{)}^2,n=d_x.$ In this example $n=d_x=4$.

1. List the per-entry errors. $0.5-0.4=0.1$; $1.0-1.2=-0.2$; $-0.5-(-0.3)=-0.2$; $2.0-1.5=0.5$. Why this step? MSE never cares about sign — the square erases it — so we can proceed to squaring.
2. Square each. $0.01,0.04,0.04,0.25$. Why this step? Squaring is why large errors ($0.5\to 0.25$) hurt far more than small ones ($0.1\to 0.01$). This is the Gaussian-noise assumption from the parent note showing up.
3. Average over $n=4$. $\frac{0.01+0.04+0.04+0.25}{4}=\frac{0.34}{4}=0.085$. Why this step? Dividing by $n$ makes the loss comparable across vectors of different lengths.

Verify. Sum $=0.34$, divide by 4 $=0.085$. The single $0.5$ error contributed $0.25/0.34\approx 74\%$ of the loss — confirming the forecast that big errors dominate. Units: same as input-squared (a "power-like" quantity), dimensionally consistent with a squared distance.

---

Case D — binary data → BCE

For a single binary pixel, BCE reduces (since $x=1$ kills the second term) to ${\mathcal{L}}_{\text{BCE}}=-[x\log \hat{x}+(1-x)\log (1-\hat{x})]=-\log \hat{x}.$

1. Confident-correct, $\hat{x}=0.9$. $-\log (0.9)=0.105$. Why this step? $\log$ of something near 1 is near 0 → tiny penalty. Being right and sure is cheap, as it should be.
2. Confident-wrong, $\hat{x}=0.1$. $-\log (0.1)=2.303$. Why this step? As $\hat{x}\to 0$ while $x=1$, $-\log \hat{x}\to \infty$. BCE screams when you are confidently wrong.
3. Compare with what MSE would say. MSE here: $(1-0.9{)}^2=0.01$ vs $(1-0.1{)}^2=0.81$ — ratio $81$. BCE ratio: $2.303/0.105\approx 21.9$. Both punish the wrong guess, but BCE's punishment grows unboundedly toward the boundary, matching the Bernoulli likelihood. Why this step? The parent note's rule: match the loss to the data's noise model. For 0/1 data the likelihood is Bernoulli ⇒ BCE.

Verify. $-\ln 0.9=0.1054$, $-\ln 0.1=2.3026$, ratio $\approx 21.85$. The wrong guess costs about 22× more — so "roughly 20×" was the right forecast.

---

Case E — the log-of-zero trap

1. Plug $\hat{x}=0$ into $-\log \hat{x}$. $-\log (0)=+\infty$. Why this step? The logarithm has a vertical asymptote at 0 — the "confidently wrong" penalty is unbounded, not just large. This is the limiting behaviour the matrix demanded we cover.
2. The mirror case: $x=0$ and $\hat{x}=1$. Then $-\log (1-\hat{x})=-\log (0)=+\infty$ too. Why this step? Symmetry — a confident-wrong output at either extreme blows up.
3. The engineering fix: clamp. Real code clamps $\hat{x}$ into $[\epsilon ,1-\epsilon ]$ with e.g. $\epsilon =10^{-7}$, so $-\log$ stays finite. Equivalently, use a sigmoid output which never exactly reaches 0 or 1. Why this step? Without clamping, one saturated pixel produces NaN gradients and destroys training. This is why the parent's MNIST decoder ends in a sigmoid.

Verify. $-\log (10^{-7})=7\log 10\approx 16.118$ — the clamped worst-case penalty, large but finite. The un-clamped value is $\infty$, confirming the "literal infinity" forecast.

---

Case F — gradient through the bottleneck

We chain the loss backward: loss ← $\hat{x}$ ← $z$ ← $w$. Here $w$ is the single encoder weight (entry of $W$) and $u$ the single decoder weight (entry of $U$).

1. $\frac{\partial \mathcal{L}}{\partial \hat{x}}=-2(x-\hat{x})$. Differentiate $(x-\hat{x}{)}^2$. Why this step? This is the "how wrong are we" signal that gets pushed backward.
2. $\frac{\partial \hat{x}}{\partial z}=u$ and $\frac{\partial z}{\partial w}=x$. Straight from $\hat{x}=uz$, $z=wx$. Why this step? Each factor is one layer's local derivative; the chain rule multiplies them.
3. Multiply, then evaluate. $\frac{\partial \mathcal{L}}{\partial w}=-2(x-\hat{x})\cdot u\cdot x$. At $x=1,w=1,u=1$ first compute $z=wx=1$ and $\hat{x}=uz=1$, so $(x-\hat{x})=(1-1)=0$, giving $\frac{\partial \mathcal{L}}{\partial w}=-2\cdot 0\cdot 1\cdot 1=0$. Why this step? The chain rule multiplies the three local derivatives into the full gradient; evaluating at the given point shows the encoder weight $u$ multiplies the signal reaching $w$. If $u$ (or $z$) shrinks toward 0, the encoder's gradient vanishes — this is the parent's warning "gradients flow through the bottleneck". Here the gradient is 0 because the fit is already perfect.

Verify. $-2(1-1)(1)(1)=0$. Zero gradient at a perfect reconstruction ⇒ we are already at a loss minimum, matching the forecast. Sanity: nudge $u\to 0.5$ (imperfect), then $\hat{x}=0.5$, gradient $=-2(0.5)(0.5)(1)=-0.5\ne 0$, so learning would resume; the negative sign says "increase $w$", which raises $\hat{x}$ toward $x=1$ ✓.

---

Case G — real-world word problem

1. Compute $d_x$. $6\text{ channels}\times 50\text{Hz}\times 2\text{s}=600$. So $\mathbf{x}\in {\mathbb{R}}^{600}$. Why this step? We must flatten the window into one vector before it enters the encoder — dimensions multiply.
2. Compression ratio $=d_x/d_z=600/24=25$. A 25× squeeze. Why this step? This ratio tells you how aggressive the bottleneck is; the parent used a similar $\sim$25× for MNIST ($784\to 32$).
3. Choose the loss. Sensor readings are continuous real numbers with roughly Gaussian noise ⇒ MSE. Why this step? BCE only makes sense for values in $[0,1]$ modelled as Bernoulli. Heart rate is not a coin flip.

Verify. $6\times 50\times 2=600$ ✓; $600/24=25$ exactly ✓. The forecast "25" matches. Units check: $d_x$ is a pure count, ratio is dimensionless — correct for a compression factor.

---

Case H — the information / capacity limit

The bound (parent note): for lossless coding you need $d_z{\log }_2(L_z)\ge I(\mathbf{x})\le d_x{\log }_2(L),$ where $L$ = intensity levels per input pixel, $L_z$ = distinguishable values per latent slot, and $I(\mathbf{x})$ = the information content in bits.

1. Input information. $d_x=8\times 8=64$ pixels, $L=16$ levels ⇒ ${\log }_{2}16=4$ bits each ⇒ $I(\mathbf{x})\le 64\times 4=256$ bits. Why this step? This counts the worst-case bits needed if pixels were totally independent (the upper bound on content).
2. Latent capacity per slot. $L_z=256\Rightarrow {\log }_{2}256=8$ bits per latent number. Why this step? Each latent slot can carry only its bit-budget; we tally the total budget next.
3. Solve for $d_z$. Need $8d_z\ge 256\Rightarrow d_z\ge 32$. Why this step? Below 32 slots there aren't enough bits to distinguish every possible patch — reconstruction must lose information. Note: real image patches have low intrinsic dimension, so in practice far fewer slots work; this is the worst-case ceiling.

Verify. $64\times 4=256$ bits; $256/8=32$. Minimum $d_z=32$ — so "more than 20" was right. Sanity: if latent slots held only $4$ bits each ($L_z=16$), you'd need $256/4=64=d_x$ slots and no compression is guaranteed.

---

Case I — the exam twist: forced (irreducible) error

The 1D linear AE reconstructs a point $\mathbf{x}$ as its projection onto $\mathbf{u}$: $\hat{\mathbf{x}}=(\mathbf{u}\cdot \mathbf{x})\mathbf{u}$, where $\mathbf{u}\cdot \mathbf{x}$ is the dot product (multiply matching entries, add them). Here $\mathbf{u}$ plays the role of both the encoder direction (it computes the latent number $z=\mathbf{u}\cdot \mathbf{x}$) and the decoder direction (it maps $z$ back with $g_{\theta }(z)=z\mathbf{u}$).

Figure s03 — the two points $A,B$ (coral) and their forced reconstructions on the mint line $\mathbf{u}$ (lavender). The dashed drops are the irreducible errors — the parts perpendicular to $\mathbf{u}$ that one latent number can never store.

1. Project $A=(1,0)$. $\mathbf{u}\cdot A=\frac{1}{\sqrt{2}}(1)=\frac{1}{\sqrt{2}}$. So $\hat{A}=\frac{1}{\sqrt{2}}\cdot \frac{1}{\sqrt{2}}(1,1)=(\frac{1}{2},\frac{1}{2})$. Why this step? With one latent number, every reconstruction is forced to land on the line $\mathbf{u}$ — see the dashed drops in Figure s03. $A$ can't be reached exactly.
2. Error for $A$. $A-\hat{A}=(1-\frac{1}{2},0-\frac{1}{2})=(\frac{1}{2},-\frac{1}{2})$, squared length $=\frac{1}{4}+\frac{1}{4}=\frac{1}{2}$. Why this step? This leftover is the irreducible error — the part of $A$ perpendicular to $\mathbf{u}$, which a 1D code can never store.
3. By symmetry, $B=(0,1)$ gives the same $\frac{1}{2}$. Total loss $=\frac{1}{2}+\frac{1}{2}=1$. Why this step? This is the exam lesson: when intrinsic dimension (here 2) exceeds $d_z$ (here 1), loss has a floor > 0. Plotting loss vs $d_z$ shows an "elbow" — beyond it, adding latent slots stops helping.

Verify. $\hat{A}=(0.5,0.5)$, error${}^2=0.5$; total $=1.0$. Nonzero, confirming the forecast that unequal-direction points cannot be compressed losslessly to 1D. (Best possible $\mathbf{u}$ = first PCA axis still leaves a positive floor here.)

---

Recall Scenario self-test (cover, then reveal)

Which loss for continuous sensor data, and why? ::: MSE — Gaussian noise model; large errors squared → dominate. Why does $d_z\ge d_x$ give zero loss but zero learning? ::: The network learns the identity map; no compression, latent code meaningless. What happens to BCE as $\hat{x}\to 0$ while $x=1$? ::: It diverges to $+\infty$; implementations clamp to $[\epsilon ,1-\epsilon ]$. A 1D linear AE on points not sharing a direction gives what loss? ::: Nonzero — an irreducible floor from the perpendicular component. In backprop, why don't we special-case negative inputs $x<0$? ::: The gradient $-2(x-\hat{x})ux$ is a signed product; sign flips cancel so descent still moves $\hat{x}$ toward $x$ in every quadrant.

Where to go next

• The regularized cousins that fix the wide-bottleneck trap: Sparse Autoencoders and Variational Autoencoders (VAE).
• The geometry of the middle: Latent Space Representations and Dimensionality Reduction Techniques.
• Structural view: Encoder-Decoder Architectures; the learning setting: Unsupervised Learning.
• Hindi walkthrough of the parent: 4.5.02 Autoencoders fundamentals (Hinglish).