Autoencoders fundamentals: Visual walkthrough

Before Step 1, let us fix the vocabulary so no symbol arrives unannounced.

Step 1 — Draw the data before drawing any network

WHAT. We plot our raw data as points in space. Take the simplest possible case: each data point is just two numbers $(x_{1}, x_{2})$ , so it is a dot on a flat page.

WHY start here. You cannot compress what you have not looked at. Compression only works if the data has shape — if the dots secretly lie on something thin. So we look first.

PICTURE. In the figure the burnt-orange dots almost all sit along one tilted line. That thin line is the whole story: even though every dot needs two numbers to write down, knowing how far along the line a dot sits — a single number — nearly pins it down.

Step 2 — Build the squeeze: the encoder

WHAT. The encoder turns the two-number dot $x = [x_{1}, x_{2}]$ into one number $z$ . In the linear case it is a weighted sum:

$z = dial on x_{1} w_{1} 1st coord x_{1} + dial on x_{2} w_{2} 2nd coord x_{2}$

Each $w$ is a knob the network gets to tune. The output $z \in R^{1}$ is a single number: the compressed code.

WHY a weighted sum. A weighted sum is the simplest way to collapse two numbers into one while still letting both inputs have a say. Geometrically, " $w_{1} x_{1} + w_{2} x_{2} = z$ " measures how far the dot projects onto a chosen direction. Choosing the weights = choosing which direction to measure along.

PICTURE. The teal arrow is the direction the encoder measures along. Dropping a dot straight down onto that arrow (the dotted plum line) gives its single coordinate $z$ — its position on the line.

Step 3 — Rebuild: the decoder

WHAT. The decoder takes the single number $z$ and stretches it back into two numbers, a guess $\hat{x}$ :

$\overset{x}{^}_{1} = stretch for coord 1 u_{1} z, \overset{x}{^}_{2} = stretch for coord 2 u_{2} z$

The $u$ 's are a second set of knobs.

WHY this shape. With one number in and two out, the only honest thing to do is place the guess somewhere on a single line — the line traced out as $z$ slides. So the decoder can only ever output points on the line spanned by the vector $[u_{1}, u_{2}]$ .

PICTURE. As $z$ sweeps from negative to positive, $\hat{x}$ walks along the plum "reconstruction line". Notice: the decoder can only reach points on that line. If the real data does not sit on it, there will be error — and that error is the lever that trains the network.

Step 4 — Measure the mistake: the reconstruction error

WHAT. For one data point we compare the truth $x$ against the rebuild $\hat{x}$ and measure the gap:

$L = miss on coord 1 (x_{1} - \overset{x}{^}_{1})^{2} + miss on coord 2 (x_{2} - \overset{x}{^}_{2})^{2} = ∥ x - \hat{x} ∥^{2}$

The symbol $∥ \cdot ∥^{2}$ (read "squared length") just means add up the squared misses in every coordinate. It is the squared straight-line distance between the true dot and its rebuild.

WHY squared, not plain distance. Two reasons, both visual. (1) Squaring removes the sign — an overshoot and an undershoot both cost. (2) Squaring is smooth: it has no sharp corner at zero, so we can slide downhill on it (Step 6). Plain absolute distance has a kink at zero that trips gradient descent.

PICTURE. The short red segment connects each true dot to its rebuild on the plum line. The loss is the sum of the squares of all these red lengths. Training will try to make every red segment as short as possible.

Step 5 — Why this formula and not another: the Gaussian argument

WHAT. We claimed the natural loss is squared distance. Here is why that specific choice, not something arbitrary. Assume the data was the ideal reconstruction $\hat{x}$ plus a little random blur — bell-curve ("Gaussian") noise. The probability of seeing the actual $x$ is then

$p (x ∣ \hat{x}) \propto exp (- \frac{1}{2 σ ^{2}} squared miss ∥ x - \hat{x} ∥^{2})$

Here $exp$ is the exponential $e^{(\dots)}$ ; $σ$ (Greek "sigma") is how wide the blur is.

WHY the exponential and the log. The bell curve is tall in the middle and falls off. We want the reconstruction that makes the observed data most probable — that means making the exponent's inside as small as possible. Because $exp$ is stubborn to work with, we take its logarithm. Logarithm is the perfect tool here for one reason: it is monotonic (bigger input → bigger output), so it never moves where the maximum is, and it turns $exp (- x^{2})$ into a clean $- x^{2}$ :

$- lo g p (x ∣ \hat{x}) = \frac{1}{2 σ ^{2}} ∥ x - \hat{x} ∥^{2} + const$

Dropping the constant scaling, minimizing this negative log is exactly minimizing squared error. The loss was not a guess — it fell out of "assume bell-curve noise".

PICTURE. The teal bell curve peaks where $\hat{x} = x$ . Below it, its upside-down log is a plum parabola with its lowest point in the same place. Same minimum, friendlier shape.

Step 6 — Slide downhill: how the knobs actually move

WHAT. We now have a number $L$ that depends on the knobs $w_{1}, w_{2}, u_{1}, u_{2}$ . Training nudges each knob a tiny bit in the direction that lowers $L$ . The rule for "which direction lowers it" is the gradient, written $\frac{\partial L}{\partial w}$ — "how much does the loss change if I wiggle this one knob".

WHY the gradient. On a hilly landscape, the gradient points straight uphill; its negative points straight downhill. Following it repeatedly is the fastest local way to the valley floor — the smallest error.

The loss depends on $w_{1}$ only through the chain: change $w_{1} \to$ changes $z \to$ changes $\hat{x} \to$ changes $L$ . So we multiply the links (the chain rule):

$\frac{\partial L}{\partial w _{1}} = loss vs rebuild \frac{\partial L}{\partial x ^} \cdot rebuild vs code \frac{\partial x ^}{\partial z} \cdot code vs knob \frac{\partial z}{\partial w _{1}}$

PICTURE. The loss surface over two knobs is a bowl; the plum path shows the knobs rolling down to the valley floor, where the red segments of Step 4 are as short as they can be.

Step 7 — The narrow neck is the whole point (the bottleneck)

WHAT. Everything above forced $z$ to be one number for a two-number input. That squeeze is the bottleneck. What if we widened it — let $z$ have as many numbers as $x$ ?

WHY narrowness matters. If the code $z$ is as wide as the input, the network can cheat: encoder copies $x$ into $z$ , decoder copies $z$ back out. Zero error, zero learning — it memorized instead of understanding. The narrow neck makes copying impossible, so the only way to score well is to discover the thin line the data lives on.

PICTURE. Left panel: wide neck — a full-color pass-through pipe, data goes straight through unchanged. Right panel: narrow neck — the pipe pinches to a single wire, forcing the data through one number and out again onto the learned line.

Step 8 — The degenerate cases (never leave the reader stranded)

WHAT & WHY. We check the corners so nothing surprises you.

Neck too wide ( $d_{z} \geq d_{x}$ ): identity map, no learning — the trap of Step 7.
Neck exactly right ( $d_{z} = d_{intrinsic}$ ): here $d_{intrinsic}$ is the true number of degrees of freedom in the data (1 for our line). Error can reach zero and the code is meaningful. Sweet spot.
Neck too narrow ( $d_{z} < d_{intrinsic}$ ): not enough wires to carry the data's real freedom. Some information is forced to be dropped; error can no longer reach zero. The rebuild collapses onto a lower shape.
All data at one point (zero spread): nothing to compress; $z$ can be constant and the decoder just outputs that point. Loss $= 0$ trivially, code carries no information — a reminder that autoencoders only earn their keep when data has variation.

PICTURE. Reconstruction error plotted against $d_{z}$ : it drops steeply, then hits a flat floor at the "elbow". The elbow marks $d_{intrinsic}$ — spending more wires past it buys almost nothing.

Recall Quick check: what is

$d_{z}$ at the elbow telling you? The elbow position ::: the data's intrinsic dimensionality — the smallest code size that still reconstructs well.

The one-picture summary

The whole machine on one canvas: input dot $x$ → squeeze through the encoder direction into a single number $z$ → decoder stretches $z$ back onto its line as $\hat{x}$ → the red gap between them is the squared-distance loss, and rolling the knobs downhill shortens every red gap until the decoder's line snaps onto the data's line.

Recall Feynman retelling — say it to a 12-year-old

Imagine you must describe where every ant sits on a nearly-straight garden path, but you're only allowed to send one number per ant to a friend. You'd send "how far along the path" each ant is, and your friend redraws the ant on the path. If your description of the path is good, the redrawn ants land almost exactly where the real ones were; the tiny distances they land off is your mistake score (that's the squared-distance loss). You keep adjusting two things — the rule for reading the position (encoder knobs) and the drawn path itself (decoder knobs) — always nudging in whatever direction makes the total mistake smaller (that's gradient descent). The one-number limit is the trick: it's exactly what stops you from just photographing the whole garden and forces you to learn the path. If you allowed two numbers, you'd cheat and photograph everything, learning nothing. That single-wire squeeze is the bottleneck, and finding its right width is finding how few numbers the data truly needs.

Where to go next: Latent Space Representations (what the code $z$ means), Variational Autoencoders (VAE) (make the code smooth and generative), Sparse Autoencoders (a different squeeze), Dimensionality Reduction Techniques and Unsupervised Learning (the family this belongs to), and Encoder-Decoder Architectures (the general pattern).