Foundations — Autoencoders fundamentals

This page is the toolbox. The parent note throws a lot of symbols at you at once — $\mathbf{x}$, $\mathbf{z}$, $f_{\varphi }$, $\Vert \cdot {\Vert }^2$, $\sigma$, $\log$, $\mathcal{N}$. Here we build each one from nothing, in an order where every symbol is earned before the next uses it. A smart 12-year-old should be able to read from line one.

1. A number vs. a list of numbers: the vector

The picture. A vector with two components is an arrow in a flat plane: $x_1$ tells you how far right, $x_2$ how far up. A vector with three components is an arrow in 3D space. Look at figure s01: the same list of numbers is both "a point" and "an arrow from the origin to that point."

Why the topic needs it. A grayscale image is 28 pixels wide and 28 tall = $784$ little brightness numbers. Line them all up and you get one giant vector $\mathbf{x}=[x_1,x_2,\dots ,x_{784}]$. So an image is a vector, and an autoencoder is a machine that eats and spits out vectors.

2. The space a vector lives in: ${\mathbb{R}}^n$

• ${\mathbb{R}}^2$ = every point in a flat plane (2 numbers each).
• ${\mathbb{R}}^3$ = every point in ordinary space (3 numbers each).
• ${\mathbb{R}}^{784}$ = every possible $784$-number list, i.e. every possible $28\times 28$ image.

The picture. Think of ${\mathbb{R}}^n$ as a room where each vector is one specific spot. $n$ is the number of directions you can move in. We can only draw up to 3 directions, but the idea works for 784.

Why the topic needs it. When the parent writes $\mathbf{x}\in {\mathbb{R}}^n$, the symbol $\in$ just means "is a member of / lives inside." So $\mathbf{x}\in {\mathbb{R}}^{784}$ reads: "$\mathbf{x}$ is one image, sitting somewhere in the room of all possible images." The whole point of an autoencoder is to move this vector into a much smaller room.

3. A function that moves between rooms: $f:\mathcal{X}\to \mathcal{Z}$

The picture (figure s02). Draw a big box labelled $\mathcal{X}$ (all images) and a small box labelled $\mathcal{Z}$ (all codes). The encoder $f_{\varphi }$ is an arrow carrying a point from the big box to the small box. The decoder $g_{\theta }$ is an arrow carrying it back. That round trip — big → small → big — is the autoencoder.

Why the tiny letters $\varphi$ and $\theta$? These Greek letters (say "fy" and "thay-ta") are the knobs of the function — its adjustable settings, called parameters. $f_{\varphi }$ means "the encoder function, currently set to knob-values $\varphi$." Training = twisting those knobs until the round trip works. We meet the actual knobs (weights) in section 6.

4. Measuring "how far apart": distance and $\Vert \cdot \Vert$

The whole reason we train is to make the rebuilt vector $\hat{\mathbf{x}}$ close to the original $\mathbf{x}$. So we need to measure closeness.

The picture (figure s03). Two arrows, $\mathbf{x}$ (blue) and $\hat{\mathbf{x}}$ (orange). The red arrow between their tips is the error vector; its length $\Vert \mathbf{x}-\hat{\mathbf{x}}\Vert$ is the single number "how far off were we?"

Why squared and not the plain length? Two reasons, and both matter:

1. Squaring kills the square root, giving a smooth sum that is easy to differentiate (section 6 needs derivatives).
2. Squaring punishes big mistakes far more than small ones — a pixel off by $2$ costs $4$, off by $10$ costs $100$. This pushes the network to fix its worst errors first.

The symbol ${\sum }_{i=1}^n$ (capital Greek sigma) just means "add up, letting the counter $i$ run from $1$ to $n$." So ${\sum }_{i=1}^n(x_i-{\hat{x}}_i{)}^2$ = "add the squared error of every component."

5. The bending rule: the activation $\sigma$

The picture (figure s04). ReLU is a flat line then a ramp; sigmoid is a smooth S-curve pinned between 0 and 1.

Why the topic needs bending. Stacking only straight-line (linear) steps still gives a straight line — a linear autoencoder can only ever do PCA (the parent's Worked Example 1). Real images live on curved shapes, so we need a curved rule to bend the mapping. That is exactly what $\sigma$ adds.

Why sigmoid at the decoder's output? Pixel brightness lives in $[0,1]$. Sigmoid guarantees the rebuilt pixels land in $[0,1]$ — which is also what the BCE loss (next section) demands.

6. Knobs, slopes, and the chain: weights, $\log$, derivatives

Three last tools the parent leans on.

Why the topic needs derivatives. Training = "twist each knob in the direction that lowers the loss." The derivative points in that direction. The chain rule is what lets the signal travel backward from the output all the way through the bottleneck to the first encoder knob — the parent's "gradient flows through the bottleneck."

Prerequisite map

This map feeds directly into the Encoder-Decoder Architectures and Latent Space Representations topics, and underpins the whole family: Variational Autoencoders (VAE), Sparse Autoencoders, Dimensionality Reduction Techniques, and the broader idea of Unsupervised Learning.

Equipment checklist

Read each question, answer in your head, then reveal.