Autoencoders fundamentals: Exercises — AI-ML deep dive

Before we start, one reminder of the three objects, since every problem leans on them:

Level 1 — Recognition

Exercise 1.1

A network maps $x \in R^{784}$ to $z \in R^{784}$ (same size, no shrinking), then back to $\hat{x} \in R^{784}$ . Is this a useful autoencoder? Answer yes/no and name the failure in one phrase.

Recall Solution

No. With $d_{z} = d_{x}$ there is no bottleneck, so the network can learn the identity mapping ( $f_{ϕ} (x) = x$ , $g_{θ} (z) = z$ ). It copies its input perfectly while learning no structure. The latent code is meaningless. The whole point is $d_{z} < d_{x}$ — the shrinking is what forces learning.

Exercise 1.2

Match each data type to the loss it should use: (a) grayscale photos with pixel values in $[0, 1]$ treated as continuous intensities; (b) strictly black-or-white pixels (each is exactly 0 or 1).

Recall Solution

(a) continuous → MSE (Mean Squared Error), because we assume Gaussian noise around the reconstruction.
(b) binary → BCE (Binary Cross-Entropy), because each pixel is a coin-flip (Bernoulli), 0 or 1.

The rule: the loss must match the assumed noise model of the data, not personal preference.

Exercise 1.3

An autoencoder trains with no labels — the target it tries to match is the input itself. Which learning paradigm is this?

Recall Solution

Unsupervised learning — see Unsupervised Learning. The "auto" in autoencoder means it makes its own targets: the label is the input.

Level 2 — Application

Exercise 2.1

A single pixel has true value $x = 1$ . The decoder outputs $\overset{x}{^} = 0.9$ . Compute the per-pixel BCE: $L_{BCE} = - [x lo g (\overset{x}{^}) + (1 - x) lo g (1 - \overset{x}{^})] .$ Then recompute for $\overset{x}{^} = 0.1$ . Which prediction is punished harder, and by roughly how many times?

Recall Solution

With $x = 1$ , the second term $(1 - x) lo g (1 - \overset{x}{^})$ vanishes, leaving $- lo g (\overset{x}{^})$ .

$\overset{x}{^} = 0.9$ : $- lo g (0.9) = 0.1054$ (using natural log $ln$ ).
$\overset{x}{^} = 0.1$ : $- lo g (0.1) = 2.3026$ . Ratio $2.3026/0.1054 \approx 21.8$ . The confident-but-wrong guess ( $0.1$ when truth is $1$ ) is punished about 22× harder. That steepness is exactly why BCE suits binary data.

Exercise 2.2

Two pixels: truth $x = [1.0, 0.0]$ , reconstruction $\hat{x} = [0.8, 0.3]$ . Compute the MSE $L_{MSE} = \frac{1}{n} \sum_{i = 1}^{n} (x_{i} - \overset{x}{^}_{i})^{2} .$

Recall Solution

Errors: $(1.0 - 0.8) = 0.2$ and $(0.0 - 0.3) = - 0.3$ . Squares: $0.04$ and $0.09$ . Sum $= 0.13$ . Divide by $n = 2$ : $L_{MSE} = \frac{0.13}{2} = 0.065.$

Exercise 2.3

Compression ratio. An MNIST autoencoder maps $d_{x} = 784$ down to $d_{z} = 32$ . Compute the compression factor $d_{x} / d_{z}$ , rounded to one decimal.

Recall Solution

$\frac{d _{x}}{d _{z}} = \frac{784}{32} = 24.5.$ So the latent code is a ~24.5× shrink. The parent note called it "~25× compression" — same thing rounded.

Level 3 — Analysis

Exercise 3.1

A linear autoencoder (no activation functions) is trained on data lying exactly on the line $x_{2} = 3 x_{1}$ in $R^{2}$ , with a 1-dimensional bottleneck. Encoder $z = w_{1} x_{1} + w_{2} x_{2}$ , decoder $\overset{x}{^}_{1} = u_{1} z, \overset{x}{^}_{2} = u_{2} z$ . What ratio $u_{2} / u_{1}$ must the network learn, and why?

Recall Solution

Every valid data point has the form $(t, 3 t)$ for some scalar $t$ — one degree of freedom. The single number $z$ must stand in for $t$ . On decode, $\overset{x}{^}_{1} = u_{1} z$ and $\overset{x}{^}_{2} = u_{2} z$ , so the decoder output always sits on the line $\overset{x}{^}_{2} = (u_{2} / u_{1}) \overset{x}{^}_{1}$ . To land on the true data line we need $\frac{u _{2}}{u _{1}} = 3.$ Geometric picture: the decoder can only produce points along one direction it picks. Training forces that direction to align with the data line — this is exactly PCA finding the first principal axis. See figure below.

Exercise 3.2

A team observes that increasing $d_{z}$ from $16$ to $256$ keeps lowering the reconstruction error smoothly toward zero, and they conclude "bigger latent is strictly better." What is the flaw in this reasoning, and what one diagnostic plot exposes it?

Recall Solution

Lower reconstruction error is not the goal — useful, compressed structure is. As $d_{z} \to d_{x}$ , error falls to zero because the net approaches the identity map, storing the input verbatim instead of learning structure. Error-goes-down is consistent with the worst outcome (memorisation). Diagnostic: plot reconstruction error vs $d_{z}$ and find the elbow — the point after which extra dimensions barely help. Choose $d_{z}$ at the elbow, well below $d_{x}$ . The elbow marks the data's intrinsic dimensionality.

Exercise 3.3

Using the capacity bound from the parent note, $d_{z} lo g_{2} (L_{z}) \geq I (x)$ : suppose an image truly needs $I (x) = 40$ bits of information to specify, and each latent unit can carry $L_{z} = 4$ distinct levels. What is the smallest integer $d_{z}$ that can still perfectly encode it?

Recall Solution

Each unit carries $lo g_{2} (4) = 2$ bits. We need $d_{z} \cdot 2 \geq 40 \Rightarrow d_{z} \geq 20.$ Smallest integer: $d_{z} = 20$ . Fewer than 20 units cannot hold 40 bits at 2 bits each — reconstruction must lose information (Shannon's source-coding floor).

Level 4 — Synthesis

Exercise 4.1

Design (on paper) a symmetric autoencoder for $28 \times 28$ binary images. Specify: input dim, one hidden layer each side, latent dim = 16, activations, and output activation. Justify the output activation and the loss.

Recall Solution

Input: $28 \times 28 = 784$ flattened to $x \in R^{784}$ .
Encoder: $784 \to 128$ (ReLU) $\to 16$ (linear).
Decoder: $16 \to 128$ (ReLU) $\to 784$ (sigmoid).
Loss: BCE. Why sigmoid on output? BCE needs each output in $(0, 1)$ to read as a probability "this pixel is on"; sigmoid squashes any real number into $(0, 1)$ . Why BCE not MSE? Pixels are binary (Bernoulli), so BCE is the matched likelihood — it punishes confident wrong pixels far more (recall Exercise 2.1). This mirrors the MNIST design in the parent note; see Encoder-Decoder Architectures.

Exercise 4.2

You have two trained latent codes $z_{A}$ and $z_{B}$ for two different digits. Write the formula for the midpoint interpolation and describe what decoding it should produce in a well-trained autoencoder.

Recall Solution

Linear interpolation with mixing weight $α \in [0, 1]$ : $z_{α} = (1 - α) z_{A} + α z_{B} .$ The midpoint is $α = 0.5$ : $z_{0.5} = \frac{1}{2} (z_{A} + z_{B})$ . Decoding $g_{θ} (z_{0.5})$ should give a plausible in-between image — a smooth morph — if the latent space is smooth. Plain autoencoders often produce blurry or nonsensical midpoints because they don't enforce smoothness; Variational Autoencoders (VAE) add that constraint.

Exercise 4.3

Convert the plain autoencoder above into a sparse autoencoder conceptually. What single new term is added to the loss and what does it encourage?

Recall Solution

Add a sparsity penalty to the loss: $L_{total} = L_{BCE} + β \cdot Ω (h),$ where $Ω$ penalises the average activation of hidden units (e.g. an L1 term $\sum_{j} ∣ h_{j} ∣$ or a KL term pushing mean activation toward a small target $ρ$ ), and $β$ controls its strength. This forces few units active at once, so each unit learns a distinct feature — the Sparse Autoencoders idea. Note it lets you keep a wide hidden layer while still avoiding the identity trap, because sparsity (not size) does the constraining.

Level 5 — Mastery

Exercise 5.1

Prove that for continuous data with Gaussian noise $p (x ∣ \hat{x}) = N (x ∣ \hat{x}, σ^{2} I)$ , maximising the likelihood is equivalent to minimising squared error, and show the noise variance $σ^{2}$ does not change the optimum.

Recall Solution

The Gaussian density (dropping the normalising constant $C$ that doesn't depend on $\hat{x}$ ) is $p (x ∣ \hat{x}) = C exp (- \frac{1}{2 σ ^{2}} ∥ x - \hat{x} ∥^{2}) .$ Step 1 (why log): maximising a product/exponential is awkward; the logarithm is monotone increasing, so $ar g max p = ar g max lo g p$ . Take $lo g$ : $lo g p = lo g C - \frac{1}{2 σ ^{2}} ∥ x - \hat{x} ∥^{2} .$ Step 2 (why negate): we minimise by convention, so flip sign. Since $lo g C$ is a constant it drops out of $ar g min$ : $ar g max_{\hat{x}} p = ar g min_{\hat{x}} \frac{1}{2 σ ^{2}} ∥ x - \hat{x} ∥^{2} .$ Step 3 (why $σ^{2}$ vanishes): $\frac{1}{2 σ ^{2}}$ is a positive constant multiplier. Scaling an objective by a positive constant never moves its minimiser. Hence $ar g min \frac{1}{2 σ ^{2}} ∥ x - \hat{x} ∥^{2} = ar g min ∥ x - \hat{x} ∥^{2} = MSE optimum . ■$

Exercise 5.2

For BCE with a single pixel and true value $x$ , show that the loss is minimised exactly at $\overset{x}{^} = x$ . (Treat $x$ as a fixed number in $(0, 1)$ and set the derivative to zero.)

Recall Solution

Per-pixel loss: $ℓ (\overset{x}{^}) = - [x lo g \overset{x}{^} + (1 - x) lo g (1 - \overset{x}{^})]$ . Differentiate with respect to $\overset{x}{^}$ (using $\frac{d}{d x ^} lo g \overset{x}{^} = 1/ \overset{x}{^}$ and chain rule on $lo g (1 - \overset{x}{^})$ ): $\frac{d ℓ}{d x ^} = - \frac{x}{x ^} + \frac{1 - x}{1 - x ^} .$ Set to zero: $\frac{x}{x ^} = \frac{1 - x}{1 - x ^} \Rightarrow x (1 - \overset{x}{^}) = (1 - x) \overset{x}{^} \Rightarrow x - x \overset{x}{^} = \overset{x}{^} - x \overset{x}{^} \Rightarrow x = \overset{x}{^} .$ The second derivative is positive (both terms are positive), so this is a minimum. Thus BCE is honestly minimised only when the guess equals the truth. $■$

Exercise 5.3

Generalise the capacity argument: a dataset has intrinsic dimensionality $d_{intrinsic}$ . Argue why setting $d_{z} < d_{intrinsic}$ guarantees some reconstruction error no matter how long you train, while $d_{z} \geq d_{intrinsic}$ permits (does not guarantee) zero error.

Recall Solution

By definition, the data lives on a manifold needing $d_{intrinsic}$ independent coordinates to specify a point. The decoder is a function $g_{θ} : R^{d_{z}} \to R^{d_{x}}$ ; its image (the set of reconstructable points) has dimension at most $d_{z}$ .

If $d_{z} < d_{intrinsic}$ : the decoder's reachable set is a lower-dimensional sheet than the data manifold, so it cannot cover every data point. Some points are unreachable ⇒ nonzero error for those, guaranteed, however long you train. This is the Shannon floor $d_{z} lo g_{2} L_{z} < I (x)$ .
If $d_{z} \geq d_{intrinsic}$ : the decoder's image can be made large enough to contain the whole manifold, so zero error is possible — but only if training actually finds such weights. Enough room ≠ guaranteed success (optimisation may still fail). Hence "permits", not "guarantees". $■$