4.5.2 · D5Generative Models
Question bank — Autoencoders fundamentals
Before we start, three words we will lean on constantly:
- bottleneck — the narrowest layer, the latent vector , whose small size is what forces compression.
- reconstruction loss — how badly the rebuilt differs from the original .
- identity mapping — the lazy "solution" where the network just copies input to output, learning nothing useful.
True or false — justify
An autoencoder is a supervised learning method because it has a target output.
False — it is unsupervised. The "target" is the input itself, so no human-provided labels are needed; the network supervises itself.
If reconstruction loss reaches zero, the autoencoder has learned a good representation.
Not necessarily — zero loss with a wide bottleneck () just means it copied the input (identity mapping). A useful representation requires the bottleneck to force it to discard redundancy.
A linear autoencoder (no activation) with a -dimensional bottleneck learns the same subspace as top- PCA.
True — minimizing squared reconstruction error with linear maps spans the same principal subspace as PCA, though the learned axes may be rotated/scaled versions of the principal components.
Making the latent dimension larger always improves reconstruction quality.
False — once the network can memorize via identity mapping, so "quality" is meaningless; the goal is a constrained bottleneck, not a large one.
Autoencoders are generative models: you can sample the latent space to create brand-new data.
Mostly false — a plain autoencoder learns where real data maps, but leaves gaps and holes in latent space, so random samples usually decode to garbage. VAEs fix this by shaping the latent space into a known distribution.
The encoder and decoder must be exact mirror images of each other.
False — symmetry is a common convention that saves hyperparameters, but the architectures are independent; only the input and output dimensions must match.
If the data already fills its space with no redundancy, an autoencoder can still compress it losslessly.
False — compression exploits redundancy (intrinsic dimensionality below ). Truly random, structureless data cannot be squeezed without loss, by Shannon's source coding theorem.
BCE loss can be used for grayscale images normalised to .
True — BCE only requires outputs in and treats each pixel as a Bernoulli-like probability; it is a valid (and common) choice for normalised grayscale, not just strictly binary pixels.
Spot the error
"I'll use MSE loss with a sigmoid output on binary pixel images — that's the safe default."
The mismatch is the trap: for binary/probability data, BCE matches the Bernoulli likelihood and penalises confident wrong predictions sharply, while MSE penalises them only linearly. Use BCE here.
"My latent vector has 800 dimensions for 784-pixel MNIST images, so it should reconstruct perfectly."
With there is no bottleneck, so it "reconstructs perfectly" by learning identity — that is the failure, not the success. It has learned no structure.
"Reconstruction loss shrinks the gradient at the bottleneck, so a tiny is always best for stable training."
Half-right, half-wrong: too small a throttles gradient flow and discards essential detail, hurting reconstruction; the bottleneck must be small enough to compress but wide enough to carry the data's intrinsic dimensionality.
"Since the target equals the input, backpropagation isn't needed — the network trivially copies."
The target equalling the input does not make copying free; the information must survive squeezing through a narrow , and backprop through decoder → bottleneck → encoder is exactly what learns to do that.
"I set BCE loss but my decoder ends in a ReLU, and losses blew up."
BCE needs outputs in ; ReLU outputs , so and go undefined/negative. Use a sigmoid output for BCE.
Why questions
Why does a bottleneck prevent the network from just memorising the training set?
A layer narrower than the input cannot store every input verbatim, so the network is forced to find shared structure (redundancy) that compresses across examples — that shared structure is the learned representation.
Why do we take the negative log of the Gaussian likelihood to get MSE?
The Gaussian density is ; maximising it equals minimising its negative log, and — the squared-error term, with constants irrelevant to optimisation.
Why can real images (784 pixels) be compressed to ~32 numbers without disaster?
Because natural images live on a low-dimensional manifold — their intrinsic dimensionality ; most of is noise-like images that never occur.
Why do similar digits cluster together in a trained latent space?
To reconstruct many "3"s efficiently the encoder maps their shared features to nearby codes, so proximity in comes to mean visual/semantic similarity — even though no labels said so.
Why is BCE but when the true pixel is 1?
BCE measures surprise: predicting for a true is nearly right (small penalty), predicting is confidently wrong (large penalty). This asymmetric, probability-aware scaling is exactly why BCE suits Bernoulli data.
Why does a plain autoencoder's latent space have "holes" that decode badly?
Nothing in the reconstruction objective forces every region of to correspond to valid data — only the points where real inputs land are constrained, so in-between/unvisited regions decode to nonsense.
Edge cases
What happens if the input has zero redundancy and ?
Perfect reconstruction becomes impossible: there is no structure to exploit, so the bottleneck must discard genuine information and loss cannot reach zero.
What does the loss surface look like for a linear autoencoder trained on 2D points lying exactly on ?
The optimum learns weights with the ratio (and ), because the single latent number only needs to encode position along that one line — its lone degree of freedom.
What if hits exactly or under BCE while the true pixel is the opposite value?
The term diverges to (infinite loss / gradient). Frameworks clamp outputs or fuse sigmoid+BCE numerically to avoid ; conceptually, the model is being punished for being infinitely confident and wrong.
What does the reconstruction-error-vs- curve tell you, and what is the "elbow"?
Error falls steeply as grows, then flattens; the elbow is where extra latent dimensions stop helping — it estimates the data's intrinsic dimensionality and a good bottleneck size.
If you feed a trained autoencoder an input unlike anything in training (out-of-distribution), what happens?
It reconstructs poorly, because the encoder maps it to some region of the decoder never learned to invert well — high reconstruction error can therefore flag anomalies.
What is the degenerate behaviour when exactly?
The network can learn the identity function (, ), achieving zero loss while learning no useful compression — the boundary case just before the bottleneck disappears entirely.