Saving and loading models (checkpoints): Question bank

Two words we'll lean on constantly:

Model parameters = the learned numbers (weights $W$ , biases $b$ ) that produce predictions.
Optimizer state = the extra bookkeeping the optimizer keeps between steps. For SGD with momentum this is one velocity tensor $v_{t}$ (a running blend of past gradients). For Adam it is two tensors, written $m_{t}$ (running average of gradients) and $s_{t}$ (running average of squared gradients). Optimizer state is not part of the model — it's part of how the model is being trained.

The picture below is the one intuition to hold onto: what actually lives inside a checkpoint, and which parts each use needs.

Figure — Saving and loading models (checkpoints)

True or false — justify

A checkpoint that saves only the model's weights is enough to deploy the model for predictions

True — predictions depend only on parameters, so weights (plus matching architecture) fully define inference behaviour. It is not enough to resume training smoothly, which also needs optimizer state.

Saving model.state_dict() also saves the model's architecture

False — the state dict is just a name→tensor mapping of parameters; it carries no layer definitions, so you must re-create the same architecture in code before loading.

If two checkpoints have identical weights, resuming from either gives identical next training steps

False — identical weights but different optimizer state (e.g. different accumulated momentum) produce different next updates, so training trajectories diverge.

torch.save(model, path) and torch.save(model.state_dict(), path) are interchangeable (PyTorch)

False — the first pickles the whole object and breaks if the class code moves or changes; the second stores only tensors and is portable across code refactors, which is why state dicts are preferred.

Calling model.eval() changes the model's weights (PyTorch)

False — eval() only flips the mode of dropout and batch-norm (stops dropping units, uses running stats); the weight tensors are untouched.

The SavedModel format lets you load a model without the original Python model-building code (TensorFlow/Keras)

True — it bundles architecture, weights, and the computational graph, so tf.keras.models.load_model reconstructs everything; that self-containment is why it suits Model Deployment.

Saving every epoch is always the safest strategy

False — safety-vs-cost is a tradeoff; per-epoch saves of a large model burn huge disk (Model Size × Frequency × Duration), so "best-only" or a rolling window is usually smarter.

restore_best_weights=True in early stopping means training continued to the best epoch (Keras)

False — training may stop later (after patience epochs of no improvement), but the flag rolls the weights back to the best-scoring epoch seen, discarding the worse tail.

Optimizer state is tiny compared to the model, so it barely affects checkpoint size

False — Adam stores two extra tensors (

m_{t}

s_{t}

) the same shape as the parameters, so a full training checkpoint is roughly 3× the weight-only size.

Loading weights saved on a GPU into a CPU-only machine works automatically (PyTorch)

False — tensors carry their device; you must pass map_location='cpu' or loading fails trying to place tensors on an absent device.

Spot the error

"I only saved model.state_dict(), then resumed training with a fresh Adam optimizer — same as if I never stopped." (PyTorch)

Adam's

m_{t}, s_{t}

reset to zero, so momentum/adaptivity is lost; the first steps behave like a cold start and can jolt or destabilise training, unlike true continuation.

"To resume, I set start_epoch = checkpoint['epoch'] and looped from there."

Off-by-one bug — the saved epoch already finished, so you'd re-run it; use start_epoch = checkpoint['epoch'] + 1 to pick up at the next epoch.

"I sorted checkpoint files with sorted(glob('checkpoint_*.pt')) and named them checkpoint_7.pt, checkpoint_10.pt."

String sort puts 10 before 7 (lexicographic), so [-1] grabs the wrong file; zero-pad names (checkpoint_0010.pt) so alphabetical order matches numeric order.

"My learning-rate scheduler jumped back to the initial rate after resuming."

The scheduler's step counter is part of training state; if you didn't save/restore it (or the optimizer/scheduler state), it restarts from step 0 — see Learning Rate Schedules for why that resets the whole decay curve.

"I changed a layer from 256 to 512 units, then loaded my old checkpoint — load_state_dict silently used it." (PyTorch)

It won't be silent; load_state_dict validates shapes and keys and raises a size-mismatch error, which is exactly the guardrail that catches architecture drift.

"I saved weights during training but forgot model.eval() before inference, and got different predictions each run." (PyTorch)

Dropout stays active in train mode, randomly zeroing units, so each forward pass differs; eval() disables that and uses batch-norm running statistics for deterministic inference.

"For deployment I shipped a checkpoint containing optimizer_state_dict."

Not wrong, but wasteful — inference never uses optimizer state, so a production checkpoint should drop it to shrink the file (keep weights, class names, metrics only).

Why questions

Why save the epoch/step counter alongside weights?

So resumption knows where in the schedule it is — this drives the learning-rate schedule position and prevents redoing or skipping epochs.

Why does momentum take about

1/ (1 - β)

steps to rebuild after a reset?

The velocity

v_{t}

is an exponential moving average controlled by

β

; roughly

1/ (1 - β)

recent gradients dominate it, so

β = 0.9

needs about

10

steps of accumulation before the average is "full" again.

Why does the "best-only" strategy usually suffice for finished experiments?

The end goal is the single highest-performing model; intermediate snapshots are only useful as fallbacks during a run, so keeping just the peak avoids storing dozens of inferior copies.

Why prefer state dicts over pickling the whole nn.Module (PyTorch)?

It decouples data (parameters) from code (architecture), so you can refactor or move your model class and still load old checkpoints as long as layer names/shapes match.

Why use torch.no_grad() during inference (PyTorch)?

It stops PyTorch from building the autograd graph, saving memory and time since you never call backward at inference — gradients would be pure waste.

Why does Transfer Learning often load only part of a checkpoint?

You reuse a pretrained backbone's weights but swap the final layer for a new task, so you deliberately allow missing/unexpected keys (strict=False in PyTorch) instead of demanding an exact match.

Why give early stopping a patience greater than 1?

Validation loss is noisy; a single bad epoch isn't real regression, so patience lets the model ride out fluctuations and escape shallow local minima before you decide to stop.

Edge cases

What if training crashes during the save (partial file written)?

A half-written checkpoint may be corrupt/unloadable; safe pipelines write to a temp file and atomically rename, and keep the previous good checkpoint as a fallback.

What if the very first epoch is your best and you resume later — does best-only ever overwrite it?

No — best-only only writes when the monitored metric improves, so an early peak is preserved until something genuinely beats it.

What if you load a checkpoint whose optimizer type differs from your current one (e.g. saved Adam, resuming with SGD)?

The state dicts hold different keys/tensors, so load_state_dict will error or mis-match; optimizer state is only transferable between the same optimizer configuration.

What about a model with zero trainable parameters (all layers frozen)?

The parameter state dict is still valid but empty of grad-tracking entries; the optimizer sees nothing to update, so a checkpoint captures only the fixed weights — resuming is trivial since nothing changes.

What if you resume but the dataloader/random seed wasn't saved?

Batch order and augmentations differ from the original run, so results won't be bit-for-bit reproducible even though weights/optimizer are restored — that's why seeds are part of full training state.

What happens if save_freq='epoch' but an epoch has millions of steps and the run dies mid-epoch?

All progress since the last epoch boundary is lost; for very long epochs, checkpoint by step count instead so the fallback window is smaller.

Recall One-line self-test

Weights answer "what does the model predict?"; optimizer state answers "how do I keep training it well?" — deploy needs the first, resume needs both.