Worked examples — TensorBoard - Weights & Biases logging

This page is a workbook. The parent note TensorBoard / W&B logging told you how to call add_scalar and wandb.log. Here we grind through every kind of situation those calls can land you in — the clean cases, the weird edge cases, and the ones that silently break your plots.

Before line one, three plain-word anchors so nothing sneaks in undefined:

Keep that picture: a chart is (tag); a dot is (step, scalar). Almost every bug on this page is a step or a tag doing something unexpected.

---

The scenario matrix

Every logging situation is one cell below. The worked examples underneath are each labelled with the cell(s) they cover, and together they touch all of them.

Axis	Cases you must handle
Step direction	(A) monotonically increasing step ✓ · (B) step resets / repeats · (C) step goes backwards
Step value	(D) step = None (auto) · (E) step is a float (illegal) · (F) two metrics on different step clocks
Value health	(G) finite number ✓ · (H) NaN / inf · (I) a tensor with grad instead of a python float
Cadence	(J) log every batch · (K) log every N batches (must scale step)
Multi-run	(L) comparing runs (same tags, separate dirs)
Non-scalar	(M) histogram for gradient diagnosis · (N) hyperparameter sweep summary
Word problem	(O) "why does my val curve look shifted left?" real debugging
Exam twist	(P) resumed training after a crash — continue the step counter correctly

Twelve examples below cover cells A–P.

---

Example 1 — The clean baseline (cells A, G, J)

Forecast: guess the five dots before reading on.

1. List the (step, value) pairs. (0, 2.30), (1, 1.80), (2, 1.40), (3, 1.15), (4, 1.02). Why this step? epoch is the step, so the x-coordinates are exactly 0..4. This is the healthy case: step increases by 1, values are finite.

2. Check the trend. Each value is smaller than the last, so the line slopes down — training is converging. Why this step? A down-sloping Loss/train is the first sanity check TensorBoard gives you; if it slopes up, jump to the gradient-explosion diagnosis (Example 6).

3. Compute the total drop. $2.30-1.02=1.28$. Why this step? Quantifies "how much did we learn", useful when comparing runs.

Verify: 5 dots, x-values 0,1,2,3,4 (strictly increasing ✓), y strictly decreasing ✓, drop $=1.28$.

---

Example 2 — Letting the writer auto-step (cell D)

Forecast: if you never say where to draw the dot, where does it land?

1. What "no step" means. add_scalar's step argument defaults to None. When it is None, the SummaryWriter uses an internal counter it increments by 1 on each call for that tag. Why this step? You must know the machine invents a step for you — it does not skip the dot.

2. Enumerate the auto steps. Calls 1–5 receive internal steps 0, 1, 2, 3, 4. Why this step? This coincidentally matches "one call per epoch". Auto-stepping is fine only when you log a tag exactly once per iteration.

3. The trap. If you log the same tag twice in one iteration (say train then a debug print), auto-step gives them 0,1 then 2,3 — your "epochs" now advance by 2, silently doubling the x-axis. That is why explicit step (Example 1) is safer. Why this step? Shows the one situation where step=None bites you, so you know when not to rely on it.

Verify: 5 auto-stepped calls produce steps [0,1,2,3,4]; two-calls-per-iteration produces [0,1,2,3,...] where iteration $i$'s train dot sits at step $2i$, not $i$.

---

Example 3 — Passing a float step (cell E)

Forecast: does TensorBoard round it, crash, or accept the float as-is?

1. What the API expects. The step (called global_step) is documented as an int. A step is a count of iterations — counts are whole numbers, so a fractional step is meaningless. Why this step? Naming the type contract explains why a float is rejected rather than rounded.

2. What actually happens. PyTorch's SummaryWriter casts the value; a clean float like 100.0 becomes int 100, but a value like 100.5 is truncated to 100, silently colliding with the real step 100. In stricter loggers (and wandb.log(..., step=100.5)) it raises a TypeError. Why this step? You need to know it fails quietly in the worst case — a collision, not an error.

3. The fix. Keep step integer by construction: use integer arithmetic step = epoch * len(train_loader) + batch_idx (Example 5), never division. Why this step? Removes the float at the source instead of relying on the caster.

Verify: int(100.0) == 100 (harmless) but int(100.5) == 100 too (collision with the true step 100); the two distinct floats 100.0 and 100.5 map to the same int step.

---

Example 4 — Logging every N batches, scaling the step (cells A, K)

The parent's training loop logs every 100 batches. If you naively pass batch_idx as the step, epoch 2's batch 5 collides with epoch 1's batch 5 (same step, different data). The fix is the global step.

Figure s01: three coloured horizontal bands, one per epoch (lavender=epoch 0, coral=epoch 1, mint=epoch 2). Each band holds four dots at its logged global steps — 0,100,200,300 then 400,500,600,700 then 800,900,1000,1100. The bands sit side by side and never overlap, showing that multiplying epoch by 400 pushes each epoch's dots past the previous block.

Forecast: how many dots total, and what is the largest step?

1. Find the logging batch indices in one epoch. batch_idx ∈ {0, 100, 200, 300} (those satisfy % 100 == 0 and are < 400). Why this step? These are the only batches where the if fires — 4 per epoch.

2. Convert to global steps for each epoch. Using $g=400e+b$:

   • Epoch 0: $0,100,200,300$
   • Epoch 1: $400,500,600,700$
   • Epoch 2: $800,900,1000,1100$ Why this step? Multiplying epoch by len(train_loader) shifts each epoch's block past the previous one — look at the figure: the three coloured bands never overlap. This is exactly why we scale the step.

3. Count and find max. $4\times 3=12$ dots; largest step $=1100$. Why this step? Confirms no collisions — 12 distinct increasing x-values.

Verify: 12 dots, all distinct, monotonically increasing, max $=1100$. Units: step is a count, dimensionless.

---

Example 5 — Two metrics on different clocks (cell F)

Forecast: true or false — the curves are comparable point-for-point.

1. Identify the two step clocks. Train uses global_step (0..1199 over 3 epochs); val was stamped with raw epoch (0,1,2). Why this step? Two different x-axes are hiding here. Train's clock ticks 400 times per epoch; the raw-epoch clock ticks once. They are not the same ruler.

2. Translate the raw-epoch stamp onto the train clock. If val is stamped with plain epoch = e, that dot lands at train-step $e$ — i.e. val "epoch 1" sits at train step 1, deep inside epoch 0's very first batches. That is the bug we are exposing: raw epoch $e$ maps to the wrong place, $e$, instead of where epoch $e$ actually ended. Why this step? This makes the mismatch concrete: the naive stamp $e$ and the correct stamp are different numbers, and step 3 says exactly what the correct one is.

3. The fix. An epoch $e$ finishes after its last batch, at global step $(e+1)\cdot \text{len(train\_loader)}-1$. So stamp val with that: writer.add_scalar('Loss/val', val_loss, (epoch+1)*len(train_loader)-1). For epoch 0 that is step $399$ — the end of the first epoch, exactly where training was when validation ran. Why this step? The discrepancy between sub-step 2's $e$ and this $(e+1)\cdot 400-1$ is the whole lesson: raw epoch counts the epoch index, but the chart's x-axis counts batches, and one epoch equals 400 batches. You must convert index → batch-count.

Verify: For epoch index e, corrected val step $=(e+1)\cdot 400-1$. Epoch 0 → 399, epoch 1 → 799, epoch 2 → 1199. Strictly increasing, aligned with train's block ends. ✓ See 2.4.7-Gradient-flowand-vanishing-exploding-gradients for why misaligned curves fooled you about overfitting.

---

Example 6 — Logging a tensor instead of a float (cell I)

Forecast: does it crash, silently leak memory, or log fine?

1. What loss actually is. It's a 0-dim tensor plus an entire computation graph pinned to it (the graph is what backward() walks). Why this step? You must see the hidden baggage: logging the tensor keeps that whole graph alive.

2. The consequence. If TensorBoard holds a reference to the graph-bearing tensor, the graph can't be freed → memory grows every step → eventual OOM. Even when it works, you're storing a fat object where a number belongs. Why this step? This is the classic "why does my training slow down and die at epoch 30" bug.

3. The fix: loss.item() — extract the plain python float, detached from the graph. Why this step? .item() reads the single number and drops the graph. One token, bug gone.

Verify: After loss.item() you hold a float; type(loss.item()) is float → the logged object has no .grad_fn, graph is collectable.

---

Example 7 — NaN / inf values (cell H)

Forecast: will the loss curve show a giant spike, then what?

1. What inf plots as. TensorBoard drops non-finite points (or shows a gap). You get a curve that stops at step 6. Why this step? A curve that "ends early" is the signature of a divergence, not a bug in logging.

2. Trace the story. Finite losses 2.3, 2.1, ... (step 0–6), then inf (step 7), then NaN (step 8+). Once a NaN appears, gradients are NaN, weights become NaN, everything downstream is NaN. Why this step? NaN is contagious: $\text{anything}\times \text{NaN}=\text{NaN}$. The disease never heals on its own.

3. Diagnose the cause with the gradient-norm log. The GradNorm scalar from the parent's loop would spike toward $\infty$ one step before the loss does. That's your early-warning line. Why this step? Exploding gradients precede exploding loss — see 2.4.7-Gradient-flowand-vanishing-exploding-gradients.

Verify: In floating point, inf + 1 == inf, inf - inf is NaN, and NaN == NaN is False (that last fact is how code detects it: x != x is True only for NaN).

---

Example 8 — Histogram for a vanishing gradient (cell M)

Figure s02: two overlaid gradient histograms. The mint curve (last layer) is a wide, low bell — many different gradient magnitudes, so that layer is learning. The coral curve (first layer) is a tall thin spike centred at 0 — almost all its gradients are ≈ 0, the vanishing-gradient signature. An arrow points at the coral spike labelled "collapsed to ~0 = vanished gradient".

Forecast: which layer has stopped learning — first or last?

1. Read a histogram. A histogram is a spread of numbers: wide = many different gradient magnitudes; a thin spike at 0 = almost all gradients ≈ 0. Why this step? One scalar can't show this — you need the whole distribution, which is why the histogram tool exists (parent note, "Why each type").

2. Interpret the collapse. Layer 1's gradients ≈ 0 means its weights barely update → it has effectively stopped learning. This is the vanishing gradient signature. Why this step? The figure shows the front-layer distribution squeezing to zero while the back layer stays fat — exactly the depth-dependent decay of gradients.

3. What to check next. Activation function (sigmoid/tanh saturate), initialization, or add residual connections. Why this step? Ties the visual straight to a fix. See 2.4.7-Gradient-flowand-vanishing-exploding-gradients.

Verify: If gradient std shrinks by factor 0.5 per layer across 6 layers, the front layer's std is $0.5^6=0.015625$ of the back layer's — a 64× collapse, matching a near-spike.

---

Example 9 — Comparing runs (cell L)

Forecast: guess the winning lr before computing anything.

1. Why they overlay. Same tag (Accuracy/val) across sibling directories → TensorBoard treats each directory as a separate run and draws both on the shared chart, colour-coded. Why this step? This is the comparison mechanism — same tag, different dir. Rename the tag and they'd split into two charts (usually not what you want).

2. Pick the winner. $0.94>0.88$, so Run A (lr = 0.01) wins by $0.06$ accuracy. Why this step? Direct read-off; the gap quantifies how much the smaller lr helped here.

3. Generalise. This is the seed of a hyperparameter sweep — see 4.1.5-Hyperparameter-tuning and record the winner for 5.2.3-Model-versioning.

Verify: winner is Run A; margin $=0.94-0.88=0.06$ (6 percentage points). $0.94>0.88$ ✓.

---

Example 10 — Summarising a hyperparameter sweep (cell N)

Forecast: guess whether "bigger lr = better" or there's a sweet spot.

1. What add_hparams logs. Unlike add_scalar (a curve over steps), add_hparams logs one summary row per run: the hyperparameters as inputs and the final metrics as outputs. TensorBoard's HPARAMS tab shows a sortable table and a parallel-coordinates plot. Why this step? A sweep summary answers a different question than a training curve — "which config won?", not "how did this one config converge?" — so it uses a different call.

2. Read off the best row. Sort by val_acc: $0.94$ (lr=0.01) is highest. So the winning lr is $0.01$, not the largest. Why this step? Sorting is exactly what the HPARAMS table does for you; we do it by hand here.

3. See the shape. Accuracy rises 0.90 → 0.94 then falls 0.88 → 0.10: an inverted-U. Too small = slow learning; too large = divergence (that 0.10 is chance level on 10 classes). The peak is the sweet spot. Why this step? Recognising the inverted-U is the whole point of a sweep — the optimum is interior, never at an extreme. Feeds directly into 4.1.5-Hyperparameter-tuning.

Verify: best val_acc $=0.94$ at lr $=0.01$; sequence [0.90,0.94,0.88,0.10] rises then falls (inverted-U), and $0.10$ equals random-guess accuracy $1/10$ on 10 classes.

---

Example 11 — Word problem: the shifted val curve (cells O, F)

Forecast: is validation really "ahead", or is a step off by one?

1. Restate as steps. Train logs at global steps ending each epoch: $399,799,1199$. Val is logged at epoch = $0,1,2$ — but the teammate multiplied by len(train_loader) using the next epoch, landing val at $400,800,1200$, one batch right of train... yet the plot shows left. Why this step? Turning a vague complaint into concrete step numbers is the whole skill.

2. Find the real mistake. They logged val before incrementing epoch, so val for "epoch 1's data" was stamped with step $0\cdot 400=0$ — pushing every val point one whole epoch left of the matching train point. Why this step? Off-by-one in the step formula is the single most common cause of "shifted" curves.

3. The fix. Stamp val with the end-of-epoch global step: $(e+1)\cdot 400-1$ (same fix as Example 5). Why this step? Re-anchors val to where training actually finished that epoch.

Verify: wrong stamp for epoch $e$: $400e$ → epoch 1 gives 400. Correct stamp: $(e+1)400-1$ → epoch 1 gives 799. Difference $=799-400=399$ ≈ one epoch of batches. That 399-step gap is the visible "shift". ✓

---

Example 12 — Exam twist: resuming after a crash (cells B, C, P)

Forecast: will the new run's steps go backwards?

1. Trace the naive restart. New loop computes global_step = epoch*400 + batch_idx starting at $0$. But the event file already has data up to step $850$. New points land at $0,100,\dots$ — behind existing ones. Why this step? Steps that go backwards (case C) make TensorBoard draw a curve that loops back on itself — a tell-tale zig-zag.

2. Why it's bad. Two different weight-states now share step $100$ (pre-crash and post-resume). The chart is meaningless. Why this step? Same step, different data = collision, exactly the sin Example 4 fought to avoid.

3. Correct resume. Save the last step to the checkpoint; on reload set start_step = 850 and compute global_step = start_step + epoch*400 + batch_idx. The first post-resume dot lands at $850$ (or the next logged multiple, $950$), always greater than everything before it, so the step axis stays monotonically increasing across the crash — the curve continues instead of rewinding. Why this step? Preserving a single ever-increasing step counter is what makes a resumed run readable; store start_step next to the checkpoint so it survives the crash — a 5.2.3-Model-versioning concern.

Verify: first post-resume logged step $=850+0\cdot 400+0=850$ (next log at $850+100=950$), all $>850$ except the boundary equal — strictly increasing beyond it ✓.

---

Recall

Recall The step is the ___ coordinate of a logged dot; the scalar is the ___ .

Step is the x-coordinate; scalar is the y (height).

Recall Why pass

loss.item() instead of loss to add_scalar? .item() extracts a plain float and drops the autograd graph, avoiding a memory leak.

Recall You log train every batch and val every epoch. What must you do so the curves line up in time?

Stamp val with the end-of-epoch global step $(e+1)\cdot N-1$, not the raw epoch.

Recall On a

Loss/train chart the curve simply stops at step 6. Most likely cause? Divergence — an inf/NaN appeared at step 7 and TensorBoard drops non-finite points.

Recall What does

add_hparams log that add_scalar does not? One summary row per run (hyperparameters in, final metrics out) — not a curve over steps.

Recall You pass

step=100.5 (a float). What is the danger? It is cast to int 100, silently colliding with the true step 100 (or raises TypeError in stricter loggers).