Exercises — TensorBoard - Weights & Biases logging

• a → 2 scalar: a single number per step → line plot → convergence.
• b → 3 histogram: distribution of many values (all weights/grads in a layer) → spot a spike near zero (vanishing) or huge tails (exploding).
• c → 5 image: literally shows the pixels the model receives.
• d → 1 graph: the computation/architecture diagram.
• e → 4 embedding: projects high-dim feature vectors to 2D/3D to see clusters.

The third argument is the step (the x-axis position of the point). It tells TensorBoard where on the horizontal axis to place this value. If you always pass 0, every point lands at horizontal position zero — you get a vertical stack of dots, no curve, no way to see the trend over time.

Here len(train_loader) = 200 (batches per epoch). Epoch 3 (zero-indexed) means 3 full epochs already passed. Batch index 150. $\text{global\_step}=3\times 200+150=600+150=750.$ Why global_step and not batch_idx alone: batch_idx resets to 0 every epoch, so points from different epochs would overwrite/overlap. Multiplying epoch by the number of batches per epoch shifts each epoch into its own horizontal slot, producing one continuous axis.

batch_idx % 50 == 0 is true for batch_idx ∈ {0, 50, 100, 150} → 4 logs per epoch. Over 5 epochs: five groups of four, which is 20 points in total. Each point has a distinct step because epoch*200 + batch_idx never repeats across the loop, so all 20 are plotted (none overwritten).

wandb.log({"train_loss": tl, "val_loss": vl}, step=epoch)

Why one call: wandb.log takes a dictionary — all keys logged at the same internal step. Why step=epoch explicitly: passing step=epoch pins these metrics to a known x-axis position instead of relying on W&B's auto-incrementing internal counter, so the two curves align exactly on the epoch axis. (You may also add "epoch": epoch as a dict key if you want epoch selectable as a custom x-axis in the UI, but the step= argument is the reliable alignment tool.) This is the structural difference from TensorBoard: TB uses one call per tag; W&B batches a whole dict per step.

This is overfitting: the model keeps memorising the training set (train loss ↓) while generalisation worsens (val loss ↑). The gap between the two curves is the tell. Most useful extra log: a validation metric (accuracy/F1), because a rising val loss with still-improving accuracy can be benign (confident-but-correct); a rising val loss and falling accuracy confirms true overfitting. Also useful: weight-norm histograms trending large (over-confident weights).

The gradient norm is shrinking by roughly one to two orders of magnitude per step: $\frac{3\times 10^{-3}}{1\times 10^{-1}}=0.03,\frac{8\times 10^{-5}}{3\times 10^{-3}}\approx \mathrm{0.0267.}$ A steady multiplicative decay toward zero is the signature of vanishing gradients. The early-layer weight histogram would barely move between epochs — a nearly frozen, static distribution — because tiny gradients produce tiny updates. (Contrast: exploding gradients give a growing norm and histograms that fan out / develop huge tails.)

Panel (ii) (the red distribution that moved) is healthy. A learning layer's weights change: updates accumulate, the distribution shifts and typically spreads. Panel (i), where nothing moved, means near-zero effective updates — a symptom of vanishing gradients, a dead layer, or a learning rate too small. The movement itself is the diagnostic, not the shape.

Grid size: three learning rates times two batch sizes, which is 6 runs.

• TensorBoard: writer.add_hparams(hparam_dict, metric_dict) → populates the HPARAMS tab, a sortable table (one row per run, columns = hyperparameters + final metrics).
• W&B: pass hyperparameters into wandb.init(config=...); the Runs table / Sweeps view sorts and parallel-coordinates them automatically.

writer = SummaryWriter('runs/exp_v1')
best_acc = 0.0
for epoch in range(E):
    tl = train_epoch()                       # scalar: convergence
    vl, va = validate()
    writer.add_scalar('Loss/train', tl, epoch)
    writer.add_scalar('Loss/val',   vl, epoch)   # gap → overfit check
    writer.add_scalar('Acc/val',    va, epoch)   # the metric we optimise
    for n, p in model.named_parameters():
        writer.add_histogram(n, p, epoch)        # weight movement → learning
        if p.grad is not None:
            writer.add_histogram(n+'/grad', p.grad, epoch)  # vanish/explode
    if va > best_acc:                            # versioning
        best_acc = va
        torch.save(model.state_dict(), f'ckpt_epoch{epoch}_acc{va:.3f}.pth')
writer.close()

Why each item: train/val loss → convergence + overfitting gap; val acc → the objective; weight histograms → confirm layers learn; grad histograms → catch vanishing/exploding early; checkpoint filename embeds epoch + accuracy so the best model is self-describing and reproducible (the core idea of 5.2.3-Model-versioning).

Values touched per epoch (order of magnitude): $10^7\text{ params}\times 1000\text{ batches}=10^{10}\text{ value-reads per epoch.}$ Histogram computation must scan all values and bin them — this is roughly $O(\text{params})$ per call, so you're doing tens of billions of extra reads plus disk writes per epoch. Training I/O and CPU are swamped; the GPU starves. Fix: log histograms per epoch, not per batch (÷1000), and optionally only for a few diagnostic layers. Log cheap scalars frequently, expensive histograms rarely. This is the general principle: match logging frequency to the cost and the signal's rate of change.

Steps total: $50\times 800=40,000$. Logged points: $40,000/10=4,000$. Bytes: $4,000\times 40=160,000\text{ bytes}\approx 160\text{ kB}.$ Scalars are tiny — disk is not the bottleneck for scalars. This confirms the L5.1 lesson from the other side: log scalars generously, they're cheap; the cost explosion comes from histograms, images, and embeddings, not scalars.

W&B batches logs client-side: many wandb.log calls accumulate in a buffer and are flushed to the cloud asynchronously in a background process, not one request per call. So 40,000 logs become a manageable number of batched uploads — training threads aren't blocked on the network. Crash-safety despite buffering: the same data is also written to a local run directory on disk immediately. If the process dies, the local records survive and are re-synced on the next wandb sync. The append-only local log is the source of truth; the cloud upload is an optimistic mirror.

L1 recognition ::: know what each logged object answers; the step argument is the x-axis position of a point. L2 application ::: global_step = epoch × len(train_loader) + batch_idx; batch W&B metrics in one dict and pass step=epoch to align curves. L3 analysis ::: read curves — overfitting is train loss down while val loss up; vanishing gradients show a steadily decaying grad norm; a healthy layer's weight histogram moves between epochs. L4 synthesis ::: combine scalars + histograms + versioned checkpoints; use the HPARAMS tab / W&B sweeps to compare runs in a sortable table. L5 mastery ::: match logging frequency to cost — scalars are cheap, histograms/images/embeddings are expensive; W&B batches uploads and keeps a local crash-safe copy.