Overview

Experiment tracking is the systematic recording of hyperparameters, metrics, and artifacts during model training. TensorBoard and Weights & Biases (W&B) are the two dominant tools for this: TensorBoard is TensorFlow's built-in visualization suite (now framework-agnostic), while W&B is a cloud-based experiment tracking platform with collaboration features.

Why this matters: Without logging, you're flying blind—you can't compare runs, debug training issues, or reproduce results. These tools transform chaotic experimentation into scientific methodology.

Core Concepts

Logging tools solve this by capturing everything automatically and making it queryable/visual.

The goal: reproducibility and comparison.

TensorBoard: Local-First Visualization

How TensorBoard Works

The event file system:

# During training, a SummaryWriter writes to disk
from torch.utils.tensorboard import SummaryWriter
 
writer = SummaryWriter('runs/experiment_1')
for epoch in range(100):
    loss = train_one_epoch()
    writer.add_scalar('Loss/train', loss, epoch)
writer.close()

Why this design?

• Event files are append-only binary logs → crash-safe, can resume
• TensorBoard server reads files independently → no coupling to training process
• Multiple experiments = multiple directories → easy comparison

What gets logged:

# Scalars (line plots)
writer.add_scalar('Loss/train', loss, step)
writer.add_scalar('Accuracy/val', acc, step)
 
# Histograms (weight/gradient distributions)
writer.add_histogram('layer1.weights', model.layer1.weight, step)
 
# Images (sample predictions, feature maps)
writer.add_image('predictions', img_grid, step)
 
# Graphs (model architecture)
writer.add_graph(model, input_sample)
 
# Embedings (t-SNE/UMAP visualization)
writer.add_embedding(features, metadata=labels, label_img=images)

Why each type?

• Scalars: Track convergence, detect overfitting
• Histograms: Diagnose vanishing/exploding gradients, dead neurons
• Images: Sanity-check data augmentation, visualize predictions
• Graphs: Verify architecture correctness
• Embeddings: Understand learned representations

Setup

writer = SummaryWriter('runs/cnn_mnist') model = SimpleCNN() criterion = nn.CrossEntropyLoss() optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

Log hyperparameters

hparams = {'lr': 0.001, 'batch_size': 64, 'architecture': 'SimpleCNN'}

(W&B style, TensorBoard supports this via add_hparams)

Training

for epoch in range(10): model.train() for batch_idx, (data, target) in enumerate(train_loader): optimizer.zero_grad() output = model(data) loss = criterion(output, target) loss.backward() optimizer.step()

    # Log every N batches
    global_step = epoch * len(train_loader) + batch_idx
    if batch_idx % 100 == 0:
        writer.add_scalar('Loss/train', loss.item(), global_step)
        # Log gradient norms (diagnostic)
        total_norm = 0
        for p in model.parameters():
            if p.grad is not None:
                total_norm += p.grad.data.norm(2).item() ** 2
        writer.add_scalar('GradNorm', total_norm ** 0.5, global_step)
# Validation
model.eval()
val_loss, correct = 0, 0
with torch.no_grad():
    for data, target in val_loader:
        output = model(data)
        val_loss += criterion(output, target).item()
        pred = output.argmax(dim=1)
        correct += pred.eq(target).sum().item()

val_loss /= len(val_loader)
val_acc = correct / len(val_loader.dataset)
writer.add_scalar('Loss/val', val_loss, epoch)
writer.add_scalar('Accuracy/val', val_acc, epoch)
# Log weight histograms
for name, param in model.named_parameters():
    writer.add_histogram(name, param, epoch)

# Log sample predictions
with torch.no_grad():
    sample_data, sample_target = next(iter(val_loader))
    sample_output = model(sample_data[:8])
    img_grid = make_grid(sample_data[:8])
    writer.add_image('Sample Predictions', img_grid, epoch)

writer.close()

**Why this step?** 
- `global_step` ensures scalars from different epochs don't overlap
- Gradient norms detect exploding/vanishing gradients early
- Weight histograms show if layers are learning (distributions should change)
- Sample images verify the model "sees" what you think it sees

### Viewing TensorBoard

```bash
tensorboard --logdir=runs/
# Opens browser at http://localhost:6006

Key views:

• SCALARS: Line plots with smoothing, compare multiple runs
• HISTOGRAMS: Weight/gradient distributions over time (3D view)
• IMAGES: Grid of logged images with slider for timesteps
• GRAPHS: Interactive model architecture (click nodes to see shapes)
• HPARAMS: Table of hyperparameters with sortable metrics

Weights & Biases: Cloud Collaboration

Why W&B exists (vs TensorBoard)

TensorBoard limitations:

1. Local files → hard to share with team (need to zip/upload)
2. No hyperparameter search tracking (which combos were tried?)
3. No dataset versioning
4. No model registry (where's the best checkpoint?)
5. Manual comparison (copy-paste directory names)

W&B solutions:

1. Cloud storage → shareable links
2. Built-in hyperparameter tracking and visualization
3. Dataset/model artifact versioning
4. Leaderboard for best runs
5. Automatic comparison

How W&B Works

Initialization:

import wandb
 
# Start a run
wandb.init(
    project="image-classification",
    config={
        "learning_rate": 0.001,
        "batch_size": 64,
        "architecture": "ResNet50",
        "dataset": "CIFAR-10",
    }
)
config = wandb.config  # Access hyperparameters

Why wandb.init? Creates a unique run ID, uploads code/git state, records system info (GPU type, library versions). This provenance tracking is automatic—you don't forget to log the environment.

Logging during training:

for epoch in range(config.epochs):
    train_loss = train_one_epoch(model, train_loader, optimizer)
    val_loss, val_acc = validate(model, val_loader)
    
    # Log metrics (auto-batched and uploaded)
    wandb.log({
        "epoch": epoch,
        "train_loss": train_loss,
        "val_loss": val_loss,
        "val_acc": val_acc,
    })
    # Log images
    wandb.log({"predictions": wandb.Image(pred_img)})
    # Log model checkpoint
    if val_acc > best_acc:
        torch.save(model.state_dict(), "best_model.pth")
        wandb.save("best_model.pth")  # Uploads to W&B

Why this step?

• wandb.log auto-syncs to cloud → immune to crashes
• wandb.Image handles PIL/numpy/torch tensors → no manual conversion
• wandb.save versions artifacts → retrievable by run ID later

1. Initialize

wandb.init(project="mnist-cnn", config={ "learning_rate": 0.001, "batch_size": 128, "epochs": 10, "optimizer": "Adam", }) config = wandb.config

2. Create dataloaders

transform = transforms.Compose([ transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,)) ]) train_dataset = datasets.MNIST('./data', train=True, download=True, transform=transform) train_loader = torch.utils.data.DataLoader( train_dataset, batch_size=config.batch_size, shuffle=True)

3. Model setup

model = SimpleCNN() wandb.watch(model, log="all", log_freq=100) # Auto-logs gradients/params criterion = nn.CrossEntropyLoss() optimizer = torch.optim.Adam(model.parameters(), lr=config.learning_rate)

4. Training

for epoch in range(config.epochs): model.train() for batch_idx, (data, target) in enumerate(train_loader): optimizer.zero_grad() output = model(data) loss = criterion(output, target) loss.backward() optimizer.step()

    # Log every 10 batches
    if batch_idx % 10 == 0:
        wandb.log({"batch_loss": loss.item()})
# Validation
model.eval()
val_loss, correct = 0, 0
with torch.no_grad():
    for data, target in val_loader:
        output = model(data)
        val_loss += criterion(output, target).item()
        pred = output.argmax(dim=1)
        correct += pred.eq(target).sum().item()

val_loss /= len(val_loader)
val_acc = correct / len(val_loader.dataset)
wandb.log({
    "epoch": epoch,
    "val_loss": val_loss,
    "val_acc": val_acc,
})

# Log confusion matrix
wandb.log({"confusion_matrix": wandb.plot.confusion_matrix(
    probs=None, y_true=all_targets, preds=all_preds,
    class_names=class_names)})

5. Save final model

torch.save(model.state_dict(), "final_model.pth") wandb.save("final_model.pth") wandb.finish()

**Why this step?**
- `wandb.watch(model, log="all")` automatically logs parameter/gradient histograms → replaces manual TensorBoard histogram code
- `wandb.plot.confusion_matrix` generates interactive plots → easier than matplotlib
- `wandb.finish()` ensures all data uploads complete

### W&B Advanced Features

> [!definition] **Sweps (Hyperparameter Search)**
> W&B's built-in hyperparameter optimization. Define a search space, W&B spawns parallel runs with different configs.

```yaml
# sweep_config.yaml
program: train.py
method: bayes  # or grid, random
metric:
  name: val_acc
  goal: maximize
parameters:
  learning_rate:
    min: 0.001
    max: 0.1
    distribution: log_uniform
  batch_size:
    values: [32, 64, 128]
  dropout:
    min: 0.1
    max: 0.5

# train.py (modified for sweps)
def train():
    with wandb.init() as run:  # Sweep controller fills config
        config = wandb.config
        model = build_model(config)
        # ... training loop ...
        wandb.log({"val_acc": final_val_acc})
 
# Run sweep
sweep_id = wandb.sweep(sweep_config, project="mnist-cnn")
wandb.agent(sweep_id, function=train, count=20)  # 20 runs

Why this matters? Manual hyperparameter search = you try5 configs and give up. Sweps = systematic exploration with early stopping (kill bad runs early via Bayesian optimization).

# Version a dataset
with wandb.init(project="data-prep", job_type="preprocess") as run:
    artifact = wandb.Artifact("cifar10-preprocessed", type="dataset")
    artifact.add_dir("processed_data/")
    run.log_artifact(artifact)
 
# Use versioned dataset in training
with wandb.init(project="training") as run:
    artifact = run.use_artifact("cifar10-preprocessed:v2")
    data_dir = artifact.download()
    # ... load data from data_dir ...

Why this step? Your dataset changes (fixed bugs, added samples). Without versioning, you can't reproduce old experiments—you don't know which data version produced which results.

Key Differences: TensorBoard vs W&B

When to use which?

• TensorBoard: Quick local debugging, no internet, privacy-sensitive work
• W&B: Team projects, hyperparameter search, model versioning, production ML

Can use both: W&B can sync TensorBoard event files (wandb.init(sync_tensorboard=True)).

Common Patterns

for epoch in range(epochs): train_loss = train_one_epoch() scheduler.step(train_loss)

# Log current learning rate
current_lr = optimizer.param_groups[0]['lr']
wandb.log({"learning_rate": current_lr, "train_loss": train_loss})

**Why this matters?** Learning rate changes explain sudden metric jumps. Without logging, you see loss drop and don't know why.

> [!example] Logging hardware utilization
> ```python
> import GPUtil

for epoch in range(epochs):
    # ... training ...
    
    # Log GPU stats
    gpus = GPUtil.getGPUs()
    wandb.log({
        "gpu_util": gpus[0].load * 100,
        "gpu_memory": gpus[0].memoryUsed,
    })

Why this step? If training is slow, check if GPU utilization is low (<80%) → bottleneck is data loading, not compute.

TensorBoard: Copy runs to same directory

runs/

exp1/

exp2/

exp3/

Then: tensorboard --logdir=runs/

W&B: Automatic in web UI

Just filter by tag or use comparison view

---

## Common Mistakes

> [!mistake] **Logging too frequently**
> **Wrong approach**: `wandb.log({"loss": loss})` inside inner loop (every batch)
> ```python
> for epoch in range(100):
>     for batch in train_loader:  # 1000 batches/epoch
>         loss = train_step(batch)
>         wandb.log({"loss": loss})  # 100,000 data points!
> ```

**Why it feels right**: More data = better visibility!

**The problem**: 
1. Slows training (network I/O overhead)
2. UI becomes lagy (too many points to render)
3. Wastes storage (batch-level noise isn't informative)

**Fix**: Log every N batches or only at epoch level
```python
if batch_idx % 100 == 0:  # Every 100 batches
    wandb.log({"batch_loss": loss})
# Or aggregate and log once per epoch

Why it feels right: "I'll remember" / "I'll check the code later"

The problem: Code changes, notebooks get overwritten, you run50 experiments and forget which config was which.

Fix: Always log config at start

wandb.init(project="..", config={
    "lr": args.lr,
    "batch_size": args.batch_size,
    "model": args.model_name,
    # ... everything that affects results
})

Why it feels right: "Logs are written immediately, right?"

The problem: TensorBoard/W&B buffer writes for performance. If process dies, buffer is lost.

Fix: Use context managers

# TensorBoard
with SummaryWriter('runs/exp1') as writer:
    # ... training ...
    
# W&B
with wandb.init(project="...") as run:
    # ... training ...
# Auto-calls finish() on exit

Why it feels right: "My code is version-controlled, that's enough"

The problem: Library versions change (PyTorch 1.10 → 1.12), random seeds differ, data preprocessing has a subtle bug.

Fix: W&B does this automatically. For TensorBoard, log manually:

writer.add_text('Info/git_commit', git_commit_hash, 0)
writer.add_text('Info/torch_version', torch.__version__, 0)

TensorBoard and Weights & Biases are like a super-detailed lab notebook that automatically writes down everything:

• What you tried (how much water, sunlight)
• What happened (plant height each day)
• Pictures of the plants

TensorBoard is like keeping your notebook on your own desk—it's private and works without internet. Weights & Biases is like a shared Google Doc—your whole team can see it, and it backs up automatically.

The cool part? These tools draw graphs for you! You can instantly see which experiments worked without reading pages of numbers. And if you want to try 1000 different combinations, W&B can even run them automatically while you sleep!

Connections

• 3.3.8-Checkpointing-and-early-stopping: Save best model based on logged metrics
• 3.3.11-Distributed-training: Logging in multi-GPU/multi-node setups
• 4.1.5-Hyperparameter-tuning: W&B sweps automate this
• 5.2.3-Model-versioning: W&B artifacts track model lineage
• 2.4.7-Gradient-flowand-vanishing-exploding-gradients: Diagnose with histogram logs

#flashcards/ai-ml

What is the primary purpose of experiment tracking in deep learning? :: Systematic recording of hyperparameters, metrics, and artifacts to enable reproducibility and comparison of training runs.

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

        # Log gradient norms (detect exploding gradients)
        total_norm = 0
        for p in model.parameters():
            if p.grad is not None:
                total_norm += p.grad.data.norm(2).item() ** 2
        writer.add_scalar('Gradients/norm', total_norm ** 0.5, global_step)

# Validation
model.eval()
val_loss, val_acc = evaluate(model, val_loader)
writer.add_scalar('Loss/val', val_loss, epoch)
writer.add_scalar('Accuracy/val', val_acc, epoch)

# Log weight histograms
for name, param in model.named_parameters():
    writer.add_histogram(name, param, epoch)

import wandb
 
# Initialize a run
wandb.init(
    project="mnist-classification",
    config={  # hyperparameters
        "learning_rate": 0.001,
        "batch_size": 64,
        "architecture": "SimpleCNN",
        "epochs": 10
    }
)
 
# Access config anywhere
config = wandb.config
 
# Log metrics (auto-syncs to cloud)
for epoch in range(config.epochs):
    train_loss = train_one_epoch()
    val_loss, val_acc = evaluate()
    
    wandb.log({
        "train_loss": train_loss,
        "val_loss": val_loss,
        "val_accuracy": val_acc,
        "epoch": epoch
    })
 
# Log model artifacts
wandb.log_artifact('model.pth', type='model')
 
wandb.finish()

# sweep_config.yaml
sweep_config = {
    'method': 'bayes',  # bayesian optimization
    'metric': {'name': 'val_accuracy', 'goal': 'maximize'},
    'parameters': {
        'learning_rate': {'min': 0.0001, 'max': 0.1},
        'batch_size': {'values': [32, 64, 128]},
        'dropout': {'min': 0.1, 'max': 0.5}
    }
}
 
# Create and run sweep
sweep_id = wandb.sweep(sweep_config, project="mnist")
wandb.agent(sweep_id, function=train, count=50)  # run 50 experiments