3.3.10 · AI-ML › Deep Learning Frameworks

Overview

Experiment tracking systematic recording hai hyperparameters, metrics, aur artifacts ka — jo model training ke dauran hota hai. TensorBoard aur Weights & Biases (W&B) is kaam ke liye do dominant tools hain: TensorBoard, TensorFlow ka built-in visualization suite hai (ab framework-agnostic ho gaya hai), jabki W&B ek cloud-based experiment tracking platform hai jisme collaboration features bhi hain.

Yeh kyun zaroori hai: Logging ke bina, aap andheron mein kaam kar rahe ho — runs compare nahi kar sakte, training issues debug nahi kar sakte, ya results reproduce nahi kar sakte. Ye tools chaotic experimentation ko scientific methodology mein badal dete hain.

Core Concepts

Logging tools yeh sab solve karte hain — sab kuch automatically capture karke, queryable/visual banana ke liye.

Goal hai: reproducibility aur comparison.

TensorBoard: Local-First Visualization

TensorBoard Kaise Kaam Karta Hai

Event file system:

# During training, a SummaryWriter writes to disk
from torch.utils.tensorboard import SummaryWriter
 
writer = SummaryWriter('runs/experiment_1')
for epoch in range(100):
    loss = train_one_epoch()
    writer.add_scalar('Loss/train', loss, epoch)
writer.close()

Yeh design kyun?

• Event files append-only binary logs hain → crash-safe, resume kar sakte hain
• TensorBoard server files independently padhta hai → training process se koi coupling nahi
• Multiple experiments = multiple directories → easy comparison

Kya kya log hota hai:

# Scalars (line plots)
writer.add_scalar('Loss/train', loss, step)
writer.add_scalar('Accuracy/val', acc, step)
 
# Histograms (weight/gradient distributions)
writer.add_histogram('layer1.weights', model.layer1.weight, step)
 
# Images (sample predictions, feature maps)
writer.add_image('predictions', img_grid, step)
 
# Graphs (model architecture)
writer.add_graph(model, input_sample)
 
# Embedings (t-SNE/UMAP visualization)
writer.add_embedding(features, metadata=labels, label_img=images)

Har type kyun?

• Scalars: Convergence track karo, overfitting detect karo
• Histograms: Vanishing/exploding gradients aur dead neurons diagnose karo
• Images: Data augmentation sanity-check karo, predictions visualize karo
• Graphs: Architecture correctness verify karo
• Embeddings: Learned representations samjho

Setup

writer = SummaryWriter('runs/cnn_mnist') model = SimpleCNN() criterion = nn.CrossEntropyLoss() optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

Log hyperparameters

hparams = {'lr': 0.001, 'batch_size': 64, 'architecture': 'SimpleCNN'}

(W&B style, TensorBoard supports this via add_hparams)

Training

for epoch in range(10): model.train() for batch_idx, (data, target) in enumerate(train_loader): optimizer.zero_grad() output = model(data) loss = criterion(output, target) loss.backward() optimizer.step()

    # Log every N batches
    global_step = epoch * len(train_loader) + batch_idx
    if batch_idx % 100 == 0:
        writer.add_scalar('Loss/train', loss.item(), global_step)
        # Log gradient norms (diagnostic)
        total_norm = 0
        for p in model.parameters():
            if p.grad is not None:
                total_norm += p.grad.data.norm(2).item() ** 2
        writer.add_scalar('GradNorm', total_norm ** 0.5, global_step)
# Validation
model.eval()
val_loss, correct = 0, 0
with torch.no_grad():
    for data, target in val_loader:
        output = model(data)
        val_loss += criterion(output, target).item()
        pred = output.argmax(dim=1)
        correct += pred.eq(target).sum().item()

val_loss /= len(val_loader)
val_acc = correct / len(val_loader.dataset)
writer.add_scalar('Loss/val', val_loss, epoch)
writer.add_scalar('Accuracy/val', val_acc, epoch)
# Log weight histograms
for name, param in model.named_parameters():
    writer.add_histogram(name, param, epoch)

# Log sample predictions
with torch.no_grad():
    sample_data, sample_target = next(iter(val_loader))
    sample_output = model(sample_data[:8])
    img_grid = make_grid(sample_data[:8])
    writer.add_image('Sample Predictions', img_grid, epoch)

writer.close()

**Yeh step kyun?**
- `global_step` ensure karta hai ki alag-alag epochs ke scalars overlap na karein
- Gradient norms exploding/vanishing gradients ko jaldi detect karte hain
- Weight histograms dikhate hain ki layers sikh rahi hain ya nahi (distributions badalni chahiye)
- Sample images verify karte hain ki model wahi "dekh" raha hai jo aap sochte hain

### TensorBoard Kaise Dekhein

```bash
tensorboard --logdir=runs/
# Opens browser at http://localhost:6006

Key views:

• SCALARS: Line plots with smoothing, multiple runs compare karo
• HISTOGRAMS: Weight/gradient distributions over time (3D view)
• IMAGES: Timesteps ke liye slider ke saath logged images ka grid
• GRAPHS: Interactive model architecture (nodes click karke shapes dekho)
• HPARAMS: Sortable metrics ke saath hyperparameters ki table

Weights & Biases: Cloud Collaboration

W&B Kyun Aaya (TensorBoard ke Muqable Mein)

TensorBoard ki limitations:

1. Local files → team ke saath share karna mushkil (zip/upload karna padta hai)
2. Hyperparameter search tracking nahi (kaun se combos try kiye gaye?)
3. Dataset versioning nahi
4. Model registry nahi (best checkpoint kahan hai?)
5. Manual comparison (directory names copy-paste karne padte hain)

W&B ke solutions:

1. Cloud storage → shareable links
2. Built-in hyperparameter tracking aur visualization
3. Dataset/model artifact versioning
4. Best runs ke liye leaderboard
5. Automatic comparison

W&B Kaise Kaam Karta Hai

Initialization:

import wandb
 
# Start a run
wandb.init(
    project="image-classification",
    config={
        "learning_rate": 0.001,
        "batch_size": 64,
        "architecture": "ResNet50",
        "dataset": "CIFAR-10",
    }
)
config = wandb.config  # Access hyperparameters

wandb.init kyun? Ek unique run ID banata hai, code/git state upload karta hai, system info record karta hai (GPU type, library versions). Yeh provenance tracking automatic hai — aap environment log karna bhoolte nahi.

Training ke dauran logging:

for epoch in range(config.epochs):
    train_loss = train_one_epoch(model, train_loader, optimizer)
    val_loss, val_acc = validate(model, val_loader)
    
    # Log metrics (auto-batched and uploaded)
    wandb.log({
        "epoch": epoch,
        "train_loss": train_loss,
        "val_loss": val_loss,
        "val_acc": val_acc,
    })
    # Log images
    wandb.log({"predictions": wandb.Image(pred_img)})
    # Log model checkpoint
    if val_acc > best_acc:
        torch.save(model.state_dict(), "best_model.pth")
        wandb.save("best_model.pth")  # Uploads to W&B

Yeh step kyun?

• wandb.log auto-sync karta hai cloud se → crashes se safe rehta hai
• wandb.Image PIL/numpy/torch tensors handle karta hai → manual conversion nahi chahiye
• wandb.save artifacts version karta hai → baad mein run ID se retrieve ho sakta hai

1. Initialize

wandb.init(project="mnist-cnn", config={ "learning_rate": 0.001, "batch_size": 128, "epochs": 10, "optimizer": "Adam", }) config = wandb.config

2. Create dataloaders

transform = transforms.Compose([ transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,)) ]) train_dataset = datasets.MNIST('./data', train=True, download=True, transform=transform) train_loader = torch.utils.data.DataLoader( train_dataset, batch_size=config.batch_size, shuffle=True)

3. Model setup

model = SimpleCNN() wandb.watch(model, log="all", log_freq=100) # Auto-logs gradients/params criterion = nn.CrossEntropyLoss() optimizer = torch.optim.Adam(model.parameters(), lr=config.learning_rate)

4. Training

for epoch in range(config.epochs): model.train() for batch_idx, (data, target) in enumerate(train_loader): optimizer.zero_grad() output = model(data) loss = criterion(output, target) loss.backward() optimizer.step()

    # Log every 10 batches
    if batch_idx % 10 == 0:
        wandb.log({"batch_loss": loss.item()})
# Validation
model.eval()
val_loss, correct = 0, 0
with torch.no_grad():
    for data, target in val_loader:
        output = model(data)
        val_loss += criterion(output, target).item()
        pred = output.argmax(dim=1)
        correct += pred.eq(target).sum().item()

val_loss /= len(val_loader)
val_acc = correct / len(val_loader.dataset)
wandb.log({
    "epoch": epoch,
    "val_loss": val_loss,
    "val_acc": val_acc,
})

# Log confusion matrix
wandb.log({"confusion_matrix": wandb.plot.confusion_matrix(
    probs=None, y_true=all_targets, preds=all_preds,
    class_names=class_names)})

5. Save final model

torch.save(model.state_dict(), "final_model.pth") wandb.save("final_model.pth") wandb.finish()

**Yeh step kyun?**
- `wandb.watch(model, log="all")` automatically parameter/gradient histograms log karta hai → manual TensorBoard histogram code ki zaroorat nahi
- `wandb.plot.confusion_matrix` interactive plots generate karta hai → matplotlib se zyada aasaan
- `wandb.finish()` ensure karta hai ki saara data upload complete ho jaaye

### W&B Advanced Features

> [!definition] **Sweeps (Hyperparameter Search)**
> W&B ka built-in hyperparameter optimization. Ek search space define karo, W&B alag-alag configs ke saath parallel runs spawn karta hai.

```yaml
# sweep_config.yaml
program: train.py
method: bayes  # or grid, random
metric:
  name: val_acc
  goal: maximize
parameters:
  learning_rate:
    min: 0.001
    max: 0.1
    distribution: log_uniform
  batch_size:
    values: [32, 64, 128]
  dropout:
    min: 0.1
    max: 0.5

# train.py (modified for sweeps)
def train():
    with wandb.init() as run:  # Sweep controller fills config
        config = wandb.config
        model = build_model(config)
        # ... training loop ...
        wandb.log({"val_acc": final_val_acc})
 
# Run sweep
sweep_id = wandb.sweep(sweep_config, project="mnist-cnn")
wandb.agent(sweep_id, function=train, count=20)  # 20 runs

Yeh kyun matter karta hai? Manual hyperparameter search = aap 5 configs try karte ho aur haath dho lete ho. Sweeps = Bayesian optimization ke saath early stopping ke saath systematic exploration (kharab runs jaldi band karo).

# Version a dataset
with wandb.init(project="data-prep", job_type="preprocess") as run:
    artifact = wandb.Artifact("cifar10-preprocessed", type="dataset")
    artifact.add_dir("processed_data/")
    run.log_artifact(artifact)
 
# Use versioned dataset in training
with wandb.init(project="training") as run:
    artifact = run.use_artifact("cifar10-preprocessed:v2")
    data_dir = artifact.download()
    # ... load data from data_dir ...

Yeh step kyun? Aapka dataset badalta rehta hai (bugs fix kiye, samples add kiye). Versioning ke bina, purane experiments reproduce nahi ho sakte — aap nahi jaante ki kaun sa data version kaun se results deta tha.

Key Differences: TensorBoard vs W&B

Kab kaun sa use karein?

• TensorBoard: Quick local debugging, internet nahi, privacy-sensitive kaam
• W&B: Team projects, hyperparameter search, model versioning, production ML

Dono use kar sakte ho: W&B, TensorBoard event files sync kar sakta hai (wandb.init(sync_tensorboard=True)).

Common Patterns

for epoch in range(epochs): train_loss = train_one_epoch() scheduler.step(train_loss)

# Log current learning rate
current_lr = optimizer.param_groups[0]['lr']
wandb.log({"learning_rate": current_lr, "train_loss": train_loss})

**Yeh kyun matter karta hai?** Learning rate changes sudden metric jumps explain karte hain. Logging ke bina, aap loss drop dekhte ho aur nahi samajhte kyun.

> [!example] Hardware utilization log karna
> ```python
> import GPUtil

for epoch in range(epochs):
    # ... training ...
    
    # Log GPU stats
    gpus = GPUtil.getGPUs()
    wandb.log({
        "gpu_util": gpus[0].load * 100,
        "gpu_memory": gpus[0].memoryUsed,
    })

Yeh step kyun? Agar training slow hai, check karo ki GPU utilization low hai ya nahi (<80%) → bottleneck compute mein nahi, data loading mein hai.

TensorBoard: Runs ko same directory mein copy karo

runs/

exp1/

exp2/

exp3/

Then: tensorboard --logdir=runs/

W&B: Web UI mein automatic

Bas tag se filter karo ya comparison view use karo

---

## Common Mistakes

> [!mistake] **Bahut zyada frequently log karna**
> **Galat approach**: `wandb.log({"loss": loss})` inner loop ke andar (har batch par)
> ```python
> for epoch in range(100):
>     for batch in train_loader:  # 1000 batches/epoch
>         loss = train_step(batch)
>         wandb.log({"loss": loss})  # 100,000 data points!
> ```

**Kyun sahi lagta hai**: Zyada data = better visibility!

**Problem**:
1. Training slow ho jaati hai (network I/O overhead)
2. UI laggy ho jaata hai (render karne ke liye bohot zyada points)
3. Storage barbad hota hai (batch-level noise informative nahi hota)

**Fix**: Har N batches par log karo ya sirf epoch level par
```python
if batch_idx % 100 == 0:  # Every 100 batches
    wandb.log({"batch_loss": loss})
# Ya aggregate karo aur ek baar per epoch log karo

Kyun sahi lagta hai: "Mujhe yaad rahega" / "Baad mein code check kar lunga"

Problem: Code badalta hai, notebooks overwrite ho jaati hain, 50 experiments run karte ho aur bhool jaate ho kaun si config kaun si thi.

Fix: Hamesha start mein config log karo

wandb.init(project="..", config={
    "lr": args.lr,
    "batch_size": args.batch_size,
    "model": args.model_name,
    # ... sab kuch jo results affect karta hai
})

Kyun sahi lagta hai: "Logs immediately likhte hain, hai na?"

Problem: TensorBoard/W&B performance ke liye writes buffer karte hain. Agar process mar jaaye, buffer bhi kho jaata hai.

Fix: Context managers use karo

# TensorBoard
with SummaryWriter('runs/exp1') as writer:
    # ... training ...
    
# W&B
with wandb.init(project="...") as run:
    # ... training ...
# Exit par auto-calls finish()

Kyun sahi lagta hai: "Mera code version-controlled hai, bas itna kaafi hai"

Problem: Library versions badal jaati hain (PyTorch 1.10 → 1.12), random seeds alag hote hain, data preprocessing mein koi subtle bug hota hai.

Fix: W&B yeh automatically karta hai. TensorBoard ke liye, manually log karo:

writer.add_text('Info/git_commit', git_commit_hash, 0)
writer.add_text('Info/torch_version', torch.__version__, 0)

TensorBoard aur Weights & Biases ek super-detailed lab notebook ki tarah hain jo automatically sab kuch likh deti hai:

• Tumne kya try kiya (kitna paani, sunlight)
• Kya hua (har din plant ki height)
• Plants ki pictures

TensorBoard is tarah hai jaise apni notebook apni desk par rakhna — private hai aur internet ke bina kaam karta hai. Weights & Biases is tarah hai jaise ek shared Google Doc — poori team dekh sakti hai, aur automatically backup hota rehta hai.

Sabse cool part? Ye tools tumhare liye graphs draw karte hain! Tum turant dekh sakte ho ki kaun se experiments kaam aaye bina numbers ke pages padhe. Aur agar tum 1000 alag combinations try karna chahte ho, toh W&B unhe automatically run kar sakta hai jab tum so rahe hote ho!

Connections

• 3.3.8-Checkpointing-and-early-stopping: Logged metrics ke basis par best model save karo
• 3.3.11-Distributed-training: Multi-GPU/multi-node setups mein logging
• 4.1.5-Hyperparameter-tuning: W&B sweeps ise automate karte hain
• 5.2.3-Model-versioning: W&B artifacts model lineage track karte hain
• 2.4.7-Gradient-flowand-vanishing-exploding-gradients: Histogram logs se diagnose karo

#flashcards/ai-ml

Deep learning mein experiment tracking ka primary purpose kya hai? :: Hyperparameters, metrics, aur artifacts ki systematic recording taaki training runs ki reproducibility aur comparison ho sake.

Concept Map