Deep Learning Frameworks

Question 1 (12 marks)

(a) Graph (2 marks):

w ──┐
     ×──> (wx) ──+──> u ──σ──> z ──> L=½(z-t)²
x ──┘          b┘

Nodes: mul(w,x), add(+b) = u, sigmoid(u) = z, loss. (1 for correct node order, 1 for showing intermediates $u,z$.)

(b) Chain rule (6 marks):

• $\dfrac{\partial L}{\partial z} = (z-t)$ (1)
• $\dfrac{\partial z}{\partial u} = \sigma'(u) = z(1-z)$ (2, using sigmoid derivative identity)
• $\dfrac{\partial u}{\partial w} = x$, $\dfrac{\partial u}{\partial b} = 1$ (1)
• Therefore: $\frac{\partial L}{\partial w} = (z-t)\,z(1-z)\,x \quad(1)$ $\frac{\partial L}{\partial b} = (z-t)\,z(1-z) \quad(1)$

(c) Numeric (4 marks):

• $u = 0.5\cdot 2.0 + 0 = 1.0$ (1)
• $z = \sigma(1.0) = \dfrac{1}{1+e^{-1}} \approx 0.7310586$ (1)
• $(z-t) = -0.2689414$; $z(1-z)=0.1966119$ (1)
• $\dfrac{\partial L}{\partial w} = (-0.2689414)(0.1966119)(2.0) \approx -0.105770$ (1)

Question 2 (14 marks)

import torch
 
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")  # (2)
model.to(device)
 
for epoch in range(num_epochs):            # (2) epoch loop
    model.train()
    running_loss = 0.0
    for xb, yb in train_loader:            # batch loop
        xb, yb = xb.to(device), yb.to(device)  # (1)
        optimizer.zero_grad()              # (4) correct order below
        out = model(xb)
        loss = criterion(out, yb)
        loss.backward()
        optimizer.step()
        running_loss += loss.item() * xb.size(0)   # (2) accumulate
    print(f"epoch {epoch} loss {running_loss/len(train_loader.dataset):.4f}")
 
    # validation (3)
    model.eval()
    val_loss = 0.0
    with torch.no_grad():
        for xb, yb in val_loader:
            xb, yb = xb.to(device), yb.to(device)
            val_loss += criterion(model(xb), yb).item() * xb.size(0)

Mark deductions: zero_grad after backward = −2; missing no_grad in val = −2; loss accumulated as tensor (memory leak, not .item()) = −1.

Question 3 (10 marks)

(a) (5 marks):

import torch.nn as nn
import torch.nn.functional as F
 
class MLP(nn.Module):
    def __init__(self):
        super().__init__()                 # (1) required super call
        self.fc1 = nn.Linear(784, 128)     # (1)
        self.fc2 = nn.Linear(128, 10)      # (1)
    def forward(self, x):
        x = F.relu(self.fc1(x))            # (1) ReLU
        return self.fc2(x)                 # (1) no softmax (CrossEntropyLoss expects logits)

(b) (3 marks): nn.Module.__setattr__ intercepts attribute assignment; when the value is an nn.Parameter or nn.Module, it registers it in the internal _parameters / _modules OrderedDicts. This lets .parameters(), .to(device), .state_dict() traverse them recursively. If you build layers inside forward, new params are created each call, never registered, and the optimizer never sees or updates them.

(c) (2 marks): Returns the learnable nn.Parameter tensors (weights and biases). The optimizer stores references to them; optimizer.step() reads each .grad and updates the tensor in place.

Question 4 (10 marks)

(a) (3): Mixed precision runs most ops (matmuls, convs) in FP16/BF16 while keeping a master copy / sensitive ops (reductions, loss) in FP32. Speedup comes from Tensor Cores executing FP16 matmuls faster and halved memory bandwidth / larger batch sizes.

(b) (3): FP16 has a small dynamic range; small gradients underflow to zero. GradScaler multiplies the loss by a large scale factor before backward() so gradients stay in representable range, then unscales them before the optimizer step, dynamically adjusting the factor if inf/NaN appears.

(c) (4):

optimizer.zero_grad()
with torch.cuda.amp.autocast():        # (1)
    out = model(xb)
    loss = criterion(out, yb)
scaler.scale(loss).backward()          # (1)
scaler.step(optimizer)                 # (1)
scaler.update()                        # (1)

Question 5 (8 marks)

(a) (3):

torch.save({
    "epoch": epoch,
    "model_state": model.state_dict(),
    "optim_state": optimizer.state_dict(),
}, "ckpt.pth")

(b) (3):

ckpt = torch.load("ckpt.pth", map_location=device)   # (1) map_location
model.load_state_dict(ckpt["model_state"])            # (1)
optimizer.load_state_dict(ckpt["optim_state"])
start_epoch = ckpt["epoch"] + 1                        # (1)

(c) (2): Saving the whole object pickles the class definition and file paths, breaking if code/paths change; state_dict is just a portable tensor dictionary decoupled from code, robust across refactors and versions.

Question 6 (6 marks)

(a) (2): Data parallelism — replicate the full model on each device, split the batch across them. Model parallelism — split one model's layers/parameters across devices when it doesn't fit on one.

(b) (2): During loss.backward(); as each gradient becomes ready, DDP triggers an all-reduce to average gradients across processes (overlapped with backward computation).

(c) (2): e.g. training/validation loss or accuracy per step (metric); learning rate, batch size, or model architecture (config at run start).

[
  {"claim":"sigmoid(1.0) approx 0.7310586","code":"z=1/(1+exp(-Integer(1))); result=abs(float(z)-0.7310586)<1e-6"},
  {"claim":"dL/dw = -0.105770 at given values","code":"z=1/(1+exp(-Integer(1))); zf=float(z); g=(zf-1.0)*zf*(1-zf)*2.0; result=abs(g-(-0.105770))<1e-4"},
  {"claim":"dL/db = -0.052885 at given values","code":"z=1/(1+exp(-Integer(1))); zf=float(z); g=(zf-1.0)*zf*(1-zf); result=abs(g-(-0.052885))<1e-4"},
  {"claim":"sigmoid derivative z(1-z) at z=0.7310586 equals 0.1966119","code":"zf=0.7310586; d=zf*(1-zf); result=abs(d-0.1966119)<1e-6"}
]