Exercises — Datasets and DataLoaders

The two required methods are:

• ==__len__()== — returns the total number of samples $N$ (an integer). The DataLoader needs this to know how many batches an epoch contains.
• ==__getitem__(idx)== — returns the single sample at position idx, usually a tuple $(x_i,y_i)$ of data and label.

Why exactly these two? Because the DataLoader only ever needs to ask "how many are there?" (__len__) and "give me number idx" (__getitem__). Everything else — shuffling, batching, parallel loading — is built on top of just those two questions.

• (a) shuffling → DataLoader (the "how it's delivered")
• (b) loading one image → Dataset (inside __getitem__)
• (c) stacking into a tensor (collation) → DataLoader
• (d) defining what a sample is → Dataset

Mnemonic: the Dataset answers what, the DataLoader answers how it arrives.

Recall from the parent [!formula]: batches = $\lceil N/B\rceil$ when we keep everything, and $\lfloor N/B\rfloor$ when we drop the incomplete tail.

First compute the raw ratio: $\frac{N}{B}=\frac{1000}{32}=31.25$.

(a) drop_last=False keeps the leftover, so we round up: $\lceil 31.25\rceil =32\text{ batches.}$ (b) drop_last=True throws the leftover away, so we round down: $\lfloor 31.25\rfloor =31\text{ batches.}$ (c) The first $31$ batches hold $31\times 32=992$ samples. The last batch holds the remainder: $1000-992=8\text{ samples},\text{i.e. }1000\mathrm{mod}32=8.$ Look at figure s02: the full bars are height 32, and the short orange stub on the right is the size-8 remainder.

$\lceil N/64\rceil =50$ means $N/64$ lands in the interval $(49,50]$ — any value above 49 but not above 50 rounds up to 50. But we're also told the last batch is full, i.e. $N$ is an exact multiple of 64, so $N\mathrm{mod}64=0$.

The only multiple of 64 with $\lceil N/64\rceil =50$ is: $N=50\times 64=3200.$ So smallest = largest = $3200$. (If the "full last batch" condition were dropped, $N$ could range from $49\times 64+1=3137$ up to $3200$.)

Serial (num_workers=0): the CPU loads, then the GPU computes, nothing overlaps: $t_{\text{serial}}=t_{\text{load}}+t_{\text{gpu}}=40+25=65\text{ ms per batch.}$ Overlapped (enough workers): while the GPU chews on batch $t$, worker processes are already loading batch $t+1$. In steady state each batch's wall-clock time is the slower of the two stages, not their sum: $t_{\text{overlap}}=\max (t_{\text{load}},t_{\text{gpu}})=\max (40,25)=40\text{ ms per batch.}$ Speed-up: $\frac{t_{\text{serial}}}{t_{\text{overlap}}}=\frac{65}{40}=1.625\times .$ Notice the ceiling on this idea: because loading (40 ms) is slower than compute (25 ms), the pipeline is data-bound — the GPU idles 15 ms each batch. More workers won't help beyond removing the serial stall; you'd instead need faster storage or lighter transforms. This overlap trick is exactly what powers efficient 3.3.08-Distributed-Training.

The mini-batch gradient is the average over the batch: $\hat{g}=\frac{1}{B}{\sum }_{i=1}^B{\nabla }_{\theta }\mathcal{L}(x_i,y_i;\theta ).$ This estimate is only a good stand-in for the true full-data gradient if the batch is a representative sample of the dataset. Engineer B's sorted data means an early batch is all cats. Its average gradient points "make everything look like a cat," then a later all-dogs batch yanks the weights the opposite way. The optimizer sees a violently biased, oscillating signal — high-variance, non-representative steps — and generalizes poorly.

Engineer A's shuffle makes each batch a random mix, so $\hat{g}$ is an unbiased, low(er)-variance estimate of the real gradient — smooth, honest steps. This is precisely why 2.2.03-Mini-Batch-Gradient-Descent assumes i.i.d. sampling.

The problem: torch.stack demands identical shapes, but sequence lengths differ. The fix is to pad the short ones with zeros up to the batch maximum, making the batch rectangular.

import torch
import torch.nn.functional as F
 
def collate_fn(batch):
    # batch: list of (seq, label), seq shape = (length_i, 16)
    max_len = max(seq.shape[0] for seq, _ in batch)   # longest in THIS batch
    seqs, labels = [], []
    for seq, label in batch:
        pad_rows = max_len - seq.shape[0]             # how many rows to add
        # F.pad args go last-dim-first: (feat_left, feat_right, len_left, len_right)
        padded = F.pad(seq, (0, 0, 0, pad_rows))      # pad only the length dim
        seqs.append(padded)
        labels.append(label)
    return torch.stack(seqs), torch.tensor(labels)

Final shape. With batch size $B$ and per-batch max length $L={\max }_i{\text{length}}_i$, the stacked tensor is: $(B,L,16).$ Note $L$ is chosen per batch, not globally — a batch of short sequences pads to a small $L$, saving memory. Figure s03 shows three ragged sequences becoming one rectangle with zero-padding.

import torch
 
def collate_drop_none(batch):
    kept = [(x, y) for (x, y) in batch if y is not None]  # filter first
    if len(kept) == 0:
        return None                                        # signal empty batch
    xs = torch.stack([x for x, _ in kept])
    ys = torch.tensor([y for _, y in kept])
    return xs, ys, len(kept)

Reasoning: we filter before stacking because torch.stack can't ignore None — it would crash. The surviving count len(kept) matters downstream: the gradient average must divide by the actual number of valid samples, not the nominal batch size $B$, or the gradient magnitude is wrong.

Whole-batch-missing case: if every label is None, kept is empty; stacking an empty list would error, so we return None and the training loop must continue (skip that step). This is the degenerate case you must handle — silently stacking would crash mid-epoch.

(a) __getitem__(idx) — lazy load + parse label from filename:

def __getitem__(self, idx):
    path = self.paths[idx]
    img = Image.open(path).convert("RGB")        # load NOW
    label = int(path.split("/")[-1].split("_")[0].replace("class", ""))
    if self.transform: img = self.transform(img) # augment per-access
    return img, label

Storing only paths in __init__ keeps RAM constant (lazy loading); label parsing is a pure string split, so __getitem__ stays deterministic.

(b) DataLoader:

DataLoader(ds, batch_size=32, shuffle=True,
           num_workers=8, pin_memory=True, drop_last=True)

• batch_size=32 — small dataset, standard trade-off between gradient noise and speed.
• num_workers=8 — one per core; hides disk+decode latency behind GPU compute (see Ex 3.1).
• pin_memory=True — page-locked RAM → faster CPU→GPU transfer (~10–20%).
• drop_last=True — keeps every batch the same size so batch-norm statistics stay stable.

(c) Batches per epoch with drop_last=True: $\lfloor \frac{5000}{32}\rfloor =\lfloor 156.25\rfloor =156\text{ batches.}$ ($156\times 32=4992$ samples used; the leftover $8$ are dropped each epoch.)

(d) Why shuffle matters more in fine-tuning: the dataset is tiny (5,000), so the model sees each sample many times across epochs. Without shuffling it would encounter them in the same order every epoch — the optimizer can memorize that sequence and overfit fast. Fresh random orderings each epoch (per 4.1.02-Transfer-Learning best practice) break this and improve generalization on the small set.

Explanation: the pipeline is GPU-bound, not data-bound. From Ex 3.1, steady-state batch time is $\max (t_{\text{load}},t_{\text{gpu}})$. If $t_{\text{gpu}}>t_{\text{load}}$ already at num_workers=2, the workers can supply batches faster than the GPU can consume them — so adding more workers changes nothing. The GPU is the ceiling.

Diagnostic: measure GPU utilization (e.g. nvidia-smi during training). If it sits near 100%, the GPU is the bottleneck and more workers can't help — you'd need a smaller model, mixed precision, or a bigger batch. If utilization is low with gaps, the data pipeline is stalling and you should profile __getitem__ (slow disk / heavy transforms) instead.

Batches with drop_last=False ::: $\lceil N/B\rceil$ Batches with drop_last=True ::: $\lfloor N/B\rfloor$ Steady-state batch time with overlap ::: $\max (t_{\text{load}},t_{\text{gpu}})$ Where randomness lives ::: in the DataLoader's index order, never in __getitem__ Padded variable-length batch shape ::: $(B,L,\text{feat})$ with $L$ the per-batch max length