Datasets and DataLoaders — Hinglish notes

3.3.4 · AI-ML › Deep Learning Frameworks

Intuition Datasets aur DataLoaders kyun zaroori hain

Neural networks ko train karna sirf model ke baare mein nahi hai—yeh efficiently data feed karne ke baare mein bhi hai. Socho kisi ko ek-ek example dikhake padhane vs. examples ko batches mein organize karna, unhe shuffle karna, aur missing data automatically handle karna. Datasets define karte hain ki aapka data kya hai, jabki DataLoaders define karte hain ki woh training ke dauran model tak kaise pahunchta hai.

Core problem: Bade datasets (images, videos, text corpora) ke liye saara data ek saath RAM mein load karna impossible hai. Humein ek aisa system chahiye jo:

Data ko on-demand load kare (lazy loading)
Examples ko efficiently batch kare
Better generalization ke liye shuffle kare
CPU cores pe data loading ko parallelize kare Yahi data pipeline hai, aur training mein yahi aksar bottleneck hoti hai.

Dataset kya hota hai?

Dataset Interface ko First Principles se Derive karna

Problem se shuru karo: Hamare paas N samples hain (images, texts, etc.). Training ke dauran humein:

Koi bhi sample index se access karna hai (batching ke liye)
Total count pata hona chahiye (epoch iteration ke liye)

Naive approach: Sab kuch ek list mein load karo.

data = [load_image(i) for i in range(N)]  # Uses N × image_size RAM!

Problem: ImageNet ke liye (1.2M images, ~150GB), yeh crash kar deta hai.

Solution: Lazy loading indexing ke zariye.

class Dataset:
    def __len__(self): return N
    def __getitem__(self, idx): return load_image(idx)  # Load on-demand

Ab hum ek baar mein ek image load karte hain, RAM constant rehti hai.

Example 1: Scratch se Custom Image Dataset

Problem: Ek folder se images load karo jahan filenames mein labels encoded hain (jaise cat_001.jpg).

import os
from PIL import Image
import torch
from torch.utils.data import Dataset
 
class CustomImageDataset(Dataset):
    def __init__(self, img_dir, transform=None):
        self.img_dir = img_dir
        self.transform = transform
        # List all image paths
        self.img_paths = [f for f in os.listdir(img_dir) if f.endswith('.jpg')]
        # WHY: We store paths, not images (lazy loading)
    def __len__(self):
        return len(self.img_paths)
        # WHY: Needed for batching and epoch calculation
    
    def __getitem__(self, idx):
        img_path = os.path.join(self.img_dir, self.img_paths[idx])
        image = Image.open(img_path)  # Load NOW (on-demand)
        # WHY: Only load when requested, not during __init__
        # Extract label from filename (e.g., "cat_001.jpg" → "cat")
        label = self.img_paths[idx].split('_')[0]
        if self.transform:
            image = self.transform(image)
            # WHY: Data augmentation (random crops, flips) happens per-access
        
        return image, label

Yeh step (lazy loading) kyun? Agar humne __init__ mein saari images load ki hoti, toh training shuru hone se pehle humein gigabytes RAM chahiye hoti. Ab hum ek baar mein ~1 image load karte hain.

DataLoader kya hota hai?

Training Requirements se DataLoader Derive karna

Training loop structure:

for epoch in range(num_epochs):
    for batch in data:  # Need batches, not individual samples
        loss = model(batch)
        loss.backward()

Requirements analysis:

Batching: SGD/Adam ko gradient estimation ke liye batches chahiye (jaise 32 samples).
- Math: Gradient estimate $\overset{g}{^} = \frac{1}{B} \sum_{i = 1}^{B} \nabla_{θ} L (x_{i}, y_{i}; θ)$
- Batches kyun? Gradient variance reduce karta hai, GPU parallelism enable karta hai.
Shuffling: Model ko sample order seekhne se rokta hai (sequence pe overfitting).
- Har epoch kyun? Model same data dekhta hai, lekin different order mein → better generalization.
Parallelism: Data loading (disk I/O, decompression) CPU-bound hota hai. GPU idle wait karta hai.
- Solution: Multiple CPU processes use karo jo next batch pre-load karte hain jab GPU current pe train kar raha ho.

Example 2: DataLoader with All Features

from torch.utils.data import DataLoader
 
# Assume we have the CustomImageDataset from before
dataset = CustomImageDataset(img_dir='./data/images')
 
dataloader = DataLoader(
    dataset,
    batch_size=32,          # Stack 32 samples per batch
    shuffle=True,           # Randomize order each epoch
    num_workers=4,          # 4 parallel CPU processes
    pin_memory=True,        # Speed up CPU → GPU transfer
    drop_last=True          # Drop incomplete last batch
)
 
# WHY each parameter:
# - batch_size=32: Trade-off between gradient accuracy and speed
#   (larger batch → less noise, but slower per-iteration)
# - shuffle=True: Breaks correlation between consecutive samples
#   (prevents overfitting to data order)
# - num_workers=4: While GPU trains batch t, CPUs load batch t+1
#   (overlap computation and I/O)
# - pin_memory=True: Allocates tensors in page-locked RAM
#   (faster DMA transfer to GPU, ~10-20% speedup)
# - drop_last=True: Ensures all batches have same size
#   (some models/layers require fixed batch size)

DataLoader use karna:

for epoch in range(10):
    for batch_idx, (images, labels) in enumerate(dataloader):
        # images: Tensor of shape (32, 3, 224, 224)
        # labels: Tensor of shape (32,)
        
        images = images.to(device)  # Move to GPU
        labels = labels.to(device)
        # Train step...
        outputs = model(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

Yeh step (enumerate) kyun? Hume batch index (logging ke liye) aur data dono milte hain. DataLoader andar se dataset.__getitem__() ko 32 baar call karta hai, results stack karta hai, aur yield karta hai.

Deep Dive: Collation

Custom Collate Function

def collate_variable_length(batch):
    # batch: list of tuples [(seq1, label1), (seq2, label2), ...]
    # where seq_i has shape (length_i, feature_dim)
    # Step 1: Find max length
    max_len = max([seq.shape[0] for seq, _ in batch])
    # WHY: Tensors must be rectangular (all rows same length)
    
    # Step 2: Pad sequences to max_len
    padded_seqs = []
    labels = []
    for seq, label in batch:
        pad_amount = max_len - seq.shape[0]
        padded = torch.nn.functional.pad(seq, (0, 0, pad_amount))
        # WHY: Pad on length dimension (dim 0), not feature dimension (dim 1)
        padded_seqs.append(padded)
        labels.append(label)
    
    # Step 3: Stack into batch tensors
    batch_seqs = torch.stack(padded_seqs)    # (batch_size, max_len, feature_dim)
    batch_labels = torch.tensor(labels)      # (batch_size,)
    
    return batch_seqs, batch_labels
 
# Use with DataLoader
dataloader = DataLoader(dataset, batch_size=32, collate_fn=collate_variable_length)

Padding kyun? GPU operations (matrix multiplication) ke liye fixed-size inputs chahiye. Padding se thodi computation waste hoti hai lekin batching enable hoti hai.

Common Patterns aur Optimizations

Worked example Example 3: Dataset mein Data Augmentation

Problem: Random crops aur flips ke saath image classifier train karo.

from torchvision import transforms
transform = transforms.Compose([
    transforms.RandomResizedCrop(224),      # Random crop and resize
    transforms.RandomHorizontalFlip(p=0.5), # 50% chance to flip
    transforms.ToTensor(),                   # Convert to tensor
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                        std=[0.229, 0.224, 0.225])  # ImageNet stats
])
dataset = CustomImageDataset(img_dir='./data', transform=transform)

Transforms Dataset mein kyun apply karein? Har __getitem__() call pe ek alag augmentation milta hai (access per randomness). Agar humne pehle se apply kar diya hota, toh har epoch mein identical augmentations milte.

Worked example Example 4: Persistent Workers ke saath Prefetching

Problem: Har epoch mein naaye worker processes spawn karne ka zyada overhead.

dataloader = DataLoader(
    dataset,
    batch_size=64,
    num_workers=8,
    persistent_workers=True,  # Keep workers alive across epochs
    prefetch_factor=2         # Each worker prefetches 2 batches
)

persistent_workers kyun? Worker initialization (libraries load karna, model state) mein seconds lagte hain. persistent_workers=True ke saath, yeh har training session mein ek baar hota hai, har epoch mein nahi.

prefetch_factor=2 kyun? Jab GPU batch $t$ process karta hai, workers batches $t + 1$ aur $t + 2$ load karte hain. I/O latency hide ho jaati hai.

Diagram Explanation

Yeh diagram data pipeline flow dikhata hai:

Dataset: On-disk data (images, CSVs) with lazy __getitem__() access
Sampler: Indices generate karta hai (shuffled ya sequential)
Workers: Parallel CPU processes indices ke zariye samples fetch karte hain
Collate: Samples ko batch tensors mein stack karta hai
GPU: Batched tensors receive karta hai training ke liye

Key insight: DataLoader ek producer-consumer system hai jahan CPU workers (producers) GPU (consumer) se aage chalte hain taaki wait time khatam ho.

Common mistake Common Mistake 1: Dataset ke andar Shuffling karna

Galat approach:

class MyDataset(Dataset):
    def __getitem__(self, idx):
        idx = random.randint(0, len(self) - 1)  # Random index
        return self.data[idx]

Kyun sahi lagta hai: "Mujhe randomness chahiye, toh main index randomize kar deta hoon."

Kyun galat hai:

DataLoader shuffling already indices randomize karta hai—aap double kar rahe ho
Multiple epochs mein, kuch samples kabhi nahi dekhoge (birthday paradox)
Reproducibility toot jaati hai (seed reliably set nahi kar sakte)

Fix: DataLoader ko shuffle=True ke zariye shuffling handle karne do. Dataset deterministic hona chahiye.

Common mistake Common Mistake 2: Bahut zyada Workers

Galat approach:

dataloader = DataLoader(dataset, num_workers=32)  # "More is better!"

Kyun sahi lagta hai: Zyada parallelism → faster loading.

Kyun galat hai:

32 processes spawn karne ka overhead hota hai (memory, context switching)
Disk I/O bottleneck hai, CPU nahi—32 workers same disk bandwidth ke liye ladte hain
Diminishing returns: 4-8 workers typically optimal hote hain

Fix: num_workers=0,2,4,8 ke saath benchmark karo. Woh value chunno jahan training speed plateau ho jaaye. Rule of thumb: num_workers = 4 * num_GPUs.

Common mistake Common Mistake 3: GPU Training ke liye pin_memory bhool jaana

Galat approach:

dataloader = DataLoader(dataset, batch_size=64)
for batch in dataloader:
    batch = batch.to('cuda')  # Slow transfer

Kyun sahi lagta hai: "Main data GPU pe tab move karunga jab zaroorat hogi."

Kyun galat hai: Default tensors pageable memory use karte hain. OS unhe disk pe swap kar sakta hai, jisse .to('cuda') ke dauran unpredictable delays aate hain.

Fix: pin_memory=True use karo:

dataloader = DataLoader(dataset, batch_size=64, pin_memory=True)
for batch in dataloader:
    batch = batch.to('cuda', non_blocking=True)  # Fast DMA transfer

Yeh kyun kaam karta hai: Pinned (page-locked) memory swap nahi ho sakti, jisse faster, asynchronous GPU transfer enable hoti hai.

Recall Ek 12-saal ke bacche ko samjhao

Socho tum ek test ki taiyaari flashcards se kar rahe ho. Tumhare paas ek bade box mein 1000 cards hain.

Dataset un cards ko organize karna hai. Har card pe ek number (index) hai, aur tum kisi bhi card ko uske number se nikal sakte ho. Lekin tum ek saath saare 1000 cards NAHI nikalte—woh toh poori desk cover kar denge!

DataLoader tumhara study buddy hai jo:

Cards shuffle karta hai taaki tum order yaad na kar lo (woh toh cheating hai!)
Tumhe ek baar mein 10 cards deta hai (ek batch) taaki tum unhe saath mein padh sako
4 dost hain (workers) jo next batch pehle se laa dete hain jab tum current padhte ho—taaki kabhi wait na karo

Magic yeh hai: Tum 1000 cards padhte ho bina ek saath desk pe rakhe, aur tumhare dost ensure karte hain ki agla batch hamesha ready ho!

Connections

3.3.01-Introduction-to-PyTorch: DataLoader PyTorch ke torch.utils.data module ka part hai
3.3.05-Training-Loop-Basics: DataLoader training loop ko batches feed karta hai
2.2.03-Mini-Batch-Gradient-Descent: Batch size parameter DataLoader se aata hai
3.2.04-Data-Augmentation: Transforms Dataset ke __getitem__() mein apply hote hain
4.1.02-Transfer-Learning: Pre-trained models specific normalization expect karte hain (Dataset mein apply)
3.3.08-Distributed-Training: Multi-GPU ke liye DataLoader with DistributedSampler

#flashcards/ai-ml

PyTorch Dataset ko kaun se do methods implement karni zaroori hain? :: __len__() (total samples return karta hai) aur __getitem__(idx) (index idx par sample return karta hai)

Datasets mein lazy loading kyun use karte hain, __init__ mein saara data load karne ki jagah?

Lazy loading data on-demand load karta hai jab __getitem__ call hota hai, RAM usage constant rakhte hue. Ek saath saara data load karna N × sample_size RAM ki demand karega, jo bade datasets ke liye (jaise ImageNet ~150GB) crash kar deta hai.

Dataset size N, batch size B, aur number of batches ke beech mathematical relationship kya hai?

Number of batches = ⌈N/B⌉ agar drop_last=False (incomplete batch rakho), ya ⌊N/B⌋ agar drop_last=True (incomplete batch discard karo). Last batch ka size N mod B hota hai jab drop nahi karte.

DataLoader ka collate function kya karta hai?

Yeh Dataset.getitem() se individual samples ki list leta hai aur unhe batch tensors mein stack karta hai. Default torch.stack() use karta hai, lekin custom collate functions variable-length data handle karte hain (jaise sequences ko max length tak pad karna).

DataLoader mein num_workers > 0 kyun use karte hain?

Data loading ko CPU processes mein parallelize karne ke liye. Jab GPU batch t pe train karta hai, CPU workers batch t+1 pre-load karte hain, I/O latency hide karte hain aur GPU ko idle rehne se bachate hain. Typical optimal value: 4-8 workers.

DataLoader mein pin_memory=True kya karta hai?

Tensors ko page-locked (pinned) RAM mein allocate karta hai, jise disk pe swap nahi kiya ja sakta. Yeh CPU-to-GPU transfer ke liye faster Direct Memory Access (DMA) enable karta hai, .to('cuda') mein non_blocking=True ke saath ~10-20% speedup deta hai.

Shuffling DataLoader mein kyun karein, Dataset.getitem() ke andar nahi?

Dataset deterministic hona chahiye (same index → same sample) reproducibility ke liye. DataLoader ka shuffle=True Dataset ko pass karne se pehle indices randomize karta hai, ensure karta hai ki har epoch mein saare samples exactly ek baar dekhe jayein aur seed-based reproducibility maintain ho.

DataLoader mein prefetch_factor kya hai? :: Har worker kitne batches pehle se load karta hai. prefetch_factor=2 aur num_workers=4 ke saath, system current GPU batch se 8 batches aage load karta hai, computation aur I/O ke beech overlap maximize karta hai.

DataLoader mein drop_last=True kyun use karein?

Ensure karta hai ki saare batches identical size ke hoon (batch_size). Kuch models/layers (jaise BatchNorm with running statistics, kuch distributed training setups) fixed batch sizes ke saath require ya better perform karte hain. Iske bina, last batch chhota ho sakta hai agar N, batch_size se divisible nahi hai.

Ek well-designed Dataset mein ek sample access karne ki time complexity kya hai?

Ideally O(1), ya indexed structures ke liye O(log N). Dataset ko ek sample dhundhne ke liye saare samples iterate nahi karne chahiye. Direct index se access ke liye file paths ya indices list mein store karo.