Foundations — Datasets and DataLoaders

Before you can read the parent note, you must be able to read its alphabet. Below is every symbol and idea it silently assumes, built from nothing, in an order where each rung of the ladder rests on the one below it.

1. A "sample" and its index $i$

We have many samples, so we number them $0,1,2,\dots$ The number is called an index and we write it $i$. Sample number $i$ is $(x_i,y_i)$ — the little $i$ hanging below is just "which one."

2. The counting symbols: $N$, $\in$, $[0,N-1]$, $\mathbb{Z}$

Because we start counting at $0$ (computers do), the valid indices run

$i=0,1,2,\dots ,N-1$

That range is written $[0,N-1]$ — a compact way of saying "from $0$ up to and including $N-1$."

3. Sets and the curly-brace notation $\{\cdot \}$

The parent writes the whole dataset as

$\mathcal{D}=\{(x_i,y_i){\}}_{i=1}^N$

Let us dismantle this piece by piece.

The little $i=1$ underneath and $N$ on top is a recipe for the collection: "let $i$ run from 1 to $N$, and for each value drop the pair $(x_i,y_i)$ into the bag." So the fancy line just says:

$\mathcal{D}=\{(x_1,y_1),(x_2,y_2),\dots ,(x_N,y_N)\}$

The curly $\mathcal{D}$ is simply a name for that whole bag of data — "D for Dataset."

4. The arrow $\to$ and "a function / a mapping"

• $\mathcal{X}$ = the space of all possible inputs (all possible images).
• $\mathcal{Y}$ = the space of all possible labels (all possible answers).
• $\mathcal{X}\times \mathcal{Y}$ (the $\times$ is "paired with") = the space of all (input, label) pairs.

Two properties fall straight out of "it's a function":

• Deterministic: ask for index 5 twice, get the same sample twice. (Randomness lives elsewhere — see §7.)
• Independent: fetching sample 5 does nothing to sample 6.

5. A "batch", the batch size $B$, and the sum $\sum$

The reason batches exist is the gradient formula the parent quotes:

$\hat{g}=\frac{1}{B}{\sum }_{i=1}^B{\nabla }_{\theta }\mathcal{L}(x_i,y_i;\theta )$

Let us earn every symbol here so it is not scary.

• $\mathcal{L}$ (curly L) = the loss: one number saying "how wrong was the model on this sample" (big = very wrong, 0 = perfect).
• $\theta$ (Greek "theta") = the weights of the model — the knobs it is allowed to turn.
• ${\nabla }_{\theta }$ (upside-down triangle, "nabla") = the gradient w.r.t. $\theta$: an arrow pointing in the direction that would increase the loss fastest; we step the opposite way to reduce it.
• $\frac{1}{B}\sum \dots$ = average the per-sample directions over the batch.
• $\hat{g}$ (a $g$ wearing a hat) = the estimate of the true gradient. The hat always means "our best guess of," not the exact thing.

6. Floor, ceiling, and modulo — counting the batches

If $N$ samples are cut into groups of $B$, how many groups? Usually $N$ doesn't divide evenly, so we need three little tools.

Now the parent's two options make total sense. With $N=100$, $B=32$:

$\lfloor 100/32\rfloor =3(\text{full batches}),100\mathrm{mod}32=4(\text{leftovers})$

• drop_last=True → throw away the 4 leftovers → $\lfloor N/B\rfloor =3$ batches, all size 32.
• drop_last=False → keep them as a small final batch → $\lceil N/B\rceil =4$ batches (last one size 4).

7. Shuffling — where the randomness lives

8. Tensors and shape — the rectangular box of numbers

The prerequisite map

Everything on the left is foundation; it all funnels into the Datasets and DataLoaders topic itself, which then powers the training loop, scales up with distributed training, and reuses pretrained data pipelines in transfer learning.

Equipment checklist

Read each question, answer in your head, then reveal.