Datasets and DataLoaders: Visual walkthrough — AI-ML deep dive

Before we start, three plain-word promises about the words we will use:

A sample = one training example: a pair (the input, its answer label). We write it $(x_{i}, y_{i})$ , where the little $i$ is just its position number.
An index = a position number, like a seat number in a cinema. Index $i$ means "the sample in seat $i$ ".
A tensor = a rectangular grid of numbers, like a spreadsheet but allowed to have more than two directions. Every number in it must line up in a straight grid — no ragged edges.

We will earn everything else as we go.

Step 1 — The raw problem: files scattered on a disk

WHAT. You start with $N$ separate files. Nothing is organised. There is no order, no batching, no numbers-in-a-grid — just files with names like cat_001.jpg.

WHY. We must be honest about the starting point. Every later trick exists to fix a specific pain from this picture: files are (a) too big to all fit in memory together, and (b) not yet numbers.

PICTURE. Below, the amber cylinder is the disk. Each white rectangle is one file. Notice there is no arrow yet — nothing is flowing. This is the "before".

![[deepdives/dd-ai-ml-3.3.04-d2