Data loading (CSV, JSON, parquet): Foundations

Before you can understand why Parquet is fast or why JSON nests, you need every word the parent note quietly assumed you already knew. This page builds each one from nothing, in an order where every idea rests on the one before it.

1. A file is a row of bytes

Look at the top strip in Figure 1: the computer never sees "a table". It sees one endless ribbon of boxes. Every format we study is just a different agreement about which boxes mean what.

Figure — Data loading (CSV, JSON, parquet)

Figure 1 — The byte ribbon. A file on disk is one left-to-right line of numbered boxes (top). The same number 25 costs two character-boxes in text (middle, red) but sits in a raw multi-byte slot in binary (bottom, green).

2. Text vs. binary — two ways to write a number

The parent note calls CSV and JSON "text-based" and Parquet "binary". What does that mean on the ribbon?

Definition Binary format, and how bytes become a number

A binary format writes numbers in the raw box-language the computer already uses. But a single box only holds 0–255, so a value like 92.5 or a big integer needs several boxes stitched together — this is a multi-byte encoding (e.g. a 4-byte integer, an 8-byte float). Two extra rules must be agreed:

Endianness: the order the boxes are read in. The number 258 = one box holding 1 and one holding 2. Big-endian writes [1][2] (most significant first); little-endian writes [2][1]. Read with the wrong order and 258 becomes 513 — silent corruption.
Type width: how many boxes one value occupies (an int32 is 4 boxes, a float64 is 8). The reader must know this from the schema, or it won't know where one value stops.

Parquet's file metadata records both, so the program can slice the ribbon into values as-is — no per-character parsing.

Figure 1 shows this: "25" in text is two character-boxes plus the cost of parsing; 25 in binary is a fixed-width slot you read straight off (once you know its width and byte order). This single difference is why the parent's comparison table says CSV/JSON read "Slow" and Parquet reads "Fast".

3. Table, row, column — the target shape

Figure 2 — Two ways to flatten one table into a line. The same 2-row table (top) written row-first (CSV/JSON, blue) scatters the ages; written column-first (Parquet, green) packs the ages into one contiguous block. The yellow boxes are the ages you want to read.

4. CSV semantics — delimiters, headers, quoting, types

The parent note shows a CSV but glosses over the exact rules that make it parseable. Here they are from zero.

Figure 4 — CSV anatomy. A yellow header row names the columns; each following line is one record ended by a newline box (blue). A field containing a comma is wrapped in quotes (red), and an inner quote is escaped by doubling.

5. JSON's world — key–value pairs, nesting, and arrays

Figure 3 — Flattening nested JSON. A nested object (left, blue outer / yellow inner) is flattened by lifting inner keys up with an underscore join into one straight table row (right, green). An array of objects instead unrolls downward into several rows.

6. Row-groups, compression, metadata, and "predicate pushdown"

These power every Parquet claim. First, the layout is more precise than "one big column".

Definition Row-group and column-chunk — how Parquet is

really laid out Parquet does not store one entire column for the whole file end-to-end. It first slices the rows into large blocks called row-groups (say, a million rows each). Within each row-group it stores the data column-by-column — each column's slice there is a column-chunk. So the ribbon reads: [row-group 1: names-chunk | ages-chunk | scores-chunk][row-group 2: names-chunk | ages-chunk | scores-chunk]…

Why bother with row-groups instead of one giant column each? Because a whole-file column would force reading start-to-end; row-groups make the file seekable and skippable in chunks (next definition), and let huge files be split across machines. Section 3's "column-first" picture is the idea inside one row-group — now you have the exact shape.

7. Memory footprint and chunking

The prerequisite map

Every arrow says "you need the left idea before the right one makes sense." The three loaders all funnel into the single skill of the parent topic.

Where these foundations go next

The grid you build here becomes the Pandas DataFrame you'll operate on.
The columns you load are really NumPy arrays under the hood.
The na_values and type-inference traps feed straight into handling missing data and data cleaning.
Clean loaded tables are the raw material for feature engineering and eventually TensorFlow/Keras pipelines.

Equipment checklist

Cover the right side and answer aloud. Reveal to check.

A file on disk is fundamentally what shape?

One long line of bytes, read strictly left to right.

What does "parse" mean and why does it cost time?

To rebuild real values from their text characters (e.g. '2','5' → 25); it's per-value work, so text formats read slowly.

Text vs binary — how many boxes does 25 take in each?

Text: two character-boxes '2','5'. Binary: a fixed-width slot (e.g. 4 boxes for int32) read as-is.

What is endianness and why does it matter?

The byte order of a multi-byte number; reading big-endian bytes as little-endian silently corrupts the value.

What is a row? A column?

A row is one horizontal record; a column is one vertical field (all values of the same kind).

Row-first vs column-first — which puts all ages together?

Column-first (Parquet, inside a row-group). Row-first (CSV/JSON) scatters ages across the file.

How does one CSV row end, and what is CRLF?

With a newline box — either \n (Unix) or the pair \r\n (CRLF, Windows).

Where do CSV column names come from?

The header, the first line by convention; a headerless file needs header=None.

How does CSV handle a comma inside a value?

Quote the whole field in double quotes; an inner quote is escaped by doubling it.

What is a delimiter and why is it needed?

The marker separating values (comma in CSV); without it the parser can't tell where one value ends.

Why does CSV need type inference?

CSV stores no types, so Pandas must peek at a sample and guess int/float/string.

What is a key–value pair?

A labelled slot like "age": 25 — key is the label, value is the content.

Nested object vs array — how does each flatten?

A nested object lifts inner keys up (address_city); an array of objects unrolls into multiple rows (record_path/explode), a scalar array stays/joins/explodes.

What is a row-group and a column-chunk?

Parquet slices rows into row-groups; inside each, one column's slice is a column-chunk.

What is metadata in Parquet and what does it enable?

Per-chunk min/max and location; it enables predicate pushdown (skipping whole row-groups unread).

State the memory-footprint relation.

footprint

\propto

rows loaded

\times

row size.

Why does chunking keep memory constant?

It loads a fixed number of rows, processes, discards them, then repeats — the desk never fills beyond one chunk.