Data loading (CSV, JSON, parquet): Question bank

Before we start, one picture to fix the vocabulary the whole page leans on: row-oriented vs columnar storage. Everything else is a consequence of this one difference.

Figure — Data loading (CSV, JSON, parquet)

Look at the two coloured blocks. On the left (row-oriented, like CSV and JSON) the bytes for one person sit together: Alice, 25, 92.5, then Bob, 30, 87.3. On the right (columnar, like Parquet) all the names sit together, then all the ages, then all the scores. Keep this image in your head — most answers below are just "which layout does this format use, and what does that force?"

True or false — justify

True or false: A CSV file records the data type of each column.

False. CSV is pure text with no schema, so "01234" could be the integer 1234 or a zip-code string — the loader must guess (infer) or you must tell it via dtype.

True or false: JSON is always larger on disk than the same data as CSV.

Usually true. JSON repeats every key on every record ({"age":25} vs just 25), so it carries roughly 2–3× the overhead of CSV for flat tabular data — the price of being self-describing.

True or false: Parquet is faster to write than CSV.

False. Parquet is slower to write — it must sort by column, compress, and build metadata/statistics. Its payoff is fast reads, small size, and preserved types, not write speed.

True or false: Reading one column from a 100-column Parquet file touches roughly 1/100th of the data.

True (approximately). Columnar layout stores each column separately and records its byte location in metadata, so the reader seeks directly to that column's block and skips the rest.

True or false: Passing usecols=['age'] to read_csv makes CSV as fast as columnar Parquet.

False. usecols discards unwanted columns after parsing them, because a row-oriented text file has no per-column index — you still pay to scan the whole file.

True or false: chunksize reduces the total amount of work done to load a file.

False. It processes the same total rows; it only caps peak memory by loading a bounded slice at a time and discarding it before the next — total CPU work is unchanged.

True or false: Parquet's filters argument still reads the whole file, just hiding rows afterward.

False. Parquet stores per-row-group min/max statistics, so a filter like age > 25 lets it skip entire row groups without decompressing them — this is predicate pushdown, done during the read.

True or false: pd.read_json(..., lines=True) and default read_json expect the same file layout.

False. lines=True expects JSONL/NDJSON — one independent JSON object per line — whereas the default expects a single JSON value (e.g. one big array) covering the whole file.

True or false: Nested JSON loads straight into a flat DataFrame with no extra step.

False. Nested keys like {"address":{"city":"NYC"}} need json_normalize to be flattened into columns such as address.city; a plain load would leave a column holding dict objects.

True or false: Because Parquet is binary, you can open and eyeball it in a text editor.

False. It's a compressed, encoded binary format needing pyarrow/fastparquet to read — that opacity is the trade-off for its size and speed advantages.

Spot the error

pd.read_csv('data.csv', dtype=int) to force everything numeric — what breaks?

A single non-numeric cell (a name, a stray "NA") raises a parse error and aborts the whole load; dtype should target only truly numeric columns, e.g. dtype={'age': int}.

df.address.city after json_normalize gives an AttributeError — why?

The normalized column is literally named address.city; dot-attribute access can't handle the dot inside the name. Use df['address.city'], or rename columns to address_city first.

Loading dates with pd.read_csv('data.csv') then sorting by the timestamp column gives wrong order — where's the bug?

Without parse_dates=['timestamp'] the dates stay as strings, so they sort lexicographically ("2023-1-9" after "2023-1-10") instead of chronologically.

na_values=['NA'] set, yet blank cells still don't become NaN — what was forgotten?

Empty strings are only auto-detected as NaN when keep_default_na=True; if you overrode it to False, or your blanks are actually spaces " ", they won't match 'NA' and stay as data.

One malformed row (extra comma) crashes the entire CSV load — what one argument prevents this?

on_bad_lines='skip' (or 'warn') lets the parser drop the offending row and continue instead of aborting the whole file over one bad line.

pd.read_parquet('data.parquet') raises "no engine" — what's missing?

Parquet support isn't built into pandas; you must install an engine (pyarrow or fastparquet). CSV/JSON need no extra engine, which is why this trap surprises people.

Why questions

Why does pre-specifying dtype in read_csv speed up loading?

Without it, pandas reads a sample, infers types, then re-reads to apply them — effectively an extra pass. Telling it the types upfront skips the inference pass entirely.

Why does columnar data compress better than row data?

A column is homogeneous — all ints, or all floats, or repeated categories — so schemes like run-length and dictionary encoding find patterns that mixed-type rows destroy.

Why is JSONL (lines=True) called "streaming friendly"?

Each line is a complete, independent JSON object, so you can read, append, or process records one line at a time without loading or re-parsing the entire file.

Why does the columnar read saving grow as tables get wider?

Reading one column costs column_size; a row-format read costs N × row_size. The wider the row (more columns you skip), the larger the fraction you avoid — for 100-column tables it approaches ~90%+.

Why can two systems disagree on whether a CSV cell is "missing"?

There's no standard: one tool writes empty, another writes "NA", "null", or "-". That's exactly why na_values lets you list every token your source uses for missing.

Why does JSON preserve nested structure while CSV cannot?

JSON's syntax (objects inside objects, arrays inside values) is inherently hierarchical, whereas CSV is a flat grid of rows and columns with no way to express a value that is itself a sub-table.

Edge cases

What happens if a CSV value legitimately contains a comma, like "Doe, John"?

It must be wrapped in quotes so the delimiter inside isn't read as a column split — and if the value also contains a quote, escaping rules (doubling "") kick in, a frequent source of corruption.

You load an empty CSV (headers only, zero rows) — what do you get?

A valid, empty DataFrame with the correct column names but no rows; downstream code that assumes at least one row (e.g. df.iloc[0]) will fail, so guard against len(df)==0.

A zip code column 01234 loads as 1234 — is the data lost, and how do you prevent it?

The leading zero is gone because it was inferred as an integer; force dtype={'zip': str} at load time so the text "01234" is preserved.

Parquet with filters=[('age','>',25)] returns all rows anyway — is the filter broken?

Not necessarily. Filters skip row groups only when statistics guarantee none match; if every group contains some age > 25 value, no group can be skipped, and remaining non-matching rows are then dropped — the result is still correct.

You call json_normalize on records where some objects lack a nested key others have — what fills the gap?

The missing keys become NaN in those rows, so the DataFrame stays rectangular; this silent filling is why you should check for unexpected nulls after normalizing heterogeneous API responses.

Recall Quick self-test

One-format-per-question — say it before revealing. Which format preserves types with zero configuration? ::: Parquet — its schema and column types are stored in the file itself. Which format lets you append a new record by writing one more line? ::: JSONL (line-delimited JSON), since each record is self-contained on its own line. Which format is the safe default for human-readable interchange despite its type ambiguity? ::: CSV — universal, editable in a spreadsheet, at the cost of no type safety.

Related paths: build on 1.4.06-Pandas-DataFrames-and-operations for what a DataFrame is, 1.4.05-NumPy-arrays-and-operations for the array backing columns, then continue to 1.5.01-Handling-missing-data and 1.4.08-Data-cleaning-and-preprocessing where these NaN and type traps get resolved. In real pipelines this feeds 2.3.05-Feature-engineeringtechniques and eventually 3.1.02-TensorFlow-and-Keras-basics.