Visual walkthrough — Pandas DataFrames and Series basics

Before we start, a promise: every word below (Series, index, dtype, alignment, NaN, boolean mask) is drawn as a picture the first time it appears. If you have never touched Python, you can still follow from line one.

---

Step 1 — A bare list has no memory of what each number is

WHAT. We begin with the most primitive container: an ordered list of four numbers.

$[15000,18000,22000,19000]$

• The square brackets $[]$ just mean "a sequence, in order".
• Each number sits at a position counted from zero: position $0$ holds $15000$, position $1$ holds $18000$, and so on. That "count from zero" habit is standard in Python data structures.

WHY. We want to talk about sales per month. But the list only knows positions, not months. If I ask "what were the March sales?", the list cannot answer — it only knows position 2, and you have to remember that position 2 means March. That memory living in your head is exactly the bug factory Pandas removes.

PICTURE. Look at the four boxes. Below each box is its position number in orange. Notice there is no month written anywhere — that missing information is the whole problem.

---

Step 2 — Glue a label to every value → a Series

WHAT. We attach a label to each value. The list of labels is called the index. A value-array plus an index is a Series.

$\underset{\text{the whole labeled column}}{\underbrace{\text{Series}}}(\underset{\text{values}}{\underbrace{[15000,18000,22000,19000]}},\underset{\text{labels}}{\underbrace{\text{index}=[\text{Jan},\text{Feb},\text{Mar},\text{Apr}]}})$

Term by term:

• values — the raw numbers, stored as a fast NumPy array. They all share one dtype (data type) — here int64, meaning "64-bit integers". One dtype per Series is why column math is fast: no type-checking each element.
• index — the labels Jan, Feb, Mar, Apr. This is the new memory the bare list lacked.

WHY. Now the Series can answer "March sales?" directly: it looks up the label Mar, finds it points to position 2, and returns $22000$. The label became a key into the data.

PICTURE. Same four value-boxes, but now each has a magenta label glued to its left. The arrow shows the two-hop lookup label → position → value. The label is what "sticks".

---

Step 3 — Two lookups, two speeds (and why we need both)

WHAT. A Series can be reached two different ways, and they are not the same operation.

$\underset{\text{by label}}{\underbrace{s[\text{‘Mar’}]}}=22000\underset{\text{by position}}{\underbrace{s.\text{iloc}[2]}}=22000$

• s['Mar'] — hand Pandas the label. It hashes the label to find the position. A hash lookup is $O(1)$: constant time, no matter how many rows.
• s.iloc[2] — hand Pandas the integer position directly. iloc = "integer location". It skips straight into the underlying array, also $O(1)$.

WHY. You want labels when you know the name ("give me March"). You want positions when you iterate or slice by order ("give me the first three months"). Same value, two questions — Pandas keeps both doors open.

PICTURE. One Series, two colored paths reaching the same box: the violet path enters through the label, the orange path enters through the position number.

---

Step 4 — Stack Series that share an index → a DataFrame

WHAT. Take several Series that all use the same index, and set them side by side. Each becomes a column. The result is a DataFrame.

$\text{DataFrame}=\{\underset{{\text{col}}_1}{\underbrace{\text{name}}}:{\text{Series}}_1,\underset{{\text{col}}_2}{\underbrace{\text{score}}}:{\text{Series}}_2,\underset{{\text{col}}_3}{\underbrace{\text{grade}}}:{\text{Series}}_3\}$

The critical condition: every ${\text{Series}}_i$ shares one common index (here rows s1, s2, s3).

WHY. Because the columns share row labels, row s1 means "the same person" in every column. That single shared index is what lets you say "the score belonging to Alice" and get an answer that is guaranteed to line up with Alice's grade. Different dtypes per column are fine (name is text, score is integer) — homogeneity is required within a column, never across columns.

PICTURE. Three Series drawn as vertical strips slide together. The shared index runs down the left in magenta; each column keeps its own dtype tag at the top.

---

Step 5 — The shared index makes math auto-align (the central result)

WHAT. Do arithmetic on two Series. Pandas does not line them up by position — it lines them up by label first. This is called alignment.

Suppose: $q_1=\{\begin{array}{l}\text{Jan}:100\\ \text{Feb}:200\\ \text{Mar}:300\end{array}{q}_2=\{\begin{array}{l}\text{Feb}:20\\ \text{Mar}:30\\ \text{Apr}:40\end{array}$

Then $q_1+q_2$ matches by label, not by slot:

$$\begin{array}{rl}\text{Jan}& :100+(\text{missing})=\text{NaN}\\ \text{Feb}& :200+20=220\\ \text{Mar}& :300+30=330\\ \text{Apr}& :(\text{missing})+40=\text{NaN}\end{array}$$

Term by term:

• Pandas takes the union of the two indices: Jan, Feb, Mar, Apr.
• Where a label exists in both, it adds the values (Feb → 220, Mar → 330).
• Where a label exists in only one, there is nothing to add against, so Pandas inserts NaN ("Not a Number") — its marker for missing. NaN forces the dtype to become float64, since NaN is defined only for floats.

WHY. This is the punchline of the whole topic. In a plain list, list_a[0] + list_b[0] blindly adds slot 0 to slot 0 — if the two lists were sorted differently, you'd silently add January to February. Pandas refuses to do that. The label carries the truth, so misaligned data can never quietly corrupt a result. You handle the gaps deliberately later in data cleaning.

PICTURE. Two Series with overlapping but not identical labels. Green links show matched labels adding; red dashed links show unmatched labels producing NaN.

---

Step 6 — Filtering falls out of the same alignment rule

WHAT. A filter is just alignment applied to a boolean mask. First build a Series of True/False:

$\text{mask}=(\text{df}[\text{‘score’}]>80)$

• df['score'] pulls out the score column as a Series (dict-style column access).
• > 80 compares every element at once — this is vectorization: no loop written, one operation over the whole column.
• The result is a boolean Series with the same index as the DataFrame.

Then apply it: $\text{df}[\text{mask}]\to \text{keep exactly the rows where mask is True}$

WHY. Because the mask shares the DataFrame's index, Pandas aligns them and keeps a row only where its label maps to True. Filtering a million rows is therefore one aligned pass — milliseconds, no Python loop.

PICTURE. The score column becomes a column of green (True) / grey (False) flags; the DataFrame on the right keeps only the green-flagged rows, index labels preserved.

---

Step 7 — Edge cases you must have seen (or you'll get surprised)

WHAT. Three degenerate inputs and what Pandas does with each.

Case A — Ragged records (missing keys). Building from a list of dicts where one record lacks a key: $[\{\text{name}:\text{Alice},\text{score}:85\},\{\text{name}:\text{Bob},\text{grade}:\text{A}\}]$ The columns become the union {name, score, grade}. Bob has no score, Alice has no grade → both holes filled with NaN. Same union rule as Step 5.

Case B — Empty Series. A Series with zero values still has a valid (empty) index and a dtype. Its length is $0$; shape is $(0,)$. Nothing breaks — operations just return empty results.

Case C — Duplicate index labels. If two rows share the label s1, then df.loc['s1'] returns both rows (a sub-DataFrame), not one value. The label is no longer a unique key. Position access iloc still works fine because positions are always unique.

WHY. Real data is messy: APIs drop fields, files come empty, join keys repeat. If you've only ever seen the tidy case, each of these will look like a bug. They are not — they are the union rule and the label rule playing out consistently.

PICTURE. Three mini-panels: (A) ragged dicts filling NaN, (B) an empty Series box, (C) a duplicate label pulling two rows.

---

The one-picture summary

Everything above is one chain: list → glue labels (Series) → stack shared-index Series (DataFrame) → all operations align by label → NaN fills the gaps.

Recall Feynman retelling — say it back in plain words

I start with four loose numbers. They're anonymous — I only know them by their slot, and I have to remember what each slot means. That's fragile. So I staple a name onto each number; the names are called the index, and the whole named strip is a Series. Now the name — not the slot — is how I find things.

I stack several of these named strips side by side, all sharing the same row names, and I get a DataFrame: a table where "row Alice" means the same person in every column. Because the names are shared, whenever I do math or filtering, Pandas matches by name first, not by slot. Names present in both sides combine; a name present on only one side has nothing to pair with, so Pandas writes NaN — its little flag for "missing". Filtering is the same trick with a column of True/False flags glued to the same names. Ragged records, empty tables, repeated names — all just this one matching rule playing out honestly. The label sticks to the value. That's the whole story.

Recall

What makes a Series different from a plain Python list? ::: A Series glues a label (the index) and a single dtype onto the values, so you access data by name, not just by position. In q1 + q2, how does Pandas decide which values to add? ::: It aligns by index label (matching names), not by position; matched labels add, unmatched labels give NaN. Why does a filter df[df['score'] > 80] keep the correct rows? ::: The boolean mask shares the DataFrame's index, so alignment keeps exactly the rows whose label maps to True. If two rows share the index label s1, what does df.loc['s1'] return? ::: Both rows (a sub-DataFrame) — the label is no longer a unique key; use iloc for unique positional access.

Related: plot a Series/column once it's aligned, and use these tables downstream for feature engineering.