Exercises — Pandas DataFrames and Series basics

A Series is three arrays glued together plus a label. The figure below stacks them so you can point at each part.

Reading them straight off:

• values — the actual data as a NumPy array: [10, 20, 30].
• index — the labels that name each value: ['a', 'b', 'c'].
• name — the optional identifier for the whole column: 'scores'.
• dtype — the shared type of every value. All three are whole numbers, so int64.

WHY dtype is single: a Series is homogeneous — one type for the whole column. That is what lets NumPy store it as one contiguous block and do fast vectorized maths.

df.shape is (4, 2).

• The first number, $4$, is the number of rows $n$ (one per person) — the vertical axis=0 direction from the intro figure.
• The second number, $2$, is the number of columns $m$ (name and score) — the horizontal axis=1 direction.

The rule from the parent note: with index length $n$ and $m$ columns, the shape is $(n,m)$. Picture a table — height first, width second, exactly like reading a NumPy 2-D array.

sales = pd.Series([15000, 18000, 22000],
                  index=['Jan', 'Feb', 'Mar'],
                  name='sales')
sales['Feb']   # 18000

WHAT we did: attached human-readable labels to raw numbers. WHY: sales['Feb'] says exactly what we want, whereas sales[1] forces the reader to remember that position 1 is February. Under the hood Pandas hashes the label 'Feb' → position 1 → array value $18000$ in $O(1)$.

(a) df['score'] extracts one column. Since a DataFrame is a dictionary of Series sharing one index, a single column is a Series:

0    85
1    92
2    78
Name: score, dtype: int64

So the type is Series.

(b) Summing the values: $85+92+78=255$.

(a) .loc is label-based. Row label 's2', column label 'score' → intersection → $92$. (b) .iloc is position-based. Row position $1$ (the 2nd row), column position $1$ (the 2nd column score) → same cell → $92$. (c) .loc wants labels. The row labels here are the strings 's1','s2','s3' — there is no label 1 (an integer). So df.loc[1, 'score'] raises KeyError: 1. WHY it feels tempting: without a custom index the labels would be integers $0,1,2$, so .loc[1,...] would work. Once you overwrite the index with strings, integers are no longer valid labels — you must switch to .iloc.

Bonus edge case — the scalar accessors .at and .iat. When you want one single cell (not a slice), Pandas gives faster twins:

• df.at['s2', 'score'] is the label version — same answer $92$ as .loc, but optimised for a single value.
• df.iat[1, 1] is the position version — same answer $92$ as .iloc.

Rule of thumb: .loc/.iloc for ranges and slices; .at/.iat for one scalar. Same label-vs-position split you already learned, just the "single cell" express lane.

(a) Pandas takes the union of all keys across records: name, score, grade. (b) Row 1 (Bob) had no 'score' key, so Pandas fills the gap with NaN (Not-a-Number, the "missing" marker). So df.loc[1, 'score'] is NaN. (c) NaN is a floating-point value. To hold both $85$ and NaN in one homogeneous column, Pandas must promote the whole column to float64 — so scores read as 85.0 and NaN. WHY: a Series is one dtype; the moment a NaN appears, integers can't represent it, so the column upgrades to float.

(a) Comparing a Series to a number is vectorized — it tests every element and returns a boolean Series with the SAME index. The figure below shades which rows survive:

0     True    (85 > 80)
1     True    (92 > 80)
2    False    (78 > 80)
3     True    (95 > 80)

(b) df[df['score'] > 80] keeps rows where the boolean is True — Alice, Bob, Diana. WHY this works: the boolean Series and df share the same index, so Pandas aligns them row-by-row and drops the False rows (the shaded-out row 2 in the figure). (c) Mean of the kept scores: the sum $85+92+95=272$ divided by $3$ items gives $272/3\approx 90.667$.

df['passed'] = df['score'] >= 90
df['passed'].sum()

WHAT: we assigned a new column built by a vectorized comparison. WHY assigning to df['passed'] creates it: bracket-assignment on a missing label adds that column (bracket-read on a missing label errors — assignment and reading differ!). The boolean column: [False, True, False, True] (only Bob=92 and Diana=95 clear $90$). .sum() on booleans counts True as $1$: $0+1+0+1=2$. 2 students passed. This is exactly the spirit of feature engineering: turning raw numbers into a new signal your model can use.

Pandas adds by matching labels, not positions. It takes the union of both indices {w, x, y, z} and lines up values by label. The figure shows this as overlapping label sets (Venn style): only the shared labels y and z get a real sum; the lonely labels become NaN.

• Label w: only in b → no partner → NaN.
• Label x: only in a → no partner → NaN.
• Label y: in both → $2+10=12$.
• Label z: in both → $3+20=23$.

So the result reads:

w     NaN
x     NaN
y    12.0
z    23.0
dtype: float64

WHY the NaNs: w exists only in b, x only in a; with no partner, the sum is undefined → NaN. WHY the dtype is float64: presence of NaN forces the float promotion (same reason as 3.2). This label-first arithmetic is the superpower over plain NumPy arrays, which would align by raw position and silently give wrong sums.

The Series rule from 5.1 now runs in both directions at once: Pandas takes the union of the row labels and the union of the column labels, then adds cell-by-cell only where BOTH a row label and a column label match in the two frames.

• Row union: {r0, r1, r2}. Column union: {x, y, z}.
• A cell gets a real number only if that (row, column) pair exists in both p and q. The only overlap is row r1, column y: p has $4$ there, q has $10$ → $14$.
• Every other cell is missing in at least one frame → NaN.

Result:

      x     y     z
r0  NaN   NaN   NaN
r1  NaN  14.0   NaN
r2  NaN   NaN   NaN

WHY only one number survives: alignment is label intersection per axis. This is the block-wise generalisation of 5.1 — same "shared label → combine, lonely label → NaN" law, applied to a 2-D grid instead of a 1-D line.

The figure contrasts the two slice brackets over the same four rows.

(a) .iloc[0:2] is positional and follows Python's rule: the stop index is excluded. So rows at positions $0$ and $1$ → labels a, b → values $10,20$. (b) .loc['a':'c'] is label-based and here the stop label is included. So a, b, c → values $10,20,30$. (c) WHY they differ: positional slicing inherits Python/NumPy's half-open convention (stop excluded, so 0:2 = 2 items). Label slicing has no notion of "one past the end", so Pandas chose the intuitive inclusive endpoint — if you name 'c', you expect 'c' in the result. This asymmetry catches everyone once; memorise: iloc excludes the stop, loc includes it.

df.values strips away labels and returns the raw 2-D NumPy array underneath: two rows [1, 3, 5] and [2, 4, 6]. .sum() over the whole array adds every entry: $1+3+5+2+4+6=21$. So total = 21. WHY this matters: it shows a DataFrame is labels on top of a NumPy block — pull the block out whenever you need pure array speed and no longer care about labels.