Pandas DataFrames and Series basics

Core Concepts from First Principles

What is a Series?

A Series is fundamentally an ordered, labeled, homogeneous array. Let's build the concept:

Start with a simple array: [10, 20, 30, 40]
Add labels (index): Now each value has a name: {'a': 10, 'b': 20, 'c': 30, 'd': 40}
Keep order: Unlike a dict, Series maintains insertion order and supports integer positional access
Add type consistency: All elements share the same dtype (int64, float64, object, etc.)

import pandas as pd
import numpy as np
 
# From list - automatic integer index
s1 = pd.Series([10, 20, 30, 40])
# Index: 0, 1, 2, 3
 
# From list with custom index
s2 = pd.Series([10, 20, 30, 40], index=['a', 'b', 'c', 'd'], name='scores')
# Index: 'a', 'b', 'c', 'd'
 
# From dictionary - keys become index
s3 = pd.Series({'Alice': 85, 'Bob': 92, 'Charlie': 78})

What is a DataFrame?

A DataFrame is a dictionary of Series with aligned indices. Let's derive this:

Start with multiple Series:
- name_series = Series(['Alice', 'Bob', 'Charlie'])
- score_series = Series([85, 92, 78])
Align them by a common index: Row 0, Row 1, Row 2
Each Series becomes a column: Column 'name', Column 'score'
Result: A2D table where each column shares the same row index

Figure — Pandas DataFrames and Series basics

Construction Methods: Why Each Exists

Worked example Example 1: Building a Series from Scratch

Problem: You have monthly sales data and want to track it by month name.

# Bad approach: just list
sales = [15000, 18000, 22000, 19000]
# Problem: What month is sales[1]? You have to remember!
# Good approach: Series with labels
sales = pd.Series(
    [15000, 18000, 22000, 19000],
    index=['Jan', 'Feb', 'Mar', 'Apr'],
    name='monthly_sales'
)
 
# Why this step? Labels make code self-documenting
print(sales['Feb'])  # 18000 - clear intent!
print(sales.name)    # 'monthly_sales'

Why this step by step:

List → Series: Adds labels and dtype tracking
Custom index: Makes access semantic (sales['Feb'] vs sales[1])
Name parameter: Documents what this Series represents (crucial for DataFrame columns)

Worked example Example 2: DataFrame from Dictionary

Problem: You have student records and want to analyze them together.

# Dictionary of lists (most common pattern)
data = {
    'name': ['Alice', 'Bob', 'Charlie', 'Diana'],
    'score': [85, 92, 78, 95],
    'grade': ['B', 'A', 'C', 'A']
}
 
df = pd.DataFrame(data)
# Why this works - internal process:
# 1. Each list becomes a Series: name → Series([Alice, Bob, Charlie, Diana])
# 2. Automatic integer index: 0, 1, 2, 3 (since no index specified)
# 3. Keys become column names
# 4. All Series aligned to shared index
 
print(df.shape)  # (4, 3 - 4 rows, 3 columns
print(df.index)  # RangeIndex(start=0, stop=4, step=1)
print(df.columns) # Index(['name', 'score', 'grade'])

Why dict-of-lists? Most natural mapping from "column-oriented" thinking. Each key is a feature/variable.

Worked example Example 3: DataFrame from List of Dicts

Problem: Data comes from JSON/API as list of records.

# List of dictionaries (row-oriented)
records = [
    {'name': 'Alice', 'score': 85, 'grade': 'B'},
    {'name': 'Bob', 'score': 92, 'grade': 'A'},
    {'name': 'Charlie', 'score': 78, 'grade': 'C'}
]
 
df = pd.DataFrame(records)
 
# Why this works:
# 1. Each dict is one row
# 2. Keys from all dicts become columns (union of all keys)
# 3. Missing keys get NaN
# 4. Automatic integer index
 
# What if keys don't match?
mixed = [
    {'name': 'Alice', 'score': 85},
    {'name': 'Bob', 'grade': 'A'}  # missing 'score'
]
df_mixed = pd.DataFrame(mixed)
# Result: df_mixed.loc[1, 'score'] = NaN (missing data handled gracefully)

Why list-of-dicts? Natural for APIs/JSON. Each record is complete.

Accessing Data: The Three Patterns

Formula Access Patterns Derivation

Why three ways exist: Different use cases need different semantics.

1. Column access (Series extraction): $\text{df}[\text{'col\_name'}] \rightarrow \text{Series}$

Derived from: DataFrame as dict-of-Series
Returns: The Series for that column
Use when: You want to operate on entire column 2. Label-based (.loc) (semantic indexing): $\text{df.loc}[\text{row\_label}, \text{col\_label}] \rightarrow \text{value}$
Derived from: Index as label mapping
Returns: Value at intersection of labeled row and column
Use when: You know semantic labels

3. Integer-based (.iloc) (positional indexing): $\text{df.iloc}[\text{row\_int}, \text{col\_int}] \rightarrow \text{value}$

Derived from: Underlying numpy array
Returns: Value at positional intersection
Use when: You need position-based slicing or iteration

Worked example Example 4: Access Patterns Compared

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie'],
    'score': [85, 92, 78],
    'grade': ['B', 'A', 'C']
}, index=['s1', 's2', 's3'])  # Custom index
 
# Pattern 1: Column access
scores = df['score']  # Series with index ['s1', 's2', 's3']
# Why: Most common - get entire column for analysis
 
# Pattern 2: Label-based
bob_score = df.loc['s2', 'score']  # 92
# Why this step? We know Bob is labeled 's2'
# Can use slices: df.loc['s1':'s2', ['name', 'score']]
 
# Pattern 3: Position-based
second_studentscore = df.iloc[1, 1]  # 92 (2nd row, 2nd col)
# Why this step? Position-based iteration or slicing
# Can use slices: df.iloc[0:2, 0:2]
# Why not df['s2']? That would look for column named 's2'!
# Why not df[1]? That would look for column named 1!

Common Operations: Derived from Structure

Filtering (Boolean Indexing)

Worked example Example 5: Filtering Students

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie', 'Diana'],
    'score': [85, 92, 78, 95],
    'grade': ['B', 'A', 'C', 'A']
})
# Step 1: Create boolean mask
high_scorers = df['score'] > 85
# Result: Series([False, True, False, True])
# Why? Element-wise comparison
 
# Step 2: Apply mask
df_high = df[high_scorers]
# Result: DataFrame with Bob and Diana only
# Why this step? Keeps rows where mask is True
# Compact form (most common)
df_high = df[df['score'] > 85]
# Multiple conditions (use & and |, not 'and'/'or')
df_filtered = df[(df['score'] > 80) & (df['grade'] == 'A')]
# Why parentheses? Operator precedence - & binds tighter than >

Adding/Modifying Columns

Worked example Example 6: Column Operations

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie'],
    'score': [85, 92, 78]
})
 
# Add new column (broadcasting)
df['passed'] = df['score'] >= 80
# Why this works: Creates boolean Series, aligned by index
# Result: Series([True, True, False]) becomes new column
 
# Add computed column
df['score_normalized'] = df['score'] / 100
# Why this step? Element-wise division, maintains index
 
# Modify existing column
df['score'] = df['score'] + 5  # Curve grades
# Why allowed? Same-index assignment, replaces Series
# Add column from function
df['grade'] = df['score'].apply(lambda x: 'A' if x >= 90 else 'B' if x >= 80 else 'C')
# Why apply? Applies function to each element, returns new Series

Common Mistakes and Steel-manning

Common mistake Mistake 1: Confusing

.loc and .iloc Wrong intuition: "I have an index [0, 1, 2], so df.loc[1] and df.iloc[1] are the same." Why it feels right: Integer indices look like positions!

The truth:

df = pd.DataFrame({'A': [10, 20, 30]}, index=[2, 4, 6])
df.loc[2]  # Row with label 2 → Series([10])
df.iloc[2]  # Row at position 2 → Series([30])
# They're different!

The fix: .loc is always label-based. .iloc is always position-based. If your index happens to be integers, use .iloc for positions to avoid confusion.

Steel-man: The confusion exists because default integer indices make .loc and .iloc coincidentally equal. The moment you have a custom index, they diverge.

Common mistake Mistake 2: Using

and/or in Boolean Indexing Wrong code: df[(df['score'] > 80) and (df['grade'] == 'A')]

Why it feels right: Natural language uses "and"!

Why it fails: and expects single boolean values. df['score'] > 80 is a Series of booleans, not one boolean. Python tries bool(Series), which raises ValueError: The truth value of a Series is ambiguous.

The fix: Use bitwise operators on Series:

& for element-wise AND
| for element-wise OR
~ for element-wise NOT

# Right
df[(df['score'] > 80) & (df['grade'] == 'A')]
 
# Also need parentheses due to operator precedence
# Without parentheses: df['score'] > (80 & df['grade']) - wrong!

Steel-man: Python's and/or are designed for scalar logic. Pandas overloads &/| for element-wise operations on arrays, following numpy conventions.

Common mistake Mistake 3: Chained Assignment (SettingWithCopyWarning)

Wrong code: df['score'] > 80]['grade'] = 'A'

Why it feels right: "Filter first, then modify."

Why it might fail: df['score'] > 80] might return a copy or a view (Pandas decides). If it's a copy, you modify the copy, not the original df.

The fix: Use .loc for chained access:

# Right
df.loc[df['score'] > 80, 'grade'] = 'A'
# Why: Single .loc call guarantees modification of original df

Steel-man: Pandas warns about this because the behavior is ambiguous. Always use .loc[row_indexer, col_indexer] = value for assignment.

Active Recall Questions

Recall Explain DataFrames to a 12-Year-Old

Imagine you have a notebook where you track all your friends' video game high scores. Each page has columns: "Friend's Name", "Game", "Score", "Date". That's basically a DataFrame!

A Series is just ONE column from that notebook. Like if you ripped out just the "Score" column - it's a list of numbers, but each number still knows which friend it belongs to (that's the index/label).

A DataFrame is the WHOLE notebook page with ALL the columns together. The cool part? The labels (friend names) stick to each row, so even if you sort by score, you never lose track of whose score is whose. It's like magic sticky notes that never fall off!

Why use Pandas instead of just writing in a notebook? Because you can ask it questions like "Show me everyone who scored over 1000" or "What's the average score?" and it answers in a millisecond. Try doing that with 10,000 friends in a paper notebook!

Connections

1.4.01-NumPy-arrays-and-vectorization - DataFrames are built on NumPy arrays; understanding vectorization helps explain Pandas operations
1.4.02-Python-data-structures - Series is like an enhanced dict; DataFrame combines list and dict concepts
1.4.07-Pandas-data-cleaningand-manipulation - Next step: using these structures for real analysis
1.5.01-Data-visualizationmatplotlib-basics - DataFrames integrate seamlessly with matplotlib for plotting
2.1.03-Feature-engineering-basics - DataFrames are the primary structure for ML feature preparation

#flashcards/ai-ml

What are the two core components of a Pandas Series? :: Values (numpy array) and Index (labels). The index makes each value identifiable even after operations.

Why does a DataFrame require all Series to share the same index?

So operations can automatically align data across columns. When you filter, sort, or merge the index ensures each row's values stay together.

What is the difference between df['col'] and df['col']]?

df['col'] returns a Series (single column). df[['col']] returns a DataFrame (single-column table). Use Series for operations, DataFrame when you need to maintain table structure.

Why must you use & instead of and for DataFrame boolean indexing?

and expects single booleans; df['score'] > 80 returns a Series of booleans. & performs element-wise AND on Series. Same for | vs or.

What does .loc[row_label, col_label] return?

The value at the intersection of the labeled row and column. Uses semantic labels (names/strings), not positions.

What does .iloc[row_int, col_int] return?

The value at the positional intersection (like 2D array indexing). Uses integer positions (0-based), not labels.

When does df.loc[1] differ from df.iloc[1]?

When the DataFrame has a custom (non-integer) index, OR when the integer index is not sequential. .loc[1] finds label 1, .iloc[1] finds position 1.

How do you safely modify a filtered DataFrame?

Use .loc with boolean indexer: df.loc[df['score'] > 80, 'grade'] = 'A'. Chained assignment like df[df['score'] > 80]['grade'] = 'A' may fail (SettingWithCopyWarning).

What is the shape of a DataFrame with 100 rows and 5 columns?

(100, 5) - shape is (n_rows, n_columns). Access via df.shape.

Why use index parameter when creating a Series?

To provide semantic labels instead of default integer index. Makes code self-documenting: sales['Feb'] is clearer than sales[1].

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Chalo simple tarike se samajhte hain. Pandas ka core idea yeh hai ki tum apne data ko ek Excel spreadsheet ki tarah handle kar sakte ho, lekin code se — bina copy-paste ke jhanjhat ke. Yahan do main cheezein hain: ek Series jo ek single labeled column hai (jaise ek array jismein har value ka ek naam hota hai), aur ek DataFrame jo actually kai Series ka collection hai jo ek common index (row labels) se aligned hote hain. Matlab, DataFrame ko soch lo ek table jaisa jismein har column ek Series hai aur sabhi columns same row numbers share karte hain.

Ismein sabse zaroori concept hai — labels stick to your data. Yeh line bahut important hai. Jab tum data ko sort karte ho, filter karte ho, ya merge karte ho, tab har value ke saath uska label chipka rehta hai. Isse tumhe wo classic "off-by-one" error nahi aata jo Excel ya plain arrays mein hota hai jab rows aage-peeche ho jaati hain. Aur speed ka bhi kamaal hai — label se value access karna ek $O(1)$ hash lookup hota hai, aur position se access karna direct array indexing, dono hi super fast. Isliye tum ek million rows ko bhi milliseconds mein filter kar sakte ho.

Yeh kyun matter karta hai? Kyunki real-world AI-ML kaam mein data hi sab kuch hai, aur woh data hamesha tables mein aata hai — CSV files, databases, sensor readings. Agar tumhe data cleaning, filtering, grouping, ya joining aati hai to aadha data science ka kaam ho gaya. Pandas tumhe SQL jaisi power deta hai, par Python ke andar, aur labeled access ke saath jo mistakes ko rokta hai. Isliye Series aur DataFrame ka basic solid hona bahut zaroori hai — yeh tumhare pure ML pipeline ki foundation hai.

Core Concepts from First Principles

What is a Series?

What is a DataFrame?

Construction Methods: Why Each Exists

Accessing Data: The Three Patterns

Common Operations: Derived from Structure

Filtering (Boolean Indexing)

Adding/Modifying Columns

Common Mistakes and Steel-manning

Active Recall Questions

Connections

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Python & Scientific Computing

Connections