Data loading (CSV, JSON, parquet) — Hinglish notes

1.4.7 · AI-ML › Python & Scientific Computing

Ye formats kya hain?

Definition CSV (Comma-Separated Values)

Ek text-based format jisme har line ek row represent karti hai, aur ek row ke andar values ek delimiter se separated hoti hain (usually comma). Schema-less: column types ko infer karna padta hai ya manually specify karna padta hai.

Structure:

name,age,score
Alice,25,92.5
Bob,30,87.3

Definition JSON (JavaScript Object Notation)

Ek hierarchical text format jo key-value pairs aur arrays use karta hai. Self-describing: keys meaning carry karte hain. Nested structures support karta hai. Human-readable hai lekin verbose hai.

Structure:

[
 {"name": "Alice", "age": 25, "score": 92.5},
 {"name": "Bob", "age": 30, "score": 87.3}
]

Figure — Data loading (CSV, JSON, parquet)

Har format ko kaise load karein

CSV loading — first principles se

CSV row-oriented kyun hai: Jab tum Alice,25,92.5 likhte ho, computer left-to-right padhta hai, ek time mein ek row. Har row independent hoti hai. "All ages" paane ke liye, parser ko har row scan karni padti hai.

Formula Pandas CSV loading

import pandas as pd
 
# Basic load
df = pd.read_csv('data.csv')
# With explicit types (avoids inference overhead)
df = pd.read_csv('data.csv', 
                dtype={'age': int, 'score': float},
                parse_dates=['timestamp'])
# For huge files: chunked reading
for chunk in pd.read_csv('huge.csv', chunksize=1000):
   process(chunk)  # memory stays bounded

Chunked reading ka derivation:

Memory usage ∝ rows loaded × row size
Agar file mein 1M rows hain, sab load karo → 1M × row_size memory
10k ek time mein load karo → max 10k × row_size memory
Har chunk process karo, discard karo → constant memory footprint

Key parameters WHY:

dtype: Pre-specifying types Pandas ki inference skip karta hai (jo ek sample padhta hai, guess karta hai, phir dobara padhta hai)—data ka ek full pass bachata hai
parse_dates: "2023-01-01" jaise strings strings hi rahenge jab tak parse na ho—comparisons fail ho jaate hain, sorting toot jaati hai
chunksize: DataFrame nahi, iterator return karta hai—RAM se bade datasets process karne deta hai

Worked example CSV loading with error handling

Scenario: Real-world CSV jisme missing values aur bad rows hain.

df = pd.read_csv('mesy.csv',
                na_values=['', 'NA', 'null '?'],  # what counts as missing
                keep_default_na=True,
                on_bad_lines='skip')  # or 'warn' or 'error'

Ye step kyun?

na_values: Alag-alag systems missing data alag-alag encode karte hain—NaN par standardize karo
on_bad_lines='skip': Ek bhi malformed row (galat column count) poora load crash kar dega—skip recovery allow karta hai

JSON loading — hierarchical access

JSON alag kyun hai: JSON nest kar sakta hai: {"user": {"name": "Alice", "prefs": {"theme": "dark"}}. Ye flat table nahi hai. Pandas ko ya toh flatten karna padega ya structure preserve karna padega.

Formula Pandas JSON loading

# Flat JSON (list of records)
df = pd.read_json('data.json', orient='records')
 
# Nested JSON → normalize (flatten)
import json
with open('nested.json') as f:
   data = json.load(f)
df = pd.json_normalize(data, sep='_')
# For huge files: line-delimited JSON (JSONL/NDJSON)
df = pd.read_json('data.jsonl', lines=True)

Normalization ka derivation:

Input: {"user": {"name": "Alice", "age": 25}, "score": 92}

Flatten with sep='_':
- Walk nested keys: user.name, user.age, score
- Replace '.' with '_': user_name, user_age, score
- Result: flat dict → DataFrame row

Orient options WHY:

orient='records': Har JSON object → ek row (objects ke arrays ke liye default)
orient='index': Keys row labels hain, values row data hain
lines=True: Har line ek alag JSON object hai (JSONL format—streaming friendly, append karna aasaan)

Worked example JSON normalization

Scenario: API nested user data return karta hai.

data = [
   {"name": "Alice", "address": {"city": "NYC", "zip": "10001"},
   {"name": "Bob", "address": {"city": "LA", "zip": "9001"}}
]
df = pd.json_normalize(data)
# Result columns: name, address.city, address.zip
# Rename for clarity
df.columns = df.columns.str.replace('.', '_')

Ye step kyun?

json_normalize dots ke saath multi-level column names create karta hai—valid hai lekin awkward
Underscores se replace karne par df.address_city kaam karta hai (names mein dots hone par dot notation fail ho jaata hai)

Parquet loading — columnar efficiency

Parquet columnar kyun hai: Traditional row format: [Alice|25|92.5][Bob|30|87.3]—"all ages" padhne ka matlab hai full rows parse karna. Columnar: [names: Alice,Bob][ages: 25,30][scores: 92.5,87.3]—"all ages" padhne ke liye sirf ages block touch karna padta hai.

Compression benefit: Column data homogeneous hota hai (saare ints, saare floats)—mixed-type rows se better compress hota hai.

Formula Pandas Parquet loading

# Requires pyarrow or fastparquet engine
df = pd.read_parquet('data.parquet')
 
# Load specific columns (columnar win!)
df = pd.read_parquet('data.parquet', columns=['age', 'score'])
 
# Filter during read (predicate pushdown)
df = pd.read_parquet('data.parquet', 
                    filters=[('age', '>', 25)])

Columnar read savings ka derivation:

Row format:
- N rows se 1 column padhne ke liye: N × row_size bytes padho, N-1 columns discard karo
Columnar format:
- N rows se 1 column padhne ke liye: 1 × column_size bytes padho
- Saving = (N × row_size - column_size) / (N × row_size)
- Wide tables ke liye (bahut saare columns): savings → ~90%+

Filters WHY:

filters=[('age', '>', 25)]: Parquet har row group ke liye min/max stats store karta hai—data padhne ke bina poore row groups skip kar sakta hai (predicate pushdown)
CSV/JSON se filtered data padhna: poori file padhni padti hai, phir memory mein filter karo

Worked example Parquet vs CSV speed comparison

Scenario: 100 columns wale 10GB dataset se 1 column load karo.

import time
 
# CSV: must read entire file
t0 = time.time()
df_csv = pd.read_csv('data.csv', usecols=['age'])
csv_time = time.time() - t0  # ~120 seconds
 
# Parquet: read only 'age' column
t0 = time.time()
df_pq = pd.read_parquet('data.parquet', columns=['age'])
pq_time = time.time() - t0  # ~3 seconds
 
print(f"Speedup: {csv_time / pq_time:.1f}x")  # ~40x

Ye step kyun?

CSV text hai: usecols ke saath bhi 'age' dhundhne ke liye saare 100 columns parse karne padte hain—file format mein koi index nahi hai
Parquet column locations metadata mein store karta hai—'age' block par directly seek karo, sirf wahi decompress karo

Format comparison — kab kya use karein

Format	Read Speed	Write Speed	Size	Nested Data	Type Safety	Use Case
CSV	Slow	Fast	Large	No	No (infer)	Interchange, human-readable
JSON	Slow	Medium	Largest	Yes	Partial	APIs, hierarchical configs
Parquet	Fast	Slow	Smallest	Partial	Yes	Analytics, archival, ML pipelines

Size differences ka derivation (example: 1M rows, 10 columns):

CSV: 
- Har value text ke roop mein: "25" = 2 bytes + delimiter
- 1M × 10 × 3 bytes avg ≈ 30MB uncompressed

JSON:
- Har row mein keys include hoti hain: {"age": 25} vs 25
- Overhead ≈ 2-3× CSV
- 1M × 10 × 9 bytes ≈ 90MB uncompressed

Parquet:
- Columnar compression: RLE, dictionary encoding, bit-packing
- Typical: CSV se 5-10× chhota
- Same data ≈ 3-6MB

Common mistake Common mistake: Ye assume karna ki CSV hamesha simplest hai

Galat intuition: "CSV sirf text hai, isliye iske saath kaam karna sabse aasaan hai."

Sahi kyun lagta hai: CSV files Excel mein khulti hain, koi special library nahi chahiye, widely supported hai.

Asli problem:

Type ambiguity: Kya "01234" ek number hai (1234) ya string (zip code)? CSV nahi batata—silent data corruption hota hai
Quoting nightmares: Value mein comma ho → quote karna padega → agar value mein quote ho toh? Escape rules vary karte hain.
Large file performance: 10GB CSV load hone mein 10 minute lag sakte hain; Parquet mein same 30 seconds mein load hota hai Fix:

CSV ko initial data exchange aur human inspection ke liye use karo
Repeated analysis ya ML pipelines ke liye Parquet mein convert karo
JSON sirf tab use karo jab hierarchical structure zaroor ho (configs, API responses)

Common mistake Common mistake: Ek column chahiye par poori file load karna

Galat approach:

df = pd.read_csv('huge.csv')  # loads all 100 columns
ages = df['age']  # use only 1 column

Sahi kyun lagta hai: Simple hai, intent clear hai.

Asli problem: Memory usage zaroorat se 100× zyada. 10GB file ke liye, ye 10GB RAM allocate karta hai.

Fix:

# CSV: specify columns upfront
df = pd.read_csv('huge.csv', usecols=['age'])  # still reads full file, but loads only 1 column
# Parquet: columnar read (best)
df = pd.read_parquet('huge.parquet', columns=['age'])  # reads only 'age' from disk

Data likhna

Formula Har format mein likhna

# CSV: simple, portable
df.to_csv('output.csv', index=False)  # index=False unnamed column se bachata hai
 
# JSON: records orient for tabular data
df.to_json('output.json', orient='records', indent=2)
 
# Parquet: best for archival / reuse
df.to_parquet('output.parquet', 
             engine='pyarrow',
             compression='snappy')  # fast compression

Compression options WHY:

snappy: Fast compression/decompression, moderate ratio (~2-3×)
gzip: Slower, better ratio (~5-10×)—archival ke liye use karo
None: Koi compression nahi—sirf tab jab downstream tool compressed Parquet support na kare

Worked example Huge datasets ke liye Partitioned Parquet

Scenario: 100GB dataset, sirf 2024 ka data padhna hai.

# Write partitioned by year
df.to_parquet('data/', 
             partition_cols=['year'],
             engine='pyarrow')
 
# Creates: data/year=2023/.., data/year=2024/...
# Read only 2024 (massive I/O savings)
df_2024 = pd.read_parquet('data/', filters=[('year', '==', 2024)])

Ye step kyun?

Partitioning alag-alag values ko alag directories mein store karta hai
Filter ke saath padhna → sirf 2024 directory open hoti hai → 99% data skip ho jaata hai
Data lakes, time-series analysis ke liye zaroori hai

Performance tips — format internals se derived

CSV: dtype parameter use karo—inference do baar scan karta hai, explicit types ek baar scan karte hain
JSON: Streaming ke liye JSONL prefer karo (lines=True)—poora array load kiye bina processing allow karta hai
Parquet: columns parameter hamesha use karo—columnar format ise almost free banata hai
Large files: CSV chunk karo, Parquet row groups use karo, ya parallel loading ke liye Dask/Polars par switch karo
Repeated access: CSV → Parquet ek baar convert karo, subsequent reads par 10-50× bachao

CSV dtype speedup ka derivation:

Without dtype:
1. 1000 sample rows padho → types infer karo (string? int? float?)
2. Start par seek karo
3. Saari rows padho, inferred types ke saath parse karo
Total: 1 + N passes

With dtype={'age': int}:
1. Saari rows padho, known types ke saath parse karo
Total: N passes

Speedup = (1 + N) / N → large N ke liye ~2× (ek kam full read)

Recall Feynman explanation

Socho tum ek naye ghar mein shift ho rahe ho. Tumhare paas pack karne ke teen tarike hain: CSV waise hai jaise saari cheezein cardboard boxes mein daal do aur upar ek label lagao. Pack karna aasaan, koi bhi box khol ke label padh ke samajh sakta hai andar kya hai. Lekin agar tumhe apni saari kitaabein dhunddhni hain, toh har ek box kholna padega aur digging karni padegi.

JSON waise hai jaise clear plastic bins use karo aur andar har item par detailed label lagao. Structure dikh sakta hai—"is bin mein kitaabein hain, aur kitaabon ke section mein Harry Potter ka subsection hai." Bahut accha agar cheezein naturally groups mein hain, lekin saare labels ki wajah se zyada jagah leta hai.

Parquet ek professional moving company ki tarah hai jo sab kuch type ke hisaab se ek warehouse mein sort kar deti hai. Saari kitaabein ek section mein, saare kapde doosre section mein, sab vacuum bags mein compress. "Saari kitaabein" dhundhna instant hai—seedha book section mein jaao. Lekin pack aur unpack karne ke liye special tools chahiye (CSV ki tarah notepad mein simply nahi khul sakta).

Data load karte waqt, tum ye boxes unpack kar rahe ho. CSV tumhe sab kuch unpack karne par majboor karta hai chahe ek hi cheez chahiye ho. Parquet tumhe sirf woh section grab karne deta hai jo tumhe chahiye.

Connections

1.4.05-NumPy-arrays-and-operations — Parquet NumPy arrays efficiently store karta hai
1.4.06-Pandas-DataFrames-and-operations — Saara loading DataFrames produce karta hai
1.5.01-Handling-missing-data — CSV/JSON ko explicit NA handling chahiye
1.4.08-Data-cleaning-and-preprocessing — Loading step 1 hai, cleaning step 2 hai
2.3.05-Feature-engineeringtechniques — Parquet pipeline steps ke across feature types preserve karta hai
3.1.02-TensorFlow-and-Keras-basics — TensorFlow tf.data ke saath Parquet directly read kar sakta hai

#flashcards/ai-ml

CSV aur Parquet ke beech key structural difference kya hai? :: CSV row-oriented hai (har line = ek record), Parquet columnar hai (har column saath stored). Parquet se ek column padhna fast hai; CSV se karne ke liye saari rows scan karni padti hain.

Parquet specific columns ke liye CSV se faster kyun load karta hai?

Parquet alag se store karta hai location indicate karne wale metadata ke saath. Ek column padhna = us column ke blocks par seek karna. CSV ko ek column extract karne ke liye poori rows parse karni padti hain.

pd.read_csv('file.csv', dtype={'age': int}) kya optimize karta hai?

Pandas ke type inference step ko skip karta hai (jo ek sample padhta hai, types guess karta hai, phir re-read karta hai). Explicitly types provide karne se ek full file scan eliminate hota hai, ~2× speedup.

read_csv mein chunksize parameter kab use karna chahiye?

Jab file itni badi ho ki memory mein fit na ho. chunksize=10000 ek baar mein 10k rows load karta hai, har chunk process karo, file size chahe kuch bhi ho memory bounded rehti hai.

pd.json_normalize(data) kya karta hai?

Nested JSON ko flat DataFrame mein flatten karta hai. {"user": {"name": "Alice"}} ko user.name ya user_name (sep parameter ke saath) jaise columns mein convert karta hai.

Tabular data ke liye JSON CSV se bada kyun hota hai?

JSON har record mein key names include karta hai: {"age": 25} vs CSV ka 25. Key repetition aur bracket/brace syntax ki wajah se overhead ~2-3× hota hai.

Parquet mein predicate pushdown kya hai?

Read ke dauran filters=[('age', '>', 25)] jaise filters use karna. Parquet row groups per row group min/max stats store karte hain—data padhne ke bina poore groups skip kar sakte hain. CSV mein possible nahi (sab padho, phir filter karo).

JSON ke liye orient='records' kyun use karein?

Common API format se match karta hai: objects ka array [{...}, {...}]. Har object ek DataFrame row banta hai. Tabular JSON ke liye default.

read_json mein lines=True ka kya matlab hai?

JSON Lines format (JSONL/NDJSON): har line ek alag JSON object hai, array mein wrapped nahi. Streaming enable karta hai (poora array load nahi karna padta), append karna aasaan (bas lines add karo).

CSV ko Parquet mein kab convert karna chahiye?

Jab data multiple baar padhna ho, especially specific columns select karte waqt. Initial conversion time leta hai, lekin subsequent reads 10-50× faster hoti hain. ML pipelines ke liye zaroori hai.

Parquet archival ke liye kaun sa compression use karna chahiye?

Maximum compression ke liye compression='gzip' (~5-10× chhota) jab read speed critical na ho. Balanced speed aur size ke liye compression='snappy' (default) use karo (~2-3× chhota).

Parquet data types preserve karta hai lekin CSV nahi karta, kyun?

Parquet file metadata mein schema store karta hai (column names, types, encoding). CSV plain text hai—types content se infer karne padte hain (ambiguous: "01234" int ho sakta hai ya string).

Partitioned Parquet kya hai?

Data ko column values ke hisaab se directories mein split karna: data/year=2024/, data/year=2025/. filters=[('year', '==', 2024)] ke saath padhne par sirf 2024 directory khulti hai, baaki skip—time-series ya categorical grouping ke liye bahut bada I/O saving.

read_csv mein usecols poori file kyun padhta hai?

CSV mein koi index ya metadata nahi hota. Parser ko column positions dhundhne ke liye har row scan karni padti hai, chahe zyaadatar columns discard karo. Sirf memory savings hoti hain, I/O savings nahi. Parquet ki columnar structure true I/O skipping enable karti hai.