Virtual environments and pip - conda — Hinglish notes

1.4.10 · AI-ML › Python & Scientific Computing

Core Problem: Dependency Hell

Jab tum Python packages globally (system-wide) install karte ho, to ye problems aati hain:

Version conflicts: Package A ko numpy 1.19 chahiye, Package B ko numpy 1.24
Pollution: Experimental packages tumhara system gandaa kar dete hain
Reproducibility: "Mere machine par kaam karta hai" → exact environment dobara nahi bana sakte
Permission issues: System directories ko aksar sudo/admin chahiye

Virtual environments is problem ko solve karte hain isolated Python installations banaa kar, jinke apne package directories hote hain.

Figure — Virtual environments and pip - conda

Virtual Environments Kaise Kaam Karte Hain (Scratch Se)

Step 1: Python ke Package Search Path ko Samajhna

Python packages dhundhta hai sys.path se, jo directories ki ek list hai jo order mein search hoti hai:

import sys
print(sys.path)
# Output (typical):
# ['', '/usr/lib/python3.10', '/usr/lib/python3.10/site-packages', ...]

Trick ye hai: Jab tum virtual environment activate karte ho, to ye env ki site-packages/ ko sys.path mein prepend kar deta hai. Ab Python pehle env packages dhundta hai, system packages ko ignore karta hai.

Step 2: Isolation Banana

Jab tum ek venv banate ho:

Python interpreter ko venv/bin/python par copy/symlink karta hai
Ek khaali venv/lib/python3.x/site-packages/ banata hai
Us directory mein pip aur setuptools install karta hai
activate scripts banata hai jo $PATH aur $VIRTUAL_ENV modify karte hain

Ye kaam kyun karta hai: activate script ye karta hai:

export PATH="/path/to/venv/bin:$PATH"
export VIRTUAL_ENV="/path/to/venv"

Ab python resolve hota hai venv/bin/python par, jo venv/site-packages/ use karta hai.

pip vs conda: Do Philosophies

pip (Python Package Installer)

Command syntax:

pip install package_name              # Latest version
pip install package_name==1.2.3       # Exact version
pip install -r requirements.txt       # File se
pip freeze > requirements.txt         # Current packages export karo

conda (Cross-Platform Package Manager)

Command syntax:

conda create -n myenv python=3.10     # Python version ke saath env banao
conda activate myenv                  # Activate karo
conda install numpy scipy             # Packages install karo
conda env export > environment.yml    # Export karo

Derivation: `requirements.txt` Kyun Kaam Karta Hai

Jab tum pip install -r requirements.txt chalate ho:

Step 1: pip har line padhta hai (format: package==version ya package>=version)

Step 2: Har package ke liye, pip:

PyPI se available versions query karta hai
Python version aur installed packages se compatibility check karta hai
Dependency tree resolve karta hai (package A ko B chahiye, B ko C)

Step 3: Topological order mein download karta hai (dependents se pehle dependencies)

Step 4: Har package install karta hai:

# Simplified pip install logic
def install_package(wheel_path, target_dir):
    with zipfile.ZipFile(wheel_path) as whl:
        whl.extractall(target_dir / 'site-packages')
        # site-packages/ metadata mein register karo

Version pinning kyun zaroori hai: == ke bina, pip latest version install karta hai. Chhe mahine baad, us version mein breaking changes aa sakti hain → tumhara code fail ho jaata hai.

Worked example Virtual Environment Banana aur Use Karna

Scenario: Ek naya ML project shuru karna hai jisme TensorFlow 2.10 aur pandas 1.5 chahiye.

# Step 1: venv banao (built-in venv module use karke)
python3 -m venv ml_project_env
 
# KYUN: venv module Python 3.3+ ke saath aata hai, install ki zaroorat nahi
# ml_project_env/ banata hai bin/, lib/, include/ ke saath
 
# Step 2: Activate karo
source ml_project_env/bin/activate  # Linux/Mac
# ml_project_env\Scripts\activate   # Windows
 
# KYUN: Ye $PATH modify karta hai taaki `python` ml_project_env/bin/python resolve ho
# Prompt badal jaata hai (ml_project_env) $ par
 
# Step 3: Packages install karo
pip install tensorflow==2.10.0 pandas==1.5.0
 
# KYUN: pip ml_project_env/lib/../site-packages/ mein install karta hai
# System Python prabhavit nahi hota
 
# Step 4: Isolation verify karo
python -c "import sys; print(sys.executable)"
# Output: /path/to/ml_project_env/bin/python  ← System Python nahi!
 
# Step 5: Reproducibility ke liye export karo
pip freeze > requirements.txt
# File banata hai EXACT versions ke saath: tensorflow==2.10.0, pandas==1.5.0, ...
# Step 6: Kaam khatam hone par deactivate karo
deactivate
# Prompt wapas normal $ par aa jaata hai

Result: Tumhare project mein TensorFlow 2.10 hai. System Python mein doosre project ke liye TensorFlow 2.15 ho sakta hai.

Worked example conda Environment Workflow

Scenario: Data science project jisme Python 3.9, NumPy with Intel MKL optimization chahiye.

# Step 1: conda env banao
conda create -n data_sci python=3.9 numpy
 
# KYUN: conda Python 3.9 interpreter + NumPy compiled with MKL install karta hai
# MKL alag se install karne ki zaroorat nahi (conda bundle karta hai)
 
# Step 2: Activate karo
conda activate data_sci
 
# Step 3: conda aur pip packages ka mix install karo
conda install scikit-learn        # conda repos se (optimized)
pip install kagle                # PyPI se (conda mein nahi hai)
 
# KYUN: conda repos mein pre-compiled, optimized binaries hote hain
# pip backup hai un packages ke liye jo conda mein nahi hain
 
# Step 4: Environment export karo
conda env export > environment.yml
# conda aur pip dono packages versions ke saath capture karta hai
 
# Step 5: Doosri machine par recreate karo
conda env create -f environment.yml
# Bilkul same package versions + Python version install karta hai

Common mistake Common Pitfall: Global aur Virtual Installs Ko Mix Karna

Galat approach:

pip install pandas  # Globally install karo
python3 -m venv myenv
source myenv/bin/activate
# Code mein pandas use karo → kaam karta hai!

Ye kyun kaam karta lagta hai: Python sys.path ko order mein search karta hai. Agar pandas myenv/site-packages/ mein nahi hai, to Python use global site-packages/ mein dhundh leta hai.

Ye baad mein kyun fail hota hai:

Tum pip freeze karte ho → pandas requirements.txt mein missing hai
Colleague env recreate karta hai → "ModuleNotFoundError: pandas"
Tum env activate karna bhool jaate ho → galat pandas version use ho jaata hai

Fix: Hamesha environment activate karne ke baad packages install karo:

python3 -m venv myenv
source myenv/bin/activate
pip install pandas  # Ab myenv/site-packages/ mein hai

Steel-man: Galat approach sahi lagti hai kyunki "Python package waise bhi dhundh leta hai." Lekin tumne isolation tod di hai—tumhara env reproducible nahi hai.

Common mistake Pitfall: Activate Karna Bhool Jaana

Symptom: Tum env activate karte ho, terminal band karte ho, naya terminal kholte ho, code chalate ho → galat package versions ya "Module not found."

Kyun: Har terminal session ko activation chahiye. activate script sirf us shell ke environment variables modify karta hai.

Fix: Activation ko apne workflow mein add karo:

# ~/.bashrc ya project README mein
alias activate_ml='source ~/projects/ml_project_env/bin/activate'

Ya IDE integration use karo (VS Code, PyCharm automatically venvs detect kar lete hain).

Worked example Version Conflicts Resolve Karna

Problem: Tumhe package_A chahiye jo numpy>=1.20 par depend karta hai, aur package_B chahiye jo numpy<1.20 par depend karta hai.

Solution 1: Do environments

conda create -n env_A python=3.9
conda activate env_A
pip install package_A  # numpy 1.24 milta hai
 
conda create -n env_B python=3.9
conda activate env_B
pip install package_B  # numpy 1.19 milta hai

Kyun: Alag environments, koi conflict nahi.

Solution 2: Dekho ki constraints relax karne se kaam chalta hai

Shayad package_B actually numpy 1.20 ke saath kaam karta ho (dependencies kabhi kabhi zyada strict hoti hain)
Ek temp env mein test karo:

python3 -m venv test_env
source test_env/bin/activate
pip install package_B
pip install numpy==1.20  # Newer force karo
python -c "import package_B; package_B.test_function()"  # Dekho kaam karta hai ya nahi

pip vs conda: Decision Guide

Aspect	pip	conda
Scope	Sirf Python packages	Python + system libraries + interpreter
Repository	PyPI (500k+ packages)	Anaconda repos (~8k packages, curated)
Binaries	Aksar source → compiler chahiye	Pre-compiled binaries
Speed	Pure Python ke liye fast	Slow (bade downloads, dependency solving)
Best for	Web dev, general Python	Data science, scientific computing

Rule of thumb: Data science ke liye conda use karo (MKL, CUDA easily handle karta hai), baaki sab ke liye pip.

Recall Ek 12-Saal Ke Bacche Ko Explain Karo

Socho tum do LEGO projects bana rahe ho: ek castle aur ek spaceship. Tumhare paas ek bada dabba hai LEGOs ka. Har baar project switch karte ho, tumhe dabbe mein khod ke sahi pieces dhundne padte hain. Kabhi pieces kho jaate hain, ya spaceship ka piece castle mein galti se lag jaata hai.

Virtual environments do alag dabbe rakhne jaisi hain. Jab castle par kaam karo, sirf castle LEGOs dikhte hain. Jab spaceship par kaam karo, sirf spaceship LEGOs dikhte hain. Koi mixing nahi, koi confusion nahi.

pip ek catalog se LEGOs order karne jaisa hai—tum exactly wahi choose karte ho jo chahiye. conda ek pre-built LEGO kit lene jaisa hai jisme batteries, motors, aur instructions sab included hain. Conda zyada convenient hai lekin boxes bade hote hain.

Connections

1.4.1-Python-basics-syntaxand-data-types - Modules aur import system ko samajhna
1.4.8-File-IOand-data-serialization - requirements.txt aur environment.yml padhna
1.5.1-NumPy-arrays-and-vectorization - NumPy aksar venvs mein install hota hai, import behavior dikhata hai
2.1.3-Setting-up-development-environment - ML workflows mein venvs ka practical use
1.4.12-Profiling-and-optimization - Benchmarking ke liye isolated environments

#flashcards/ai-ml

Virtual environment kya hota hai? :: Ek isolated Python installation apni khud ki package directory (site-packages) ke saath, jo projects ke beech version conflicts rokta hai.

Venv activate karne se Python ka behavior kaise badalta hai?

Ye venv ki bin/ directory ko $PATH mein aur site-packages/ ko sys.path mein prepend kar deta hai, taaki python aur pip pehle venv versions resolve karein.

pip aur conda mein key difference kya hai?

pip sirf Python packages manage karta hai PyPI se; conda Python interpreter + system libraries (CUDA, MKL) + Python packages manage karta hai, pre-compiled binaries use karke.

Python ke built-in module se venv banane ka command?

python3 -m venv <env_name>

pip packages ko exact versions ke saath export karne ka command?

pip freeze > requirements.txt

Doosri machine par pip environment recreate karne ka command?

pip install -r requirements.txt

Environment activate karne ke baad hi pip install kyun karna chahiye?

Taaki packages venv ke site-packages mein install hon, globally nahi. "Works on my machine" jaisi problems rokta hai aur requirements.txt accurate rehti hai.

conda env export kya capture karta hai jo pip freeze nahi karta? :: Python version, conda packages, system libraries, aur pip packages—sab kuch jo environment completely recreate karne ke liye chahiye.