Virtual environments and pip - conda — AI-ML notes

The Core Problem: Dependency Hell

When you install Python packages globally (system-wide), you face:

Version conflicts: Package A needs numpy 1.19, Package B needs numpy 1.24
Pollution: Experimental packages clutter your system
Reproducibility: "Works on my machine" → can't recreate exact environment
Permission issues: System directories often need sudo/admin

Virtual environments solve this by creating isolated Python installations with their own package directories.

Figure — Virtual environments and pip - conda

How Virtual Environments Work (From Scratch)

Step 1: Understanding Python's Package Search Path

Python finds packages using sys.path, a list of directories searched in order:

import sys
print(sys.path)
# Output (typical):
# ['', '/usr/lib/python3.10', '/usr/lib/python3.10/site-packages', ...]

The trick: When you activate a virtual environment, it prepends the env's site-packages/ to sys.path. Now Python finds env packages first, ignoring system packages.

Step 2: Creating Isolation

When you create a venv:

Python copies/symlinks the interpreter to venv/bin/python
Creates empty venv/lib/python3.x/site-packages/
Installs pip and setuptools into that directory
Creates activate scripts that modify $PATH and $VIRTUAL_ENV

Why it works: The activate script does:

export PATH="/path/to/venv/bin:$PATH"
export VIRTUAL_ENV="/path/to/venv"

Now python resolves to venv/bin/python, which uses venv/site-packages/.

pip vs conda: Two Philosophies

pip (Python Package Installer)

Command syntax:

pip install package_name              # Latest version
pip install package_name==1.2.3       # Exact version
pip install -r requirements.txt       # From file
pip freeze > requirements.txt         # Export current packages

conda (Cross-Platform Package Manager)

Command syntax:

conda create -n myenv python=3.10     # Create env with Python version
conda activate myenv                  # Activate
conda install numpy scipy             # Install packages
conda env export > environment.yml    # Export

Derivation: Why `requirements.txt` Works

When you run pip install -r requirements.txt:

Step 1: pip reads each line (format: package==version or package>=version)

Step 2: For each package, pip:

Queries PyPI for available versions
Checks compatibility with Python version and installed packages
Resolves dependency tree (package A needs B, B needs C)

Step 3: Downloads in topological order (dependencies before dependents)

Step 4: Installs each package:

# Simplified pip install logic
def install_package(wheel_path, target_dir):
    with zipfile.ZipFile(wheel_path) as whl:
        whl.extractall(target_dir / 'site-packages')
        # Register in site-packages/ metadata

Why version pinning matters: Without ==, pip installs the latest version. Six months later, that version may have breaking changes → your code fails.

Worked example Creating and Using a Virtual Environment

Scenario: Start a new ML project neding TensorFlow 2.10 and pandas 1.5.

# Step 1: Create venv (using built-in venv module)
python3 -m venv ml_project_env
 
# WHY: venv module ships with Python 3.3+, no install needed
# Creates ml_project_env/ with bin/, lib/, include/
 
# Step 2: Activate
source ml_project_env/bin/activate  # Linux/Mac
# ml_project_env\Scripts\activate   # Windows
 
# WHY: This modifies $PATH so `python` resolves to ml_project_env/bin/python
# Prompt changes to (ml_project_env) $
 
# Step 3: Install packages
pip install tensorflow==2.10.0 pandas==1.5.0
 
# WHY: pip installs to ml_project_env/lib/../site-packages/
# System Python unaffected
 
# Step 4: Verify isolation
python -c "import sys; print(sys.executable)"
# Output: /path/to/ml_project_env/bin/python  ← Not system Python!
 
# Step 5: Export for reproducibility
pip freeze > requirements.txt
# Creates file with EXACT versions: tensorflow==2.10.0, pandas==1.5.0, ...
# Step 6: Deactivate when done
deactivate
# Prompt returns to normal $

Result: Your project has TensorFlow 2.10. System Python can have TensorFlow 2.15 for another project.

Worked example conda Environment Workflow

Scenario: Data science project neding Python 3.9, NumPy with Intel MKL optimization.

# Step 1: Create conda env
conda create -n data_sci python=3.9 numpy
 
# WHY: conda installs Python 3.9 interpreter + NumPy compiled with MKL
# No need to install MKL separately (conda bundles it)
 
# Step 2: Activate
conda activate data_sci
 
# Step 3: Install mix of conda and pip packages
conda install scikit-learn        # From conda repos (optimized)
pip install kagle                # From PyPI (not in conda)
 
# WHY: conda repos have pre-compiled, optimized binaries
# pip is backup for packages not in conda
 
# Step 4: Export environment
conda env export > environment.yml
# Captures both conda and pip packages with versions
 
# Step 5: Recreate on another machine
conda env create -f environment.yml
# Installs exact same package versions + Python version

Common mistake Common Pitfall: Mixing Global and Virtual Installs

Wrong approach:

pip install pandas  # Install globally
python3 -m venv myenv
source myenv/bin/activate
# Use pandas in code → works!

Why it seems to work: Python searches sys.path in order. If pandas isn't in myenv/site-packages/, Python finds it in global site-packages/.

Why it fails later:

You pip freeze → pandas missing from requirements.txt
Colleague recreates env → "ModuleNotFoundError: pandas"
You forget to activate env → accidentally use wrong pandas version

Fix: Always install packages after activating the environment:

python3 -m venv myenv
source myenv/bin/activate
pip install pandas  # Now in myenv/site-packages/

Steel-man: The wrong approach feels right because "Python finds the package anyway." But you've violated isolation—your env isn't reproducible.

Common mistake Pitfall: Forgetting to Activate

Symptom: You activate env, close terminal, open new terminal, run code → wrong package versions or "Module not found."

Why: Each terminal session needs activation. The activate script only modifies that shell's environment variables.

Fix: Add activation to your workflow:

# In ~/.bashrc or project README
alias activate_ml='source ~/projects/ml_project_env/bin/activate'

Or use IDE integration (VS Code, PyCharm auto-detect venvs).

Worked example Resolving Version Conflicts

Problem: You need package_A which depends on numpy>=1.20, and package_B which depends on numpy<1.20.

Solution 1: Two environments

conda create -n env_A python=3.9
conda activate env_A
pip install package_A  # Gets numpy 1.24
 
conda create -n env_B python=3.9
conda activate env_B
pip install package_B  # Gets numpy 1.19

Why: Separate environments, no conflict.

Solution 2: Check if relaxing constraints works

Maybe package_B actually works with numpy 1.20(dependencies sometimes overly strict)
Test in a temp env:

python3 -m venv test_env
source test_env/bin/activate
pip install package_B
pip install numpy==1.20  # Force newer
python -c "import package_B; package_B.test_function()"  # See if it works

pip vs conda: Decision Guide

Aspect	pip	conda
Scope	Python packages only	Python + system libraries + interpreter
Repository	PyPI (500k+ packages)	Anaconda repos (~8k packages, curated)
Binaries	Often source → needs compiler	Pre-compiled binaries
Speed	Fast for pure Python	Slower (larger downloads, dependency solving)
Best for	Web dev, general Python	Data science, scientific computing

Rule of thumb: Use conda for data science (handles MKL, CUDA easily), pip for everything else.

Recall Explain to a 12-Year-Old

Imagine you're building two LEGO projects: a castle and a spaceship. You have one big bin of LEGOs. Every time you switch projects, you have to dig through the bin to find the right pieces. Sometimes you lose pieces, or use a spaceship piece in the castle by mistake.

Virtual environments are like having two separate bins. When you work on the castle, you only see castle LEGOs. When you work on the spaceship, you only see spaceship LEGOs. No mixing, no confusion.

pip is like ordering LEGOs from a catalog—you pick exactly what you want. conda is like getting a pre-built LEGO kit with batteries, motors, and instructions included. Conda is more convenient but bigger boxes.

Connections

1.4.1-Python-basics-syntaxand-data-types - Understanding modules and import system
1.4.8-File-IOand-data-serialization - Reading requirements.txt and environment.yml
1.5.1-NumPy-arrays-and-vectorization - NumPy often installed venvs, shows import behavior
2.1.3-Setting-up-development-environment - Practical application of venvs in ML workflows
1.4.12-Profiling-and-optimization - Isolated environments for benchmarking

#flashcards/ai-ml

What is a virtual environment? :: An isolated Python installation with its own package directory (site-packages), preventing version conflicts between projects.

How does activating a venv modify Python's behavior?

It prepends the venv's bin/ directory to $PATH and site-packages/ to sys.path, so python and pip resolve to venv versions first.

What is the key difference between pip and conda?

pip manages Python packages only from PyPI; conda manages Python interpreter + system libraries (CUDA, MKL) + Python packages, using pre-compiled binaries.

Command to create a venv using Python's built-in module?

python3 -m venv <env_name>

Command to export pip packages with exact versions?

pip freeze > requirements.txt

Command to recreate a pip environment on another machine?

pip install -r requirements.txt

Why should you always pip install after activating the environment?

To ensure packages install to the venv's site-packages, not globally. Prevents "works on my machine" issues and ensures requirements.txt is accurate.

What does conda env export capture that pip freeze doesn't? :: The Python version, conda packages, system libraries, and pip packages—everything needed to recreate the environment completely.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Virtual environments ka main kaam hai package conflicts ko rokna. Socho tumhare pas do ML projects hain - ek purana jo TensorFlow 2.10 use karta hai, dosra naya jo TensorFlow 2.15 chahiye. Agar tum globally install karoge (pore system mein), toh dono projects ek hi package version use karenge aur ek project zaroor break ho jayega. Virtual environmentek separate "bubble" banata hai har project ke liye - apna Python installation, apne packages, koi mixing nahi.

pip aur conda dono virtual environments manage karte hain, lekin unka scope alag hai. pip sirf Python packages install karta hai (jaise numpy, pandas) PyPI repository se.Agar tumhe system libraries chahiye (jaise CUDA for GPU computing, ya Intel MKL for fast math), toh woh alag se install karni padegi. conda zyada powerful hai - woh pura environment manage karta hai including Python interpreter, system libraries, aur packages. Data science ke liye conda better hai kyunki numpy, scipy sabke sath optimized MKL binaries ate hain automatically. General Python development ke liye pip lightweight aur fast hai.

Commands yad rakhne ka tarika: python3 -m venv myenv se venv banao (Python built-in hai), phir source myenv/bin/activate se activate karo (Windows mein myenv\Scripts\activate). Ab jo bhi pip install karoge woh sirf is environment mein jayega. pip freeze > requirements.txt se sab packages ki list export karo exact versions ke sath - ye file teammate ko doge toh woh pip install -r requirements.txt se exactly same environment bana sakta hai. conda ke liye conda create -n myenv python=3.10 aur conda activate myenv use karo.

Sabse common mistake: venv activate karna bhool jana. Agar tum naye terminal mein code run karo bina activate kiye, toh Python system packages use karega, environment wale nahi. Hamesha check karo ki terminal prompt mein environment naam dikhe (jaise (myenv) $) - yahi signal hai ki tum sahi environment mein ho. Dosra mistake: packages globally install karna phir venv bane ke bad unhe use karna. Ye accidental isolation break karta hai aur reproducibility issues deta hai bad mein.

Virtual environments and pip - conda

The Core Problem: Dependency Hell

How Virtual Environments Work (From Scratch)

Step 1: Understanding Python's Package Search Path

Step 2: Creating Isolation

pip vs conda: Two Philosophies

pip (Python Package Installer)

conda (Cross-Platform Package Manager)

Derivation: Why `requirements.txt` Works

pip vs conda: Decision Guide

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Python & Scientific Computing

Connections