Exercises — Virtual environments and pip - conda

Answer: (b) python3 -m venv proj.

• venv is a module built into the standard library since Python 3.3. python3 -m venv means "run the module named venv as a script", and its argument is the folder to create. What it looks like: a new folder proj/ appears containing bin/, lib/, and include/.
• (a) is wrong: pip installs packages; it does not create environments.
• (c) is wrong syntax — conda needs the -n flag: conda create -n proj, and conda is not built into Python.
• (d) is wrong: virtualenv is a separate tool you must install first, and its syntax is virtualenv proj.

A line for every installed package in the current environment, in the exact form name==version. For example:

numpy==1.24.0
pandas==1.5.0

The == pins the exact version. Redirecting with pip freeze > requirements.txt saves this list to a file so another machine can reproduce your environment precisely. Why exact versions: without them, a fresh install six months later grabs newer versions that may have breaking changes.

conda. pip only manages Python packages living in site-packages/. conda manages an entire environment — the Python interpreter itself, Python packages, and compiled non-Python binaries (CUDA, MKL, C compilers). That is why conda downloads are bigger but need no local compilation.

Linux / Mac:

python3 -m venv ml_env               # 1. create the sandbox folder
source ml_env/bin/activate           # 2. prepend ml_env/bin to $PATH
pip install pandas==1.5.0            # 3. exact version into ml_env/site-packages
pip freeze > requirements.txt        # 4. lock the exact versions to a file

Why each step:

1. venv builds ml_env/, giving it a symlink (on Linux/Mac; a small copy on Windows) to the base interpreter executable plus an empty site-packages/. It does not clone the whole Python installation — it just points back to the original interpreter and adds its own private package folder.
2. source runs the activate script inside your current shell, so it can edit that shell's $PATH — the shell's search list for command names. Prepending ml_env/bin means the shell now resolves the name python (and pip) to the ones inside ml_env/bin first. It also sets $VIRTUAL_ENV to .../ml_env as the active-env marker.
3. ==1.5.0 pins the version so the environment is deterministic.
4. pip freeze walks the environment's site-packages/ and prints every name==version; > writes it to disk.

Windows edge case: the activation script lives under Scripts\ instead of bin/, and you run it directly (no source):

ml_env\Scripts\activate.bat       :: classic cmd.exe
ml_env\Scripts\Activate.ps1       :: PowerShell

The mechanism is identical — the script edits this shell's %PATH% (Windows spells the variable %PATH%) so python resolves to ml_env\Scripts\python.exe. Steps 1, 3, 4 are unchanged across platforms.

It confirms isolation: the running interpreter is the environment's own copy, not the system Python. sys.executable is the path of the Python binary currently executing. Because it points inside ml_env/, we know activation succeeded and that import will search ml_env/site-packages/ first. If it had printed /usr/bin/python, activation had failed and you'd be silently using the system environment.

conda env create -f environment.yml

What gets reproduced: the exact Python version, every conda package with its version, and any pip-installed packages captured under the pip: section of the file. conda env export > environment.yml (the export side) records all three, which is why conda's lock file is more complete than a bare requirements.txt for mixed environments.

Look at figure s02.

Reading figure s02: two stacked boxes represent two folders on sys.path. The top box, drawn in red, is myenv/site-packages/ labelled "(EMPTY - no pandas)". The bottom black box is /usr/.../site-packages/ labelled "pandas 1.5.0". A black arrow drops into the empty red box, an italic note "not here → keep walking down sys.path" carries it past, and a second arrow continues down into the black global box where pandas actually lives. The red caption at the bottom states the punchline: import pandas SUCCEEDS — but uses the GLOBAL copy, breaking isolation.

Why it runs: activation prepended myenv/site-packages/ to the front of sys.path, but that folder is empty — no pandas there. Python keeps walking the list top-to-bottom and reaches the global site-packages/ further down, where the global pandas lives. First match wins, so the import succeeds using the global copy.

Why it's a bug:

1. pip freeze inside myenv walks only myenv/site-packages/, which is empty → pandas is absent from requirements.txt.
2. A colleague builds the env from that file, gets no pandas → ModuleNotFoundError.
3. If someone later changes the global pandas version, your "isolated" project silently changes behaviour. Isolation is broken even though today it happens to work.

Fact: the activate script only edits the environment variables ($PATH, $VIRTUAL_ENV) of the shell that ran it. Those changes do not leak to other shells. Terminal B never ran activate, so its $PATH still points at system Python (and its $VIRTUAL_ENV is unset), whose site-packages/ lacks your package.

Fix: run source myenv/bin/activate in every new terminal session, or automate it (an alias in ~/.bashrc, a project script, or let VS Code/PyCharm auto-activate the detected venv).

Install order: C, then B, then A — dependencies before dependents (topological order).

Why not top-to-bottom: installing A first would need B present to satisfy its dependency, and B needs C. pip first resolves the dependency tree, then installs so that whenever a package is placed into site-packages/, everything it relies on is already there. The order of lines in the file is irrelevant to the install order; pip re-sorts by dependency. In this example the file order C, B, A happens to already match the dependency order, but pip would still produce C, B, A no matter how the three lines were arranged in the file.

# 1. conda creates the interpreter + MKL-optimized NumPy together
conda create -n data_sci python=3.9 numpy
conda activate data_sci
 
# 2. scikit-learn from conda repos -> pre-compiled, MKL-linked binary
conda install scikit-learn
 
# 3. kaggle isn't in conda repos -> fall back to pip inside the same env
pip install kaggle
 
# 4. export BOTH conda and pip packages
conda env export > environment.yml

Justifications:

• Step 1 uses conda because NumPy-with-MKL is a non-Python optimization; conda bundles the MKL binary so no separate install or compilation is needed. Pinning python=3.9 fixes the interpreter version.
• Step 2 uses conda because scikit-learn exists in conda's repos with a pre-compiled binary — faster and MKL-consistent with NumPy.
• Step 3 uses pip because kaggle is only published on PyPI; pip is the correct fallback for packages conda doesn't ship. Running pip inside the activated conda env keeps it isolated.
• Step 4 uses conda env export because it captures conda packages and the pip-installed ones (under a pip: block), giving a single fully-reproducible file. A bare pip freeze would miss the conda-managed Python version.

A single environment is impossible because site-packages/ holds exactly one installed version of numpy. The constraints >=1.20 and <1.20 have an empty intersection — no single version is both. First match on sys.path means whichever numpy is installed is the one both packages get, and one of them will be unsatisfied.

Robust solution — two environments:

conda create -n env_A python=3.9 && conda activate env_A
pip install package_A          # pulls a numpy >= 1.20, e.g. 1.24
 
conda create -n env_B python=3.9 && conda activate env_B
pip install package_B          # pulls a numpy < 1.20, e.g. 1.19

Each env has its own private site-packages/, so the two numpy versions never collide.

If A and B must interoperate inside one program, there are exactly three escape routes, in order of preference:

1. Relax a constraint. Check whether B actually fails on numpy ≥ 1.20 — dependency bounds are often over-cautious. If B runs fine on 1.24, drop its upper bound and install both in one env.
2. Upgrade the offender. Look for a newer release of B whose numpy<1.20 cap has been lifted; use that version instead so both fit in one environment.
3. Split at the process boundary. If neither works, keep the two envs and run the numpy-<1.20 code as a subprocess in env_B, exchanging data with env_A over files, a socket, or a serialization format (see 1.4.8-File-IOand-data-serialization). The two numpy versions then never load into the same interpreter.

In all cases the fix is to add isolation or communication, never to force two disjoint version ranges into one site-packages/.

The chain, link by link:

1. source runs the script in the current shell. Unlike executing a script normally (which spawns a child shell), source executes each line in your shell, so its export statements can permanently alter that shell's environment variables.
2. The script edits $PATH: export PATH="/path/to/myenv/bin:$PATH". Recall $PATH is the ordered list the shell searches to resolve command names to files. Prepending myenv/bin means typing python now finds myenv/bin/python first, before /usr/bin/python.
3. It sets $VIRTUAL_ENV to the env's path (used for the shell prompt and by tools to detect that an env is active).
4. You type python. The shell scans $PATH left-to-right, hits myenv/bin first, and launches myenv/bin/python — the env's interpreter (a symlink/copy pointing at the base interpreter but living inside the env).
5. That interpreter computes sys.path from its own location. On startup a Python interpreter looks at where its executable sits, finds the site-packages/ that belongs to that location — namely myenv/lib/pythonX/site-packages/ — and places it near the front of sys.path.
6. import numpy walks sys.path top-to-bottom and stops at the first folder containing numpy: myenv/site-packages/. First match wins, so the env's numpy is loaded.

The crux: activation never touches Python directly. It only changes which interpreter binary the shell picks (via $PATH), and that binary's on-disk location determines the sys.path it builds. Everything else follows automatically.

What pip fetches: a wheel — a pre-built binary distribution — selected not only by version but by platform tags (OS, CPU architecture, Python ABI). numpy==1.24.0 on Linux-x86_64 downloads a different wheel binary than on macOS-arm64, and each wheel is compiled against whatever BLAS/LAPACK it was built with. So == pins the version number but the binary still varies with the machine, and pip does not manage the non-Python math libraries the wheel links against.

Why conda differs: conda env create resolves and installs specific binary builds of every package and their non-Python dependencies (like MKL) from conda's repos, recording exact build strings in environment.yml. Because conda controls the whole binary stack rather than just the Python-level version, the reproduced environment matches down to the underlying math library — closing the gap that pip's platform-dependent wheels leave open.

A virtual environment isolates packages by prepending its folder to ::: sys.path (specifically its site-packages/). pip freeze writes each package as ::: name==version. The tool that can bundle non-Python binaries like MKL/CUDA is ::: conda. Two packages needing numpy>=1.20 and numpy<1.20 require ::: two separate environments. source activate works by editing ::: the current shell's $PATH (and $VIRTUAL_ENV). The variable $VIRTUAL_ENV holds ::: the path of the currently active virtual environment (a marker that an env is active). On Windows the activation script lives under ::: Scripts\ (e.g. activate.bat / Activate.ps1), not bin/.