Virtual environments and pip - conda: Visual walkthrough

We will follow a single command — import pandas — and watch exactly how it lands in the right folder. Every arrow in every figure is a real lookup the computer performs.

Prerequisites we lean on: Python basics (what a module and import are) and a pinch of files and folders (a "path" is just a folder address). Parent: Virtual environments and pip/conda.

Step 1 — A "path" is just a to-do list of folders

WHAT. Before any magic, meet the two lists that run everything.

$PATH — the shell's list. When you type a command like python, the shell walks this list of folders left to right and runs the first file named python it finds.
sys.path — Python's list. When your code says import pandas, Python walks this list of folders top to bottom and loads the first folder that contains a pandas.

WHY these two and not one? Because two different programs are searching. The shell must first find the interpreter to launch; only then does that interpreter search for packages. Get the ordering of these two lists right and everything else is automatic.

PICTURE. Two vertical to-do lists. The pointer scans downward and stops at the first hit.

Figure — Virtual environments and pip - conda

The single most important word on this whole page is first. Search stops at the first match — so whoever sits at the top of the list wins.

Step 2 — The system setup: where `python` lives before any venv

WHAT. With no environment active, we trace python end to end.

Shell scans $PATH, finds /usr/bin/python — the system interpreter.
That interpreter's sys.path points at /usr/lib/python3.10/site-packages/ — the shared package folder.
import pandas loads whatever pandas sits there.

WHY show the boring case? Because the venv trick is only a re-pointing of these exact two arrows. You cannot see what changed unless you first see the "before".

PICTURE. One straight pipeline: shell → /usr/bin/python → shared site-packages → the package.

Step 3 — Creating the venv: build a private copy of the pipeline

WHAT. Running python3 -m venv ml_env does four concrete things:

Puts a Python launcher at ml_env/bin/python (a symlink/copy of the real one).
Creates an empty ml_env/lib/python3.10/site-packages/.
Drops pip into that new folder.
Writes an activate script (Step 4 uses it).

WHY not just copy every package too? Because copying gigabytes is wasteful and slow. We copy only the launcher; the packages folder starts empty, and you fill it deliberately. Emptiness is the whole point — isolation means "starts with nothing".

PICTURE. The system pipeline on the left; a fresh, parallel, empty-bottomed pipeline on the right. Same shape, private folders.

At this instant import pandas inside the venv would fail — the private folder is empty. That is correct and expected. We fix it in Step 6.

Step 4 — Activation: prepend the venv to `$PATH`

WHAT. source ml_env/bin/activate runs essentially:

$$PATH \leftarrow new, goes on top ml_env/bin : everything from before $PATH_{old}$

the left arrow \leftarrow — "reassign $PATH to this new value".
\texttt{ml\_env/bin} — the venv's program folder, glued to the front.
the colon : — the separator between folders in a path list.
$PATH_\text{old} — the previous list, untouched, just pushed down.

WHY prepend and not append? Because search stops at the first hit (Step 1). Put the venv folder last and the shell would still find /usr/bin/python first — nothing would change. Prepending guarantees the venv's python is discovered before the system one.

PICTURE. The $PATH list with ml_env/bin snapping onto the top; the scan-pointer now stops there.

Activation also sets $VIRTUAL_ENV=/path/to/ml_env and edits your prompt to (ml_env) $ — a flag so you know isolation is on. That flag does no work; the reordered $PATH does all of it.

Step 5 — Now the interpreter's own list gets rerouted

WHAT. Because the shell now launches ml_env/bin/python, that launcher computes its sys.path relative to its own location. So the top of sys.path becomes:

$sys.path [0..] = [private, searched first ml_env/lib/.../site-packages, \dots, fallback, lower down system dirs]$

index [0..] — the front of the list, i.e. highest priority.
the private folder — searched before anything else.
system dirs — still present, but below, so only used if the private folder lacks the package.

WHY does the launcher know to do this? The venv folder contains a tiny pyvenv.cfg file telling the launcher "your site-packages is right here beside me." The launcher reads it and puts that folder on top of sys.path automatically.

PICTURE. Two chained redirects now: shell picks the venv launcher (Step 4) → venv launcher puts its private site-packages on top of sys.path.

Step 6 — Install lands in the private folder; import finds it there

WHAT. Now pip install pandas==1.5.0:

pip itself is found via $PATH → it's the venv's pip.
It writes pandas into ml_env/.../site-packages/ (the private folder).
import pandas walks sys.path, hits the private folder first, loads that exact 1.5.0.

WHY this closes the loop. Install-location and import-location are now the same folder, and it sits first. Reproducibility follows: pip freeze reads that one folder and lists exactly what's there — see NumPy projects pinning versions this way, and dev-environment setup automating it.

PICTURE. pip writes into the private box; the import arrow reads from the very same box. Circle closed.

Worked example Watch the redirect with your own eyes

python -c "import sys; print(sys.executable)"
# /home/you/ml_env/bin/python   ← the venv launcher, not /usr/bin/python
python -c "import sys; print(sys.path[0:2])"
# ['', '/home/you/ml_env/lib/python3.10/site-packages']  ← private folder on top

sys.executable proves Step 4 worked; sys.path[1] proves Step 5 worked.

Step 7 — The degenerate cases (so you never hit a surprise)

Every case below is just Steps 4–5 succeeding, failing, or half-done.

Case A — Forgot to activate. $PATH is unchanged → shell finds /usr/bin/python → system sys.path → your project package is missing → ModuleNotFoundError. Nothing broke; the redirect simply never happened.

Case B — Installed globally, then made a venv. The private folder is empty, but system site-packages still sits lower on sys.path. import pandas misses the private folder and falls through to the system copy — so it "works" but pip freeze on the empty private folder lists nothing. Your env is a lie. (Parent note's first pitfall — now you see why: the fallback tier caught it.)

Case C — New terminal. Each shell gets its own copy of $PATH. A fresh terminal never ran activate, so it's back to Case A until you activate again.

Case D — conda instead of venv. Same two lists, one extra swap: conda also replaces the launcher itself, so it can hand you a different Python version (e.g. 3.9 vs system 3.11). venv can only reuse the Python it was built from; conda ships the interpreter too. The $PATH/sys.path reordering is identical.

The one-picture summary

Everything above collapses into a single decision every lookup makes: is a private folder sitting on top of the list? If yes, use it; if not, fall through to the shared one. Two lists, two arrows, one rule — first match wins.

Recall Feynman retelling — say it back in plain words

The computer keeps two to-do lists of folders. One tells the shell where to find the program called python; the other tells that Python where to find packages. Normally both lists point at shared, system-wide folders, so every project fights over the same packages. "Activating" a virtual environment does one humble thing: it glues your project's private folders to the top of both lists. Since every search stops at the first folder it matches, your private Python and private packages now win, and the shared ones only get used as a backup. pip then installs into that same private folder, so installing and importing agree perfectly — that's why the setup is reproducible. Forget to activate, and the lists are never rewritten, so you silently get the shared versions again. conda does the identical trick but also swaps the interpreter itself, letting it hand you a whole different Python version. No magic anywhere — just reordering two lists and remembering that first match wins.

Recall Quick self-check

Why must the venv folder be prepended to $PATH, not appended? ::: Search stops at the first match; appended, the system python would still be found first, so nothing changes. In Case B, why does import succeed but pip freeze list nothing? ::: The package sits in the shared folder (a lower fallback tier of sys.path), while the private folder that freeze reads is empty. What single extra thing does conda change that venv cannot? ::: conda swaps the interpreter itself, so it can provide a different Python version per environment.

Related: profiling runs inside whichever env is active — measure the code you actually ship, not the system one.