Level 3 — ProductionPython & Scientific Computing

Python & Scientific Computing

⏱ 45 minutes60 marksprintable — key stays hidden on paper

Level: 3 — Production (from-scratch derivations, code-from-memory, explain-out-loud) Time limit: 45 minutes Total marks: 60

Instructions: Write code from memory. Where asked to "explain out loud", give the reasoning a reviewer would accept. Assume import numpy as np, import pandas as pd are available unless stated.

Question 1 — Broadcasting derivation (10 marks)

Given A with shape (4, 1, 3) and B with shape (3,):

(a) State NumPy's broadcasting rules step by step and derive the shape of A + B. If it fails, say why. (4)

(b) You want to compute, for a matrix X of shape (n, d), the pairwise squared Euclidean distance matrix D of shape (n, n) without any Python loops. Write the vectorized expression and explain how broadcasting produces each intermediate shape. (6)

Question 2 — Comprehensions & generators (10 marks)

(a) Rewrite the following using a single dict comprehension: (3)

result = {}
for w in words:
    if len(w) > 3:
        result[w] = len(w)

(b) Explain out loud the difference between a list comprehension and a generator expression in terms of memory and evaluation. Give one situation where the generator is strictly preferable. (4)

(c) Write a generator function running_mean(iterable) that yields the cumulative mean after each element, from memory. (3)

Question 3 — Classes & functions from memory (10 marks)

Implement a class Standardizer with fit, transform, and fit_transform methods that z-score-normalizes a 2D NumPy array column-wise (subtract mean, divide by std). Requirements:

fit stores per-column mean and std. (3)
transform raises a clear error if called before fit. (3)
Guard against division by zero for constant columns. (2)
fit_transform reuses the other two methods. (2)

Question 4 — Pandas + vectorization (10 marks)

You load sales.csv with columns date (string), region, revenue, units.

(a) Write code to load the CSV parsing date as datetime, then add a column price = revenue / units handling units == 0 by producing NaN (no crash). (4)

(b) Produce total revenue per region sorted descending, using a groupby. (3)

(c) Explain out loud why chained indexing like df[df.region=='X']['revenue'] = 0 may fail silently, and give the correct .loc form. (3)

Question 5 — Debugging & environments (10 marks)

(a) You get ValueError: operands could not be broadcast together with shapes (100,3) (100,). Explain out loud the likely cause and two fixes. (4)

(b) Describe the exact commands to create a virtual environment, activate it, install numpy==1.26, and freeze dependencies to requirements.txt. (3)

(c) A colleague's notebook "works on their machine". List three concrete reasons reproducibility can break and one Git-hygiene practice that mitigates it. (3)

Question 6 — Plotting & Git (10 marks)

(a) From memory, write matplotlib code to plot two lines (y1, y2 vs x) on the same axes with labels, a legend, axis titles, and save to out.png at 150 dpi. (5)

(b) You committed a large data file by mistake and already ran git commit. Give the sequence of Git commands to remove it from the last commit without deleting the local file, and prevent re-committing it. (5)

Answer keyMark scheme & solutions

Question 1

(a) Broadcasting rules (align shapes from the right, pad missing dims with 1, dims must be equal or one of them = 1): (1 for rules)

A: (4, 1, 3), B: (3,) → pad B to (1, 1, 3). (1)
Compare per axis: (4 vs 1)→4, (1 vs 1)→1, (3 vs 3)→3. (1)
Result shape = (4, 1, 3). Broadcast succeeds. (1)

(b) Using the identity $\|x_i - x_j\|^2 = \|x_i\|^2 + \|x_j\|^2 - 2 x_i\cdot x_j$ :

G = X @ X.T                       # (n,n) Gram matrix
sq = np.sum(X**2, axis=1)         # (n,)  squared norms
D = sq[:, None] + sq[None, :] - 2*G
D = np.maximum(D, 0)              # clip tiny negatives

(2 formula, 2 code)

Shape explanation: sq[:,None] is (n,1), sq[None,:] is (1,n); broadcasting to (n,n); G already (n,n). Sum is (n,n). (2)

Question 2

(a) (3)

result = {w: len(w) for w in words if len(w) > 3}

(b) List comprehension builds the whole list in memory eagerly; generator expression is lazy, producing items one at a time (constant memory), evaluated on iteration. (2) Generator strictly preferable when streaming/large or infinite data, or when you only consume part of it (e.g. next(), any()). (2)

(c) (3)

def running_mean(iterable):
    total, n = 0.0, 0
    for x in iterable:
        total += x
        n += 1
        yield total / n

Question 3 (10)

import numpy as np
 
class Standardizer:
    def __init__(self):
        self.mean_ = None
        self.std_ = None
 
    def fit(self, X):
        X = np.asarray(X, dtype=float)
        self.mean_ = X.mean(axis=0)
        std = X.std(axis=0)
        std[std == 0] = 1.0          # guard constant columns
        self.std_ = std
        return self
 
    def transform(self, X):
        if self.mean_ is None:
            raise RuntimeError("Call fit before transform.")
        return (np.asarray(X, float) - self.mean_) / self.std_
 
    def fit_transform(self, X):
        return self.fit(X).transform(X)

Marks: fit stores mean/std (3), transform guard (3), div-by-zero guard (2), fit_transform reuse (2).

Question 4

(a) (4)

df = pd.read_csv("sales.csv", parse_dates=["date"])
df["price"] = df["revenue"] / df["units"].replace(0, np.nan)

(Division by NaN yields NaN, no crash.)

(b) (3)

df.groupby("region")["revenue"].sum().sort_values(ascending=False)

(c) Chained indexing creates a temporary copy; assignment mutates the copy not df, triggering SettingWithCopyWarning and no real change. (1.5) Correct: (1.5)

df.loc[df.region == 'X', 'revenue'] = 0

Question 5

(a) Cause: a (100,3) array combined with a (100,) vector — the trailing dims 3 vs 100 don't match. (2) Fixes: reshape the vector to (100,1) so it broadcasts across columns (v[:,None]), or transpose/align operands correctly. (2)

(b) (3)

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install numpy==1.26
pip freeze > requirements.txt

(c) Any three: different package versions; unpinned dependencies; Python version mismatch; hidden global state / hardcoded paths; hidden execution order in the notebook. (2) Git hygiene: commit a pinned requirements.txt/environment.yml (and add .gitignore for data/venv). (1)

Question 6

(a) (5)

import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.plot(x, y1, label="y1")
ax.plot(x, y2, label="y2")
ax.set_xlabel("x"); ax.set_ylabel("y")
ax.set_title("y1 & y2 vs x")
ax.legend()
fig.savefig("out.png", dpi=150)

(b) (5)

git rm --cached bigdata.csv          # unstage, keep file locally
echo "bigdata.csv" >> .gitignore
git add .gitignore
git commit --amend                   # rewrite last commit without the file

(1 for --cached, 1 for keeping local, 1 for .gitignore, 1 for add, 1 for amend.)

[
  {"claim":"A(4,1,3)+B(3,) broadcasts to (4,1,3)","code":"a=ones((4,1,3)); b=ones((3,)); result = ((a+b).shape==(4,1,3))"},
  {"claim":"Pairwise distance identity gives correct D","code":"X=Matrix([[0,0],[3,4],[1,1]]); import numpy as np; Xn=np.array(X).astype(float); G=Xn@Xn.T; sq=(Xn**2).sum(1); D=sq[:,None]+sq[None,:]-2*G; import itertools; brute=np.array([[((Xn[i]-Xn[j])**2).sum() for j in range(3)] for i in range(3)]); result = bool(np.allclose(np.maximum(D,0),brute))"},
  {"claim":"running_mean yields cumulative means","code":"def rm(it):\n t=0.0;n=0\n for x in it:\n  t+=x;n+=1;yield t/n\nout=list(rm([2,4,6])); result = out==[2.0,3.0,4.0]"},
  {"claim":"Standardizer produces zero mean unit std per column","code":"import numpy as np\nX=np.array([[1.,10.],[2.,20.],[3.,30.]]);m=X.mean(0);s=X.std(0);s[s==0]=1;Z=(X-m)/s; result = bool(np.allclose(Z.mean(0),0) and np.allclose(Z.std(0),1))"}
]