Foundations — Data leakage identification and prevention

Before we can talk about how leakage happens, we must agree on what every squiggle and word in the parent note means. This page assumes nothing. If you have never seen a summation sign or the letter $\mathbb{E}$, start here at line one.

The cast of characters (built one at a time)

1. A dataset: rows and columns

Picture a spreadsheet. Every row is one thing you observed (one transaction, one house, one day of the stock market). Every column is one measurement about that thing.

Why the topic needs this: leakage is entirely about which columns are allowed to be features. If a column secretly contains target information, disaster follows. So we must be crystal clear on the feature/target split first.

2. The symbols $X$ and $y$

Instead of writing "the features" and "the target" in English every time, we use letters.

The picture: $X$ is the left block of the spreadsheet, $y$ is the single column on the right. The subscript $i$ points at a single row across both blocks.

Why: the whole topic is about the relationship between $X$ (what you're allowed to use) and $y$ (what you want). Leakage means $X$ was contaminated by knowledge of $y$.

3. A model, written $f$

A model is a machine that turns features into a guess for the target.

The picture: a box with an arrow in (the row $X_i$) and an arrow out (a number ${\hat{y}}_i$).

Why the "hat": the hat $\hat{}$ always means "our estimate of", never the true value. $y_i$ is the truth; ${\hat{y}}_i$ is the guess. Keeping them separate is the whole point of measuring error.

4. The summation sign $\sum$

We need to add up something over every sample. Writing $a_1+a_2+\cdots +a_n$ is tedious, so mathematicians invented shorthand.

The picture: a "for every row, grab this number and drop it in a bucket; at the end, report the bucket's total."

Why this tool and not another? We use $\sum$ (not, say, an integral) because a dataset is a finite list of separate rows — you add discrete things, you don't smoothly integrate. It answers the question "what is the total across my whole dataset?"

5. The loss $\mathcal{L}$ — measuring wrongness

We need a number that says how badly a single guess missed.

A common choice is the squared difference: $\mathcal{L}(\hat{y},y)=(\hat{y}-y{)}^2$

Why squared? Two reasons. First, squaring makes the penalty positive whether you guessed too high or too low (a miss of $-3$ and a miss of $+3$ both cost $9$). Second, squaring punishes big misses much harder than small ones — a miss of $10$ costs $100$, a miss of $1$ costs only $1$. That answers "how do I turn 'right or wrong' into a single comparable number?"

6. Empirical risk $\hat{R}$ — average wrongness on YOUR data

One loss is for one sample. To score the whole model, average the loss over every sample you have.

Let's read this left to right, out loud:

• WHAT: ${\sum }_{i=1}^n$ — go through every sample $1$ to $n$.
• For each, compute $\mathcal{L}(f(X_i),y_i)$ — how wrong the guess $f(X_i)$ was versus truth $y_i$.
• $\frac{1}{n}$ — divide the total by $n$ to get the average wrongness per sample.

Why the hat on $\hat{R}$? Same rule as before: it is our estimate of true error, computed only from the data we happen to hold. It is not the truth about how the model behaves everywhere.

7. Expectation $\mathbb{E}$ and true risk $R$ — wrongness in the whole world

Your dataset is a small sample of reality. What you really care about is the average error over all possible future data, not just the rows you own.

Why we need both $\hat{R}$ and $R$: the entire disaster of leakage is that $\hat{R}$ (what you measure on your data) becomes tiny while $R$ (what happens in the real world) stays huge. The gap between the hat-version and the true-version is the villain of this whole topic.

8. A probability distribution $P(X,y)$

The picture: imagine a cloud of dots, one per possible $(X,y)$ pair; denser regions are more likely. Three clouds — train, test, production — that we hope are the same shape.

Why this is the keystone: machine learning only works if the past resembles the future in shape. Leakage secretly deforms $P_{\text{train}}$ (by adding columns that won't exist later) so it no longer matches $P_{\text{production}}$. Every type of leakage is a specific way this equation gets broken. See Train-testvalidation split strategies for how we build the test cloud honestly.

9. Function composition $f\circ g$ and the leak function $g$

The parent note writes features as $X^{\prime }=g(X,y,\text{future info})$. Let's unpack that.

The picture: two boxes in a row. If box $g$ has a hidden wire feeding it the answer $y$, then box $f$ downstream is effectively being handed the answer. That's target leakage in one image.

Why: this notation lets us say precisely where the leak lives — inside $g$, before the model $f$ ever runs. See Causal inference basics for why the direction of the wire (does $y$ cause the feature, or vice versa?) decides guilt.

10. Greek helpers: $\mu$, $\sigma$, $ϵ$, $\Delta$

These appear in the scaling and target-leakage derivations.

The scaling formula from the parent note: $\mu =\frac{1}{n}{\sum }_{i=1}^n{X}_i,{\sigma }^2=\frac{1}{n}{\sum }_{i=1}^n(X_i-\mu {)}^2,X_i^{\prime }=\frac{X_i-\mu }{\sigma }$

Reading it: $\mu$ is the average (there's our $\sum$ and $\frac{1}{n}$ again). ${\sigma }^2$ averages the squared distances from the mean (there's squaring, to keep distances positive). The last line rescales each value: subtract the mean so the column centres on zero, divide by the spread so it has a standard size.

Why the topic needs $\mu$ and $\sigma$: scaling leakage happens when you compute $\mu$ and $\sigma$ using the test rows too — the test cloud leaks into the recipe. More on the honest procedure in Feature scaling and normalization.

Why $\Delta$: the parent writes $\Delta =|R_{\text{production}}(f)-\hat{R}(f)|$ — the gap between real-world error and measured error. The bars $|\cdots |$ mean absolute value (drop the minus sign, keep the size). Leakage makes $\Delta$ large.

How these feed the topic

Everything funnels into the parent topic Data leakage: leakage is precisely the situation where the honest chain (rows → features → model → risk) is corrupted by a recipe $g$ that used $y$ or the future, so measured risk $\hat{R}$ no longer predicts true risk $R$.

Worked mini-check: reading a formula aloud

Recall drills

Equipment checklist

Read the left side; say the answer before revealing.