Feature stores: Worked examples — AI-ML deep dive

This page drills the Feature stores parent note into concrete numbers. The parent gave you the point-in-time join rule and the skew condition. Here we grind every case where those rules bite — including the weird corners where a naive engineer ships a bug.

The scenario matrix

Every cell below is a distinct thing that can go wrong (or must be handled). Each worked example is tagged with the cell it covers.

Cell	Case class	What varies	Danger if mishandled
A	Normal as-of join	$t_{e}$ falls between two observations	none — baseline
B	Future-leak corner	an observation exists after $t_{e}$	data leakage
C	Exact-tie corner	an observation lands exactly on $t_{e}$	off-by-one (" $\leq$ vs $<$ ")
D	Empty-past / cold start	no observation before $t_{e}$	null must be returned, not crash
E	TTL expiry	freshest value is older than $τ$	stale feature lies to model
F	Multi-entity join	several users, one query	wrong-key cross-contamination
G	Skew (offline ≠ online)	two code paths compute differently	silent production degradation
H	Word problem (fraud)	full pipeline end-to-end	integrating all rules at once
I	Exam twist	latest-value shortcut vs as-of	reveals the classic trap

We reuse two prerequisite ideas: Data leakage (cell B) and Training-serving skew (cell G).

The picture we compute against

Look at the figure. The horizontal axis is time. Each mint dot is a feature observation $(t_{i}, v_{i})$ — a moment when we recorded a value. The coral vertical line at $t_{e}$ is the label event: the instant we must make a prediction. The as-of rule says: sweep leftward from the coral line and grab the first mint dot you meet. Everything to the right of the coral line is the forbidden future.

Throughout, we use one running table for user U:

$t_{i}$	value $v_{i}$
09:00	$100
10:30	$40
12:00	$90

Cell A — the normal as-of join

Cell B — the future-leak corner

Worked example The bug a naive join creates

Event at $t_{e} = 11 : 00$ . A careless engineer sorts by value and grabs the largest balance ever seen, or grabs the chronologically last row of the table. What do they get, and why is it wrong?

Forecast: which value does the buggy path pick?

Buggy "last row" pick returns $90 (from 12:00). Why this happens? They forgot the $t_{i} \leq t_{e}$ filter — they read the whole table, whose last row is 12:00.
Compare to the correct as-of answer $40 (Cell A). Why this matters? $90 was recorded at 12:00, one hour after the prediction moment. Using it means the model trains on knowledge it could not have had. That is textbook Data leakage.
Symptom in the wild: offline accuracy looks fantastic; production accuracy collapses. Why? At serving time 12:00 hasn't happened yet, so the leaked signal simply isn't available.

Verify: correct $=\$ 40 $, b ug g y$ =$90 $, an d$ $40 \ne $90 $, so t h e l e ak c han g es t h e an s w er . T h e l e ak e d o b ser v a t i o n^{'} s t im es t am p$ 12{:}00 > 11{:}00 = t_e$ confirms it's from the future. ✓

Cell C — the exact-tie corner ( $\leq$ vs $<$ )

Worked example Observation lands exactly on the event

Event at $t_{e} = 10 : 30$ — the same instant as an observation. Do we include $40 or not?

Forecast: included or excluded?

Apply the constraint literally: $t_{i} \leq t_{e}$ . Since $10 : 30 \leq 10 : 30$ is true, the $40 observation is allowed. Why $\leq$ and not $<$ ? A value recorded at the event time was known at the moment of prediction — it is legal information, not the future. We only forbid strictly-later values.
Take the freshest allowed: 10:30 → $40.

Verify: candidates are 09:00 and 10:30 (both $\leq 10 : 30$ ); 12:00 excluded. Max is 10:30 → $40. ✓

Cell D — empty past / cold start

Cell E — TTL expiry (stale kill)

Cell F — multi-entity join (don't cross the keys)

Cell G — the skew corner (offline ≠ online)

Worked example Same feature, two code paths

The feature is avg_balance = mean of a user's recorded balances. Offline (batch, Spark) computes it over all three of U's observations: 100, 40, 90. Online (real-time) mistakenly averages only the two most recent it has cached: 40, 90. What is the skew?

Forecast: by how much do the two paths disagree?

Offline value: $(100 + 40 + 90) /3 = 230/3 = 76.67$ .
Online value: $(40 + 90) /2 = 130/2 = 65.00$ . Why this step? We evaluate each code path on the same raw data to expose that $f_{off} (x) \neq = f_{on} (x)$ .
Skew $= ∣76.67 - 65.00∣ = 11.67$ . Why it's fatal? The model was trained on 76.67-style numbers but served 65.00-style numbers — the input distribution shifts silently. This is exactly the Training-serving skew the parent note warns about.

Verify: offline $\approx 76.6667$ , online $= 65$ , difference $\approx 11.6667 \neq = 0$ . A feature store fixes this by materializing the offline result into the online store (Redis/DynamoDB), so both paths read one number and skew $= 0$ by construction. ✓

Cell H — real-world word problem (fraud, full pipeline)

Worked example End-to-end fraud feature at prediction time

A fraud model uses feature hours_since_last_topup. User U's top-ups happened at 09:00 and 10:30. A transaction (the label event) fires at $t_{e} = 11 : 00$ . TTL $τ = 24$ h. Compute the served feature.

Forecast: guess the hour count before deriving.

As-of the last top-up $\leq 11 : 00$ : that is 10:30. Why this step? We need what was known at 11:00; the 12:00 balance top-up (if any) is the future.
TTL check: age $= 11 : 00 - 10 : 30 = 0.5$ h $\leq 24$ h ✓ — fresh, keep it.
Compute the feature: $hours_since_last_topup = 11 : 00 - 10 : 30 = 0.5$ h. Why this form? The transformation $f (x) = t_{e} - t_{last}$ must be the same registered definition used offline in training, or we re-introduce Cell G skew.

Verify: $t_{e} - t_{last} = 0.5$ h, positive (event after top-up) ✓, and within TTL ✓. Units are hours ✓. Served value 0.5 h.

Cell I — the exam twist (latest-value trap)

Worked example "Just use the latest value for the training set" — quantify the error

An exam asks: for a historical label at $t_{e} = 10 : 00$ , a student reuses the online store's latest value (which today is $90 from 12:00) instead of doing an as-of join. What value should they have used, and what is the error?

Forecast: correct value? magnitude of the mistake?

Correct as-of at $t_{e} = 10 : 00$ : candidates $\leq 10 : 00$ are just 09:00 → $100. Why? 10:30 and 12:00 are both after 10:00, forbidden.
Student's latest-value pick: $90 (from 12:00, three hours in the future).
Error: $∣100 - 90∣ = 10$ , and worse, it's a leak (the $90 encodes future behaviour). Why the trap is seductive? The online store makes "latest" trivially easy — but "latest today" ≠ "latest as of that past label." This is the parent note's Mistake #2 in numbers.

Verify: as-of value $=\$ 100 $(o n l y 09 : 00 q u a l i f i es f or$ t_e=10{:}00 $), l a t es t$ =$90 $, d i f f er e n ce$ =$10\ne 0 $, an d 12 : 00$ > 10{:}00$ proves it's a future leak. ✓

Recall Which cell breaks each way?

Future leak comes from forgetting ::: the $t_{i} \leq t_{e}$ filter (Cell B, Cell I) A tie at $t_{i} = t_{e}$ should be ::: included, because $\leq$ is used, not $<$ (Cell C) No past observation means the feature is ::: null / default, never fabricated (Cell D) A value older than TTL is treated as ::: unknown / expired (Cell E) Offline mean 76.67 vs online mean 65 is an example of ::: training–serving skew (Cell G)