Feature stores: Exercises — AI-ML deep dive

Prerequisite links: Training-serving skew · Data leakage · Batch vs streaming pipelines · Redis · DynamoDB · Data versioning · Feature engineering · Model deployment

L1 — Recognition

Recall Solution L1.1

Online store. One row, right now, latency-critical → a low-latency key-value store such as Redis or DynamoDB. Why: the offline store (Parquet on S3, BigQuery) is built for scanning millions of rows cheaply, not for a single sub-10ms lookup.

Recall Solution L1.2

(b) Training-serving skew. Two code paths ( $f_{off}$ and $f_{on}$ ) produce different values from the same raw input. Detectable offline? No — offline metrics only see $f_{off}$ . The model looks fine until it meets $f_{on}$ in production.

Recall Solution L1.3

"Among all feature observations recorded at or before the label time $t_{e}$ , pick the most recent one." The $t_{i} \leq t_{e}$ part forbids the future; the $ar g max$ part picks the freshest survivor.

L2 — Application

Worked example L2.1 — Basic point-in-time join

Feature account_balance for user U:

$t_{i}$	value
08:00	$200
09:15	$50
11:40	$500

A label event happens at $t_{e} = 10 : 00$ . Give the correct feature value.

Recall Solution L2.1

Step 1 (filter, $t_{i} \leq t_{e}$ ): keep 08:00 and 09:15. Drop 11:40 — it is in the future of the event (using $500 would be Data leakage). Step 2 ( $ar g max$ ): of {08:00, 09:15}, the largest is 09:15. Answer: $50.

Recall Solution L2.2

Allowed window: $t_{e} - τ \leq t_{i} \leq t_{e}$ , i.e. $09 : 30 \leq t_{i} \leq 10 : 00$ . Check each survivor from L2.1: 09:15 is $45$ min old $> 30$ ; 08:00 is $120$ min old. Neither is inside the window. Answer: null / default. The model must explicitly handle "no fresh feature," rather than trusting a stale $50.

Recall Solution L2.3

The filter is $t_{i} \leq t_{e}$ — the $\leq$ is inclusive, so $t_{i} = t_{e}$ is allowed (it was known at the instant of the event). The 12:00 value is rejected. With two equal timestamps at 10:00, ties are broken by an ingestion order / row id chosen by the feature store; both are equally "freshest," so implementations pick a deterministic one (e.g. last-written). Answer: one of the 10:00 values (not 12:00).

L3 — Analysis

Recall Solution L3.1

Apply filter-then- $ar g max$ per event. See the step-function figure below — the served value is a staircase that only ever steps at an observation time and holds flat until the next one.

$t_{e} = 09 : 30$ : allowed = {09:00}. Answer $100.
$t_{e} = 11 : 00$ : allowed = {09:00, 10:30}. Freshest = 10:30. Answer $40.
$t_{e} = 13 : 00$ : allowed = {09:00, 10:30, 12:00}. Freshest = 12:00. Answer $90.

Recall Solution L3.2

Window per event: $t_{e} - 90 min \leq t_{i} \leq t_{e}$ .

$09 : 30$ : window $[08 : 00, 09 : 30]$ → 09:00 valid → $100 (unchanged).
$11 : 00$ : window $[09 : 30, 11 : 00]$ → 10:30 valid (30 min old) → $40 (unchanged).
$13 : 00$ : window $[11 : 30, 13 : 00]$ → 12:00 is 60 min old, valid → $90 (unchanged).

Trick answer: none change at $τ = 90$ . Now push TTL to $τ = 30$ :

$13 : 00$ : 12:00 is 60 min old $> 30$ → null. This is the one that flips.

Recall Solution L3.3

Access pattern mismatch. Training scans millions of rows of history (throughput-bound); serving needs one key, <10 ms (latency-bound). No single engine is best at both — see Batch vs streaming pipelines.
Cost / storage shape. Offline keeps full history cheaply (columnar Parquet/BigQuery); online keeps only the latest value per entity in memory-fast KV (Redis/DynamoDB). Storing full history in Redis is wasteful; scanning Parquet per request is too slow. What syncs them: materialization — the job that copies computed feature values from offline into online, so both derive from one definition.

L4 — Synthesis

Recall Solution L4.1

Zero skew $⟺ f_{off} (x) = f_{on} (x) \forall x .$ A feature store enforces this by registering a single feature definition and either (a) running that one definition in both paths, or (b) materializing the offline result straight into the online store so serving is a pure lookup, not a recomputation. Equality then holds structurally — you never rely on two hand-written implementations happening to agree.

Recall Solution L4.2

Define the entity user with join key user_id.
Register each feature (e.g. 7d_login_count) inside a feature view with a data source and TTL — one definition, so Feature engineering logic isn't duplicated.
Build the training set with a point-in-time join: for each churn label at $t_{e}$ , attach features with $t_{i} \leq t_{e}$ (freshest). This blocks future leakage.
Materialize the same definitions' latest values into the online store.
Bundle the model's features into a feature service; at inference, look them up — no recompute → offline path $=$ online path, so skew $= 0$ . Ties into Model deployment.

Recall Solution L4.3

In-memory KV store (Redis) for <5 ms at high QPS (DynamoDB also viable at scale). Because the streaming source updates continuously, a gap (upstream outage) can leave the last value stale — set a TTL so a value older than $τ$ returns null instead of silently serving a frozen click rate.

L5 — Mastery

Worked example L5.1 — Full end-to-end trace with TTL

Feature avg_txn_amount for user U:

$t_{i}$	value
00:00	20
02:00	35
05:00	60
09:00	15

TTL $τ = 180$ min. Label events at $t_{e} \in {01 : 30, 04 : 00, 08 : 30, 09 : 00}$ . Produce the served value (or null) for each event, showing the allowed window.

Recall Solution L5.1

Window: $t_{e} - 180 min \leq t_{i} \leq t_{e}$ ; take freshest inside.

$t_{e} = 01 : 30$ : window $[22 : 3 0_{prev}, 01 : 30]$ → only 00:00 (90 min old, ok). 20.
$t_{e} = 04 : 00$ : window $[01 : 00, 04 : 00]$ → 02:00 valid (120 min); 00:00 excluded. Freshest = 02:00. 35.
$t_{e} = 08 : 30$ : window $[05 : 30, 08 : 30]$ → 05:00 is 210 min old $> 180$ → excluded; 02:00 older still. No survivor. null.
$t_{e} = 09 : 00$ : window $[06 : 00, 09 : 00]$ → 09:00 valid ( $t_{i} = t_{e}$ , 0 min old, inclusive $\leq$ ). Freshest = 09:00. 15.

Recall Solution L5.2

With $t_{i} < 11 : 00$ : allowed = {09:00, 10:30}. Freshest = 10:30 → $40. But 10:30 is after the event at 10:00 — that observation did not exist at prediction time. The correct value is $100 (the 09:00 one). Failure: future leakage / Data leakage. Widening the upper bound past $t_{e}$ always risks pulling in post-event observations. The upper bound must be exactly $t_{e}$ ( $\leq$ ), never $t_{e} + ε$ .

Recall Solution L5.3

No — the 12:20 value hasn't been materialized (next batch run is 13:00), so serving returns the pre-12:20 value. This is materialization lag. Lever: switch that feature to a streaming materialization path (Batch vs streaming pipelines) so writes to the online store happen near-real-time. Trade-off: streaming adds infrastructure complexity and cost, and can reintroduce skew if the streaming transform differs from the offline one — so the streaming path must reuse the same registered definition.

Recall Self-test checklist

The filter uses which comparison, strict or inclusive? ::: Inclusive, $t_{i} \leq t_{e}$ — a value known at the event instant is allowed. After filtering, which observation do you keep? ::: The freshest, i.e. the one maximising $t_{i}$ ( $ar g max$ ). What makes zero skew structural rather than tested? ::: One registered definition + materialization, so $f_{off} = f_{on}$ by construction. When does TTL return null? ::: When even the freshest allowed observation is older than $τ$ (i.e. $t_{e} - t_{i} > τ$ ).