Feature stores: Question bank — AI-ML deep dive

Before you start, keep three anchor pictures in your head:

The recipe box — one shared definition serving both a huge batch (training) and a single cookie (serving).
The timeline — a label event sits at time $t_{e}$ ; feature observations sit as dots on a line; you may only reach left (past), never right (future).
The two shelves — an offline shelf holding all of history, an online shelf holding only today's latest.

True or false — justify

A feature store is basically just another database.

False — it is an abstraction layer over an offline store and an online store, plus a registry, a materialization engine, and a serving API. The consistency and single definition are the point, not the raw storage.

If two teams write two different SQL queries that both produce "user 7-day click rate", there is no skew as long as the numbers match today.

False — the risk is that they diverge on some future edge case (nulls, timezones, late data). A single registered definition removes the possibility of divergence, which is the guarantee, not "they happen to agree now".

The offline and online store hold the same data.

False — the offline store holds the full history of every feature value; the online store holds only the latest value per entity. They are synced from one definition but are not copies of each other.

Point-in-time correctness only matters at serving time.

False — it matters when building the training set: you attach each label the feature value as it was before the event. At serving time the "latest" value is already correct because you are predicting the present.

Using

t_{i} \leq t_{e}

(allowing equality) is a mistake because a value at exactly

t_{e}

leaks the future.

False — a value stamped at exactly

t_{e}

was known at prediction time, not after it, so it is legitimately available. Leakage is strictly

t_{i} > t_{e}

Setting a TTL can only ever remove useful information, so a longer TTL is always safer.

False — a too-long TTL lets a stale feature serve silently (a 3-month-old "recent activity"), which lies to the model. TTL trades freshness against coverage; both extremes hurt.

A feature store guarantees your model will be accurate.

False — it guarantees consistency (

f_{off} (x) = f_{on} (x)

), i.e. the model sees the same number it trained on. Whether that number predicts well is a modelling question, not a storage one.

If you materialize offline results into the online store, skew is eliminated by construction.

True — serving becomes a lookup of a value the offline pipeline already computed, so no second code path exists to disagree. See Training-serving skew.

Spot the error

"For the training set, I'll grab each user's current balance from the online store — it's already there and fast."

The online store holds today's value; a historical label needs the value as of that past moment. This is future leakage — the classic "join today's value onto yesterday's label" bug.

"To serve a prediction I recompute the 30-day average live from the raw DB — always fresh."

Recomputing a 30-day aggregate live blows the latency budget and runs different code than training, reintroducing Training-serving skew. Serve a pre-materialized lookup instead.

"My as-of join picks the observation with the largest

t_{i}

, full stop."

The constraint

t_{i} \leq t_{e}

is missing. Without it you may pick a future observation; "largest

t_{i}

" alone selects the newest overall, which can be after the event.

"There's no feature within the TTL window, so I'll just use the last observation anyway — some value beats no value."

When nothing is within TTL the correct output is null / a default, forcing explicit missing-data handling. Silently using an expired value is exactly the failure TTL exists to prevent.

"Offline store should be low-latency key-value like Redis so training is fast."

Training scans millions of rows of history, so the offline store optimizes throughput and cost (Parquet/BigQuery), and slowness is fine. Low-latency KV stores like Redis/DynamoDB are for the online single-row lookup.

"Two feature stores in two projects — I'll just copy the feature definition into both."

Copying re-creates duplication and the chance of divergence, the very problems the store solves. One registered definition should be referenced, tied to Data versioning, not pasted.

"My feature values look great offline — 0.99 AUC — so deployment will be fine."

Suspiciously perfect offline metrics are the fingerprint of Data leakage from a bad as-of join. Verify point-in-time correctness before trusting the number.

Why questions

Why do we need two stores instead of one great database?

No single database is excellent at both millions-of-rows throughput (training) and sub-10ms single-row latency (serving), so we split responsibilities and keep them synced from one definition.

Why does the point-in-time rule maximize

t_{i}

among allowed observations, rather than average them?

The freshest value

\leq t_{e}

is the most informative snapshot of what was actually known at decision time; averaging would blend in older, less relevant state and blur the "as-of" snapshot.

Why is training–serving skew called a silent killer?

Offline metrics stay high because the training pipeline is self-consistent; the model only degrades in production where the other code path runs, and no single test catches the mismatch. See Training-serving skew.

Why does materialization exist if the online store could recompute features?

Recomputation at serving time is slow and duplicates logic; copying pre-computed values into the online store makes serving a fast lookup and guarantees it equals the training-time value.

Why must a feature view carry a TTL rather than the model deciding freshness?

The store must return null when data is stale at lookup time, before the model sees anything; freshness is a property of the feature's serving contract, not a downstream afterthought.

Why doesn't consistency (

f_{off} = f_{on}

) automatically prevent Data leakage?

Consistency ensures both paths compute the same transformation; leakage is about which timestamps you join, a separate as-of-join concern. You can be perfectly consistent and still leak the future.

Edge cases

A label event occurs before any feature observation exists for that entity.

The allowed set

{t_{i} \leq t_{e}}

is empty, so the point-in-time join returns null / default. This is correct: nothing was known yet, and the model must handle the missing feature.

Two feature observations share the exact same timestamp

t_{i} = t_{e}

Both satisfy

t_{i} \leq t_{e}

with the maximal

t_{i}

; the definition needs a deterministic tie-break (e.g. last-written, or an event-order key) so training and serving pick the same one and stay consistent.

The event time equals a feature timestamp to the millisecond, but they arrived out of order (late data).

The join must use event time, not arrival/processing time; otherwise a value that logically preceded the event but landed late gets wrongly excluded or included, silently reintroducing skew.

Feature has observations only in the future relative to

t_{e}

(backfilled after the event).

Every

t_{i} > t_{e}

, so the allowed set is empty and the value is null — backfilled future data must never be pulled left across the event boundary.

An entity has never appeared in the online store when a prediction is requested.

The serving lookup misses; the serving path must return a default or fall back, matching how a null was treated in training so the two paths still agree.

TTL is set to zero.

Only an observation stamped exactly at

t_{e}

(within a zero-width window) qualifies, so almost every lookup returns null — effectively disabling the feature. This is a degenerate setting, useful only to prove a feature is unavailable.

A streaming source updates the online store faster than the offline store materializes history.

Serving may briefly use a value the offline store hasn't recorded, so the next training set won't contain that exact snapshot — a subtle skew requiring the offline store to log the same event stream (see Batch vs streaming pipelines).

Recall The one trap to never forget

Which single mistake causes 0.99 offline AUC that collapses in production? Answer ::: Joining today's (or any post-event) feature value onto a past label — future leakage from skipping the $t_{i} \leq t_{e}$ constraint in the point-in-time join.

Flashcards

Is a feature store just a database?

No — it is an abstraction over offline + online stores plus a registry, materialization engine, and serving API; consistency is the point.

When does point-in-time correctness matter most?

When building the training set, so each label gets the feature value known before the event, not today's value.

What should an as-of join return when no observation satisfies

t_{i} \leq t_{e}

Null / a default value, which the model must be trained to handle.

Why split into offline and online stores?

One database can't be great at both bulk historical throughput and sub-10ms single-row latency, so we split and sync.

Can perfect consistency still leak the future?

Yes — consistency equalizes the transformation; leakage is about which timestamps you join, a separate concern.