Memory systems for agents: Exercises — AI-ML deep dive

Before we start, we define the one tool used again and again on this page: cosine similarity, written $cos_sim (a, b)$ . When two pieces of text are turned into lists of numbers (embeddings — see Vector Databases), $cos_sim$ measures how "close in meaning" they are. Picture two arrows starting at the same point:

Definition Cosine similarity — the "same-direction score"

For two arrows (vectors) $a$ and $b$ , cosine similarity is $cos_sim (a, b) = \frac{a \cdot b}{∥ a ∥ ∥ b ∥} = cos θ$ where $θ$ is the angle between the two arrows.

$a \cdot b$ (the dot product) is $a_{1} b_{1} + a_{2} b_{2} + \dots$ — multiply matching entries, add them up. It is large when the arrows point the same way.
$∥ a ∥ = a_{1}^{2} + a_{2}^{2} + \dots$ is the arrow's length. Dividing by both lengths cancels out "how long" and leaves only "which direction", so the score is always between $- 1$ and $1$ .
Why cosine and not plain distance? Because meaning is about direction in embedding space, not magnitude. A short and a long arrow pointing the same way mean the same thing. $cos_sim = 1$ means identical direction (same meaning); $0$ means unrelated (perpendicular); $- 1$ means opposite.
Edge case — the zero vector. If either arrow has length $0$ (i.e. $∥ a ∥ = 0$ or $∥ b ∥ = 0$ ), the formula divides by zero and $θ$ is undefined — a zero-length arrow points nowhere, so "same direction" is meaningless. In code you must guard against this: return $0$ (treat as "unrelated") or skip the comparison. Never feed an all-zeros embedding into $cos_sim$ .

Level 1 — Recognition

L1.1

Name the three memory layers of the standard agent architecture and, in one word each, what time-scale they cover.

Recall Solution

Working memory — short-term (the current task/turn).
Episodic memory — mid-term (past interactions, "what happened last Tuesday").
Semantic memory — long-term (timeless facts and world knowledge).

L1.2

List the four memory primitives (operations) every memory system implements.

Recall Solution

Write (store), Read (retrieve), Update (modify), Forget (delete). Read as flashcards: Store new info under a key ::: Write Get info matching a query ::: Read Change an existing entry's value ::: Update Remove an entry (decay/privacy/capacity) ::: Forget

L1.3

Which memory layer would each of these live in? (a) "User Alice prefers 9–11 am meetings." (b) "Two turns ago the user pasted this stack trace." (c) "On June 26 we issued a refund to user_123."

Recall Solution

(a) Semantic — a timeless preference/fact. (b) Working — immediate context for the current task. (c) Episodic — a specific dated event.

Level 2 — Application

L2.1 — Working-memory budget

A model has an 8000-token context window. The system prompt costs 1200 tokens and you reserve 600 tokens as an output buffer. How many tokens are left for working memory?

Recall Solution

Working memory is whatever is left of the window after the fixed costs: $Working Memory = 8000 - 1200 - 600 = 6200 tokens$ What we did: subtracted the non-negotiable pieces (system prompt + output room) from the total window. Why: those tokens are spoken for before any conversation content can go in — see LM Context Window Management.

L2.2 — Cosine similarity by hand

An episode embedding is $v_{old} = (3, 4)$ and the incoming query embedding is $v_{new} = (4, 3)$ . Compute their cosine similarity.

Recall Solution

Step 1 — dot product (multiply matching entries, add): $v_{new} \cdot v_{old} = 4 \cdot 3 + 3 \cdot 4 = 12 + 12 = 24.$ Step 2 — lengths: $∥ v_{new} ∥ = 4^{2} + 3^{2} = 25 = 5, ∥ v_{old} ∥ = 3^{2} + 4^{2} = 5.$ Step 3 — divide: $cos_sim = \frac{24}{5 \cdot 5} = \frac{24}{25} = 0.96.$ What it looks like: two arrows almost overlapping (angle $\approx 16°$ ). See figure s01 — a score this high means "nearly the same meaning".

L2.3 — Recency forgetting

Retention window is $T_{window} = 30$ days and "now" is day 100. Apply the keep-rule $Keep (e) = {10 t_{now} - t_{e} < 30 otherwise$ to episodes stamped at days $95, 71, 70, 40$ . Which survive?

Recall Solution

Compute the age $t_{now} - t_{e} = 100 - t_{e}$ and compare with 30:

day 95: age $5 < 30$ → keep
day 71: age $29 < 30$ → keep
day 70: age $30$ → this is the boundary case. The rule uses a strict $< 30$ , so age exactly $30$ is not less than $30$ → forget. (Had the rule been $\leq 30$ , this episode would survive — the choice of strict-vs-inclusive is a design decision you must make explicit, because it flips exactly the episode sitting on the fence.)
day 40: age $60$ → forget Survivors: days 95 and 71.

Level 3 — Analysis

L3.1 — Redundancy pruning threshold

Your merge rule: if a new episode's embedding has $cos_sim > 0.95$ with an existing one, delete the older near-duplicate. New embedding $v_{new} = (1, 0)$ . Three stored episodes have embeddings $(0.9, 0.1)$ , $(0.7, 0.7)$ , $(0.99, 0.14)$ . Which get merged/deleted?

Recall Solution

First, why the formula simplifies here. For a stored embedding $(x, y)$ against $v_{new} = (1, 0)$ :

dot product: $(1, 0) \cdot (x, y) = 1 \cdot x + 0 \cdot y = x$ — the $y$ -term is multiplied by $0$ and vanishes, so only the first coordinate survives.
length of $v_{new}$ : $1^{2} + 0^{2} = 1$ , so dividing by it changes nothing.
length of $(x, y)$ : $x^{2} + y^{2}$ .

Putting these together, $cos_sim = \frac{x}{1 \cdot x ^{2} + y ^{2}} = \frac{x}{x ^{2} + y ^{2}}$ . That is why the whole thing collapses to "first coordinate over length".

Now evaluate each:

$(0.9, 0.1)$ : $\frac{0.9}{0.81 + 0.01} = \frac{0.9}{0.82} \approx \frac{0.9}{0.9055} \approx 0.9939 > 0.95$ → delete.
$(0.7, 0.7)$ : $\frac{0.7}{0.98} \approx \frac{0.7}{0.9899} \approx 0.707 < 0.95$ → keep (that's a $45°$ angle — genuinely different).
$(0.99, 0.14)$ : $\frac{0.99}{0.9801 + 0.0196} = \frac{0.99}{0.9997} \approx 0.9902 > 0.95$ → delete. So two are pruned, one survives.

L3.2 — Importance ranking

Importance score is $Importance (e) = w_{1} \cdot Access + w_{2} \cdot Recency + w_{3} \cdot Feedback$ with weights $w_{1} = 0.5, w_{2} = 0.3, w_{3} = 0.2$ . You may keep only the top 2 of three episodes:

Episode	Access	Recency	Feedback
A	10	0.2	0
B	2	0.9	1
C	4	0.5	1

Which two survive?

Recall Solution

Plug each row into the weighted sum:

$A = 0.5 (10) + 0.3 (0.2) + 0.2 (0) = 5 + 0.06 + 0 = 5.06$
$B = 0.5 (2) + 0.3 (0.9) + 0.2 (1) = 1 + 0.27 + 0.2 = 1.47$
$C = 0.5 (4) + 0.3 (0.5) + 0.2 (1) = 2 + 0.15 + 0.2 = 2.35$ Ranking: $A (5.06) > C (2.35) > B (1.47)$ . Keep A and C. Analysis: Access dominates because $w_{1}$ is largest and Access values are on a scale of tens while Recency/Feedback sit in $[0, 1]$ . That scale mismatch is itself a design smell (see next mistake).

Level 4 — Synthesis

L4.1 — Retrieval budget under a token cap

Each retrieved episode summary costs 300 tokens. After the system prompt and output buffer you have 6200 tokens of working memory (from L2.1), but you must also leave 2000 tokens for the live user turn and reasoning. How many past episodes can you inject into context?

Recall Solution

Step 1 — room for retrieval: subtract the reserved live-turn budget: $6200 - 2000 = 4200 tokens available for retrieved memory.$ Step 2 — divide by cost per episode and floor: $⌊ \frac{4200}{300} ⌋ = ⌊ 14 ⌋ = 14 episodes.$ Why floor? You cannot inject a fraction of an episode; overshooting the budget truncates the prompt. Exactly $14$ fit ( $14 \times 300 = 4200$ ).

L4.2 — Design a memory-routing rule

Sketch how an agent decides which layer to consult when a new user message arrives. Cover the case where nothing relevant is found. The rendered decision diagram is shown below.

Recall Solution

The agent tries the cheapest, most-current source first and falls back outward:

Working memory — is the answer in the current turn's context already? If yes, respond.
Semantic memory — is this a general fact question ("how do I read a CSV")? Query the knowledge store (Knowledge Graphs).
Episodic memory — does this refer to a past interaction ("is my refund done")? Embed the query, similarity-search past episodes (Vector Databases), load the best match into working memory.
Nothing found — either ask the user a clarifying question or call an external tool; then Write the result back so next time it's remembered.

Why this order? Working memory is free (already loaded); semantic lookups are cheap key/graph hits; episodic search costs an embedding + vector query; external tools are slowest and priciest. Try cheap-and-fresh before expensive-and-stale.

Level 5 — Mastery

L5.1 — End-to-end trace

A support agent stores this Day-1 episode:

{ "timestamp": "day1", "user_id": "u42",
  "summary": "charged twice for June subscription; refund issued",
  "outcome": "resolved" }

On Day 6 the user asks: "Did my refund go through?" The query embeds to $q = (2, 1)$ ; the Day-1 episode embeds to $e = (4, 2)$ ; an unrelated "weather" episode embeds to $w = (0, 3)$ . (a) Which episode is retrieved? (b) Walk the full memory pipeline that produces the answer.

Recall Solution

(a) Rank by cosine similarity.

$cos_sim (q, e) = \frac{2 \cdot 4 + 1 \cdot 2}{5 \cdot 20} = \frac{10}{100} = \frac{10}{10} = 1.0$ (identical direction — $e$ is just $q$ doubled).
$cos_sim (q, w) = \frac{2 \cdot 0 + 1 \cdot 3}{5 \cdot 3} = \frac{3}{3 5} = \frac{1}{5} \approx 0.447.$ The Day-1 billing episode wins ( $1.0 ≫ 0.447$ ).

(b) Pipeline.

Read (working): query not answerable from current turn alone.
Read (episodic): embed query → similarity search → retrieve Day-1 episode (score $1.0$ ).
Update (working): load "user had double-charge; refund issued Day 1" into working memory — no need to re-ask the user.
Tool call: check live refund-status API.
Respond with the status.
Write: append this Day-6 follow-up as a new episode so future queries have it. What it looks like: $q$ and $e$ lie on the same ray from the origin (angle $0°$ ), while $w$ points off to the side — cosine $1.0$ vs $0.447$ makes the billing episode the clear winner.

L5.2 — Contradiction handling in semantic memory

Semantic memory holds ("u42", "prefers_time", "09:00-11:00", confidence=0.85). The user now says, twice, "actually I prefer afternoons." Describe the correct Write/Update/Forget behaviour and why you would not just append a second fact.

Recall Solution

Why not append a second fact. Appending a new row ("u42","prefers_time","afternoon",...) alongside the old one leaves two facts sharing the same (entity, relation) key ("u42","prefers_time") but with contradictory values ("09:00-11:00" vs "afternoon"). Future retrievals would surface morning or afternoon at random, and the prompt you build (Prompt Engineering) would contain a self-contradiction. Semantic memory must answer "what is true now", so it can hold at most one live value per key.

The correct sequence, in the four primitives:

Read the existing fact by its key ("u42","prefers_time") → find value "09:00-11:00", confidence $0.85$ .
Update that same entry's value to "afternoon" — because the key already exists, this is an in-place Update, not a new Write. The old value is overwritten.
Adjust confidence upward. The user stated the new preference twice, so the evidence is stronger than a single mention. Raise confidence, e.g. from $0.85$ toward $0.90$ . (A repeated, consistent claim earns more trust.)
Forget the stale morning value completely — it is now contradicted by newer evidence, which is exactly the semantic-memory pruning rule "remove facts contradicted by newer evidence." After the Update overwrites the value there is nothing left to keep; if your store versions values, explicitly delete the old version.

Result: semantic memory now holds the single fact ("u42","prefers_time","afternoon",confidence≈0.90).

What still gets appended — but elsewhere. You do keep a record that the preference changed, but that record belongs in episodic memory (an audit trail of "on Day N the user changed their preferred time"), not in semantic memory. Episodic memory remembers what happened; semantic memory remembers what is currently true. Splitting them this way keeps retrieval unambiguous while losing no history.

Recall Quick self-check

Cosine of two identical-direction arrows ::: 1.0 Cosine of two perpendicular arrows ::: 0 Cosine when one arrow has length zero ::: undefined (division by zero) — guard and return 0 Which primitive fixes a contradicted fact ::: Update the value (then Forget the old one) Why divide the dot product by the two lengths ::: to remove magnitude and leave only angle/direction