Memory systems for agents: Visual walkthrough

This page is the visual companion to Memory systems for agents (index 6.2.5). It leans on Vector Databases (where these numbers get stored) and LM Context Window Management (why we retrieve at all).

Step 1 — Turn a memory into an arrow

WHAT. We take a memory (a sentence) and represent it as a point, then draw an arrow from the origin $(0, 0)$ to that point. That arrow is a vector. We write it $v = (v_{x}, v_{y})$ .

WHY. Computers cannot compare sentences directly, but they are excellent at comparing arrows. If we turn every memory into an arrow, "how similar are two memories?" becomes "how much do two arrows point the same way?" — a geometry question we can measure.

PICTURE. Below, one memory becomes one blue arrow. The two numbers $v_{x}$ and $v_{y}$ are just how far right and how far up the arrow reaches.

Step 2 — Two memories, two arrows, one angle

WHAT. Now draw two memories as two arrows from the same origin: $a$ (the new query, pink) and $b$ (an old stored memory, blue). Between them sits an angle we call $θ$ (the Greek letter "theta", just a name for that opening).

WHY. Here is the key insight of the whole page: similarity is direction, not length. A short arrow and a long arrow pointing the same way mean the same thing ("both about refunds"). What differs between similar and unsimilar memories is the angle between their arrows. So the quantity we actually want to measure is $θ$ .

PICTURE. The pink and blue arrows, and the wedge $θ$ between them. Small wedge = same topic. Wide wedge = different topics.

Step 3 — We need a number, not a picture. Enter cosine.

WHAT. We replace the angle $θ$ with the number $cos θ$ .

WHY THIS TOOL AND NOT ANOTHER? We could report $θ$ directly (in degrees), but three things make $cos θ$ better:

It's already normalised into a tidy $[- 1, + 1]$ range — great for thresholds like the parent's $0.95$ .
It is largest when arrows agree and shrinks as they diverge, so "bigger number = more similar" — the natural direction for ranking.
As we'll see in Step 4, it can be computed directly from the arrow's numbers without ever measuring an angle with a protractor.

PICTURE. A dial: as $θ$ sweeps from $0^{\circ}$ to $18 0^{\circ}$ , watch the value slide from $+ 1$ to $- 1$ . This is the "meaning meter."

Recall Undo check

Similarity equals 1 — what is the angle, and what does it mean? ::: The angle is $0^{\circ}$ : the arrows point the exact same way, so the memories mean the same thing.

Step 4 — The dot product: how a computer reads the angle off the numbers

WHAT. We compute one number by multiplying the arrows coordinate-by-coordinate and summing.

WHY (deriving the bridge, not just asserting it). The dot product looks like pure arithmetic, but it secretly measures the shadow. Here is why, in three plain moves:

Shadow times length. Take arrow $a$ and the shadow of $b$ on it. Multiply that shadow, $∥ b ∥ cos θ$ , by how long $a$ is, $∥ a ∥$ . That product is $∥ a ∥ ∥ b ∥ cos θ$ — pure geometry, all lengths and one angle.
The same product from coordinates. Now measure that same "shadow $\times$ length" quantity from the raw numbers. Line the arrows up along the axes and it factors coordinate by coordinate: the amount of $a$ and $b$ that agree along $x$ is $a_{x} b_{x}$ , and along $y$ is $a_{y} b_{y}$ . Their agreement adds up to $a_{x} b_{x} + a_{y} b_{y}$ — which is exactly the dot product.
Two names, one quantity. Steps 1 and 2 measured the same geometric thing two different ways, so they must be equal: $from coordinates (Step 2) a_{x} b_{x} + a_{y} b_{y} = shadow \times length (Step 1) ∥ a ∥ ∥ b ∥ cos θ$

That is the bridge — and now you know why it holds. Written with the dot-product symbol: $pure arithmetic a \cdot b = the two lengths ∥ a ∥ ∥ b ∥ the agreement we want cos θ$ The left side is easy to compute from raw numbers; the right side contains the $cos θ$ we actually crave, tangled up with the two lengths. If we divide the lengths out, we're left with exactly $cos θ$ . That's Step 5.

PICTURE. The dot product shown as "project the pink arrow $b$ onto the blue one $a$ (its shadow, length $∥ b ∥ cos θ$ ), then multiply by how long $a$ is." When they align, the shadow is long → big dot product. When perpendicular, the shadow is zero → dot product $0$ .

Step 5 — Divide the lengths out: the cosine similarity formula

WHAT. Take the bridge from Step 4 and solve it for $cos θ$ by dividing both sides by the two lengths.

WHY. Dividing by $∥ a ∥ ∥ b ∥$ cancels the length information — exactly the thing we decided (Step 2) not to care about — leaving only the direction agreement.

$cosine_sim (a, b) = cos θ = \frac{a \cdot b}{∥ a ∥ ∥ b ∥} = \frac{a _{x} b _{x} + a _{y} b _{y}}{a _{x}^{2} + a _{y}^{2} b _{x}^{2} + b _{y}^{2}}$

Term by term:

Numerator $a_{x} b_{x} + a_{y} b_{y}$ — the raw agreement (the dot product from Step 4).
Denominator $a_{x}^{2} + a_{y}^{2} b_{x}^{2} + b_{y}^{2}$ — the two lengths, which we cancel out so only direction survives.

PICTURE. Same two arrows as Step 2, but now both scaled to length $1$ (unit arrows). Once both are length one, the dot product is the cosine — the denominator becomes $1 \times 1$ .

Step 6 — The edge cases (never leave a gap)

Retrieval breaks in subtle ways. Cover all of them.

Case A — perfect match ( $+ 1$ ). New query is a copy of an old memory. Arrows overlap, $θ = 0$ , score $= 1$ . Retrieval returns it first.

Case B — orthogonal ( $0$ ). "Weather" query vs a "refund" memory. Right angle, dot product $0$ , score $0$ . The memory is correctly ignored.

Case C — opposite ( $- 1$ ). Rare in text but real for signed features: score $- 1$ means maximally unlike. Never retrieved.

Case D — the zero vector (degenerate!). If a memory embeds to $0 = (0, 0)$ — an empty string, or a failed embedding call — its length is $0 = 0$ , and the formula divides by zero. Undefined. Real systems guard against this: skip zero vectors, or define their similarity as $0$ . Never let this crash retrieval.

PICTURE. Four mini-panels: overlap, right-angle, opposite, and the collapsed dot (the zero vector) with a "⚠ divide-by-zero" flag.

Step 7 — From one score to a ranked retrieval

WHAT. Retrieval isn't one comparison — it's many. Embed the query once, compute cosine similarity against every stored memory, then sort from highest to lowest and keep the top $K$ .

WHY. This is the whole point of the arrow trick: comparing directions is cheap, so we can score thousands of memories fast and hand the agent only the few most relevant ones — small enough to fit the context budget from LM Context Window Management. The heavy lifting (nearest-arrow search) is what a vector database is built to do.

Definition What

$K$ is, and how you pick it == $K$ == is simply how many memories you keep after ranking — the size of the shortlist you actually feed back to the agent. It is a knob you choose, not something the maths hands you:

Too small (say $K = 1$ ): you risk missing a relevant memory that ranked just below the winner.
Too big (say $K = 50$ ): you flood the context window with weakly-related memories, adding cost and noise.
In practice $K$ is often a small number like $3$ – $10$ , sometimes combined with a score cutoff ("keep the top $K$ , but only those scoring above $0.7$ ") so that when nothing is truly similar you return fewer — or none — rather than padding the shortlist with junk. So $K$ trades recall (catching every relevant memory) against precision and budget (not drowning the agent in near-misses).

PICTURE. The pink query arrow surrounded by many blue memory arrows; the closest three (smallest angle, i.e. $K = 3$ ) are highlighted — those are the top- $K$ retrieved memories.

The one-picture summary

Everything on one board: memory → arrow → angle → dot product ÷ lengths → one number in $[- 1, 1]$ → rank → retrieve.

Recall Feynman retelling — say it plainly

We wanted an agent to find the right old memory. First, we turned every memory into an arrow made of a few numbers — that's the embedding. Then we noticed that similar memories point the same way, so what we really want is the angle between two arrows (always between $0^{\circ}$ and $18 0^{\circ}$ ): small angle means same topic. To get a clean number instead of an angle, we use its cosine, which runs neatly from $+ 1$ (same direction) to $- 1$ (opposite). The magic trick is that a computer can get that cosine without any protractor: the dot product (multiply coordinate-by-coordinate and add) secretly measures the shadow one arrow casts on the other, which is $∥ b ∥ cos θ$ times the length of $a$ — so dividing the dot product by both lengths leaves exactly the cosine. That single number is cosine similarity. We scored $0.96$ for two near-identical memories — above the parent's $0.95$ line — so we'd merge them. We checked the corners: aligned gives $1$ , perpendicular gives $0$ , opposite gives $- 1$ , and an all-zero arrow is undefined (guard against it!). Finally, we score the query against every memory, sort, and hand the agent the top $K$ (a shortlist size we choose). That's how remembering becomes geometry.