Foundations — Memory systems for agents

Before you can read the parent note comfortably, you need to own every symbol and word it throws at you. This page builds each one from nothing, in an order where every idea rests on the one before it. Nothing here assumes you have seen the notation before.

0. The picture we keep returning to

Every idea on this page lives inside one loop: a user speaks, the agent thinks with a language model, and a memory store sits on the side that the agent writes to and reads from. Hold this picture in your head.

Reading the figure: the yellow box on the left is the user; the blue box in the middle is the agent (the program running the loop); the red box on the right is the language model. Yellow arrows carry the user's message into the agent; the blue arrow labelled prompt is the text the agent sends to the model; the red arrow labelled reply is the text coming back. The green box at the bottom is the memory store, with two green arrows — write (agent saves something) and read (agent looks something up). The red note on the right reminds you the model itself forgets after every reply, which is precisely why the green filing cabinet has to exist.

1. Text as tokens — the unit everything is measured in

Before symbols, we need the thing being counted.

Why the topic needs it: every memory limit in the parent note is measured in tokens, not words. When it says "8K token window," it means the model can only look at about 8000 chunks at once.

2. The context window — the model's field of view

Reading the figure, step by step: along the bottom runs a strip of little squares — each square is one token of text, laid out left to right in reading order. The yellow triangle is the context window, the "beam" the model can actually see. The blue (lit) squares sit inside the beam: these are the tokens the model reads and can use for its reply. The grey squares on the left are labelled "lost / out of view" — they were part of the conversation earlier but have scrolled out of the beam and are gone unless something saved them. The grey squares on the right ("not yet seen") are future text that hasn't entered the beam yet. The one takeaway: only the blue tokens inside the beam influence the reply. Everything outside must live in the memory store or be lost.

Now we can define the three chunks that share the beam.

With those defined, the parent's constraint reads cleanly: $\text{Working Memory Size}\le \text{Context Window}-\text{Prompt}-\text{Output Buffer}$ In words: your scratchpad (working memory) can be at most the whole beam minus the always-present system-prompt instructions minus the reserved reply space. Managing this budget is the job of LM Context Window Management.

3. State vs stateless — the whole reason for the topic

4. A record and its curly braces { }

The parent writes an episode as $\{\text{timestamp},\text{user\_id},\text{summary},\dots \}$. You must know what the braces mean here.

Why the topic needs it: episodic and semantic memory both store bundles of named fields. The braces are just the notation for "these things travel together as one item."

5. An embedding — turning meaning into an arrow

This is the single most important new object in the topic. Take it slowly.

Reading the figure: the two grey axes are two of the (hundreds of) "meaning dimensions" — we only draw two so it fits on a page. Each labelled arrow is one short text turned into an embedding. Notice the green arrows "refund status" and "billing error" point almost the same way and land near each other, while the red arrows "cat photo" and "kitten picture" cluster somewhere else entirely, and "weather today" sits off on its own. Nearby arrows = related meanings. That clustering is the whole reason embeddings are useful.

Why the topic needs it: the parent stores episodes in a vector database and retrieves them by finding embeddings that point in a similar direction to the query's embedding. No embeddings → no semantic search.

6. Cosine similarity — measuring "same direction"

Now that meaning is an arrow, how do we score "these two arrows point the same way"?

Reading the figure: the two axes are again meaning-dimensions. The blue arrow is ${\mathbf{v}}_{\text{old}}$ and the yellow arrow is ${\mathbf{v}}_{\text{new}}$ — two embeddings drawn from the same origin. The red wedge between them is the angle $\theta$. Cosine similarity is a number that depends only on that wedge, not on how long either arrow is: squeeze the wedge toward zero and the score climbs toward $1$; open it to a right angle and the score drops to $0$. The two lines of white text spell out the two anchor cases so you can read any picture at a glance.

First we need the length symbol used in the formula.

So the parent's rule $\text{cosine\_sim}({\mathbf{v}}_{\text{new}},{\mathbf{v}}_{\text{old}})>0.95⟹\text{merge or delete}$ reads: "if the new arrow points almost exactly the same way as an old one (angle nearly zero), they mean nearly the same thing — don't store both."

7. Small symbols and the weights $w_1,w_2,w_3$

Here is a concrete forgetting rule from the parent, with the dials filled in so you can see them work. Suppose we score how much an episode $e$ is worth keeping: $\text{Importance}(e)=w_1\cdot \text{AccessCount}(e)+w_2\cdot \text{Recency}(e)+w_3\cdot \text{Feedback}(e)$

8. Semantic search, knowledge graph, and facts as triples

Why the topic needs it: the parent's semantic memory layer is exactly a knowledge graph of such triples — timeless facts, stored once, reused forever.

The prerequisite map

The diagram below is written in Mermaid, a tiny text-to-diagram language. You do not need to write it, only read it: each NAME["label"] is a box, and each --> is an arrow meaning "feeds into / is needed for." Follow the arrows upward and you see how the small ideas at the edges build toward the topic in the centre-bottom.

Equipment checklist

Cover the right side and test yourself. If any line surprises you, reread that section.