Memory systems for agents

What Is Agent Memory?

The Three-Layer Memory Architecture

Modern agent memory systems use a three-tier hierarchy, inspired by human memory:

1. Working Memory (Short-Term)

Formula Working Memory Capacity

Purpose: Hold the immediate context for the current task. WHY: LMs have finite context windows. Working memory is the "active" portion of context that the agent uses for the current reasoning step.

Capacity constraint: $\text{Working Memory Size} \leq \text{Context Window} - \text{Prompt} - \text{Output Buffer}$

For a model with 8K token window: $\text{Working Memory} \approx 8000 - 1000\,(\text{system}) - 500\,(\text{output}) = 6500\text{ tokens}$

HOW IT WORKS:

Stores the current conversation turn + recent history (last 3-5 exchanges)
Managed via a sliding window or token budget
Contents: user query, intermediate reasoning steps, tool outputs, partial results

Worked example Working Memory in Action

User: "Find me articles about climate change published after 2020." Working Memory Holds:

User Query: climate change articles, after 2020
Tool Call: search_arxiv(query="climate change", year_start=2021)
Tool Output: [5 papers with titles/abstracts]
Next Step: Filter by relevance, summarize top 3

WHY THIS STRUCTURE? The agent needs the original request AND the search results together to complete the filtering step. Working memory keeps them "in mind."

2. Episodic Memory (Mid-Term)

Formula Episodic Memory as Event Log

Purpose: Store sequences of past interactions as retrievable episodes.

WHY: Not everything fits in working memory. Episodic memory archives past "episodes" (conversations, tasks) so the agent can recall "what happened last Tuesday."

Structure: Each episode is a record: $\text{Episode} = \{\text{timestamp},\, \text{user\_id},\, \text{summary},\, \text{key\_facts},\, \text{outcome}\}$

Retrieval mechanism:

Semantic search: Embed the current query, find similar past episodes via cosine similarity
Temporal lookup: "Retrieve all episodes from last week"
Entity-based: "All episodes involving user Alice" Storage: Vector database (Pinecone, Chroma, Weaviate) or append-only log with embedings.

Worked example Episodic Memory: Customer Support Agent

Scenario: User contacts support on Day 1 about a billing error. On Day 5, they follow up. Day 1 Episode:

{
  "timestamp": "2026-06-26T10:00:00Z",
  "user_id": "user_123",
  "summary": "Billing error: charged twice for June subscription",
  "actions": ["issued_refund", "flaged_billing_system"],
  "outcome": "resolved"
}

Day 5 Query: "Is my refund processed?" Agent Retrieval:

Embed the query → search episodic memory
Find Day 1 episode (high similarity to "refund")
Load into working memory: "User had billing issue, refund issued 5 days ago"
Check refund status tool → respond with update WHY EPISODIC HELPS: The agent doesn't ask the user to re-explain the problem. It picks up where it left off.

3. Semantic Memory (Long-Term)

Formula Semantic Memory as Knowledge Graph

Purpose: Store general facts, procedures, and world knowledge independent of specific episodes.

WHY: Episodic memory is event-specific. Semantic memory holds timeless knowledge:

"Python uses 0-based indexing"
"Our API rate limit is 100 requests/minute"
"User Alice prefers morning meeting slots" Structure: Knowledge graph or key-value store $\text{Fact} = \{\text{entity},\, \text{relation},\, \text{value},\, \text{confidence}\}$

Example:

("Python", "uses_indexing", "0-based", confidence=1.0)
("user_alice", "prefers_time", "09:00-11:00", confidence=0.85)

HOW IT UPDATES:

Extraction: After each interaction, extract facts from episodic memory
Deduplication: Merge with existing facts (update confidence if repeated)
Pruning: Remove facts contradicted by newer evidence

Worked example Semantic Memory: Code Assistant Agent

Scenario: User asks, "How do I read a CSV in Python?" Semantic Memory Contains:

("pandas", "function_for", "CSV_reading", "pd.read_csv()")
("pandas.read_csv", "requires", "import pandas as pd")
("CSV", "common_parameters", "delimiter, header, dtype")

Agent Response Construction:

Query semantic memory: "CSV reading in Python"
Retrieve fact: pandas.read_csv
Retrieve related facts: import statement, common params

Synthesize answer:

import pandas as pd
df = pd.read_csv('file.csv')  # common params: delimiter=', header=0

WHY SEMANTIC? The agent doesn't need to remember "which conversation taught me about pandas." It's general knowledge, stored once, used forever.

Memory Operations: The Four Primitives

All memory systems implement four core operations:

Formula Memory Operations

Write (Store) $\text{Write}(m, k, v) : \text{Memory} \times \text{Key} \times \text{Value} \to \text{Memory}'$ Add new information to memory store $m$ with key $k$ and value $v$ .
Read (Retrieve) $\text{Read}(m, q) : \text{Memory} \times \text{Query} \to \text{Value}$ Retrieve information from memory $m$ matching query $q$ (exact match or similarity search).
Update (Modify) $\text{Update}(m, k, v') : \text{Memory} \times \text{Key} \times \text{NewValue} \to \text{Memory}'$ Change existing entry with key $k$ to new value $v'$ .
Forget (Delete) $\text{Forget}(m, k) : \text{Memory} \times \text{Key} \to \text{Memory}'$ Remove entry with key $k$ (implements decay, privacy, or capacity management).

Derivation: When to Forget?

Problem: Infinite memory → cost explosion + noise (irrelevant old data).

Solution: Implement a forgetting policy.

Approach 1: Recency-Based (Sliding Window) $\text{Keep}(e) = \begin{cases} 1 & \text{if } t_{\text{now}} - t_e < T_{\text{window}} \\ 0 & \text{otherwise} \end{cases}$ Where $t_e$ is timestamp of episode $e$ , $T_{\text{window}}$ is retention period (e.g., 30 days).

WHY: Older data is less likely to be relevant. Simple implement.

Approach 2: Importance-Based $\text{Importance}(e) = w_1 \cdot \text{AccessCount}(e) + w_2 \cdot \text{Recency}(e) + w_3 \cdot \text{UserFeedback}(e)$ Keep top-K by importance score.

WHY THIS STEP? AccessCount → frequently used facts are valuable. UserFeedback → user-starred items should persist.

Approach 3: Semantic Redundancy If new episode $e_{\text{new}}$ has embedding $\mathbf{v}_{\text{new}}$ and existing episode $e_{\text{old}}$ has $\mathbf{v}_{\text{old}}$ : $\text{cosine\_sim}(\mathbf{v}_{\text{new}}, \mathbf{v}_{\text{old}}) > 0.95 \implies \text{Merge or Delete } e_{\text{old}}$

WHY: Avoid storing near-duplicate information. Saves space + reduces retrieval noise.

Retrieval Strategies: How to Find the Right Memory

Formula Memory Retrieval as Ranking Problem

Goal: Given current context $C$ , rank all memory items $\{m_1, m_2, \ldots, m_N\}$ and return top-K. Scoring function: $\text{Score}(m_i, C) = \alpha \cdot \text{Relevance}(m_i, C) + \beta \cdot \text{Recency}(m_i) + \gamma \cdot \text{Importance}(m_i)$

Where:

$\text{Relevance}(m_i, C) = \text{cosine\_similarity}(\text{embed}(m_i), \text{embed}(C))$
$\text{Recency}(m_i) = e^{-\lambda(t_{\text{now}} - t_{m_i})}$ (exponential decay)
$\text{Importance}(m_i)$ = access frequency or user rating

Hyperparameters: $\alpha, \beta, \gamma$ control trade-off (typically $\alpha=0.7, \beta=0.2, \gamma=0.1$ ).

Derivation from First Principles:

WHY RELEVANCE? If the agent is discussing "Python errors," retrieving memories about "dinner recipes" wastes context. Semantic similarity ensures topical match.
WHY RECENCY? Information decays. The user's preference from yesterday is more reliable than from a year ago. Exponential decay $e^{-\lambda t}$ models human forgetting curve.
WHY IMPORTANCE? Some facts are evergreen. "Our company policy on X" should persist even if not recently accessed. Access frequency signals this.
HOW TO COMBINE? Linear combination is simple, interpretable, and fast. Normalize each term to [0,1], then weight.

Step-by-step scoring:

\begin{align} \text{Relevance}(m_i, C) &= \frac{\mathbf{v}_{m_i} \cdot \mathbf{v}_C}{\|\mathbf{v}_{m_i}\| \|\mathbf{v}_C\|} \quad \in [0, 1] \\ \text{Recency}(m_i) &= e^{-0.1 \cdot \text{days\_ago}} \quad \in [0, 1] \\ \text{Importance}(m_i) &= \frac{\text{access\_count}(m_i)}{\max_j \text{access\_count}(m_j)} \quad \in [0, 1] \\ \text{Score}(m_i, C) &= 0.7 \cdot \text{Relevance} + 0.2 \cdot \text{Recency} + 0.1 \cdot \text{Importance} \end{align}

Worked example Retrieval Scoring Walkthrough

Current Context (C): "How do I handle authentication errors in our API?"

Memory Items: | $m_i$ | Content | Relevance | Recency | Importance | Score | |-------|---------|-----------|----------|--------| | $m_1$ | "API auth uses OAuth2" | 0.92 | 0.81 (5 days) | 0.90 (high access) | $0.7(0.92)+0.2(0.81)+0.1(0.90)=\mathbf{0.896}$ | | $m_2$ | "Lunch menu for Friday" | 0.05 | 0.95 (1 day) | 0.10 | $0.7(0.05)+0.2(0.95)+0.1(0.10)=\mathbf{0.235}$ | | $m_3$ | "Auth error codes: 401, 403" | 0.88 | 0.37 (30 days) | 0.60 | $0.7(0.88)+0.2(0.37)+0.1(0.60)=\mathbf{0.750}$ | Top-K (K=2): Return $m_1$ and $m_3$ .

WHY THIS ORDER? $m_1$ wins: high relevance + recency + importance. $m_3$ is relevant but older. $m_2$ fails: irrelevant despite being recent.

Memory Architectures in Practice

Pattern 1: Conversation Buffer Memory

Definition Buffer Memory

Conversation buffer memory keeps the last N messages in working memory. Simplest form.

Implementation:

class BufferMemory:
    def __init__(self, max_messages=10):
        self.buffer = []
        self.max max_messages
    
    def add(self, message):
        self.buffer.append(message)
        if len(self.buffer) > self.max:
            self.buffer.pop(0)  # FIFO

Pros: Simple, fast, no retrieval needed. Cons: Fixed size, no semantic selection, forgets everything beyond N.

Pattern 2: Summary Memory

Pattern 3: Vector Store Memory

Pattern 4: Hybrid Memory (Combining All Three)

Formula Unified Memory Architecture

Design: Layer all three memory types, with smart routing.

\text{Memory System} = \begin{cases} \text{Working (buffer)} & \text{last 5 turns, always loaded} \\ \text{Episodic (vector store)} & \text{retrieve top-K from past 1000 episodes} \\ \text{Semantic (knowledge graph)} & \text{query facts/entities relevant to context} \end{cases}

Retrieval algorithm:

Load working memory (cheap, always relevant)
Query episodic memory: retrieve3-5 past episodes via semantic search
Query semantic memory: extract facts mentioned in (1) or (2)
Merge all into single context, rank by relevance, fit within context window

Worked example Hybrid Memory: Personal Research Assistant

User Query: "Summarize the paper on attention mechanisms I read last month."

Memory Retrieval:

Working Memory: Current query (just now)

Episodic Memory: Search "attention mechanisms" → finds episode from2026-06-01:

{
  "date": "2026-06-01",
  "action": "read_paper",
  "paper": "Attention Is All You Need (Vaswani et al.)",
  "notes": "Transformers use self-attention, no RNs..."
}
``` 3. **Semantic Memory**: Query "Vaswani et al." → retrieves:
- "Transformer architecture has encoder-decoder"
- "Self-attention computes Q, K, V matrices"

Agent Response: Combines episodic (which paper) + semantic (what it contains) → produces summary.

Common Mistakes & How to Avoid Them

Connections

Retrieval-Augmented Generation (RAG): Memory systems use RAG principles to fetch relevant context.
Vector Databases: Episodic and semantic memory often stored in vector DBs (Pinecone, Chroma).
LM Context Window Management: Working memory fits within context window constraints.
Knowledge Graphs: Semantic memory often structured as knowledge graphs (entities + relations).
Prompt Engineering: Memory retrieval results must be formatted into effective prompts.
Agent Planning and Reasoning: Memory provides the "experience" that planning relies on.
Multi-Agent Systems: Agents can share semantic memory (common knowledge base) while maintaining separate episodic memory.

Recall Explain to a 12-Year-Old

Imagine you're building a robot assistant for your homework. Every time you ask it a question, it answers, then forgets everything. So next time you ask "What was my math homework?", it goes "I don't know, I just woke up!" That would be annoying, right? Memory systems are like giving your robot a notebook:

Working memory = the page the robot is writing on right now (today's homework)
Episodic memory = the old pages in the notebook (what homework you did last week)
Semantic memory = a cheat sheet the robot made of useful facts (like "math homework is always on page 45")

When you ask a question, the robot:

Looks at the current page (working memory)
Flips back through old pages to find related stuff (episodic memory)
Checks the cheat sheet for general rules (semantic memory) This way, the robot remembers what you talked about before, learns from past mistakes, and gets smarter over time. It's like having a study buddy who actually pays attention!

Flashcards

#flashcards/ai-ml

What are the three layers of agent memory systems? :: Working memory (short-term, current context), Episodic memory (mid-term, past interactions), Semantic memory (long-term, general knowledge).

Why do agents need memory beyond just conversation history?

To maintain state across interactions, learn from experience, build long-term context, and avoid repeating questions or mistakes.

What is working memory in agent systems?

The immediate context for the current task, stored within the LM's context window, typically the last few conversation turns plus active reasoning steps.

What is episodic memory?

A store of past interaction sequences (episodes) that can be retrieved via semantic search, temporal lookup, or entity-based queries. Archives "what happened when."

What is semantic memory in agents?

Long-term storage of general facts, procedures, and world knowledge independent of specific episodes. Structured as knowledge graphs or key-value stores.

What are the four core memory operations?

Write (store), Read (retrieve), Update (modify), Forget (delete).

Why is forgetting important in memory systems?

To manage storage costs, reduce retrieval noise from irrelevant old data, improve performance, and respect privacy requirements.

What is the memory retrieval scoring formula?

Score(m, C) = α·Relevance(m, C) + β·Recency(m) + γ·Importance(m), where α, β, γ are weights (typically 0.7, 0.2, 0.1).

What is conversation buffer memory?

The simplest memory type that keeps the last N messages in working memory using a FIFO queue.

What is summary memory?

A memory system that stores compressed summaries of conversations instead of raw messages, saving space at the cost of detail.

What is vector store memory?

Memory stored as embedings in a vector database, enabling semantic search via nearest-neighbor retrieval.

What is hybrid memory architecture?

A system combining working memory (buffer), episodic memory (vector store), and semantic memory (knowledge graph) with smart retrieval routing.

What is the capacity constraint for working memory?

Working Memory Size ≤ Context Window - System Prompt - Output Buffer

How does recency-based forgetting work?

Keep(e) = 1 if t_now - t_e < T_window, else 0. Deletes memories older than the retention window.

How does importance-based memory ranking work?

Importance(e) = w1·AccessCount(e) + w2·Recency(e) + w3·UserFeedback(e). Keeps top-K items by importance score.

What is the relevance component in retrieval scoring?

Relevance(m, C) = cosine_similarity(embed(m), embed(C)), measuring semantic similarity between memory and current context.

What is the recency component in retrieval scoring? :: Recency(m) = e^(-λ(t_now - t_m)), an exponential decay modeling human forgetting curves.

What is semantic redundancy pruning?

If cosine_sim(v_new, v_old) > threshold (e.g., 0.95), merge or delete the old memory to avoid storing duplicates.

Why not stuff all history into the LM prompt?

Context windows are limited, irrelevant history adds noise and degrades performance, and costs scale linearly with input token count.

Why use hybrid scoring instead of just embedding similarity?

Pure mises recency (outdated info) and importance (rare but critical facts). Hybrid scoring balances semantic match with temporal and usage signals.

What happens if memories never get updated?

Stale facts persist (user preferences change, APIs update, facts become outdated), causing the agent to contradict itself or provide wrong information.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, agent memory ka core idea bilkul human conversation jaisa hai. Socho ki agar tum kisi se baat kar rahe ho aur har 5 second baad tum sab kuch bhool jaao, toh kya hoga? Tum baar-baar same cheez repeat karoge, context miss karoge, aur kaam aage nahi badhega. AI agents ke saath bhi exactly yahi problem hai. Bina memory ke, har agent call ek fresh start hota hai—jaise har baar naya banda aa raha ho. Memory system agent ko yeh power deta hai ki woh past interactions yaad rakhe, experience se seekhe, overall goal ka context banaye, aur multi-step task mein apni position track kare. Isse agent ek simple "tool" se badalkar ek real "partner" ban jaata hai.

Ab yeh memory kaam kaise karti hai? Iska architecture human brain se inspired hai aur teen layers mein bata hai. Pehli hai Working Memory (short-term)—yeh current task ka immediate context hold karti hai, jaise abhi ka user query, tool outputs aur recent 3-5 exchanges. Problem yeh hai ki LM ka context window limited hota hai (maano 8K tokens), toh system ko formula se calculate karna padta hai ki kitna working memory bache ga after system prompt aur output buffer nikaal ke—roughly 6500 tokens. Doosri hai Episodic Memory (mid-term)—yeh past conversations ko "episodes" ki tarah store karti hai with timestamp, user_id, summary aur outcome. Jab kuch chahiye toh agent semantic search (cosine similarity), temporal lookup ya entity-based search se relevant episode nikaal leta hai, usually vector database (Pinecone, Chroma) use karke.

Yeh why-it-matters isliye hai kyunki real applications mein context bahut zaroori hota hai. Jaise customer support agent ka example lo—Day 1 pe user billing error report karta hai aur refund issue hota hai. Day 5 pe jab woh puchta hai "Is my refund processed?", toh agent apni episodic memory se woh purana episode retrieve karta hai, working memory mein load karta hai, aur seedha relevant answer deta hai. Bina memory ke, agent poora context dobara maangta aur user frustrate ho jaata. Toh basically, yeh multi-layer architecture agent ko smart, consistent aur genuinely helpful banata hai—jo asli intelligence ki nishaani hai.

What Is Agent Memory?

The Three-Layer Memory Architecture

1. Working Memory (Short-Term)

2. Episodic Memory (Mid-Term)

3. Semantic Memory (Long-Term)

Memory Operations: The Four Primitives

Derivation: When to Forget?

Retrieval Strategies: How to Find the Right Memory

Memory Architectures in Practice

Pattern 1: Conversation Buffer Memory

Pattern 2: Summary Memory

Pattern 3: Vector Store Memory

Pattern 4: Hybrid Memory (Combining All Three)

Common Mistakes & How to Avoid Them

Connections

Flashcards

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — AI Agents & Tool Use

Connections