Foundations — Agent architectures and reasoning loops

This page assumes you have seen none of the notation. We build each symbol in order, anchor it to a picture, and say why the topic cannot live without it. Start reading at line one.

Parent: Agent architectures and reasoning loops.

0. The mental picture we keep returning to

Everything below is a label on one of the boxes in this diagram. Keep it in your eye.

What figure s01 shows: four labelled boxes arranged in a ring — Observe (magenta, holding the state $s_t$), Reason (violet, choosing an action $a$), Act (orange, running the action), and Update (navy, producing the next state $s_{t+1}$). Curved arrows run box-to-box in a clockwise cycle, and in the middle the note "$t:=t+1$ (stamp next turn)" reminds you the crank counter ticks up on every full loop.

The agent is a crank you turn again and again. One full turn = observe, then reason, then act, then update. The parent note calls one turn a cycle or a step. We now name each part of the machine.

1. The subscript $t$ — "which turn are we on?"

The picture: in figure s01, imagine a tiny number stamped on the crank each time you pull it: turn 1, turn 2, turn 3. That stamp is $t$.

Why the topic needs it: the loop repeats, so the same quantity (the state, the context) exists in many versions — one per turn. Without a counter we could not say "the state before this turn" versus "the state after". The whole idea of a loop is "the new thing depends on the old thing", and $t$ vs $t+1$ is how we write "old" and "new".

2. State — the symbol $s$

The picture: in figure s02 the state is the whole scroll of paper the agent is holding — every note written on it so far.

Why the topic needs it: a decision has to be about something. The state is that something. When the parent writes

$s_0\stackrel{\text{observe}}{\to }{s}_1$

it is saying: "the starting picture $s_0$, after observing, becomes an updated picture $s_1$." The arrow with a word on top means "turns into, by doing that word".

3. Action — the symbol $a$

The picture: the single card the agent plays face-up from its hand (figure s02, right side). Exactly one per turn.

Why the topic needs it: thinking alone changes nothing in the world. The action is the only part that reaches outside the agent. The parent's phrase "tool call or response" is the two flavours of $a$: either poke the outside world, or stop and answer.

4. Observation — the symbol $o$

The picture: the reply slip that slides back through the slot after you post an action (figure s01, "Act → Observe" arrow).

Why the topic needs it: this is how the loop learns anything new. Without $o$, every turn would think about the same stale state forever. The observation is the fresh fact that makes the next state different from this one.

5. Thought — the symbol ${\text{Thought}}_t$

The picture: in figure s02, before the agent plays the action card it scribbles a sticky note ("what should I do next?") and clips it onto the scroll. That sticky note is ${\text{Thought}}_t$.

Why the topic needs it: the ReAct pattern (see Chain-of-Thought Prompting) forces the model to speak its reasoning aloud before acting. So one turn actually produces three text pieces in order: a thought, then the action it implies, then the observation that comes back. Written with the turn counter these are $({\text{Thought}}_t,{\text{Action}}_t,{\text{Observation}}_t)$ — the same $a$ and $o$ from Sections 3–4, now paired with the reasoning that chose them. Keep this trio in mind; Section 8 glues a whole history of them together.

6. A function — the symbol $f(\cdot )$

The picture: a vending machine — coin in, snack out (figure s03).

Why the topic needs it, and why this tool: the parent writes things like $\text{action}=f(\text{current percept}).$ We use function notation instead of prose because it answers one precise question: "the output depends on exactly these inputs, and nothing hidden." A simple reflex agent's $f$ takes only the current percept — that single-input picture is why it has no memory. A goal-based agent's $f$ takes more inputs. The notation makes the difference visible at a glance.

7. Two arrow symbols — telling $⟶$ from $\to$ by their heads

The picture: figure s01 shows the long labelled transition arrows going round the loop ("observe", "reason", "act", "update"). Figure s04 shows a short bare flow arrow: a pipe pouring a fact into the memory bucket.

Why the topic needs both: the loop transitions between states; memory stores facts. Two jobs, so we keep two visibly different arrows — the transition arrow is longer and always dressed with a word, the flow arrow is short and naked.

8. The summation sign $\sum$ — "glue all the turns together"

The picture: figure s04 — a stack of index cards, one per past turn, clipped into one thick bundle.

Why the topic needs it, and why this tool: the ReAct prompt is $\text{Prompt}=I+{\sum }_{i=1}^{t-1}({\text{Thought}}_i,{\text{Action}}_i,{\text{Observation}}_i)+\text{"Thought:"}$ Read left to right: start with the fixed brief $I$, then glue on every past turn's trio (the $({\text{Thought}}_i,{\text{Action}}_i,{\text{Observation}}_i)$ from Sections 3–5), and finally add the bare word "Thought:" as a cue that tells the model "your turn — reason next." Here "+" is not arithmetic — it's concatenation, gluing text end to end. We use $\sum$ because there are a variable number of past turns (it grows every loop), and $\sum$ is the compact way to say "all of them, however many there are". Writing them out one by one is impossible when the count keeps changing.

9. $\arg \max$ — "which choice wins?"

The picture: figure s03 — several actions on a shelf, each with a score tag; $\arg \max$ is the hand pointing at the highest-scoring one.

Why the topic needs it, and why this tool: a goal-based agent must pick an action. We write $\text{action}=\arg {\max }_a\text{goal\_achieved}(a,\text{state}).$ In words: "look at every possible action $a$, score each with $\text{goal\_achieved}$, and keep the action with the top score." We choose $\arg \max$ (and not plain $\max$) because we want the action itself, not its score. $\max$ would hand us the number; $\arg \max$ hands us the move to make. An agent needs the move.

All cases covered:

• If two actions tie for the highest score, $\arg \max$ picks one (implementations break ties by order or randomly) — the agent still acts, never freezes.
• If every action scores zero (goal unreachable this turn), $\arg \max$ still returns some $a$; a well-built agent then checks its terminate condition instead of looping forever.

10. The reward $r$ — the RL cousin of the score

The picture: imagine, in figure s01, that the reply slip coming back on the "observe" arrow carries two things: the observation $o$ (the facts) and a score stamp $r$ (a thumbs-up or thumbs-down on the last action).

Why we mention it here: the parent says the $s,a,o$ vocabulary "comes straight from" reinforcement learning — and RL's native loop is actually $s\to a\to r\to s^{\prime }$ (state, action, reward, next state). Agent architectures borrow the state/action/observation skeleton but usually run without a numeric $r$: an LM agent judges success from the observation text, not from an environment-supplied reward. Knowing $r$ exists tells you what agent loops leave out — and why RL-trained agents (that do use $r$) can learn better action choices over time, while a plain ReAct agent cannot.

11. top-$k$ and similarity — the memory-recall symbols

The picture: figure s04, lower half — a query magnet pulls the $k$ nearest cards out of a filing drawer.

Why the topic needs it: long-term memory can hold thousands of facts, but the context window (next symbol) is small. So we retrieve only the $k$ most relevant. Choosing $k$ (say 5) is choosing how much to remember at once — a tuning knob, not a fixed law. See Retrieval-Augmented Generation.

12. The set braces $\{\}$ — "a bag of named things"

Why the topic needs it: the parent writes $\text{Memory}=\{\text{Short-term},\text{Long-term},\text{Working}\}.$ This just says "memory comes in these three named kinds" — nothing is being added or multiplied; the braces are a labelled bag.

13. The context window — a length limit, not a symbol

The picture: the scroll from figure s02 has a maximum length; past that, old writing falls off the top.

Why the topic needs it: it's why short-term memory can't just grow forever, and why long-term memory + top-$k$ retrieval exist at all. See LM Context Windows for the full story, and Retrieval-Augmented Generation for how retrieval sidesteps the limit.

How these feed the topic

Each short node label is spelled out in the legend directly below the diagram — read them together.

Read it top-down: the counter $t$ lets state, action, observation take turns; the function and argmax (fed by the goal score) live inside the reasoning engine that chooses the action; the sum, the instruction $I$, and the thoughts build the ReAct prompt; set braces, similarity, top-k, and the window organise memory, which flows back into the state; reward $r$ is the RL-only extra that a plain agent leaves out. Together they form the loop.

Related patterns to explore next

The strategies for choosing $a$ each get their own note: Chain-of-Thought Prompting, Tool Use and Function Calling, Prompt Engineering, Reinforcement Learning Basics (the $s,a,r$ vocabulary comes straight from here), and Multi-Agent Systems.

Equipment checklist

Cover the right side and answer aloud. If any fails, re-read that section.