Agent architectures and reasoning loops: Exercises

These exercises build from recognizing the pieces of an agent to designing your own reasoning loop. Every problem has a full solution hidden in a collapsible callout — try first, then reveal.

The state at step $t$ is written $s_{t}$ (everything the agent currently knows). An action $a$ is a tool call or a final answer. An observation $o$ is what the world returns after an action. Keep these three letters in mind — every exercise reuses them.

Level 1 — Recognition

Exercise 1.1 — Name the phase

For each line of an agent's log, name which loop phase it belongs to: Observe, Reason, Act, or Update.

Thought: I need the capital of France first.
search("capital of France")
Observation: "Paris is the capital."
Context grows to include the new observation before the next cycle.

Recall Solution

Reason — the model is thinking, producing a plan before doing anything.
Act — an actual tool is invoked (search(...)).
Observe — the world's reply enters the agent. (It is consumed at the start of the next cycle.)
Update — the observation is appended to $s_{t}$ to form $s_{t + 1}$ .

One clean sentence to memorize: Reason makes a plan, Act runs it, Observe catches the result, Update remembers it.

Exercise 1.2 — Match architecture to formula

Match each agent variant to its decision rule.

Variant	Rule
A. Simple reflex	1. $a = ar g max_{a} goal_achieved (a, s)$
B. Goal-based	2. $a = f (current percept)$
C. Utility-based	3. $a = ar g max_{a} U (a, s)$

Recall Solution

A → 2. A reflex agent maps the current percept straight to an action. No memory, no lookahead. Think thermostat: "too cold → turn on heat."
B → 1. A goal-based agent asks "does this action get me closer to the goal?" and picks any action that satisfies it.
C → 3. A utility-based agent scores how good each outcome is (not just goal/no-goal) and maximizes utility $U$ . It can trade off speed vs. cost vs. quality.

The escalation is: reflex (no memory) → goal (binary success) → utility (graded preference).

Level 2 — Application

Exercise 2.1 — Count the cycles

A ReAct agent solves "Is the population of the capital of France larger than London's?" It must (a) find France's capital, (b) find that city's population, (c) find London's population, (d) compare and answer. If each sub-goal needs exactly one tool call and the final comparison needs no tool, how many full loop cycles run before termination?

Recall Solution

Count one cycle per Reason→Act→Observe→Update pass:

Cycle 1: find capital → "Paris"
Cycle 2: find Paris population → "12.3M"
Cycle 3: find London population → "9.6M"
Cycle 4: Reason "Paris > London", Act = respond(...), then terminate.

Answer: 4 cycles. The final answer is an action (respond), so it still counts as a cycle — it just carries no tool observation. Total = 4.

Exercise 2.2 — ReAct prompt length

In ReAct the prompt at step $t$ is $Prompt_{t} = instruction + \sum_{i = 1}^{t - 1} (Thought_{i}, Action_{i}, Observation_{i}) + "Thought:"$ Suppose the instruction is 50 tokens, and each completed triple (Thought+Action+Observation) is 80 tokens, and the trailing "Thought:" is 2 tokens. What is the prompt length just before cycle 4 begins (i.e. 3 triples already in history)?

Recall Solution

Before cycle 4, three triples are complete ( $t - 1 = 3$ ): $Prompt = 50 + 3 \times 80 + 2 = 50 + 240 + 2 = 292 tokens.$ Answer: 292 tokens. Notice the prompt grows linearly — each cycle adds one 80-token triple. This is exactly why context windows eventually force summarization: history accumulates every loop.

Exercise 2.3 — Per-cycle token growth

Using the numbers in 2.1/2.2, how many tokens does the prompt grow per cycle, and after how many cycles would a 128,000-token context window be exceeded (starting from the 50-token instruction, ignoring the model's own output)?

Recall Solution

Each cycle appends one triple = 80 tokens per cycle. After $n$ cycles the input is $50 + 80 n$ tokens (dropping the trailing "Thought:" for the bound). Set this above the window: $50 + 80 n > 128000 ⟹ 80 n > 127950 ⟹ n > 1599.375.$ So at $n = 1600$ cycles the window is exceeded ( $50 + 80 \times 1600 = 128050 > 128000$ ). Answer: grows 80 tokens/cycle; overflows on the 1600th cycle. In practice you never reach this — you retrieve or summarize old triples long before.

Level 3 — Analysis

Exercise 3.1 — Why the "Thought:" prefix?

ReAct forces the model to emit Thought: text before every Action:. A colleague proposes deleting the thoughts to "save tokens." Explain, from the loop's mechanics, what breaks and give one concrete failure it would cause.

Recall Solution

What the prefix does: it makes the model write its reasoning into the context before choosing a tool. Because ReAct feeds the whole history back in (see 2.2), that written reasoning becomes visible on the next cycle — the model literally reads its own justification.

What breaks without it: the model jumps straight to actions with no recorded rationale. Two failures:

No error recovery — if a tool returns a surprising result, there's no prior thought to reconcile it against, so the agent can't notice it went off-plan (this is what Chain-of-Thought and Reflexion rely on).
Sub-goal collapse — compound questions ("capital and compare") get answered as one blind tool call, because the decomposition lived in the thought text you deleted.

Verdict: the thoughts aren't decoration; they are working memory written into the prompt. Deleting them removes the scratchpad the next cycle depends on.

Exercise 3.2 — Choose the reasoning strategy

For each task, pick ReAct, Chain-of-Thought (CoT), Tree-of-Thoughts (ToT), or Reflexion, and justify in one line.

"What's the weather in Tokyo right now, then suggest an outfit."
"If 3 apples cost $5, how much for 7?" (no tools available)
"Write the opening of a mystery novel — explore several possible hooks and keep the best."
"This code throws a TypeError; fix it, and if the fix fails, try again."

Recall Solution

ReAct — needs a live tool (weather API), so reasoning must interleave with actions.
CoT — pure stepwise arithmetic, no external tool: compute $5/3 per apple, then $\times 7$ .
ToT — generate $k$ candidate hooks, score them, expand the best branch; creative tasks with many viable paths.
Reflexion — the loop learns from failure: run → read error → self-critique → retry with the fix.

Rule of thumb: need a tool → ReAct; pure logic → CoT; many branches → ToT; must recover from mistakes → Reflexion.

Exercise 3.3 — Reflexion arithmetic check

In the Reflexion debug example, attempt 1 fails with TypeError: int + str. The agent's fix casts input with int(...). Verify the corrected computation: if the intended output is 4 + 2 where user input "2" should become integer 2, and the base is 4, what does the fixed function return?

Recall Solution

Before fix: 4 + "2" → TypeError (can't add int and str). After fix: int("2") = 2, then 4 + 2 = 6. Answer: 6. The Reflexion loop's value is that the agent diagnosed the type mismatch from the error string itself and edited the plan — no human intervention.

Level 4 — Synthesis

Exercise 4.1 — Design a memory policy

An agent has a 128k-token context and runs long conversations. Design a short-term memory eviction policy that keeps the loop working past the window limit. State (a) what you keep, (b) what you move to long-term memory, and (c) how you retrieve it back.

Recall Solution

(a) Keep in short-term (context): the instruction/system prompt, the current goal, and the most recent $m$ triples (the loop can't reason without recent history).

(b) Move to long-term: older triples that contain durable facts (e.g. "Paris population ≈ 12.3M"). Embed each fact into a vector: $embed (fact) \to vector DB .$ This is exactly RAG applied to the agent's own history.

(c) Retrieve back on demand: when a new query arrives, fetch the most similar stored facts: $retrieved = top- k (similarity (query, DB)),$ and splice them into the context. Net effect: short-term stays small and fresh; long-term is effectively unbounded and searchable. This is the standard fix for the overflow computed in 2.3.

Exercise 4.2 — Build a loop for a booking agent

Sketch the reasoning loop (as a state trace) for: "Book me the cheapest flight from Delhi to London next Friday under $600." List the phases for at least two cycles, and name where a utility-based decision (not just goal-based) appears.

Recall Solution

Cycle 1

Observe: user request parsed (origin=DEL, dest=LON, date=next Fri, budget=$600).
Reason: "I need candidate flights first."
Act: search_flights(DEL, LON, date).
Observe/Update: list of flights with prices returned; append to $s$ .

Cycle 2

Observe: candidate list in context.
Reason: filter to price < $600, then choose the cheapest — this is a utility step, because among all goal-satisfying flights we maximize $U = - price$ (lower price = higher utility). A pure goal-based agent would accept any flight under $600; the utility-based one picks the best.
Act: book(cheapest_valid_flight).
Terminate: goal met, confirmation returned.

Key insight: "under $600" is a goal constraint; "cheapest" is a utility objective. Real agents blend both — feasibility from goal logic, ranking from utility.

Level 5 — Mastery

Exercise 5.1 — Diagnose a broken loop

An agent never terminates on the task "What is 2+2?" Its trace:

Thought: I should search for the answer.
Action: search("what is 2+2")
Observation: "Various results about arithmetic..."
Thought: I should search for the answer.
Action: search("what is 2+2")
...

Identify two distinct architectural faults and propose a fix for each.

Recall Solution

Fault 1 — wrong reasoning strategy. 2+2 needs no tool; this is a pure CoT task. The agent is using ReAct where CoT suffices, so it keeps reaching for a search tool. Fix: let the reasoning engine answer directly when the sub-goal is self-contained arithmetic — no action needed.

Fault 2 — no terminate/loop-guard. The trace repeats identical Thought+Action forever. The terminate check ("goal met? max steps hit?") is missing or broken. Fix: add a max-step cap and a loop detector — if the current (Thought, Action) equals a previous one, force termination or escalate. This is the standard "max iterations" safety valve.

Together: the CoT fix removes the cause; the loop-guard fix bounds the damage if any future loop misbehaves.

Exercise 5.2 — Prove the context growth bound

Using the ReAct prompt-length model from 2.2 (instruction $I$ tokens, each triple $T$ tokens, trailing prompt $p$ tokens), derive a closed-form for prompt length $L_{n}$ entering cycle $n$ , then compute $L_{n}$ for $I = 50, T = 80, p = 2, n = 10$ .

Recall Solution

Entering cycle $n$ , exactly $n - 1$ triples are complete: $L_{n} = I + (n - 1) T + p .$ This is linear in $n$ with slope $T$ — confirming the "80 tokens per cycle" from 2.3.

For $I = 50, T = 80, p = 2, n = 10$ : $L_{10} = 50 + 9 \times 80 + 2 = 50 + 720 + 2 = 772 tokens.$ Answer: 772 tokens. The formula is the mathematical backbone of every memory/summarization decision: because $L_{n}$ grows without bound, some eviction policy (Exercise 4.1) is mandatory, not optional.

Exercise 5.3 — Full-stack scenario

Design a single agent for: "Research the three largest cities in Japan, store their populations for future questions, and tell me which is biggest." Name (a) the reasoning strategy, (b) which memory tiers you use, (c) the number of tool cycles, and (d) the terminate condition.

Recall Solution

(a) Strategy: ReAct — needs live search tools, so reasoning interleaves with actions.

(b) Memory tiers:

Short-term: the three population facts held in context for the immediate comparison.
Long-term: embed each city → population fact into the vector DB ("store for future questions" is an explicit long-term requirement — see RAG).
Working: a scratch tally to track the running maximum as results arrive.

(c) Tool cycles: one search per city = 3 search cycles, then 1 respond cycle = 4 cycles (same counting rule as 2.1 — the answer is an action).

(d) Terminate: all three populations retrieved and the maximum identified and the respond(...) action emitted. A max-step guard (Exercise 5.1) backs this up.

This ties together every module: perception (parse request), reasoning engine (ReAct), action execution (search/store), memory (all three tiers), and the terminate check.

Recall Quick self-test

Loop phases in order ::: Observe → Reason → Act → Update Prompt length entering cycle n (ReAct) ::: $L_{n} = I + (n - 1) T + p$ Strategy for a pure-arithmetic no-tool task ::: Chain-of-Thought Strategy that learns from failed attempts ::: Reflexion Where "store for future questions" data goes ::: Long-term (vector DB) memory Two fixes for a non-terminating loop ::: right strategy (CoT for arithmetic) + max-step/loop guard