Exercises — Agentic frameworks (LangChain, LlamaIndex)

• Chain → (b) deterministic fixed sequence. No reasoning about order — the steps are hard-coded.
• Tool → (c) a callable function wrapped with a description + argument schema.
• Memory → (a) the state store persisting across turns.
• Retriever → (d) the interface to a vector DB / search index (see 5.3.02-Vector-databases).

Why these and not each other? A Chain is fixed; an Agent (not listed here) is the thing that chooses. Don't confuse the recipe with the cook.

False — it is the reverse.

• LlamaIndex is data-centric: how do I get the right context out of 10,000 pages?
• LangChain is orchestration-centric: how do I chain LM calls, tools and memory?

Mnemonic: Llama = Library (of documents). Chain = Chaining calls.

Thought: → Action: → Action Input: → Observation:, looping until Final Answer:.

• Thought/Action/Action Input come from the LM (it writes text).
• Observation is injected by the framework after running the tool.

See 6.2.04-ReAct-(Reasoning-and-Acting) for the origin of this pattern.

This is the white slice of the s01 bar. Subtract everything fixed from $C$:

$\text{used}=\underset{\text{sys}}{\underbrace{300}}+\underset{\text{memory}}{\underbrace{6\times 80}}+\underset{\text{reply}}{\underbrace{500}}=300+480+500=1280$

$\text{free for docs}=C-\text{used}=4096-1280=\boxed{2816\text{ tokens}}$

Why subtract the reply reservation? The model's output also lives in the window. If you forget it, generation gets truncated mid-sentence.

$k\le \lfloor \frac{2816}{400}\rfloor =\lfloor 7.04\rfloor =\boxed{7\text{ chunks}}$

We floor (round down) because a partial 8th chunk (the 8th block would need tokens $2800$–$3200$, past the $2816$ free) would overflow the window. This is exactly the "stuff" step of a RetrievalQA chain — the free budget sets your $k$. In the s01 bar, this is asking how many 400-token blocks fit in the white slice.

Tool(
    name="CurrencyConverter",
    func=convert,
    description="Convert an amount between currencies. "
                "Input: 'AMOUNT FROM_CODE TO_CODE', e.g. '100 USD EUR'."
)

Why the wording matters: the LM never sees the Python code — it decides whether and how to call the tool purely from the description string (see 6.2.05-Tool-calling-and-function-calling-APIs). Vague description → the tool is never chosen or is called with malformed input.

Trace the ReAct loop (each LM turn = one call):

That is 3 LM calls: $\text{cost}=3\times \$0.002=\boxed{\$0.006}$

Why 3 and not 2? After the last tool's observation, the LM must be called again to compose the Final Answer. Each observation costs you one more round trip — this is why chatty agents get expensive.

Keep the last 4 verbatim, summarise the other 16:

$\text{summary memory}=\underset{\text{summary}}{\underbrace{120}}+\underset{\text{recent}}{\underbrace{4\times 80}}=120+320=440\text{ tokens}$

$\text{savings}=1600-440=\boxed{1160\text{ tokens saved}}$

The tradeoff: the 16 summarised messages lose detail — an exact quote from message 3 may vanish. You trade fidelity for budget. That is why VectorStoreMemory exists: it retrieves the exact old message only when relevant.

Why it hurts: duplicate chunks consume two slots of your finite budget for one idea. With only $k=5$ slots, you have effectively wasted 20% of your context on redundant information, crowding out a different relevant chunk that never made the cut. Fix: apply maximal marginal relevance (MMR) — re-rank so each new chunk must be relevant to the query and dissimilar to already-chosen chunks. This is standard in 5.3.03-Retrieval-Augmented-Generation-(RAG) retrievers.

Both — a Chain wrapped as a Tool inside an Agent.

• The fixed FAQ recipe (embed → search → answer) has a known order ⇒ a RetrievalQA Chain. Deterministic, cheap, no "which step?" LM calls.
• The conditional weather branch requires a run-time decision ⇒ that decision belongs to an Agent.
• Architecture: expose the FAQ chain and the weather API each as a Tool; let a ReAct Agent choose between them.

Principle: fixed sub-recipes → Chains; run-time branching → Agent. Push determinism as deep as possible to save LM calls (recall Exercise 3.1: each decision costs a round trip).

MAX_STEPS = 8
steps = 0
while steps < MAX_STEPS:
    out = llm(prompt + history)
    parsed = parse(out)
    if isinstance(parsed, AgentFinish):
        return parsed.output
    result = tools.run(parsed.tool, parsed.tool_input)
    history.append((parsed, result))
    steps += 1
return "Stopped: max steps reached."   # graceful fallback

Failure mode prevented: an infinite reasoning loop — the LM keeps emitting Action: and never emits Final Answer:, e.g. re-querying the same tool. Each iteration is a paid LM call, so without a cap this is an unbounded bill and a hung request. LangChain's real max_iterations parameter does exactly this.

Symbols: $D$ is the set of all documents, $|D|$ its total token size, and $C$ is the context window (the fixed token budget from the intro). $|D|\gg C$ just means "the library is far too big to paste into one prompt." Let $m$ be the number of sub-questions the query is split into.

Two-stage design:

1. Router / sub-question stage — 1 LM call: one LM call reads the query and emits all $m$ sub-questions at once (a single generation listing them). This is 1 call regardless of $m$.
2. Retrieve stage — 0 LM calls: run one vector search per sub-question. Vector searches are pure database lookups, not LM calls, so these $m$ searches cost 0 LM calls.
3. Synthesise stage — 1 LM call: one final LM call fuses all retrieved chunks into the answer.

Exact cost (this design): $\text{LM calls}=\underset{\text{decompose}}{\underbrace{1}}+\underset{\text{retrievals}}{\underbrace{0\cdot m}}+\underset{\text{synthesise}}{\underbrace{1}}=2(\text{independent of }m).$

Caveat — when $m$ does enter the cost: if instead you answer each sub-question with its own LM call before fusing (a "sub-answer per branch" design), the count becomes $1+m+1=m+2$ LM calls. The cheap design above deliberately keeps it at 2 by decomposing and synthesising in single batched calls. That decomposition step is what buys cross-document reasoning flat top-k cannot give.

Build the growing prompt. Base prompt each call = 350 tokens; each completed iteration appends 150 tokens of observation that persists into all later calls.

Sum the input tokens across all four calls: $\text{total input tokens}=350+500+650+800=\boxed{2300\text{ tokens}}$

Convert to dollars at $0.01 per 1000 input tokens: $\text{cost}=\frac{2300}{1000}\times \$0.01=2.3\times \$0.01=\boxed{\$0.023}$

The key insight (super-linear cost): history accumulates, so each extra iteration is billed again on every later call. Total cost grows roughly with the square of the number of steps, not linearly. This is the quantitative reason to cap iterations and to summarise memory.

Same s01 bar, two different amber (memory) slices. Free document tokens $=C-\text{overhead}-\text{memory}$.

Buffer memory (all 30 messages carried verbatim): $\text{mem}=30\times 80=2400,\text{free}=4096-800-2400=\boxed{896}$

Summary memory (120-token summary + last 4 verbatim): $\text{mem}=120+4\times 80=120+320=440,\text{free}=4096-800-440=\boxed{2856}$

Why these numbers? Buffer memory pays 80 tokens for every message forever, so a long chat's amber slice grows without bound and squeezes the white document slice. Summary memory caps the old history at a fixed 120 tokens no matter how long the chat runs, so its amber slice stays small.

What this implies: summary memory frees $2856-896=1960$ extra document tokens — about 3.2× the buffer's document budget ($2856/896\approx 3.19$). On long chats, buffer memory starves retrieval: the documents that actually ground the answer get squeezed out by verbatim chat history. You trade some fidelity of old turns (an exact early quote may be lost in the summary) for far richer grounding context — usually a winning trade for RAG-style assistants.

(a) Everything agentic (4 calls each): $100,000\times 4\times \$0.002=\$800/\text{day}$

(b) Routed:

• FAQ (80% = 80,000) as a 1-call chain: 80{,}000 \times 1 \times \0.002 = $160$
• Agentic (20% = 20,000) at 4 calls: 20{,}000 \times 4 \times \0.002 = $160$
• Total = \160 + $160 = $320/\text{day}$

$\text{savings}=800-320=\boxed{\$480/\text{day}}(60\%\text{ cheaper})$

The lesson: cheap routing (send the easy 80% down the deterministic path) is where real production savings live — the same "push determinism deep" principle from Exercise 4.1, now with a price tag.

Each observation in a ReAct loop forces one extra ::: LM call, so cost grows super-linearly with steps. The retrieval budget for documents = window − (system + memory + reply reservation) ::: everything fixed is subtracted first. Summary memory trades ::: fidelity of old turns for a smaller token footprint. Fixed recipes should be ::: Chains; run-time branching should be an Agent.