AI Agents & Tool Use

Question 1 (12 marks)

(a) ReAct trace (4 marks) — 1 mark per correct triple structure (max 4):

1. Thought: "I need the order details to answer." Action: lookup_order(id=...) Observation: {order status, items}
2. Thought: "Order shows delayed shipment; check KB for policy." Action: search_kb("shipping delay policy") Observation: policy text
3. Thought: "I can now answer the customer." Action: Finish(answer) (final reasoning step)

Accept any coherent trace with the mandated tool order (reason → lookup_order → reason → search_kb → reason). Structure must alternate Thought/Action/Observation.

(b) Latency & cost (4 marks)

Reasoning steps = 3 → 3 × 800 ms = 2400 ms; 3 × $0.01 =$0.03 lookup_order = 500 ms, $0.004 `search_kb` = 200 ms,$0.001

• Total latency = 2400 + 500 + 200 = 3100 ms = 3.1 s (2 marks)
• Total cost = 0.03 + 0.004 + 0.001 = $0.035 (2 marks)

(c) SLA analysis (4 marks)

• Escalated queries add escalate_human = 60 s (plus prior steps). Latency ≥ 60 s ≫ 3 s → SLA impossible for escalated queries (2 marks).
• But only 8% escalate. SLA requires 95% < 3 s. The 92% non-escalated already sit at ~3.1 s > 3 s, so even non-escalated fail! Note base trace 3.1 s > 3 s SLA. So SLA is violated for effectively all queries → must reduce reasoning steps or parallelize (1 mark for spotting 3.1 s > 3 s).
• Architectural fix (1 mark): async/background escalation returning an acknowledgment immediately; or caching KB lookups; or reducing reasoning steps by merging plan+interpret; or streaming partial answers so perceived latency drops below 3 s.

Question 2 (14 marks)

(a) Function spec + constraints (5 marks) — 3 for a valid schema, 2 for two constraints:

{
  "name": "transfer_funds",
  "parameters": {
    "type": "object",
    "properties": {
      "from_account": {"type":"string","pattern":"^ACC-[0-9]{3,}$"},
      "to_account":   {"type":"string","pattern":"^ACC-[0-9]{3,}$"},
      "amount":       {"type":"number","minimum":0.01,"maximum":10000},
      "currency":     {"type":"string","enum":["USD","EUR","GBP"]}
    },
    "required":["from_account","to_account","amount","currency"]
  }
}

Constraints: (i) amount must be > 0 and ≤ limit; (ii) currency restricted to enum; (iii) from ≠ to (custom rule).

(b) Failure modes + guardrail category (6 marks) — 2 marks each (mode + correct category):

1. Negative / oversized amount (hallucinated or manipulated value) → input validation.
2. Invalid/unsupported currency code → input validation (schema enum enforcement).
3. Unauthorized transfer (agent transfers from an account the user doesn't own / prompt injection) → action authorization (permission check + human confirmation). Also acceptable: leaking account numbers in the answer → output filtering.

(c) Reject the malicious call (3 marks)

• from="ACC-001", to="ACC-001" → rejected by the from ≠ to self-transfer rule (1).
• amount=-50 → rejected by minimum ≥ 0.01 constraint (1).
• currency="XYZ" → rejected by enum constraint (1).

Question 3 (12 marks)

(a) Dependency graph (5 marks)

• T1 fetch — prereq: none
• T2 clean — prereq: T1
• T3 chart1 — prereq: T2
• T4 chart2 — prereq: T2
• T5 chart3 — prereq: T2
• T6 summary — prereq: T3, T4, T5
• T7 email — prereq: T6

1 mark each for correct prereqs of fetch/clean, charts, summary, email (structure). Max 5.

(b) Parallelism & critical path (4 marks)

• Parallel: T3, T4, T5 (all three charts run concurrently after clean) (1 mark).
• Critical path: fetch(6) → clean(4) → max(chart)=3 → summary(5) → email(1)
• = 6 + 4 + 3 + 5 + 1 = 19 seconds (3 marks).

(c) Plan-then-execute vs interleaved (3 marks)

• Plan-then-execute advantage: known static pipeline → can schedule parallel charts, optimize resources, verify plan upfront (1.5).
• Interleaved (ReAct) advantage: adapts if a step fails (e.g., empty data after fetch) or if data shape is unknown until observed (1.5).

Question 4 (12 marks)

(a) Short vs long-term (4 marks)

• Short-term/working memory = current dialogue + intermediate results held in the context window; volatile, bounded by token limit (2).
• Long-term memory = persisted knowledge across sessions in an external store (vector DB, key-value, SQL); retrieved on demand into context (2).

(b) Retrieval budget (4 marks)

• Available for history = 8000 − 2000 reserved = 6000 tokens, minus room for the model's response (say ~1000). ≈ 5000 tokens of history can be injected (2 marks; accept 4000–6000 with reasoning).
• Strategy: embed the current query, do semantic (top-k) retrieval over chunked history, optionally combined with recency weighting / a running summary of older turns (2 marks).

(c) Failure modes (4 marks)

• Summarization compression: lossy — drops a specific detail (e.g., user's exact address) that becomes needed later; hallucinated summary. Scenario: user mentioned allergy 30 turns ago, summary omitted it (2).
• Embedding retrieval: retrieves semantically similar but wrong context, or misses info phrased differently from the query (lexical/semantic gap); no temporal ordering. Scenario: "my last order" fails when phrased as "the thing I bought Tuesday" (2).

Question 5 (10 marks)

(a) Pass rates (3 marks)

• A: 118/200 = 59% (1)
• B: 154/200 = 77% (1)
• Absolute improvement = 77 − 59 = 18 percentage points (1)

(b) Cost per successful task (4 marks)

• A: total cost = 200 × 0.05 = $10; successes = 118 →$10/118 = $0.0847/success (2)
• B: total cost = 200 × 0.14 = $28; successes = 154 →$28/154 = $0.1818/success (2)
• A is more cost-efficient per success (≈$0.085 vs$0.182).

(c) Additional metrics (3 marks) — any three, 1 each: latency/wall-clock time; number of tool calls / API errors; robustness to adversarial inputs; hallucination or invalid-action rate; human-intervention rate; reproducibility/variance across runs; code quality/security of output. Justify one, e.g., variance matters because a single high pass-rate run may not be reproducible.

[
  {"claim":"Q1 total latency = 3100 ms","code":"lat=3*800+500+200; result=(lat==3100)"},
  {"claim":"Q1 total cost = 0.035","code":"cost=3*0.01+0.004+0.001; result=abs(cost-0.035)<1e-9"},
  {"claim":"Q3 critical path = 19 s","code":"cp=6+4+3+5+1; result=(cp==19)"},
  {"claim":"Q5 pass rate improvement = 18 pts","code":"a=118/200*100; b=154/200*100; result=(b-a==18)"},
  {"claim":"Q5 cost per success: A cheaper than B","code":"A=(200*0.05)/118; B=(200*0.14)/154; result=(A<B)"}
]