6.2.7 · AI-ML › AI Agents & Tool Use

Overview

Agentic frameworks software libraries hain jo AI agents banane ke liye abstractions aur tools provide karte hain – aise systems jahan language models autonomously complex tasks accomplish karne ke liye reason, plan, tools use, aur external data sources ke saath interact kar sakte hain.

Do dominant frameworks hain:

• LangChain: LM calls ko tools, memory, aur external APIs ke saath chaining karne par focus
• LlamaIndex: Large document collections par indexing aur querying karne par focus

Agentic frameworks ye pre-built components (prompt templates, tool interfaces, memory stores, retrieval systems) deke solve karte hain jo ek saath snap ho jaate hain. 500 lines ka parsing logic likhne ki jagah, tum 20 lines mein framework configure karte ho.

Isko web frameworks (Flask, Django) vs raw HTTP sockets ki tarah socho. Tum could sab kuch scratch se bana sakte ho, lekin frameworks boilerplate eliminate karte hain aur best practices encode karte hain.

LangChain: Chains, Agents, aur Tools

Core Architecture: Agent Loop Derive Karna

Chaliye derive karte hain ki LangChain ka agent design aisa kyun dikhta hai.

Starting constraint: Ek LM sirf text output kar sakta hai. Ye directly functions call ya tools use nahi kar sakta.

Problem: Hum ise tool-use capability kaise dein?

Solution derivation:

1. Step 1: Structured output format

   • Hume LM ko "main tool X use karna chahta hun input Y ke saath" ko parseable tarike se express karna hai
   • Ek text format define karo: Action: tool_name\nAction Input: {"arg": "value"}
   • LLM ye format output karta hai jab wo tool use karne ka decide karta hai

2. Step 2: Parsing layer

   • Hume code chahiye jo ye format parse kare aur tool name + arguments extract kare
   • LangChain ka AgentOutputParser ye karta hai: parse(llm_output) → AgentAction

3. Step 3: Tool execution

   • Hume ek registry chahiye jo tool names ko actual Python functions se map kare
   • LangChain ka ToolRegistry karta hai: execute(tool_name, args) → tool_result

4. Step 4: Observation injection

   • Tool result LLM ke paas wapas jaana chahiye taaki wo reasoning continue kar sake
   • Hum append karte hain: Observation: <tool_result> prompt mein aur LM ko phir se call karte hain

5. Step 5: Loop until done

   • Steps 1-4 repeat karo jab tak LLM Final Answer: <response> output nahi karta

Ye hai ReAct (Reasoning + Acting) code mein formalize hoke.

Code mein:

prompt = initial_prompt
conversationhistory = []
 
while True:
    # LM decides next action
    llm_output = llm(prompt + format_history(conversation_history))
    # Parse what the LLM wants
    parsed = output_parser.parse(llm_output)
    
    if isinstance(parsed, AgentFinish):
        return parsed.output  # Done!
    
    # Execute the tool
    tool_result = tool_registry.run(parsed.tool, parsed.tool_input)
    
    # Add observation to history
    conversation_history.append({
        "action": parsed,
        "observation": tool_result
    })
    
    # Loop continues...

Ye design kyun?

• Minimal LM modifications: Hume LLM fine-tune ya architecture change karne ki zarurat nahi
• Extensibility: Naya tool add karna = Python function likhna + register karna
• Debugability: Hum loop ke har step ko inspect kar sakte hain

Key Components

Chains kyun? Kuch tasks ka ek known recipe hota hai. Jab steps pata hain toh agent reasoning ki zarurat nahi.

calculator = Tool( name="Calculator", func=lambda x: eval(x), # Unsafe! Real code uses safer parsing description="Useful for math. Input: a math expression like '2+'" )

**Description kyun?** LM isse decide karta hai ki ye tool *kab* call karna hai. Buri description = agent kabhi use nahi karta ya galat use karta hai.

> [!definition] Memory
> Ek ==state store== jo agent turns ke across persist karta hai. Types:
> - **ConversationBufferMemory**: Last N messages rakhna
> - **ConversationSummaryMemory**: LM purane messages summarize karta hai tokens bachane ke liye
> - **VectorStoreMemory**: Past messages embed karo, relevant ones retrieve karo

**Memory kyun?** Iske bina, agent conversation ke earlier parts bhool jaata hai. Limited context windows ke saath, hume smart retrieval chahiye.

### Example: Research Agent Banana

> [!example] LangChain Agent with Tools
> **Task**: Agent jo Wikipedia search kar sake aur math kar sake.

```python
from langchain.agents import initialize_agent, Tool, AgentType
from langchain.llms import OpenAI
from langchain.utilities import WikipediaAPIWrapper

# Define tools
wikipedia = WikipediaAPIWrapper()
tools = [
    Tool(
        name="Wikipedia",
        func=wikipedia.run,
        description="Search Wikipedia. Input: search query"
    ),
    Tool(
        name="Calculator",
        func=lambda x: str(eval(x)),
        description="Do math. Input: expression like '123 * 456'"
    )
]

# Initialize agent
llm = OpenAI(temperature=0)
agent = initialize_agent(
    tools=tools,
    llm=llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True
)

# Run
result = agent.run("What is the population of Tokyo? Multiply it by 2.")

Har step kyun?

1. Tool definitions: Hum agent ko batate hain ki wo kya kar sakta hai. description crucial hai – LLM isko padhta hai decide karne ke liye ki kab kaun sa tool use karna hai.

2. ZERO_SHOT_REACT_DESCRIPTION: Ye agent type ka matlab hai:

   • Zero-shot: Prompt mein koi example demonstrations nahi
   • ReAct: Reasoning + Acting pattern use karta hai
   • Description: Descriptions ke basis par tools decide karta hai

3. Agent execution (andar kya hota hai):

   LM: "I need to know Tokyo's population. I'll use Wikipedia."
   → Action: Wikipedia, Input: "Tokyo population"
   → Tool executes, returns: "~14 million"
   → Observation: "14 million"
   LM: "Now I'll multiply by 2."
   → Action: Calculator, Input: "14000 * 2"
   → Tool returns: "28000000"
   → Final Answer: "28 million"
   

verbose=True kyun? Taaki tum agent ki reasoning debug kar sako. Tum har Action/Observation pair dekhte ho.

LlamaIndex: Data Connectors aur Query Engines

LangChain se key difference: LlamaIndex data-centric hai (sahi context kaise laana hai), LangChain orchestration-centric hai (LM calls kaise chain karna hai).

Core Concept: Index

Problem: Tumhare paas 10,000 pages ki documentation hai. Ek LLM sirf ~8k tokens handle kar sakta hai. Tum questions kaise answer karte ho?

Naive solution: Docs chunk karo, chunks embed karo, top-k similar chunks search karo, prompt mein stuff karo.

Naive kyun?

• Chunks mein context ki kami ho sakti hai
• Top-k relevant info miss kar sakta hai jo multiple chunks mein spread hai
• Koi reasoning nahi ki kaun se chunks combine karne hain

LlamaIndex solution: Multiple index types with different tradeoffs.

Constraint 1: Context window limit $C$ tokens.
Constraint 2: Document corpus $D=\{d_1,..,d_n\}$ with total size $|D|>>C$.
Goal: Query $q$ given, subset $S\subset D$ select karo such that $|S|\le C$ aur $S$ answer quality maximize kare.

Different indices ise alag tarike se solve karte hain:

1. Vector Store Index (sabse common): $\text{similarity}(q,d_i)=\cos (\text{embed}(q),\text{embed}(d_i))$ Similarity se top-k select karo. Kyun? Semantic search fast hai aur paraphrases ke across kaam karta hai.

2. List Index (sequential): Sabhi chunks iterate karo, LLM se pucho "Kya ye relevant hai?" Kyun? Comprehensive hai lekin slow (N LM calls).

3. Tree Index (hierarchical): Ek tree banao jahan leaves chunks hain, nodes summaries hain. Root se start karo, relevant leaves tak traverse karo. Kyun? Logarithmic search time $O(\log n)$.

4. Keyword Table Index: Har chunk se keywords extract karo, mapping banao keyword → chunk_ids. Kyun? Fast exact-match retrieval.

Tradeoff table:

Example: LlamaIndex ke saath RAG

from llama_index import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms import OpenAI
 
# Step 1: Load documents
documents = SimpleDirectoryReader("./data").load_data()
# Why? Abstracts away PDF parsing, text extraction
 
# Step 2: Build index
index = VectorStoreIndex.from_documents(documents)
# What happens here?
# 1. Chunk documents (default: 1024 tokens with 20 token overlap)
# 2. Embed each chunk (default: OpenAI text-embedding-ada-002)
# 3. Store in vector DB (default: in-memory)
 
# Step 3: Create query engine
query_engine = index.as_query_engine()
# Why? Wraps retrieval + synthesis logic
 
# Step 4: Query
response = query_engine.query("What are the main arguments?")
print(response)

query() ke dauran kya hota hai? (Flow derive karna):

1. Query embed karo: $\mathbf{q}=\text{embed}(\text{"What are the main arguments?"})$

2. Cosine similarity se top-k chunks retrieve karo: $S=\text{top-k}(\{d_i\},\lambda d_i:\cos (\mathbf{q},\text{embed}(d_i)))$ Default k=2.

3. Prompt banao:

   Context:
   [Chunk 1 text]
   [Chunk 2 text]
   Question: What are the main arguments?
   Answer:
   

4. LM context se answer generate karta hai.

Ye kyun kaam karta hai? Chunks query ke semantically close hote hain, isliye unke andar answer hone ki likelihood zyada hoti hai.

Agentic Retrieval: Query Transformations

Problem: User poochta hai "X aur Y compare karo". Single retrieval X aur Y dono ke baare mein info fetch nahi kar sakta.

Solution: LlamaIndex ke query engines queries transform kar sakte hain:

Create sub-question engine

query_engine = SubQuestionQueryEngine.from_defaults( query_engine_tools=[ index1.as_query_engine(), # Technical docs index2.as_query_engine(), # User reviews ] )

Query: "How does product A compare to B in performance and price?"

What happens?

1. LM decomposes into sub-questions:

- "What is the performance of product A?"

- "What is the performance of product B?"

- "What is the price of product A?"

- "What is the price of product B?"

2. Routes each sub-question to appropriate index

3. Synthesizes final answer from all sub-answers

**Decomposition kyun?** Complex queries ko multiple retrieval steps chahiye. LLM reason karta hai ki *kya information chahiye*, phir retrieve karta hai.

---

## LangChain vs LlamaIndex: Kaun Sa Kab Use Karein?

> [!intuition] Mental Model: Orchestration vs Data

**LangChain** = "Mere paas ek LLM hai aur mujhe ise tools/APIs/logic se connect karna hai"
- Chatbots with tool use (search, calculator, API calls)
- Multi-step workflows (summarize → translate → email)
- Agents jo external actions ke baare mein reason karte hain

**LlamaIndex** = "Mere paas bahut saare documents hain aur mujhe unhe query karna hai"
- Internal docs, research papers, codebases par Q&A
- Context stitching ke saath semantic search
- Multi-document comparison aur synthesis

**Practice mein**: Inhe aksar saath use kiya jaata hai!
```python
# LlamaIndex query engine as a LangChain tool
from langchain.tools import Tool

doc_tool = Tool(
    name="DocumentSearch",
    func=lambda q: llamaindex_query_engine.query(q),
    description="Search company documentation"
)

# Now the LangChain agent can search docs!
agent = initialize_agent([doc_tool, calculator, ...], llm)

Common Mistakes

Ye sahi kyun lagta hai: Defaults usually kaam karte hain, aur tuning extra kaam hai.

Ye galat kyun hai:

• Bahut bade chunks: Relevance dilute hoti hai (retrieved chunks mein irrelevant text)
• Bahut chote chunks: Context toot jaata hai (chunks ke across connections miss hote hain)

Example: Legal documents mein long clauses hote hain. 1024 tokens ek clause ko mid-sentence split kar sakta hai, meaning kho jaata hai.

Fix:

• Chunk sizes experiment karo (512, 1024, 2048)
• Boundaries par context preserve karne ke liye overlap use karo (chunk size ka 20%)
• Apne specific data par retrieval quality measure karo

Code:

from llama_index import ServiceContext
from llama_index.text_splitter import SentenceSplitter
 
# Custom chunk size
text_splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)
service_context = ServiceContext.from_defaults(text_splitter=text_splitter)
index = VectorStoreIndex.from_documents(docs, service_context=service_context)

Ye sahi kyun lagta hai: Ye literally ek search tool hai, aur kya kehna hai?

Ye galat kyun hai: LLM descriptions use karta hai decide karne ke liye ki tool kab use karna hai. Vague description ka matlab hai:

• Agent kabhi use nahi karta (samajh nahi aata kab useful hai)
• Agent galat use karta hai (wrong input format)

Example:

# Bad
Tool(name="Search", func=search, description="A search tool")
 
# Good  
Tool(
    name="Search",
    func=search,
    description="Search Wikipedia for factual information about people, places, events. Input should be a specific search query like 'Barack Obama presidency' not a question."
)

Fix: Include karo:

1. Kya karta hai: "Search Wikipedia for factual information"
2. Kab use karna hai: "for people, places, events"
3. Input format: "specific search query like 'X' not a question"

Ye sahi kyun lagta hai: LMs smart hote hain, ye errors se recover kar sakte hain.

Ye galat kyun hai:

• Tool failures (API timeouts, parsing errors) agent crash kar dete hain
• Agent baar baar same broken action retry karta rehta hai
• Har retry ke saath costs accumulate hoti hain

Example: Agent Calculator ko invalid input "what is 2+2" (expression ki jagah words) ke saath call karta hai. Handling ke bina:

Action: Calculator
Action Input: "what is 2+2"
→ Error: invalid syntax
→ Agent retries same input
→ Error: invalid syntax
→ Loop forever (or hit max iterations)

Fix:

from langchain.agents import initialize_agent
 
agent = initialize_agent(
    tools=tools,
    llm=llm,
    max_iterations=5,  # Prevent infinite loops
    max_execution_time=60,  # Timeout after 60s
    handle_parsing_errors=True,  # Catch tool parsing errors
    early_stopping_method="generate"  # Return partial answer on timeout
)

Aur bhi better: Tools ko try-except se wrap karo:

def safe_calculator(expr):
    try:
        return str(eval(expr))
    except Exception as e:
        return f"Error: {e}. Please provide a valid math expression."

Ab agent ko feedback milta hai aur wo apna input correct kar sakta hai.

Advanced: Scratch se Custom Agents

Kyun? Frameworks opinionated hote hain. Kabhi kabhi tumhe custom logic chahiye (specialized prompting, novel tool-use patterns, proprietary systems ke saath integration).

import openai
import re
 
# Define tools
TOOLS = {
    "calculator": lambda x: eval(x),
    "search": lambda x: f"Search result for {x}"  # Stub
}
 
# ReAct prompt template
REACT_PROMPT = """
You can use these tools: {tools}
Format: 
Thought: <your reasoning>
Action: <tool_name>
Action Input: <input>
Observation: <result>
... (repeat Thought/Action/Observation as needed)
Thought: I now know the answer
Final Answer: <answer>
 
Question: {question}
"""
 
def parse_action(text):
    """Extract Action and Action Input from LM output."""
    action_match = re.search(r"Action: (\w+)", text)
    input_match = re.search(r"Action Input: (.+)", text)
    
    if action_match and input_match:
        return action_match.group(1), input_match.group(1).strip()
    return None, None
 
def agent_loop(question, max_steps=5):
    """Run the ReAct loop."""
    prompt = REACT_PROMPT.format(
        tools=", ".join(TOOLS.keys()),
        question=question
    )
    
    for step in range(max_steps):
        # Call LM
        response = openai.ChatCompletion.create(
            model="gpt-4",
            messages=[{"role": "user", "content": prompt}]
        )
        output = response.choices[0].message.content
        print(f"Step {step}:\n{output}\n")
        
        # Check if done
        if "Final Answer:" in output:
            answer = output.split("Final Answer:")[1].strip()
            return answer
        
        # Parse action
        tool_name, tool_input = parse_action(output)
        if not tool_name:
            return "Error: Could not parse action"
        
        # Execute tool
        if tool_name in TOOLS:
            result = TOOLS[tool_name](tool_input)
            prompt += f"\n{output}\nObservation: {result}\n"
        else:
            prompt += f"\n{output}\nObservation: Tool '{tool_name}' not found\n"
    
    return "Max steps reached"
 
# Test
answer = agent_loop("What is 123 * 456?")
print(f"Final: {answer}")

Ise build karke humne kya seekha:

1. Prompt template scaffolding hai – ye LM ko format sikhata hai
2. Parsing brittle hai – regex fail ho sakta hai agar LM format exactly follow nahi karta (frameworks isko retry logic se better handle karte hain)
3. Tool execution sirf ek dict lookup + function call hai
4. Observations ko prompt mein append karna feedback loop create karta hai

Phir frameworks kyun use karein? Wo add karte hain:

• Robust parsing (LLM output mein variations handle karna)
• Streaming (agent run hone par progress dikhana)
• Memory management (lambi conversations summarize/truncate karna)
• Logging aur debugging
• Pre-built integrations (vector DBs, APIs)

Key Formulas aur Concepts

jahan $a_t=\text{LM}(\text{prompt}+{\text{history}}_t)$.

Translation: Agent ka state (conversation history) tool results ke basis par update hota rehta hai jab tak wo final answer produce nahi karta.

jahan $\text{retrieve}(D,q)=\text{top-k}(D,\text{similarity}(q,\cdot ))$.

Interpretation: Sirf LLM ki parametric knowledge se generate karne ki jagah, hum retrieved documents par condition karte hain.

Connections

• 6.2.01-Definition-and-characteristics-of-AI-agents – Frameworks perceive-reason-act cycle implement karte hain
• 6.2.04-ReAct-(Reasoning-and-Acting) – LangChain ke agent types ReAct pattern implement karte hain
• 6.2.05-Tool-calling-and-function-calling-APIs – Frameworks function calling APIs ko easy integration ke liye wrap karte hain
• 5.3.02-Vector-databases – LlamaIndex semantic search ke liye vector DBs use karta hai
• 5.3.03-Retrieval-Augmented-Generation-(RAG) – LlamaIndex essentially ek RAG framework hai

LangChain aur LlamaIndex aise hain jaise tumhare dost ko superpowers de rahe ho:

LangChain = Apne dost ko ek toolbox dena. Ab wo kar sakta hai:

• Google use karo (search tool)
• Calculator use karo (math tool)
• Tumhari conversation ke earlier parts yaad rakho (memory)

Wo figure out karta hai kab kaun sa tool use karna hai descriptions padhke. Agar tum pucho "25 times 30 kya hai?", wo sochta hai "Mujhe calculator tool chahiye" aur use karta hai.

LlamaIndex = Apne dost ko ek magical library dena. Tumhare paas 100 textbooks hain lekin wo ek baar mein sirf 5 pages padh sakta hai (attention span!). LlamaIndex:

• Saari books ka ek map banata hai (index)
• Jab tum question poochte ho, 5 sabse useful pages dhundh leta hai (retrieval)
• Answer dene ke liye sirf wahi pages padhta hai (synthesis)

Saath mein, wo ek smart-but-limited dost ko ek superhero assistant mein badal dete hain jo search, calculate, remember, aur huge libraries padh sakta hai!

LLAMAIndex = Load Lots And Map All (data-focused)

Flashcards

#flashcards/ai-ml

Agentic framework kya hai? :: Ek library jo AI agents banane ke liye abstractions (chains, agents, tools, memory, retrievers) provide karti hai bina common patterns jaise prompt templating, tool parsing, aur conversation management reimplementing ke.

LlamaIndex mein query decomposition kya hai? :: Ek complex query ko sub-questions mein todna, har ke liye answers retrieve karna, phir ek final answer synthesize karna. Comparison queries ("X aur Y compare karo") ya multi-part questions ke liye useful hai.

Concept Map