Retrieval-Augmented Generation (RAG) for Agents

The problem RAG solves

Large language models have knowledge cutoffs. Claude's training data ends in early 2024. GPT-4's training ended in April 2024. This means:

• No proprietary information: Your company's internal policies, wikis, customer data, or product docs are not in the model's training data.
• No private knowledge: Personal notes, private documents, or internal discussions the model never saw.
• No recent information: Events after the training cutoff date.

A model alone cannot answer questions using this information, even if the information exists and is available. It will hallucinate or say "I don't know."

RAG (Retrieval-Augmented Generation), introduced by Lewis et al. in 2020, solves this by:

1. Retrieving relevant documents from a database at query time
2. Augmenting the model's prompt with those documents
3. Generating an answer based on both the model's training and the retrieved documents

This lets the model answer questions using information it never saw during training.

Walking through the pipeline

Step 1: Query A user or agent asks a question. This is the input that drives everything downstream.

User: "What is our return policy for software products?"

Step 2: Retrieve The system embeds the query into a vector (using an embedding model) and searches a vector database to find the most similar chunks of stored text.

query_embedding = embedding_model.encode("What is our return policy?")
retrieved_chunks = vector_db.search(query_embedding, top_k=5)

# Results might be:
# - Chunk 1: "Return Policy: Software purchases... 30-day return window..."
# - Chunk 2: "For SaaS products, returns are processed within 5 business days..."
# - Chunk 3: "Refund eligibility depends on..."

Step 3: Augment The retrieved chunks are inserted into the model's prompt alongside the original query. The model is instructed to base its answer on these chunks.

System prompt:
"Answer the user's question using ONLY the provided context.
If the context doesn't contain the answer, say so.
Do not use your training knowledge."

Context:
<insert retrieved chunks here>

User question: "What is our return policy for software products?"

Step 4: Generate The model generates an answer, instructed to stay within the bounds of the retrieved context.

Model answer:
"Based on our return policy, software purchases have a 30-day return window.
Refunds are processed within 5 business days of approval.
[model cites the specific policy chunk it drew from]"

The model is constrained: it can only draw from the retrieved chunks plus its general reasoning about how to apply them. It's not using general knowledge about return policies from its training; it's applying the specific policy you retrieved.

Why chunking is the quiet, decisive variable

Before RAG runs, source documents have to be split into chunks small enough to embed and retrieve individually. This is a design decision that often determines RAG quality more than the embedding model or vector database choice.

Chunk size tradeoff:

Too large (e.g., 2000 tokens):

• Pro: Retains context (related facts stay together)
• Con: Dilutes relevance (irrelevant sentences waste tokens); a single query might retrieve a huge chunk of which only 50 tokens are relevant; the model has to filter signal from noise

Too small (e.g., 100 tokens):

• Pro: High relevance (most retrieved text is actually useful)
• Con: Loses context (a sentence might not make sense without the paragraph before it); related facts get split apart; the model has to stitch them together

Most effective (e.g., 300-500 tokens):

• Balances context and relevance
• Can be tuned per document type

Chunking strategy varies by document type:

• Structured documents (code, APIs): Chunk by function or section; a Python function + its docstring is one chunk, not split mid-function.
• Legal documents: Chunk by clause or numbered section; an ambiguous clause needs its full context to be understood.
• Articles/blogs: Chunk by paragraph or subsection; splitting mid-paragraph creates confusion.
• Chat logs: Chunk by conversation turn or by semantic meaning, not arbitrary token boundaries.

Here's a practical chunking example:

def chunk_document(text, chunk_size=400, overlap=100):
    """Split a document into overlapping chunks."""
    chunks = []
    start = 0

    while start < len(text):
        # Take a chunk
        end = start + chunk_size
        chunk = text[start:end]

        # Include overlap for context
        if end < len(text):
            # For next iteration, back up to include overlap
            start = end - overlap
        else:
            start = end

        chunks.append(chunk)

    return chunks

# Example
policy_text = "Our return policy... [long text]"
chunks = chunk_document(policy_text, chunk_size=500, overlap=100)

# Each chunk is 500 tokens, with 100 tokens of overlap with the previous chunk.
# This keeps related information together while avoiding duplicates.

Common chunking failures:

Most production RAG systems tune chunk size and overlap specifically for their document type, treating this as a tuning problem like learning rate in a neural network.

RAG retrieval effectiveness comparison

Here's how different chunking and retrieval strategies compare in practice (illustrative estimates):

| Approach | Chunk size | Chunk overlap | Retrieval accuracy | False positives | Avg. tokens in top-5 | Best for | |---|---|---|---|---|---| | No chunking (full docs) | ~5000+ | N/A | 70-80% | 30-40% | 8,000+ | Small corpus (<100 docs) | | Large chunks | 1000-2000 | 200 | 75-85% | 25-35% | 5,000-8,000 | Broad questions, low specificity | | Standard chunks | 400-600 | 100 | 85-92% | 10-15% | 2,000-3,000 | Most use cases | | Small chunks | 200-300 | 50 | 80-88% | 5-10% | 1,000-1,500 | Precise queries, high specificity | | Hierarchical (chunk + parent context) | 300-500 + parent | Variable | 88-95% | 8-12% | 2,500-4,000 | Complex documents, context-dependent |

The "accuracy" column represents what percentage of retrieved chunks are actually relevant. False positives are chunks that seem relevant (high embedding similarity) but don't actually answer the query. Larger chunks have more false positives because they contain more unrelated content.

For RAG in agents specifically, standard chunks (400-600 tokens) are the sweet spot for most document types. Hierarchical chunking (chunk + parent context) is increasingly popular for long documents but requires more implementation complexity.

RAG inside an agent, not instead of one

In agent systems, RAG is best understood as one tool the agent can call, not a separate fixed architecture. An agent decides when it needs to retrieve, calls a retrieval tool, reads the result the same way it would read any other tool's observation, and continues its loop. That framing matters because it means an agent can:

• Combine RAG with other tools: Retrieve context, then call an API, then retrieve more context
• Make multiple retrieval passes: If initial retrieval doesn't work, retrieve again with a refined query
• Decide dynamically: Based on intermediate results, the agent decides whether it needs more retrieval or can proceed to generate an answer

Here's what an agent-based RAG flow looks like:

def rag_research_loop(user_query, vector_db, llm):
    """Agent loop that uses RAG as one tool among many."""
    context = []

    # Step 1: Initial retrieval
    retrieved = vector_db.search(user_query, top_k=5)
    context.append(f"Initial search results:\n{retrieved}")

    # Step 2: Agent decides if initial results are sufficient
    agent_decision = llm.decide(
        query=user_query,
        context=context,
        question="Do the retrieved documents answer the user's question?",
        options=["answer_now", "refine_search", "need_more_info"]
    )

    if agent_decision == "refine_search":
        # Step 3: Agent refines the query based on initial results
        refined_query = llm.generate(
            prompt=f"User asked: {user_query}. Initial results mentioned [X] but not [Y]. "
                   f"Write a refined search query to find [Y]."
        )
        # Re-retrieve with refined query
        retrieved = vector_db.search(refined_query, top_k=5)
        context.append(f"Refined search results:\n{retrieved}")

    # Step 4: Generate answer from accumulated context
    answer = llm.generate(
        prompt=f"Answer based on this context:\n{context}\n\nQuestion: {user_query}"
    )

    return answer

This is much more flexible than a fixed retrieve-then-generate pipeline. The agent can decide dynamically whether retrieval is working.

A worked example: an internal-policy agent

An HR-support agent gets asked: "How many vacation days do I carry over into next year if I have 12 days unused?"

Step 1: Initial retrieval Agent retrieves "carryover" from policy docs. Gets a paragraph: "Employees may carry over up to 5 unused vacation days into the next year."

Step 2: Agent checks sufficiency The retrieved chunk answers "how many" (up to 5). But the user asked about their specific situation (12 days unused). The chunk says "up to 5," so logically the answer is 5, but the user has 12, which is more. The agent should check if there are exceptions.

Step 3: Refined retrieval Agent decides to search again: "vacation carryover exceptions maximum unused days employee"

Gets additional chunks about exceptions:

• "Employees hired mid-year may carry over more"
• "Departments may grant exceptions with manager approval"

Step 4: Agent generates answer Using both the initial chunk and the exception chunks, the agent answers: "Standard carryover is 5 days. You have 12 unused, so 5 days would carry over under standard policy. Check with your manager about exceptions or mid-year hire status, which may allow more."

If the initial retrieval had been sufficient, the agent would have answered immediately in step 2. But because it detected ambiguity, it refined and retrieved again.

This is different from a fixed RAG pipeline, which would have stopped after step 1 and given a confident but possibly incomplete answer: "You can carry over 5 days."

Advanced RAG: multi-pass and re-ranking

Simple RAG retrieves once and generates. Better RAG systems use iterative retrieval:

Multi-pass RAG:

def multi_pass_rag(query, vector_db, llm):
    """RAG with multiple retrieval passes and re-ranking."""

    # Pass 1: Initial retrieval
    initial_results = vector_db.search(query, top_k=10)

    # Pass 2: Re-rank based on relevance
    reranked = llm.rerank(
        query=query,
        candidates=initial_results,
        # "Which of these are most relevant to the query?"
    )
    top_5 = reranked[:5]

    # Pass 3: Generate answer
    answer = llm.generate(
        context=top_5,
        query=query
    )

    # Pass 4: Check if answer has gaps
    gaps = llm.identify_gaps(answer, query)
    # "You answered X, but the question also asked about Y. Did you miss Y?"

    if gaps:
        # Pass 5: Targeted retrieval for gaps
        targeted_results = vector_db.search(gaps, top_k=5)
        # Re-generate with both initial and targeted results
        answer = llm.generate(
            context=top_5 + targeted_results,
            query=query
        )

    return answer

This is more expensive (2x tokens due to re-ranking and gap detection) but significantly improves answer completeness for complex questions.

Edge cases in RAG: when retrieval fails silently

Vocabulary mismatch: A company's docs use "refund" but users query "get money back." The embedding model might not bridge this gap perfectly. Solution: include synonym lists or train domain-specific embeddings.

Temporal information: Policies change over time. If docs aren't versioned, RAG might retrieve an outdated policy. If versioning is present, the retrieval might pick the wrong version (e.g., "return policy from 2023" when the user wants "current return policy"). Solution: include metadata (date, version, status) and filter by recency.

Context fragmentation: A complex topic is split across many chunks. The top-5 retrieved chunks cover fragments but no single chunk is a complete answer. The model has to synthesize, but synthesis can introduce errors. Solution: include a "hierarchy" in your chunks (chunk + parent section + document title) so related chunks are discoverable.

Here's a robust RAG pipeline with error handling:

def robust_rag_with_fallback(query, vector_db, text_db, llm):
    """RAG with fallback to full-text search if retrieval fails."""

    # Step 1: Vector search
    results = vector_db.search(query, top_k=5)

    # Step 2: Quality check
    if not results or results[0]['score'] < 0.6:
        # Low confidence; try full-text search
        results = text_db.keyword_search(query, top_k=5)
        retrieval_method = "keyword"
    else:
        retrieval_method = "vector"

    # Step 3: Generate answer
    answer = llm.generate(
        context=results,
        query=query,
        instruction="If retrieved documents don't answer the query, say so."
    )

    # Step 4: Verify answer references the documents
    answer_sources = llm.extract_citations(answer)
    if not answer_sources:
        # Model didn't cite anything; it might have hallucinated
        return {
            "answer": answer,
            "confidence": "low",
            "warning": "Answer is not grounded in retrieved documents"
        }

    return {
        "answer": answer,
        "confidence": "high",
        "retrieval_method": retrieval_method,
        "sources": answer_sources
    }

Case study: RAG deployment at a legal services firm

Scenario: A legal firm deployed RAG to help junior attorneys find relevant case law. The system indexed 10,000 legal opinions and was supposed to answer queries like "What's the precedent for contract breach in software licensing?"

Initial approach (naive RAG):

• Chunk all opinions into 1000-token chunks
• Retrieve top-5 for each query
• Generate answer

Problem: Retrieval accuracy was only ~60%. Junior attorneys still had to manually verify every recommendation. Root causes:

1. Legal language is specific: "breach of contract" in software law is different from "breach of contract" in construction. Generic embeddings couldn't distinguish.
2. Chunks were too large: a 1000-token chunk might contain 3 separate case holdings. The top-5 results included unrelated holdings.
3. No re-ranking: vector similarity alone isn't enough; a case from the same jurisdiction and era is more relevant than a case from 50 years ago in a different jurisdiction.

Improved approach:

1. Domain-specific embeddings: Trained an embedding model on legal opinions, improving relevance for law-specific language.
2. Hierarchical chunks: Top level is "case holding" (50-100 tokens), mid level is "full section" (300 tokens), leaf level is "full opinion." Retrieval searches at the holding level, then includes context.
3. Metadata filtering: Before retrieval, filter by jurisdiction and recency. "Software license breach in California after 2015."
4. Re-ranking: LLM re-ranks results, prioritizing cases with similar fact patterns.
5. Citation tracking: Extracted all case-to-case citations. If Query returns case A, also include cases that cite A (they might be more recent interpretations).

Results:

• Retrieval accuracy improved to 87%
• Juniors now trusted recommendations 80% of the time (up from 40%)
• Query latency increased (more complex pipeline) but was acceptable (<2 seconds per query)
• The system needed 8GB of embeddings storage (increased from 3GB due to more granular chunks), but this was acceptable

Common mistake

Treating a single retrieval pass as sufficient for any question. Multi-step or comparative questions ("how has this policy changed across three versions?") often need multiple, differently-worded retrieval calls. An agent that can decide to retrieve again with a refined query, rather than a fixed one-shot RAG pipeline, handles these cases far better.

Also common: providing retrieved chunks to the model without explicitly telling it "answer based ONLY on these chunks." Without that constraint, the model sometimes ignores the retrieved context and relies on its training knowledge instead, defeating the purpose of RAG.