Multi-Hop and Agentic Retrieval

When a single retrieval pass isn't enough

Most RAG systems retrieve once, then generate. This works when the question is answerable from a single neighborhood of the corpus. But some questions require combining facts from two or more disconnected places in your documents. Examples:

• "What is the impact of the Q3 marketing campaign on the sales team that shipped feature X?" — Requires finding the campaign details, then finding the Q3 sales team, then connecting the two.
• "Are there any open bugs in the code module that was refactored last month?" — Requires finding the refactoring PR, identifying the module, then searching for bugs in that specific module.
• "Did the person who approved the contract also approve the payment?" — Requires finding who approved the contract, then checking if that same person appears in payment records.

A single embedding search often doesn't surface all the necessary pieces because they may use completely different vocabulary. The embedding model has no way to know that "campaign impact" and "sales performance" are related to the question you asked.

Multi-hop retrieval solves this by turning a question into a sequence of smaller sub-questions, each of which can be answered by a single retrieval pass. The results from one hop become context for the next.

A typical multi-hop loop

Here's the core logic of a multi-hop system:

def multi_hop_retrieval(
    initial_question: str,
    vector_db,
    llm,
    max_hops: int = 3,
    context_limit_tokens: int = 8000
) -> dict:
    """
    Retrieve, reason about what's missing, decide if another hop is needed.
    Returns accumulated context and the final answer.
    """

    accumulated_context = []
    current_question = initial_question
    hop_count = 0

    while hop_count < max_hops:
        # Step 1: Retrieve for the current question/sub-question
        results = vector_db.search(current_question, top_k=5)

        # Store the raw chunks for the final answer
        accumulated_context.append({
            "hop": hop_count + 1,
            "subquestion": current_question,
            "retrieved_chunks": results
        })

        # Step 2: Ask the model: "Do we have enough to answer the original question?"
        reasoning_prompt = f"""
Original question: {initial_question}

Retrieved so far:
{format_chunks_for_display(accumulated_context)}

Can you answer the original question with what we have so far?
If not, what are we still missing?

Respond with:
- YES if you have enough
- NO and a single follow-up subquestion if more information is needed
"""

        reasoning = llm.generate(reasoning_prompt)

        # Step 3: Check if the model says we have enough
        if "YES" in reasoning:
            # Sufficient context gathered; break the loop
            break

        # Step 4: Extract the follow-up subquestion
        # (In production, use structured extraction, not string parsing)
        subquestion = reasoning.split("follow-up subquestion:")[-1].strip()
        current_question = subquestion

        # Safety check: stop if we're running out of context window
        total_tokens = sum(chunk.get("token_count", 100) for ctx in accumulated_context for chunk in ctx["retrieved_chunks"])
        if total_tokens > context_limit_tokens:
            break

        hop_count += 1

    # Step 5: Generate the final answer using all accumulated context
    answer_prompt = f"""
Question: {initial_question}

Here is all the information we found across {hop_count + 1} retrieval passes:
{format_accumulated_context(accumulated_context)}

Synthesize a complete answer. If the information is contradictory, note the contradiction.
"""

    final_answer = llm.generate(answer_prompt)

    return {
        "answer": final_answer,
        "hops": hop_count + 1,
        "context_chunks_total": sum(len(ctx["retrieved_chunks"]) for ctx in accumulated_context),
        "context": accumulated_context
    }

def format_chunks_for_display(accumulated_context: list) -> str:
    """Format all accumulated chunks for the LLM's reasoning step."""
    output = []
    for ctx in accumulated_context:
        output.append(f"\nHop {ctx['hop']}: {ctx['subquestion']}")
        for i, chunk in enumerate(ctx['retrieved_chunks'], 1):
            output.append(f"  Result {i}: {chunk['text'][:200]}...")
    return "\n".join(output)

This loop embodies the three core decisions of multi-hop retrieval:

1. Retrieve for the current sub-question — Cast as wide a net as the embedding model allows.
2. Reason about what's missing — Explicitly ask the model: "Is this enough?"
3. Decide to loop or stop — If not enough, decompose a follow-up question; if enough, synthesize an answer.

Decomposing questions into sub-questions

The quality of multi-hop retrieval depends entirely on how well you decompose the original question. A poor decomposition wastes retrieval passes and compounds errors.

Good decomposition is explicit, not implicit:

Weak: "What happened with the sales data?"

• This is so vague the model has no way to know what sub-questions to ask.

Better:

1. "What sales targets were set in Q3?"
2. "What actual sales were recorded in Q3?"
3. "What is the variance between target and actual?"

Good decomposition follows information dependencies:

If hop 1 depends on hop 2's result, retrieve in the right order.

Weak order:

1. Retrieve "variance analysis"
2. Retrieve "sales data" (too late to inform hop 1)

Better:

1. Retrieve "sales targets Q3" → get the baseline
2. Retrieve "Q3 actual sales" → compare to baseline
3. Retrieve "variance explanations" → explain the gap

Good decomposition avoids premature narrowing:

Weak:

1. Retrieve "bug in the payment module"

• If the bug is actually in the billing module, this fails immediately.

Better:

1. Retrieve "open bugs in systems that touch payments"
2. Narrow down by component based on results
3. Retrieve specific bug details

Error compounding across hops

Every retrieval pass introduces retrieval error. In multi-hop systems, errors compound.

Consider a two-hop question: "Did the team that shipped feature X also own the Q2 incident?"

• Hop 1 error rate: You search for "feature X" and get the wrong team 20% of the time (80% precision).
• Hop 2 error rate: You search for "incident Q2" using hop 1's result. If hop 1 was wrong, hop 2's search is working from bad context — likely even more error-prone.
• Overall accuracy: 80% × 80% = 64% in the best case, often worse because hop 2 compounds hop 1's mistakes.

A three-hop system with 80% accuracy per hop drops to 51% overall. A five-hop system reaches 33%.

Ways to mitigate error compounding:

1. Keep a "confidence score" at each hop. If hop 1 returns results with low confidence, either expand the search or escalate to a human rather than proceeding to hop 2.

# Simplified confidence scoring
confidence = results[0]['similarity_score']  # 0 to 1
if confidence < 0.6:
    return {
        "answer": "Low confidence in retrieval. Escalating to human review.",
        "hops": hop_count + 1,
        "confidence": confidence
    }

2. Run parallel retrieval paths instead of sequential. If you're unsure about hop 1's result, retrieve for both possibilities and proceed in parallel.

3. Verify hop transitions with the model. Before hop 2, ask: "Based on hop 1, is our understanding of X correct?" and let the model refine it.

4. Cap the number of hops. Beyond 3-4 hops, compounding error usually dominates any benefit from additional information.

Cost and latency tradeoffs

Multi-hop retrieval is expensive. Every hop costs tokens (retrieval + reasoning), API calls (vector search), and latency.

Cost per hop (illustrative estimate):

• Embedding query: ~$0.00001-0.0001 per call (mostly negligible)
• LLM call for reasoning: ~$0.001-0.005 per hop (depends on model size and context length)
• Retrieval from vector DB: ~$0.0001-0.001 per call (if you're paying per query; often included in subscription)

Total cost for a 3-hop retrieval:

• Single-pass RAG: $0.002-0.01 (one embedding call, one LLM call)
• 3-hop RAG: $0.006-0.05 (three embedding calls, three LLM "reasoning" calls, one final generation call)

That's 3-5x the cost for a 3-hop system. In production, at scale, this adds up quickly.

Latency impact:

• Single-pass RAG: ~500-1000 ms (one retrieval, one generation)
• 3-hop RAG: ~1500-3000 ms (three sequential retrievals + three reasoning calls + one final generation)

For interactive applications (chat, search), the jump from 500 ms to 3 seconds is noticeable. For batch processing, it's acceptable but multiplies the overall time to process a large queue.

When multi-hop is worth the cost:

• High-value questions (financial decisions, medical diagnosis, legal analysis) where getting the right answer justifies the extra cost.
• Offline analysis where latency isn't critical but accuracy is.
• Questions that genuinely require combining facts from disconnected parts of your corpus.

When to stick with single-hop:

• High-volume, low-value queries (customer support, FAQ search) where cost and latency matter more than perfect accuracy.
• Questions that a simple retrieval can answer adequately.
• Early-stage products where you're optimizing for simplicity and speed to market, not perfection.

Controlling runaway multi-hop: cost and complexity budgets

Without guard rails, a naive multi-hop system can loop endlessly or hit token limits mid-reasoning.

Implement hard limits:

def multi_hop_with_budgets(
    initial_question: str,
    vector_db,
    llm,
    max_hops: int = 3,
    max_total_tokens: int = 10000,
    max_cost_cents: float = 5.0  # Stop if we exceed 5 cents per query
) -> dict:
    """Multi-hop with cost and token budgets."""

    total_tokens_used = 0
    total_cost_cents = 0.0
    hop_count = 0
    accumulated_context = []
    current_question = initial_question

    while hop_count < max_hops:
        # Estimate cost before retrieving
        estimated_cost_this_hop = 0.3  # cents (empirical, tune for your setup)
        if total_cost_cents + estimated_cost_this_hop > max_cost_cents:
            break  # Over budget

        # Retrieve
        results = vector_db.search(current_question, top_k=5)
        total_tokens_used += sum(r.get('token_count', 100) for r in results)
        total_cost_cents += estimated_cost_this_hop

        if total_tokens_used > max_total_tokens:
            # We've accumulated too much context; stop
            break

        # Reason about next hop (simplified)
        reasoning = llm.generate(...)
        total_tokens_used += 500  # Approximate for reasoning call

        if should_stop(reasoning):
            break

        hop_count += 1

    # Generate final answer
    return { "answer": ..., "hops": hop_count, "cost_cents": total_cost_cents }

A good heuristic: Stop at the first hop that doesn't significantly improve confidence. If hop 2's results add only 5% more relevant information, don't bother with hop 3.

Practice: A real two-hop example

Scenario: You're building a support chatbot for a SaaS company. A user asks: "Can I export data from the mobile app, and if so, in what formats?"

This is genuinely two-hop:

• Hop 1: "Does the mobile app support data export?"
• Hop 2: (If yes) "What formats does the mobile app export support?"

Hop 1 retrieval:

Query: "mobile app data export feature"
Results:
- "The mobile app v3.2+ includes an export feature for notes, tasks, and calendar events."
- "Desktop and web support CSV, JSON, and XLSX exports."
- "See mobile app docs for platform-specific limitations."

Model reasoning: "Yes, the mobile app does support export. But we don't know the formats yet."

Hop 2 retrieval:

Query: "mobile app export formats CSV JSON XLSX Android iOS"
Results:
- "Mobile app export: Notes and tasks export as CSV only. Calendar export is JSON."
- "Limitations: Large datasets (>10k items) may timeout on mobile. Use desktop for full export."

Model reasoning: "Now we have enough."

Final answer synthesis:

"Yes, the mobile app supports exporting notes, tasks, and calendar events.
Formats are limited compared to the desktop app: notes and tasks export as CSV,
calendar as JSON. For large datasets, we recommend using the desktop or web app
to export as XLSX. See [link] for more details."

Compare this to a single-pass RAG with query "mobile app export formats":

• It might miss the fact that export exists at all (if the first few results are about desktop export).
• Or it might mention formats that only work on desktop.

The two-hop approach is more reliable here, even though it costs 3-5x more.

When to stop multi-hop early

A pragmatic heuristic for production systems: stop at the first hop that doesn't improve decision-making confidence. You can estimate this by asking the model itself: "Given the information retrieved so far, how confident are you in answering this question on a scale of 0-10?"

If confidence is ≥ 7, stop. If it's < 7, retrieve once more. If after hop 2 it only improves to 7.2, don't bother with hop 3.

This simple check prevents runaway retrieval loops while still capturing the multi-hop benefit for genuinely multi-step questions. It also pairs well with cost budgets: if a hop adds 2 cents of cost but only 0.1 confidence, skip it.

Common mistake

Attempting multi-hop when single-hop + better prompt engineering would work. Before building a multi-hop system, try:

1. A better chunking strategy. Maybe the question isn't truly multi-hop; maybe the chunks are just too fragmented.
2. Query rewriting / expansion. Synonyms and related terms added to a single query often surface the same information as two hops.
3. Hybrid search (keyword + vector). Some questions are really asking for exact matches, not semantic similarity.

Multi-hop is powerful but expensive. Only use it when you've exhausted simpler alternatives and your analysis shows the added accuracy justifies the cost.