GraphRAG and Knowledge-Graph-Augmented Retrieval

The limits of chunk-to-chunk retrieval

In a standard RAG system, you search for chunks similar to the query and return the top-5. This works well for instance-level questions that can be answered from a single neighborhood of text:

• "What is our return policy?" → Find the return policy chunk. Done.
• "Who is the product manager for the dashboard team?" → Find an org chart or mention of the PM. Done.

But it struggles with relationship and synthesis questions that require pulling facts from multiple disconnected corners of your corpus:

• "What is the organizational impact of the mobile app team's decision to deprecate the old API?" — You need to find (1) the mobile app team, (2) the deprecation decision, (3) the downstream teams that depend on the old API, (4) the expected impact on those teams. A single chunk might contain one of these, but rarely more than one or two.

• "Summarize the incidents this team has owned in the past year and the most common root cause." — You need to collect all incidents where this team is listed as the owner, extract their root causes, and synthesize patterns. Chunk-to-chunk retrieval returns "incident A" and "incident B," but you still have to manually sift through.

• "What are the dependencies between our backend services?" — You need to traverse a web of service-to-service calls. A single chunk describes one service; finding all its dependencies requires multiple retrieval passes or a graph traversal.

Graphs solve this by making relationships explicit and traversable.

Building a graph: entities and relationships

A knowledge graph represents your corpus as a network of entities (nodes) and relationships (edges).

Simple example:

Entities:
- Mobile App Team (team)
- Old API (service)
- Dashboard Team (team)
- Payment Service (service)

Relationships:
- Mobile App Team -> OWNS -> Old API
- Dashboard Team -> DEPENDS_ON -> Old API
- Payment Service -> DEPENDS_ON -> Old API
- Mobile App Team -> DEPRECATES -> Old API

When you query "What is the impact of deprecating the Old API?", the graph reveals:

1. Start at "Old API"
2. Traverse outgoing edges: "DEPRECATES" → Mobile App Team (who owns/deprecates it)
3. Traverse incoming edges: "DEPENDS_ON" ← Dashboard Team, Payment Service (who will be affected)
4. Aggregate: "The deprecation affects Dashboard and Payment teams."

No single chunk contains all this; the graph synthesizes it.

Building a graph from text (sketch):

import json

def extract_entities_and_relations(text: str, llm) -> dict:
    """
    Use an LLM to extract named entities and their relationships.
    This is a simplified sketch; production systems are more sophisticated.
    """
    prompt = f"""
Extract all entities (people, teams, services, features, etc.) and relationships
from this text. Format as JSON.

Text: "{text}"

Respond in this format:
{{
  "entities": [
    {{"name": "...", "type": "team|service|person|feature"}},
    ...
  ],
  "relationships": [
    {{"source": "...", "relation_type": "OWNS|DEPENDS_ON|DEPRECATES|..."edge", "target": "..."}},
    ...
  ]
}}
"""

    result_json = llm.generate(prompt)
    return json.loads(result_json)

# Example
text = """
The Mobile App Team decided to deprecate the old API by Q4 2026.
The Dashboard Team and Payment Service both depend on this API.
The Payment Service team has already begun migration.
"""

graph_data = extract_entities_and_relations(text, llm)
# Returns:
# {
#   "entities": [
#     {"name": "Mobile App Team", "type": "team"},
#     {"name": "old API", "type": "service"},
#     {"name": "Dashboard Team", "type": "team"},
#     {"name": "Payment Service", "type": "service"}
#   ],
#   "relationships": [
#     {"source": "Mobile App Team", "relation_type": "DEPRECATES", "target": "old API"},
#     {"source": "Dashboard Team", "relation_type": "DEPENDS_ON", "target": "old API"},
#     {"source": "Payment Service", "relation_type": "DEPENDS_ON", "target": "old API"}
#   ]
# }

In production, you'd use a dedicated NER (Named Entity Recognition) model or a more structured extraction approach, but the idea is the same: convert document text into graph nodes and edges.

Instance-level retrieval: one entity, one hop

The simplest graph retrieval is: find the entity mentioned in the query, return its 1-hop neighborhood.

Query: "What does the Mobile App Team own?"

def graph_instance_retrieval(query: str, graph, llm, hops: int = 1) -> dict:
    """
    1. Find the entity in the query.
    2. Traverse the graph for N hops.
    3. Return the neighborhood.
    """

    # Step 1: Extract the main entity from the query
    entity_prompt = f"""
Extract the main entity (person, team, service, etc.) from this question:
"{query}"

Respond with just the entity name.
"""

    entity_name = llm.generate(entity_prompt).strip()

    # Step 2: Look up the entity in the graph
    entity = graph.find_entity(entity_name)

    if not entity:
        return {"error": f"Entity '{entity_name}' not found in graph"}

    # Step 3: Traverse edges for N hops
    neighborhood = graph.traverse(entity, hops=hops)

    # Step 4: Return the neighborhood (edges and connected entities)
    return {
        "entity": entity_name,
        "neighbors": [
            {
                "target_entity": rel["target"],
                "relationship": rel["relation_type"],
                "evidence": rel["source_chunk"]  # The original chunk where this relationship was found
            }
            for rel in neighborhood
        ]
    }

# Example
result = graph_instance_retrieval("What does the Mobile App Team own?", graph, llm)
# Returns:
# {
#   "entity": "Mobile App Team",
#   "neighbors": [
#     {
#       "target_entity": "old API",
#       "relationship": "DEPRECATES",
#       "evidence": "The Mobile App Team decided to deprecate..."
#     },
#     ...
#   ]
# }

Advantage: Fast and precise. No embedding search needed; just graph traversal.

Disadvantage: Only works if the entity is explicitly named in the query. If the query is "What are we shutting down?", you'd need the LLM to infer "old API" first, which adds a step.

Community-level retrieval: summarizing clusters

For global questions ("What are the biggest operational risks across all teams?"), traversing one entity's 1-hop neighborhood isn't enough. You need to see clusters of entities and synthesize across them.

Microsoft's GraphRAG approach (2024) uses community detection to group densely-connected entities, then pre-computes a summary for each community.

Example structure:

Community 1: "Backend Services & Dependencies"
  Entities: Payment Service, Dashboard Team, Notification Service, Database
  Relationships: 5 DEPENDS_ON edges, 2 OPERATES edges
  Pre-computed summary: "The Payment Service and Dashboard depend heavily
                         on the Database and Notification Service. Single
                         points of failure: Database (used by all 3),
                         Notification Service (used by 2)."

Community 2: "Mobile Development & Deprecation"
  Entities: Mobile App Team, old API, new API, Mobile Developers
  Relationships: 4 DEPRECATES edges, 3 MIGRATES edges
  Pre-computed summary: "The Mobile App Team is deprecating the old API
                         by Q4 2026. Mobile Developers are migrating to
                         the new API. Expected completion: 6 weeks."

Retrieval flow:

def graph_global_retrieval(query: str, communities: list, llm) -> dict:
    """
    For a global question, find relevant communities, return their summaries.
    """

    # Step 1: Which communities are relevant to this query?
    relevance_scores = []

    for community in communities:
        prompt = f"""
Does this community summary contain information relevant to the question?
Community: {community['summary']}
Question: "{query}"

Respond with YES or NO.
"""

        is_relevant = "YES" in llm.generate(prompt)
        if is_relevant:
            relevance_scores.append(community)

    # Step 2: Return the top-K most relevant community summaries
    return {
        "query": query,
        "relevant_communities": relevance_scores[:3],  # Top 3
        "community_summaries": [c["summary"] for c in relevance_scores[:3]]
    }

# Example
result = graph_global_retrieval(
    "What are the biggest operational risks across all teams?",
    communities,
    llm
)
# Returns: "High risk: Database is used by all backend services (single
#           point of failure). Medium risk: old API deprecation timeline
#           is aggressive and could break downstream integrations."

Advantage: Scales to large corpora. Instead of retrieving 50 chunks, you retrieve 2-3 community summaries. Summarization is already done, no need to synthesize on the fly.

Disadvantage: Requires pre-computing communities and summaries. If your corpus changes frequently, this is expensive to maintain.

Cost and complexity: when GraphRAG is worth it

GraphRAG is a multi-phase investment:

1. Extraction phase: NER on every document to find entities and relationships. ~$0.001-0.01 per document (depends on size and model).
2. Graph construction: Merging entities, deduplicating, resolving aliases (e.g., "Mobile App Team" vs "Mobile Team"). Manual + automated. Weeks of engineering.
3. Community detection: Running clustering algorithms (e.g., Leiden algorithm). One-time or periodic. Minutes to hours depending on graph size.
4. Summarization phase: Generating summaries for each community. ~$0.01-0.05 per community. One-time, then cached.

Comparison: Plain RAG vs GraphRAG

| Dimension | Plain RAG | GraphRAG | |---|---|---| | Setup cost | Low (chunk + embed) | High (extract, deduplicate, cluster, summarize) | | Retrieval latency | ~200-500ms (vector search) | ~100-300ms (graph traversal or index lookup) | | Query latency | ~800ms-2s (retrieve + generate) | ~1-2s (retrieve communities + generate) | | Accuracy on instance queries | 80-90% (embedding similarity) | 95%+ (exact entity + graph) | | Accuracy on global queries | 40-60% (requires synthesizing many chunks) | 75-85% (community summary already synthesized) | | Handling new documents | Minutes (re-embed) | Hours (re-extract, re-cluster) | | Storage | Embeddings only (~1 GB per 100k chunks) | Graph + embeddings + summaries (~2-3 GB per 100k entities) |

When to use GraphRAG:

• Large corpus (100k+ documents) with complex relationships (org charts, service dependencies, incident ownership).
• Global questions are common ("Summarize all incidents in Q3", "What are our dependencies?").
• High-value use case (financial, operational, legal) where accuracy justifies the setup cost.
• Corpus is relatively stable (doesn't change daily).

When to stick with plain RAG:

• Small corpus (< 10k documents), where a good vector search is fast enough.
• Instance queries dominate ("How do we calculate tax?", "What's the return policy?").
• Corpus changes frequently (e.g., real-time chat logs, live product docs).
• Early-stage product where speed and simplicity matter more than accuracy.

Hybrid approach: Graph + semantic search

Many production systems use both. The graph captures relationships and structure, while embeddings capture semantic similarity.

def hybrid_graph_semantic_retrieval(query: str, graph, vector_db, llm) -> dict:
    """
    1. Try graph-based retrieval first (fast, exact).
    2. If graph doesn't have the entity, fall back to semantic search.
    3. Combine results.
    """

    # Step 1: Extract entities from the query
    entities = extract_entities_from_query(query, llm)

    if entities:
        # Step 2a: Graph-based retrieval for known entities
        graph_results = []
        for entity_name in entities:
            entity = graph.find_entity(entity_name)
            if entity:
                neighborhood = graph.traverse(entity, hops=1)
                graph_results.extend(neighborhood)

        if graph_results:
            return {
                "method": "graph",
                "results": graph_results,
                "explanation": f"Found {len(graph_results)} related entities via graph traversal."
            }

    # Step 2b: Fall back to semantic search
    semantic_results = vector_db.search(query, top_k=5)

    return {
        "method": "semantic",
        "results": semantic_results,
        "explanation": "Graph retrieval not applicable; using semantic search."
    }

Practice: Building a small knowledge graph for a team

Scenario: You manage a small engineering organization with 4 teams and 8 services. You want to build a simple knowledge graph to answer operational questions.

Step 1: Define entity types and relationships

Entity types:

• Team
• Service
• Document (incident reports, RFCs, etc.)

Relationship types:

• OWNS (Team owns Service)
• DEPENDS_ON (Service depends on Service)
• AUTHORED (Team authors Document)
• REFERENCES (Document references Service)

Step 2: Extract from documents

For each document (incident report, RFC, architecture doc), run NER:

Document: "Incident Report: Q2 Database Outage"
Entities: Database Service, Backend Team, Frontend Team, Payment Service
Relationships:
- Backend Team OWNS Database Service
- Payment Service DEPENDS_ON Database Service
- Frontend Team DEPENDS_ON Backend Team

Step 3: Build the graph

Nodes:
- Database Service
- Backend Team
- Frontend Team
- Payment Service
- API Gateway Service
- Cache Service

Edges:
Backend Team OWNS Database Service
Backend Team OWNS API Gateway Service
Backend Team OWNS Cache Service
Payment Service DEPENDS_ON Database Service
Frontend Team DEPENDS_ON API Gateway Service
Cache Service DEPENDS_ON Database Service

Step 4: Query examples

Query: "What will the impact be if the Database Service goes down?"

Start at: Database Service
Incoming edges (who depends on it):
- Payment Service DEPENDS_ON Database Service
- Cache Service DEPENDS_ON Database Service
- And transitively, Frontend Team DEPENDS_ON API Gateway, which might depend on Database

Answer: "If the Database Service fails, the Payment Service and Cache Service
will be unavailable. The Frontend Team may also be impacted if API Gateway uses the cache."

Compare this to plain RAG: You'd have to search for "Database Service failure impact", retrieve 5 chunks, and manually synthesize. The graph gives you the answer directly.

Common mistake

Building a graph and then ignoring it. Some teams invest in knowledge graph infrastructure but still rely on semantic search for every query, treating the graph as a side project. The real win is using the graph as the primary retrieval method for relationship and synthesis questions, and only falling back to semantic search for novel, unstructured questions.

Also common: Over-extracting relationships. If you extract every possible relationship from every document, the graph becomes a noisy hairball with 100k edges, and traversal becomes slow and fuzzy. Be selective: extract only relationships that answer your key questions. If you never ask "Who is the VP of Engineering?", don't extract org structure relationships.