Hybrid Search: Combining BM25 and Vector Retrieval

The limits of pure vector search

A user queries your support database: "Can I return error code 502?"

A vector embedding model encodes "error code 502" into a 768-dimensional vector. The model was trained on natural-language text, so it learns that "error code 502" and "server error" are semantically related. But a chunk that contains the exact string "error code 502" sits next to chunks about HTTP status codes, network timeouts, and other error scenarios. The embedding model might conflate all these error codes into a similar region of the embedding space.

Result: A pure vector search might retrieve chunks about "error code 500" or "error code 503" (high embedding similarity) before retrieving "error code 502" (which should be rank 1).

BM25 (Best Matching 25), a keyword-based retrieval algorithm, solves this. It rewards exact-match terms with high weight. A BM25 search for "502" returns chunks containing the literal string "502" ranked at the top, regardless of semantic context.

The problem: BM25 is purely lexical. If a user types "What is your return policy?" and a chunk says "Refunds: Our policy allows returns within 30 days," BM25 doesn't connect "return" in the question to "refunds" in the answer. (It can, with synonyms, but that requires preprocessing.) Vector search, by contrast, captures the semantic equivalence instantly.

Solution: Hybrid search. Run both retrievers in parallel. Merge their results using a ranking algorithm (reciprocal rank fusion). Let each retriever do what it does best, then combine.

BM25: Exact-match retrieval explained

BM25 is a statistical ranking algorithm. For a query q and document d, the BM25 score is:

BM25(q, d) = Σ(IDF(term) * (f(term, d) * (k1 + 1)) / (f(term, d) + k1 * (1 - b + b * (len(d) / avglen))))

Don't memorize this. Intuitively:

• IDF (Inverse Document Frequency): A term that appears in 1% of documents has higher weight than a term in 50% of documents. Rare terms are more discriminative.
• f(term, d): How many times the term appears in the document. More occurrences = higher score.
• len(d) / avglen: Document length normalization. Longer documents naturally contain more terms; we penalize them to avoid bias.
• k1, b: Tuning parameters (default k1=1.5, b=0.75; rarely changed).

Example:

Query: "error code 502"

Candidate chunks:

1. "Error code 502: Server error. Common causes include..."
2. "Error codes 500-599 are server-side errors. Code 502 means..."
3. "HTTP status: 503 Service Unavailable. Related codes: 502, 501..."
4. "How to debug timeout issues in production servers"

BM25 scores (illustrative):

1. Chunk 1: 4.5 (contains exact phrase "error code 502")
2. Chunk 2: 4.2 (contains all three terms, but spread across sentence)
3. Chunk 3: 2.1 ("502" appears once, as part of a list)
4. Chunk 4: 0.1 ("error" and "code" missing; "server" appears but no "502")

Ranking: 1 > 2 > 3 > 4. Excellent for exact-match intent.

BM25's strengths:

• Exact terms rank high
• Handles acronyms, codes, IDs perfectly
• Fast (single pass over inverted index)
• No tuning needed (defaults work well)

BM25's weaknesses:

• "Return policy" and "refunds" don't match unless you add synonyms
• "What does RAG stand for?" fails if the document says "Retrieval-Augmented Generation (RAG)" but BM25 doesn't parse that the query is asking for a definition
• Requires tokenization and preprocessing; multilingual handling is tricky

Vector search: Semantic retrieval

Query: "How do I get my money back?"

Vector embedding of the query captures the intent: "customer wants refund or return."

Candidate chunks:

1. "Return policy: We offer a 30-day return window for most products"
2. "Refunds are processed within 5 business days of approval"
3. "Money back guarantee: If you're not satisfied, we'll refund you"
4. "Our shipping policy: orders ship within 2 business days"

Cosine similarity (illustrative):

1. Chunk 1: 0.87 (high semantic match on "return")
2. Chunk 2: 0.84 (high semantic match on "refund")
3. Chunk 3: 0.85 (exact phrase "money back" and "refund")
4. Chunk 4: 0.12 (no semantic connection to refund intent)

Ranking: 1 > 3 > 2 > 4. Excellent for intent-based queries.

But pure vector search on exact-match queries:

Query: "What is error code 502?"

Vector embedding treats the query as "user asking about an error code." The model embeds it to a point in embedding space. Chunks about all error codes cluster nearby because the model hasn't learned fine-grained distinctions between "502" vs. "503" vs. "timeout."

Result: Top-3 chunks are about different error codes, with "502" somewhere in position 5-10.

Reciprocal Rank Fusion (RRF)

RRF is a simple, effective way to combine two ranked lists into one. For each document d:

RRF(d) = Σ 1 / (k + rank(d))

where k is a constant (usually 60) and rank(d) is the rank of document d in a retriever's result list (1-indexed). If a document doesn't appear in a retriever's top-N, it contributes 0.

Example:

Query: "error code 502"

BM25 results (top-5):

1. "Error code 502: Server error..."
2. "Error codes 500-599 are server-side errors..."
3. "HTTP status codes reference..."
4. (no match)
5. (no match)

Vector results (top-5):

1. "HTTP error: what does it mean when your connection fails"
2. "Error code 502: Bad gateway..."
3. "Debugging server issues: timeouts, 500s, and 503s"
4. "Server monitoring and alerting best practices"
5. (no match)

RRF scores (k=60):

• "Error code 502: Server error...": 1/(60+1) + 1/(60+2) = 0.0164 + 0.0159 = 0.0323 (top in both)
• "Error codes 500-599...": 1/(60+2) + 0 = 0.0159 (top in BM25, miss in vector)
• "Error code 502: Bad gateway...": 0 + 1/(60+2) = 0.0159 (miss in BM25, top in vector)
• "Debugging server issues...": 0 + 1/(60+3) = 0.0152 (only in vector)
• "HTTP error: connection fails": 0 + 1/(60+1) = 0.0164 (top in vector, miss in BM25)

Final ranking:

1. "Error code 502: Server error..." (0.0323) — Retrieved by both
2. "HTTP error: connection fails" (0.0164) — Retrieved by vector
3. "Error codes 500-599..." (0.0159) — Retrieved by BM25
4. "Error code 502: Bad gateway..." (0.0159) — Retrieved by vector

Implementing hybrid search in Python

Here's a realistic end-to-end example using BM25 and vector search together:

from rank_bm25 import BM25Okapi
import numpy as np
from sentence_transformers import SentenceTransformer

class HybridRetriever:
    def __init__(self, chunks, embedding_model='all-MiniLM-L6-v2'):
        """Initialize hybrid retriever with chunks."""
        self.chunks = chunks  # List of chunk dicts: {'id': str, 'text': str}

        # Initialize BM25
        tokenized_corpus = [chunk['text'].lower().split() for chunk in chunks]
        self.bm25 = BM25Okapi(tokenized_corpus)

        # Initialize vector embeddings
        self.model = SentenceTransformer(embedding_model)
        self.embeddings = self.model.encode(
            [chunk['text'] for chunk in chunks],
            convert_to_numpy=True
        )
        # Normalize for cosine similarity
        self.embeddings = self.embeddings / np.linalg.norm(
            self.embeddings, axis=1, keepdims=True
        )

    def bm25_retrieve(self, query, top_k=10):
        """Retrieve using BM25 keyword search."""
        tokenized_query = query.lower().split()
        scores = self.bm25.get_scores(tokenized_query)

        # Get top-k
        top_indices = np.argsort(scores)[::-1][:top_k]
        results = [
            {
                'id': self.chunks[idx]['id'],
                'text': self.chunks[idx]['text'],
                'score': scores[idx],
                'rank': i + 1
            }
            for i, idx in enumerate(top_indices)
        ]
        return results

    def vector_retrieve(self, query, top_k=10):
        """Retrieve using vector semantic search."""
        query_embedding = self.model.encode(query, convert_to_numpy=True)
        query_embedding = query_embedding / np.linalg.norm(query_embedding)

        # Cosine similarity
        scores = np.dot(self.embeddings, query_embedding)

        # Get top-k
        top_indices = np.argsort(scores)[::-1][:top_k]
        results = [
            {
                'id': self.chunks[idx]['id'],
                'text': self.chunks[idx]['text'],
                'score': scores[idx],
                'rank': i + 1
            }
            for i, idx in enumerate(top_indices)
        ]
        return results

    def reciprocal_rank_fusion(self, bm25_results, vector_results, k=60):
        """Fuse two ranked lists using RRF."""
        rrf_scores = {}

        # Add BM25 contributions
        for result in bm25_results:
            chunk_id = result['id']
            rrf_scores[chunk_id] = rrf_scores.get(chunk_id, 0) + 1 / (k + result['rank'])

        # Add vector contributions
        for result in vector_results:
            chunk_id = result['id']
            rrf_scores[chunk_id] = rrf_scores.get(chunk_id, 0) + 1 / (k + result['rank'])

        # Sort by RRF score
        sorted_results = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)

        # Return top-10
        fused_results = []
        for rank, (chunk_id, score) in enumerate(sorted_results[:10], 1):
            chunk_text = next(
                chunk['text'] for chunk in self.chunks if chunk['id'] == chunk_id
            )
            fused_results.append({
                'id': chunk_id,
                'text': chunk_text,
                'rrf_score': score,
                'rank': rank
            })

        return fused_results

    def retrieve(self, query, top_k=10, method='hybrid'):
        """Unified retrieve interface."""
        if method == 'bm25':
            return self.bm25_retrieve(query, top_k)
        elif method == 'vector':
            return self.vector_retrieve(query, top_k)
        elif method == 'hybrid':
            bm25_results = self.bm25_retrieve(query, top_k)
            vector_results = self.vector_retrieve(query, top_k)
            return self.reciprocal_rank_fusion(bm25_results, vector_results)
        else:
            raise ValueError(f"Unknown method: {method}")

# Usage
chunks = [
    {'id': 'doc-1', 'text': 'Error code 502: Bad Gateway. Usually caused by upstream server issues.'},
    {'id': 'doc-2', 'text': 'HTTP status codes: 200 (OK), 404 (Not Found), 500 (Server Error)...'},
    {'id': 'doc-3', 'text': 'Return policy: We offer returns within 30 days of purchase.'},
    # ... more chunks
]

retriever = HybridRetriever(chunks)

# Test on exact-match query
query1 = "error code 502"
hybrid_results = retriever.retrieve(query1, method='hybrid')
bm25_results = retriever.retrieve(query1, method='bm25')
vector_results = retriever.retrieve(query1, method='vector')

print(f"Query: {query1}")
print(f"\nBM25 top-3: {[r['id'] for r in bm25_results[:3]]}")
print(f"Vector top-3: {[r['id'] for r in vector_results[:3]]}")
print(f"Hybrid top-3: {[r['id'] for r in hybrid_results[:3]]}")

# Test on semantic query
query2 = "How do I return something?"
hybrid_results2 = retriever.retrieve(query2, method='hybrid')
print(f"\nQuery: {query2}")
print(f"Hybrid top-1: {hybrid_results2[0]['id']} - {hybrid_results2[0]['text']}")

Output (illustrative):

Query: error code 502
BM25 top-3: ['doc-1', 'doc-2', ...]
Vector top-3: ['doc-2', 'doc-1', ...]
Hybrid top-3: ['doc-1', 'doc-2', ...]

Query: How do I return something?
Hybrid top-1: doc-3 - Return policy: We offer returns within 30 days of purchase.

Hybrid search retrieves "doc-1" (the exact 502 error) in position 1, merging the best of both approaches.

When to use hybrid, pure vector, or pure BM25

| Query type | Pure BM25 | Pure Vector | Hybrid | |---|---|---|---| | Exact code/ID: "SKU-12345" | ✓ Excellent | ✗ Poor | ✓ Excellent | | Acronym: "What is RAG?" | ✗ Poor | ✓ Good | ✓ Excellent | | Error message: "file not found" | ✓ Good | ✓ Good | ✓ Excellent | | Semantic intent: "How do I refund?" | ✗ Poor | ✓ Excellent | ✓ Excellent | | Multilingual: "¿Cómo devuelvo?" | ✗ Very poor | ✓ Good (if model supports language) | ✓ Good |

General rule: Default to hybrid search. The small overhead of running two retrievers is worth the robustness gain. Only stick to pure BM25 or pure vector if your queries are homogeneous (all exact-match or all semantic).

Measuring hybrid search performance

Create a labeled query set and measure recall and precision for each method:

def evaluate_retrieval(retriever, test_queries, method='hybrid'):
    """Evaluate retrieval method on test set."""
    recalls = []
    precisions = []

    for query, relevant_ids in test_queries:
        results = retriever.retrieve(query, top_k=10, method=method)
        retrieved_ids = set(r['id'] for r in results)
        relevant_set = set(relevant_ids)

        recall = len(retrieved_ids & relevant_set) / max(1, len(relevant_set))
        precision = len(retrieved_ids & relevant_set) / max(1, len(retrieved_ids))

        recalls.append(recall)
        precisions.append(precision)

    avg_recall = np.mean(recalls)
    avg_precision = np.mean(precisions)

    print(f"{method.upper()}: Recall={avg_recall:.2%}, Precision={avg_precision:.2%}")

# Test on 50 labeled queries
evaluate_retrieval(retriever, test_queries, method='bm25')
evaluate_retrieval(retriever, test_queries, method='vector')
evaluate_retrieval(retriever, test_queries, method='hybrid')

# Expected output:
# BM25: Recall=78%, Precision=72%
# VECTOR: Recall=82%, Precision=75%
# HYBRID: Recall=88%, Precision=81%

Hybrid search typically gains 5-10% recall over the better of the two single methods.

Common mistake

Assuming hybrid search is strictly better than either method alone and doesn't need tuning. The RRF fusion can be tuned: the constant k=60 is a default, but for your corpus, k=30 or k=100 might be better. Additionally, if one retriever is much better than the other, weighting them unequally (e.g., RRF = 1.5 × vector_contribution + bm25_contribution) can improve results.

Also common: Treating vector and BM25 retrieval as independent. If you can featurize your corpus (e.g., mark "error codes" as a special type), you can route queries to the best retriever. Queries matching a code pattern go to BM25-first. Semantic queries go to vector-first. This selective routing can be more effective than always fusing.