Re-ranking: Cross-Encoders and Late Interaction

Bi-encoders vs. cross-encoders: The architecture difference

Bi-encoder (e.g., standard sentence-transformers models):

• Encodes the query independently into a vector.
• Encodes each document/chunk independently into a vector.
• Compares vectors using distance (cosine similarity, Euclidean).
• Speed: O(1) per candidate (just a dot product after pre-computing embeddings).
• Accuracy: Good, but misses fine-grained interaction between query and document.

Cross-encoder (e.g., sentence-transformers CrossEncoder):

• Takes both query and document as a single pair input.
• Passes the concatenated input through a transformer encoder.
• Outputs a relevance score for that specific (query, document) pair.
• Speed: O(d) per candidate (must run inference for each pair).
• Accuracy: Excellent. The transformer can attend to both query and document tokens, capturing complex interactions.

Analogy: Bi-encoders are like comparing two images by their color histograms. Fast, but missing details. Cross-encoders are like a human expert comparing two images side-by-side, considering spatial relationships, texture, and context. Slower, but much more accurate.

Why cross-encoders can't rank a full corpus: If you have 1M chunks and receive a query, running a cross-encoder on all 1M (query, chunk) pairs means 1M forward passes through a transformer. On GPU, each pass takes ~10-50ms, so 1M pairs = 10,000-50,000 seconds. Infeasible.

Why cross-encoders are perfect for re-ranking: If you've already retrieved the top 100 candidates with a fast bi-encoder, running a cross-encoder on those 100 pairs is cheap: ~1-5 seconds. The accuracy gain is worth it.

The two-stage retrieve-and-rerank pipeline

The standard pattern:

1. Retrieve broad (bi-encoder): Get top 50-100 candidates fast.
2. Rerank narrow (cross-encoder): Score each candidate more carefully, return top 5-10.

This combines speed (fast retrieval of many candidates) with accuracy (careful re-ranking of the final candidates).

from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np

class RetrieveAndRerank:
    def __init__(self, chunks):
        """Initialize with a corpus of chunks."""
        self.chunks = chunks  # List of {'id': str, 'text': str}

        # Stage 1: Fast bi-encoder for initial retrieval
        self.bi_encoder = SentenceTransformer('all-mpnet-base-v2')  # 768 dims
        self.chunk_embeddings = self.bi_encoder.encode(
            [chunk['text'] for chunk in chunks],
            convert_to_numpy=True
        )
        self.chunk_embeddings = self.chunk_embeddings / np.linalg.norm(
            self.chunk_embeddings, axis=1, keepdims=True
        )

        # Stage 2: Expensive cross-encoder for re-ranking
        self.cross_encoder = CrossEncoder('cross-encoder/mmarco-mMiniLMv2-L12-H384-v1')
        # This model expects (query, passage) pairs and outputs a relevance score [0, 1]

    def retrieve_and_rerank(self, query, retrieve_k=100, rerank_k=10):
        """Two-stage pipeline: retrieve 100, rerank to top-10."""
        import time

        # Stage 1: Retrieve top-100 with bi-encoder (fast)
        start = time.time()
        query_embedding = self.bi_encoder.encode(query, convert_to_numpy=True)
        query_embedding = query_embedding / np.linalg.norm(query_embedding)

        similarities = np.dot(self.chunk_embeddings, query_embedding)
        top_indices = np.argsort(similarities)[::-1][:retrieve_k]

        retrieve_time = time.time() - start
        print(f"Retrieve stage: {retrieve_time*1000:.1f}ms")

        # Stage 2: Rerank top-100 with cross-encoder (more expensive but accurate)
        start = time.time()
        candidate_chunks = [self.chunks[i] for i in top_indices]

        # Prepare pairs for cross-encoder
        pairs = [(query, chunk['text']) for chunk in candidate_chunks]

        # Get relevance scores (higher = more relevant)
        rerank_scores = self.cross_encoder.predict(pairs, batch_size=32)

        rerank_time = time.time() - start
        print(f"Rerank stage: {rerank_time*1000:.1f}ms")

        # Sort by rerank score
        rerank_indices = np.argsort(rerank_scores)[::-1][:rerank_k]
        reranked_chunks = [
            {
                'id': candidate_chunks[i]['id'],
                'text': candidate_chunks[i]['text'],
                'rerank_score': rerank_scores[i],
                'rank': j + 1
            }
            for j, i in enumerate(rerank_indices)
        ]

        return {
            'results': reranked_chunks,
            'retrieve_time_ms': retrieve_time * 1000,
            'rerank_time_ms': rerank_time * 1000
        }

# Usage
chunks = [
    {'id': 'doc-1', 'text': 'Our return policy allows 30-day returns on all items.'},
    {'id': 'doc-2', 'text': 'Refunds are processed within 5 business days.'},
    # ... 1000s more
]

rar = RetrieveAndRerank(chunks)

query = "How long do I have to return something?"
result = rar.retrieve_and_rerank(query, retrieve_k=100, rerank_k=10)

print(f"\nTop-3 after reranking:")
for r in result['results'][:3]:
    print(f"{r['rank']}. {r['id']}: score={r['rerank_score']:.3f}")
    print(f"   {r['text']}\n")

print(f"Total time: {result['retrieve_time_ms'] + result['rerank_time_ms']:.1f}ms")

Expected performance (illustrative estimates on 1M-chunk corpus):

| Stage | Method | Time | Accuracy | Notes | |---|---|---|---|---| | Retrieve | Bi-encoder (HNSW index) | 10-20ms | 85% recall | Fast, good enough | | Rerank | Cross-encoder on top-100 | 1-3 seconds | 95%+ precision on final 10 | Expensive but worth it | | Total | Two-stage pipeline | 1-3.5 seconds | 90%+ precision on final 10 | Balanced | | Alternative | Cross-encoder on all 1M | 10,000+ seconds | ~98% precision | Infeasible |

The two-stage pipeline is the standard production pattern because it balances latency and accuracy.

Measuring re-ranking's impact

To quantify the improvement, measure metrics like precision and MRR (Mean Reciprocal Rank) before and after re-ranking.

def evaluate_reranking(test_queries, rar, retrieve_k=100, rerank_k=10):
    """Measure how much re-ranking improves retrieval precision."""

    precisions_before = []
    precisions_after = []
    mrr_before = []
    mrr_after = []

    for query, relevant_ids in test_queries:
        relevant_set = set(relevant_ids)

        # Before reranking: measure precision@10 on bi-encoder results
        result = rar.retrieve_and_rerank(query, retrieve_k, rerank_k)

        # Get bi-encoder top-10 (without reranking)
        query_embedding = rar.bi_encoder.encode(query, convert_to_numpy=True)
        query_embedding = query_embedding / np.linalg.norm(query_embedding)
        similarities = np.dot(rar.chunk_embeddings, query_embedding)
        top_10_bi = np.argsort(similarities)[::-1][:10]
        retrieved_ids_before = set([rar.chunks[i]['id'] for i in top_10_bi])

        # After reranking
        retrieved_ids_after = set([r['id'] for r in result['results'][:10]])

        # Precision@10
        precision_before = len(retrieved_ids_before & relevant_set) / 10
        precision_after = len(retrieved_ids_after & relevant_set) / 10

        precisions_before.append(precision_before)
        precisions_after.append(precision_after)

        # MRR: position of first relevant result
        for rank, result_id in enumerate(list(retrieved_ids_before), 1):
            if result_id in relevant_set:
                mrr_before.append(1 / rank)
                break
        else:
            mrr_before.append(0)

        for rank, r in enumerate(result['results'], 1):
            if r['id'] in relevant_set:
                mrr_after.append(1 / rank)
                break
        else:
            mrr_after.append(0)

    avg_precision_before = np.mean(precisions_before)
    avg_precision_after = np.mean(precisions_after)
    avg_mrr_before = np.mean(mrr_before)
    avg_mrr_after = np.mean(mrr_after)

    print("Retrieval Quality Improvement from Reranking:")
    print(f"  Precision@10: {avg_precision_before:.1%} → {avg_precision_after:.1%} "
          f"({(avg_precision_after - avg_precision_before)*100:+.1f}%)")
    print(f"  MRR: {avg_mrr_before:.3f} → {avg_mrr_after:.3f} "
          f"({(avg_mrr_after - avg_mrr_before)*100:+.1f}%)")

# Test on 30 labeled queries
evaluate_reranking(test_queries, rar)

# Expected output:
# Precision@10: 72% → 85% (+13%)
# MRR: 0.72 → 0.88 (+16%)

A well-tuned cross-encoder reranker typically improves precision@10 by 10-20% over the bi-encoder alone.

Why the two-stage pipeline is essential

The bi-encoder + cross-encoder combination is the dominant pattern in production because it solves a fundamental problem: speed doesn't scale with accuracy in a single-model approach.

If you use a cross-encoder as your primary retriever, scoring all 1M chunks takes hours. If you use a bi-encoder alone, you can retrieve in 10ms, but accuracy caps out around 80-85%. The two-stage pipeline gets you both: 10ms retrieval narrowing to top-100, then 1-2s re-ranking of those 100 to get 90%+ accuracy on the final top-10.

Why precision matters more on the final results: Users see only the top-5 or top-10 results. A retrieval recall of 95% sounds good, but if 5% of relevant chunks are missing from the top-100, and your corpus has 100 relevant documents total, then ~5 relevant documents get no chance to be re-ranked at all. These forever-lost documents usually include some of the hardest-to-retrieve ones. For this reason, it's worth over-retrieving in stage 1 (get top-100 instead of top-50) to maximize the chances that stage 2's re-ranker sees all the good candidates.

Calibrating retrieve_k and rerank_k:

A common mistake is setting retrieve_k=10 and rerank_k=10 (stage 1 and stage 2 retrieve the same number). This defeats the purpose. Better settings:

• retrieve_k=50-100 (stage 1 retrieves broad)
• rerank_k=5-10 (stage 2 returns narrow)

This 5-10x gap allows the cross-encoder to see multiple good candidates and pick the best.

Late-interaction models: A middle ground

ColBERT and similar "late-interaction" models offer a speed-accuracy tradeoff between bi-encoders and cross-encoders.

How late-interaction works (ColBERT example):

Instead of computing one embedding per chunk (bi-encoder), compute a multi-vector representation: for each token in the chunk, compute an embedding. Then, at query time, compute embeddings for each query token. Finally, match query tokens to chunk tokens and take the maximum similarity per query token.

Chunk: "Return policy: 30-day returns on all items"
  Token embeddings: [return_emb, policy_emb, 30_emb, day_emb, returns_emb, ...]

Query: "How long to return?"
  Token embeddings: [how_emb, long_emb, return_emb]

Matching:
  how → {best match in chunk tokens}
  long → {best match in chunk tokens}
  return → return_emb (exact match!) → similarity = 0.95

Score = average or maximum of these token-level similarities

Speed: Still requires computing embeddings for every chunk (like bi-encoders), but because you're matching tokens, not whole documents, the matching step is parallelizable.

Accuracy: Much closer to cross-encoders than bi-encoders because it captures token-level interactions.

Tradeoff: Requires storing multiple embeddings per chunk (maybe 10-100 tokens per chunk = 10-100 vectors), so 10-100x more storage than bi-encoders. But can be indexed and searched efficiently.

Late-interaction models are less commonly used in RAG because:

1. Cross-encoder re-ranking on top-100 is already good enough.
2. Storage overhead is significant.
3. Implementation complexity is higher (no library as simple as sentence-transformers).

They're valuable if you need something between bi-encoder retrieval and cross-encoder re-ranking (e.g., you can't afford 1-3s re-ranking latency but want higher accuracy).

Practical tuning: Finding your optimal retrieve-rerank balance

The choice of which cross-encoder to use and how many candidates to rerank is corpus-dependent. Here's how to optimize:

def tune_retrieve_rerank_pipeline(test_queries, biencoder, crossencoder, corpus_size):
    """Find optimal retrieve_k and rerank_k for your corpus and latency budget."""
    import time

    retrieve_k_values = [10, 25, 50, 100, 200]
    rerank_k_values = [5, 10, 20]

    results = []

    for retrieve_k in retrieve_k_values:
        for rerank_k in rerank_k_values:
            if rerank_k > retrieve_k:
                continue  # Skip invalid combinations

            precisions = []
            latencies = []

            for query, relevant_docs in test_queries:
                # Stage 1: Retrieve
                start = time.time()
                candidates = biencoder.retrieve(query, top_k=retrieve_k)
                retrieve_time = time.time() - start

                # Stage 2: Rerank
                start = time.time()
                pairs = [(query, c['text']) for c in candidates]
                scores = crossencoder.predict(pairs)
                top_k_indices = np.argsort(scores)[::-1][:rerank_k]
                final_results = [candidates[i] for i in top_k_indices]
                rerank_time = time.time() - start

                # Measure precision
                retrieved_docs = set([r['doc_id'] for r in final_results])
                precision = len(retrieved_docs & relevant_docs) / rerank_k

                precisions.append(precision)
                latencies.append(retrieve_time + rerank_time)

            avg_precision = np.mean(precisions)
            avg_latency = np.mean(latencies)

            results.append({
                'retrieve_k': retrieve_k,
                'rerank_k': rerank_k,
                'avg_precision': avg_precision,
                'avg_latency_ms': avg_latency * 1000
            })

    # Display results sorted by precision
    results.sort(key=lambda x: x['avg_precision'], reverse=True)

    print("Retrieve-Rerank Tuning Results (sorted by precision):")
    print("retrieve_k | rerank_k | precision | latency_ms")
    for r in results:
        print(f"{r['retrieve_k']:>10} | {r['rerank_k']:>8} | {r['avg_precision']:>9.1%} | {r['avg_latency_ms']:>10.1f}")

    # Pick the best trade-off (e.g., highest precision within a latency budget of 2s)
    LATENCY_BUDGET = 2000  # milliseconds
    candidates = [r for r in results if r['avg_latency_ms'] < LATENCY_BUDGET]
    if candidates:
        best = max(candidates, key=lambda x: x['avg_precision'])
        print(f"\nRecommended (within {LATENCY_BUDGET}ms budget):")
        print(f"  retrieve_k={best['retrieve_k']}, rerank_k={best['rerank_k']}")
        print(f"  Precision: {best['avg_precision']:.1%}, Latency: {best['avg_latency_ms']:.0f}ms")

# Run tuning on your corpus
tune_retrieve_rerank_pipeline(test_queries, bi_encoder, cross_encoder, corpus_size=1_000_000)

# Expected output:
# Retrieve-Rerank Tuning Results (sorted by precision)
# retrieve_k | rerank_k | precision | latency_ms
#        100 |        5 |      88.3% |      1234.2
#         50 |        5 |      87.1% |       612.5
#        100 |       10 |      88.9% |      1543.1
#         ...
#
# Recommended (within 2000ms budget):
#   retrieve_k=100, rerank_k=10
#   Precision: 88.9%, Latency: 1543.1ms

This grid search takes a few minutes but gives you concrete data on precision-latency tradeoffs. In the output above, you'd see that retrieve_k=100 + rerank_k=10 hits nearly 89% precision in under 2 seconds, while retrieve_k=200 + rerank_k=10 might give 89.5% precision but takes 2.5+ seconds (exceeding budget).

Choosing between retrieval strategies

| Scenario | Best approach | Why | |---|---|---| | <10K documents | Bi-encoder only | Fast enough even at full scale | | 10K-1M documents, <1s latency budget | Bi-encoder only | Re-ranking pushes you over latency | | 10K-1M documents, 1-3s latency budget | Bi-encoder + cross-encoder reranking | Optimal: speed + accuracy | | 1M+ documents, high precision required | Bi-encoder + cross-encoder reranking | Standard production pattern | | 1M+ documents, extreme latency constraint | Bi-encoder + late-interaction | Tradeoff if cross-encoder too slow | | Multilingual at scale | Bi-encoder (multilingual model) + cross-encoder (multilingual) | Both need multilingual versions |

Practice: Build and measure a reranking pipeline

1. Create 30 labeled test queries with ground-truth relevant chunks.
2. Build a bi-encoder retrieval system (HNSW or IVF index).
3. Add a cross-encoder reranker on top-100 candidates.
4. Measure precision and MRR before and after reranking.
5. Estimate latency (retrieval + reranking time).

Common mistake

Assuming a cross-encoder is always better and using it as the primary retriever on a large corpus. Cross-encoders are excellent for re-ranking but prohibitively slow for ranking millions of candidates. Always use a two-stage approach: fast bi-encoder for broad retrieval, cross-encoder for narrow re-ranking.

Also common: Not measuring re-ranking's impact. The cross-encoder adds latency; you should verify that the precision gain is worth the latency cost. On some corpora, a well-tuned bi-encoder might already achieve 85% precision, and re-ranking buys only 2-3% improvement. In that case, skip re-ranking and invest in better bi-encoder selection instead.