Choosing and Benchmarking an Embedding Model

Embedding models are not all equivalent

When you build a RAG pipeline, the embedding model is the gatekeeper: it determines whether your retrieval system can even find the relevant chunks. Two embedding models trained on the same data, same architecture, and same objective can still differ dramatically on your specific domain. Public benchmarks like MTEB show broad trends, but they don't predict performance on your documents.

The dimensions vs. accuracy tradeoff

Embedding models vary from 384 to 4096+ dimensions. More dimensions theoretically encode more information, but they come with three concrete costs.

Storage cost: A vector database must store, replicate, and index every embedding. A 384-dim embedding uses ~1.5 KB per vector. A 3072-dim embedding uses ~12 KB per vector. At 1 million chunks, that's 1.5 GB vs. 12 GB. Cloud storage is cheap, but the vector database's own indexes (HNSW, IVF) scale with dimension, and in-memory serving gets slower.

Search latency: Searching high-dimensional spaces is geometrically slower. A 384-dim exact nearest-neighbor search on 1M vectors takes ~5 ms on modern hardware. A 3072-dim search takes ~15 ms. Approximate nearest-neighbor algorithms (HNSW, IVF) help, but the advantage persists.

Cost per token: Embedding services charge by token. OpenAI's small embedding model (384 dims) costs $0.02 per 1M tokens. The larger model (1536 dims) costs $0.10 per 1M tokens. At query time, re-embedding the user's question is cheap (one query per user). At index time, embedding your entire corpus can cost hundreds or thousands of dollars if re-indexing frequently.

In practice, most production RAG systems use 384-768 dimensions and achieve >85% retrieval recall on domain-matched queries. Going to 2048+ dimensions is rare unless you're optimizing for a specific hard case (multilingual, very large corpus, or extreme precision requirements).

Public benchmarks are a starting point, not a guarantee

The Massive Text Embedding Benchmark (MTEB) evaluates 150+ embedding models across retrieval, clustering, pair classification, and other tasks. It's valuable for a rough ranking: a model that scores 50/100 on MTEB retrieval is almost certainly worse than one that scores 80/100. But MTEB includes no proprietary documents, no domain-specific language, and no adversarial queries.

Example: A legal firm evaluated three models on MTEB:

• Model A: 75/100 on retrieval
• Model B: 78/100 on retrieval
• Model C: 80/100 on retrieval

They picked Model C based on MTEB ranking. But on their internal 500 labeled legal queries, the results inverted:

• Model A: 84% recall on legal queries
• Model B: 79% recall on legal queries
• Model C: 71% recall on legal queries

Model C's general-domain strength was a liability: it learned broad, generic embeddings that conflate "contract breach" in software law with "contract breach" in construction law. Model A was trained on legal data and maintained finer distinctions.

MTEB's blind spots:

• No proprietary domains (legal, medical, finance, internal wikis)
• No conversational intent matching (when users ask a question in their own words, not document phrasing)
• No metadata filtering (filtering by date, author, category before re-ranking)
• No multi-hop reasoning (question A is answered by embedding B, then B's answer enriches a query for C)

Treat MTEB as a pre-screening tool. Narrow down to 2-3 candidates, then evaluate them on your own domain.

Building a labeled validation set

Creating a representative test set takes work, but it's the only reliable way to measure embedding quality on your domain.

Step 1: Mine real queries

Start with production data (if you have any), support logs, or user interviews. Aim for 50-200 representative queries. Avoid synthetic queries until you've validated that your domain is well-covered.

# Example: extract queries from support logs
import re
from datetime import datetime, timedelta

def mine_support_queries(log_file, days_back=30):
    """Extract support questions from logs, deduplicate by semantic similarity."""
    cutoff = datetime.now() - timedelta(days=days_back)
    queries = []

    with open(log_file) as f:
        for line in f:
            # Parse log line (format depends on your logging system)
            timestamp, message = parse_log_line(line)
            if timestamp < cutoff:
                continue

            # Filter for question-like patterns
            if any(marker in message.lower() for marker in ["how", "why", "what", "does", "can i"]):
                queries.append(message)

    # Deduplicate: use embeddings to remove near-duplicates
    from sentence_transformers import SentenceTransformer
    model = SentenceTransformer('all-MiniLM-L6-v2')
    embeddings = model.encode(queries)

    # Cluster and pick one representative per cluster
    from sklearn.cluster import DBSCAN
    clustering = DBSCAN(eps=0.2, min_samples=1).fit(embeddings)

    deduplicated = []
    for label in set(clustering.labels_):
        cluster_indices = [i for i, l in enumerate(clustering.labels_) if l == label]
        # Pick the shortest query as representative (usually the clearest)
        best_idx = min(cluster_indices, key=lambda i: len(queries[i]))
        deduplicated.append(queries[best_idx])

    return deduplicated

queries = mine_support_queries("support_logs.txt")
print(f"Extracted {len(queries)} unique queries")

Step 2: Curate for coverage

Sort the mined queries by difficulty and coverage. Stratify by:

• Common cases (50%): High-frequency, straightforward
• Edge cases (30%): Rare but important ("how do I use this in China?" / "refund policy for canceled plans")
• Adversarial (10%): Deliberately hard or off-domain ("completely unrelated question")
• Unanswerable (10%): No supporting document exists in your corpus

# Curate 100 queries with balanced coverage
common = queries[:60]  # Frequency-ranked
edge_cases = [q for q in queries if any(keyword in q for keyword in ["except", "rare", "edge", "special"])][:30]
adversarial = [
    "explain quantum computing",
    "how is banana",
    "why is my car broken"
]
unanswerable = ["what's the CEO's personal favorite color"]

validation_set = common + edge_cases + adversarial + unanswerable

Step 3: Label with ground truth

For each query, identify which documents or chunks should be retrieved (if any). This is the expensive step: one person reads each query and marks which chunks are relevant.

{
  "query": "how do I reset my password if I don't remember my security question",
  "relevant_document_ids": ["doc-4521", "doc-4522"],
  "relevant_chunk_ids": ["doc-4521-chunk-3", "doc-4522-chunk-1"],
  "is_answerable": true,
  "difficulty": "edge_case",
  "notes": "user might have multiple reset paths; both docs cover different methods"
}

For 100 queries, this takes ~2-4 hours of expert time. It's worth it: this is your ground truth.

Evaluating candidate models

Once you have 100 labeled queries, rank each candidate model using retrieval metrics. Code below embeds all queries and all chunks, then measures recall and precision:

from sentence_transformers import SentenceTransformer
import json
import numpy as np

def evaluate_embedding_model(model_name, validation_set, chunk_database):
    """Compare a candidate embedding model against labeled validation set."""

    model = SentenceTransformer(model_name)

    # Embed all chunks once (happens at index time)
    all_chunk_texts = [chunk['text'] for chunk in chunk_database]
    chunk_embeddings = model.encode(all_chunk_texts, normalize_embeddings=True)

    results = []

    for item in validation_set:
        query = item['query']
        query_embedding = model.encode(query, normalize_embeddings=True)

        # Find top-10 nearest chunks by cosine similarity
        similarities = np.dot(chunk_embeddings, query_embedding)
        top_10_indices = np.argsort(similarities)[::-1][:10]

        retrieved_chunk_ids = [chunk_database[i]['id'] for i in top_10_indices]
        retrieved_scores = [similarities[i] for i in top_10_indices]

        # Compare against ground truth
        relevant_ids = set(item['relevant_chunk_ids'])
        retrieved_set = set(retrieved_chunk_ids)

        # Compute metrics
        true_positives = len(relevant_ids & retrieved_set)
        false_positives = len(retrieved_set - relevant_ids)
        false_negatives = len(relevant_ids - retrieved_set)

        recall = true_positives / max(1, len(relevant_ids))
        precision = true_positives / max(1, len(retrieved_set))

        # MRR: position of first relevant result (or 0 if none in top-10)
        mrr = 0
        for rank, chunk_id in enumerate(retrieved_chunk_ids, 1):
            if chunk_id in relevant_ids:
                mrr = 1 / rank
                break

        results.append({
            'query': query,
            'recall': recall,
            'precision': precision,
            'mrr': mrr,
            'relevant_count': len(relevant_ids),
            'retrieved_count': len(retrieved_set)
        })

    # Aggregate metrics
    avg_recall = np.mean([r['recall'] for r in results])
    avg_precision = np.mean([r['precision'] for r in results])
    avg_mrr = np.mean([r['mrr'] for r in results])

    print(f"\n{model_name}")
    print(f"  Avg Recall@10:    {avg_recall:.2%}")
    print(f"  Avg Precision@10: {avg_precision:.2%}")
    print(f"  Avg MRR:          {avg_mrr:.3f}")

    return {
        'model': model_name,
        'avg_recall': avg_recall,
        'avg_precision': avg_precision,
        'avg_mrr': avg_mrr,
        'case_results': results
    }

# Evaluate three candidates
candidates = [
    'all-MiniLM-L6-v2',           # 384 dims, fast
    'all-mpnet-base-v2',          # 768 dims, slower but more accurate
    'bge-large-en-v1.5'           # 1024 dims, slow but highest quality
]

results = []
for model_name in candidates:
    result = evaluate_embedding_model(model_name, validation_set, chunks)
    results.append(result)

# Rank by average recall
results.sort(key=lambda r: r['avg_recall'], reverse=True)
print("\nRanking by domain recall:")
for i, r in enumerate(results, 1):
    print(f"{i}. {r['model']}: {r['avg_recall']:.2%}")

This evaluation is domain-specific and realistic. The model that wins here is the one to use.

Cost comparison across models

Embedding cost compounds at scale. Here's how to estimate total cost of ownership:

The choice depends on your query volume, document churn, and whether you're fine-tuning. If you embed documents once at index time and query re-embedding is rare, the cost is front-loaded. If you're constantly re-indexing (documents change, models are updated), embedding cost is ongoing.

Multilingual considerations

If your corpus spans multiple languages, test the embedding model's cross-lingual performance. Some models handle multilingual well; others don't.

Multilingual models worth testing:

• sentence-transformers/multilingual-e5-large (supports 100+ languages)
• sentence-transformers/xlm-r-100langs-bert-base-nli-stsb-mean-tokens (older, but robust)
• OpenAI text-embedding-3-large (officially supports multiple languages)

For a financial services firm with documents in English, German, and Japanese, a truly multilingual model is non-negotiable. Testing on translation-heavy cases is important:

{
  "query_english": "how much is the premium",
  "query_german": "wie viel kostet die prämie",
  "query_japanese": "プレミアムはいくらですか",
  "relevant_chunk_ids": ["pricing-chunk-5"],
  "notes": "all three should retrieve the same pricing section despite language difference"
}

When to fine-tune embeddings

Fine-tuning an embedding model on your domain is expensive but sometimes necessary. It's worth considering if:

1. Your domain has specialized vocabulary that a pre-trained model doesn't handle (legal, medical, proprietary jargon).
2. Your validation-set recall is <75% with all pre-trained candidates, and your corpus is large enough (>10K documents) to fine-tune on.
3. You have labeled (query, relevant_chunks) pairs — at least 500-1000 pairs to fine-tune on without overfitting.

Fine-tuning tools:

• Sentence Transformers' MultipleNegativesRankingLoss: Minimizes distance between queries and matching chunks while pushing apart non-matches. ~2-4 hours to fine-tune on 1000 pairs on a single GPU.
• OpenAI Fine-tuning API: Limited to their models, but simpler to set up. ~$5-15 per 1M training tokens.

from sentence_transformers import SentenceTransformer, InputExample, losses
from torch.utils.data import DataLoader

# Load base model
model = SentenceTransformer('all-mpnet-base-v2')

# Prepare training examples
train_examples = []
for item in labeled_query_set:
    # Positive: (query, matching chunk)
    positive_example = InputExample(
        texts=[item['query'], item['matching_chunk']],
        label=1.0
    )
    train_examples.append(positive_example)

    # Negatives: (query, non-matching chunks)
    for non_match in item['non_matching_chunks'][:3]:
        negative_example = InputExample(
            texts=[item['query'], non_match],
            label=0.0
        )
        train_examples.append(negative_example)

# Fine-tune
train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=32)
train_loss = losses.MultipleNegativesRankingLoss(model)

model.fit(
    train_objectives=[(train_dataloader, train_loss)],
    epochs=1,
    warmup_steps=100,
    show_progress_bar=True
)

# Evaluate the fine-tuned model
fine_tuned_result = evaluate_embedding_model(model, validation_set, chunks)

Fine-tuning typically improves recall by 5-15% on your domain. It's most valuable for specialized fields.

Practice: Pick and validate an embedding model for your domain

1. Mine 50 real queries from your logs or users.
2. Label 20-30 of them with ground truth chunks (1-2 hours of work).
3. Evaluate 3 candidate models: a small/cheap one (384 dims), a medium one (768 dims), and a larger one (1536 dims).
4. Measure recall, precision, and MRR on your labeled set.
5. Estimate total cost including indexing and query embedding.
6. Pick the winner: highest recall for your domain, not highest MTEB score.

Common mistake

Picking an embedding model based purely on public benchmarks (MTEB) without testing on your own domain. MTEB winners are often over-engineered for a general web-text use case and can underperform on specialized domains like legal, medical, or internal knowledge bases. Always measure on real queries before committing.

Also common: Choosing a model based on cost alone. The cheapest embedding model that retrieves only 60% of relevant chunks will hurt downstream answer quality and cause users to distrust the system. Spend the 2-4 hours to test before deciding.