Query Transformation: Rewriting, Expansion, and HyDE

Why raw queries often fail

A user asks your support chatbot: "Hi, I'm having trouble with my account. Can you help? I'm trying to reset my password but it says something is wrong. Does anyone know what's happening?"

This is a real user query. It's conversational, vague, and buried in pleasantries. Your embedding model ingests it as-is and searches the vector database. The embedding captures "reset password" + "trouble" + "something is wrong," but also "help," "hi," "can you," which are noise.

Candidate retrieval results:

1. "Account Troubleshooting FAQ" (high embedding similarity to "trouble," not specific enough)
2. "How to contact support" (high similarity to "help")
3. "Password Reset Guide" (buried at rank 5, even though it's the answer)

The core intent is lost in conversational filler. The user's real question is: "How do I reset my password?"

Query transformation fixes this by:

1. Rewriting: Strip conversational filler, extract core intent.
2. Expansion: Add synonyms and related terms (reset → initialize, change, recover; password → account access, login credentials).
3. HyDE: Generate a hypothetical answer first, then embed that instead of the raw question.

Each transformation produces a slightly different query. Running retrieval on all of them and merging results captures more relevant chunks than a single raw query.

Query rewriting

Query rewriting uses an LLM to clean and clarify the user's intent.

from anthropic import Anthropic

def rewrite_query(user_query):
    """Rewrite a raw user query to extract core intent."""
    client = Anthropic()

    system_prompt = """You are a query refinement assistant. Your job is to extract the core intent
from raw user questions, removing conversational filler, ambiguity, and noise.

Input: A raw user question (may be conversational, vague, or multi-part)
Output: A single concise question that captures the core intent, suitable for document retrieval

Keep the output as one sentence. Remove "hi", "can you", "please", "I think", etc.
Keep domain-specific terms and concrete nouns."""

    message = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=100,
        system=system_prompt,
        messages=[
            {
                "role": "user",
                "content": f"Rewrite this query:\n\n{user_query}"
            }
        ]
    )

    return message.content[0].text.strip()

# Test
raw_query = "Hi, I'm having trouble with my account. Can you help? I'm trying to reset my password but it says something is wrong. Does anyone know what's happening?"
rewritten = rewrite_query(raw_query)
print(f"Original:  {raw_query}")
print(f"Rewritten: {rewritten}")

# Output:
# Original:  Hi, I'm having trouble with my account. Can you help? I'm trying to reset my password but it says something is wrong. Does anyone know what's happening?
# Rewritten: How do I reset a forgotten password when an error appears?

Benefits:

• Cleaner embeddings (no noise from "hi," "can you").
• Better retrieval of direct answers.
• Faster execution (fewer tokens to embed).

Limitations:

• Requires an LLM call (adds latency, ~100-500ms).
• Might over-simplify multi-part questions (e.g., "I want to reset my password and also check my billing history" becomes just "reset password").
• Not always necessary (some queries are already clean).

Query expansion

Query expansion adds synonyms and related terms. It's a lighter-weight alternative to rewriting.

def expand_query(query, embedding_model):
    """Expand query with synonyms and related terms using embedding similarity."""
    from sentence_transformers import SentenceTransformer
    import numpy as np

    model = SentenceTransformer('all-MiniLM-L6-v2')

    # Seed terms: the key nouns/verbs in the query
    seed_terms = ['reset', 'password', 'account']  # Extracted manually or via NLP

    # Expansion terms: synonyms, related words (from a thesaurus or model-generated)
    synonym_map = {
        'reset': ['initialize', 'restore', 'recover', 'change', 'set'],
        'password': ['credentials', 'passphrase', 'access code', 'login token', 'authentication'],
        'account': ['profile', 'user account', 'account settings', 'login']
    }

    expanded_terms = []
    for term in seed_terms:
        expanded_terms.append(term)  # Include original
        if term in synonym_map:
            expanded_terms.extend(synonym_map[term])

    # Filter: keep only terms that are semantically close to the original query
    query_embedding = model.encode(query)
    term_embeddings = model.encode(expanded_terms)

    similarities = np.dot(term_embeddings, query_embedding) / (
        np.linalg.norm(term_embeddings, axis=1) * np.linalg.norm(query_embedding)
    )

    # Keep top synonyms (threshold: similarity > 0.5)
    relevant_synonyms = [t for t, sim in zip(expanded_terms, similarities) if sim > 0.5]

    # Construct expanded query
    expanded_query = query + " OR " + " OR ".join(relevant_synonyms)
    return expanded_query

# Test
query = "How do I reset my password?"
expanded = expand_query(query, embedding_model)
print(f"Original:  {query}")
print(f"Expanded:  {expanded}")

# Output:
# Original:  How do I reset my password?
# Expanded:  How do I reset my password? OR initialize OR restore OR recover OR change OR credentials OR passphrase

Benefits:

• Improves recall on queries with domain-specific vocabulary (especially useful if your embedding model is weak on synonyms).
• No LLM call (fast, <10ms).
• Captures intent variations.

Limitations:

• Requires a good synonym source (thesaurus, manual curation, or LLM-generated at index time).
• Over-expansion can retrieve noise (too many synonyms = diluted results).
• Language-dependent (different languages have different synonym structures).

HyDE: Hypothetical Document Embeddings

HyDE is a more sophisticated approach. Instead of improving the query, generate a plausible hypothetical answer first, then embed that.

Intuition: If the user asks "How do I reset my password?", the model generates a hypothetical answer: "To reset your password, go to the login page and click 'Forgot Password'. Enter your email, and we'll send you a recovery link." This hypothetical answer contains the exact terms, context, and phrasing that will appear in real documents. Embedding this is more effective than embedding the raw question.

def hyde_retrieval(query, embedding_model):
    """Generate a hypothetical answer, embed it, use that for retrieval."""
    from anthropic import Anthropic

    client = Anthropic()

    # Step 1: Generate hypothetical answer
    system_prompt = """You are a helpful support agent. Given a user question,
write a 1-2 sentence hypothetical answer that might appear in a support document.
Be specific and natural. Do not say "I don't know" or qualify with uncertainty."""

    message = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=200,
        system=system_prompt,
        messages=[
            {
                "role": "user",
                "content": f"User question: {query}\n\nWrite a hypothetical answer:"
            }
        ]
    )

    hypothetical_answer = message.content[0].text.strip()

    # Step 2: Embed the hypothetical answer, not the raw query
    query_embedding = embedding_model.encode(hypothetical_answer, convert_to_numpy=True)

    return query_embedding, hypothetical_answer

# Test
query = "How do I reset my password?"
embedding_model = SentenceTransformer('all-MiniLM-L6-v2')
hyp_embedding, hyp_answer = hyde_retrieval(query, embedding_model)

print(f"Question: {query}")
print(f"Hypothetical answer: {hyp_answer}")
print(f"Embedding used for search: {hyp_embedding[:5]}...")  # First 5 dims

Example output:

Question: How do I reset my password?
Hypothetical answer: To reset your password, visit the login page and click "Forgot Password". We'll send you a recovery link to the email on file. Click the link and create a new password.
Embedding used for search: [0.1234, -0.0567, 0.2341, ...]

Why this works: The hypothetical answer contains phrases like "recovery link," "email on file," "create a new password" that will likely appear in real support documents. Embedding these phrases gives a better starting point for retrieval than embedding "How do I" and "password."

Benefits:

• High-quality embeddings (hypothetical answers are more like real documents).
• Improves recall significantly (often +10-20% on hard queries).
• Works well across languages and domains.
• Particularly effective on domain-specific queries where paraphrasing helps.

Limitations:

• Requires an LLM call (adds latency, ~500ms per query).
• The hypothetical answer might be wrong or hallucinated (if it doesn't match your actual docs, retrieval gets worse, not better).
• Need to validate that generated answers are reasonable (spot-check 20-30 hypothetical answers against your corpus).
• If your documents use very different phrasing than the hypothetical answer, the embedding mismatch can hurt rather than help.

When HyDE helps most:

• Complex, multi-step questions ("What's the process for requesting a budget exception if I've already used my allocation?")
• Domain-specific terminology where synonyms matter
• Queries that would normally retrieve 50-70% recall (room to improve)

When HyDE doesn't help:

• Simple factual questions ("What's the return window?") — raw query already works well
• When your corpus uses very distinctive terminology that the model might not generate
• Very low-latency requirements (<500ms per query)

Merging results from multiple transformed queries

The full query-transformation pipeline transforms the query into multiple variants and retrieves with all of them, then merges:

def multi_query_retrieval(query, retriever, embedding_model):
    """Transform query multiple ways, retrieve with each, merge results."""
    from anthropic import Anthropic

    client = Anthropic()
    queries = [query]  # Start with original

    # Transformation 1: Rewrite for clarity
    rewrite_response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=100,
        system="Extract the core intent of this query, removing conversational filler.",
        messages=[{"role": "user", "content": f"Query: {query}"}]
    )
    rewritten = rewrite_response.content[0].text.strip()
    queries.append(rewritten)

    # Transformation 2: HyDE
    hyde_response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=200,
        system="Write a 1-2 sentence hypothetical answer to this question.",
        messages=[{"role": "user", "content": f"Question: {query}"}]
    )
    hypothetical_answer = hyde_response.content[0].text.strip()
    queries.append(hypothetical_answer)

    # Now retrieve with all three queries
    all_results = []
    seen_chunk_ids = set()

    for q in queries:
        results = retriever.retrieve(q, top_k=10)
        for r in results:
            # Deduplication: only add if we haven't seen this chunk before
            if r['id'] not in seen_chunk_ids:
                all_results.append(r)
                seen_chunk_ids.add(r['id'])

    # Optional: re-rank by frequency (how many queries retrieved this chunk?)
    chunk_freq = {}
    for result in all_results:
        chunk_freq[result['id']] = chunk_freq.get(result['id'], 0) + 1

    all_results.sort(key=lambda r: chunk_freq[r['id']], reverse=True)

    return all_results[:10]  # Return merged top-10

# Usage
query = "How do I fix my password?"
top_10 = multi_query_retrieval(query, retriever, embedding_model)

print(f"Retrieved {len(top_10)} merged results from multiple query transformations")

Merging strategy:

• Deduplication: Don't return the same chunk twice.
• Ranking by frequency: A chunk that appeared in the top-10 of multiple query variants is likely relevant (votes from different angles).
• Alternative: RRF (reciprocal rank fusion): Same fusion as hybrid search. Chunk that ranked high in multiple variants gets a high RRF score.

When each transformation adds value

| Transformation | Latency overhead | Recall gain | Best for | Cost per query | |---|---|---|---|---| | Raw query | ~10ms | Baseline | Clean queries, simple intent | ~0.001 API cost | | Query rewriting | +200ms | +5-10% | Conversational, multi-part questions | ~$0.0001 | | Query expansion | +50ms | +3-7% | Vocabulary-heavy domains, weak embeddings | ~$0.00001 | | HyDE | +500ms | +10-20% | Hard questions, domain mismatch | ~$0.0003 | | Multiple transformations + merge | +750ms | +15-25% | Complex questions, high precision required | ~$0.0005 |

Decision matrix: Which transformations to use?

Your baseline recall:
  <75%? → Use at least one transformation (rewriting or HyDE)
  75-85%? → Use if latency budget allows (HyDE if you have 500ms, rewriting if you have 200ms)
  >85%? → Skip transformations; gains are minimal and latency isn't worth it

Is your query typically:
  Conversational ("I'm having trouble...")? → Rewriting mandatory
  Technical/precise ("error code 502")? → Skip rewriting (no gain)
  Domain-specific intent ("budget exception request")? → HyDE valuable
  Simple factual ("what is X")? → Expansion only; no rewriting or HyDE needed

Practical implementation in production

Most production systems implement a simpler version: query rewriting only (or HyDE only), not all three.

def simple_rag_with_query_rewriting(user_query, retriever, client):
    """Rewrite query once, retrieve with rewritten version."""
    # Rewrite
    rewrite_message = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=100,
        system="Extract the core question from this input.",
        messages=[{"role": "user", "content": user_query}]
    )
    clean_query = rewrite_message.content[0].text.strip()

    # Retrieve once with clean query
    results = retriever.retrieve(clean_query, top_k=10)

    # Generate answer using retrieved context
    answer = generate_answer(user_query, results, client)

    return answer

This adds ~200ms latency but significantly improves retrieval quality for conversational inputs. It's worth adding if:

• Your users ask questions conversationally (not structured).
• You can afford 200ms extra latency.
• Your baseline retrieval is <80% recall (room to improve).

Practice: Measure query transformation's impact

1. Create 30 test queries with known relevant chunks.
2. Measure baseline retrieval (raw query only).
3. Add query rewriting and measure recall improvement.
4. Add HyDE and measure recall improvement.
5. Compare latency vs. recall gain. Is it worth the extra LLM calls?

Common mistake

Applying all query transformations to every query. Rewriting, expansion, and HyDE all add latency. If your baseline retrieval is already >85% recall, transformation adds cost with minimal gain. Use transformations selectively:

• Rewrite if the raw query is conversational.
• Expand if your embedding model is weak on synonyms.
• HyDE only if you need to push past 85% recall.

Also common: Hallucinating in HyDE. If the model generates a plausible-sounding but incorrect hypothetical answer, retrieval will search for the wrong thing. Validate HyDE by spot-checking the generated answers against your actual document content.