Embeddings: How Meaning Becomes Vectors

What an embedding actually is

An embedding is a list of numbers—a vector—that represents the meaning of a piece of text. A sentence like "How do I cancel my subscription?" gets converted by an embedding model into a vector like:

[0.2341, -0.1523, 0.8904, -0.4521, ..., 0.1234]

The vector has a fixed number of dimensions, typically 384, 768, or 1536, depending on the embedding model. Each dimension is meaningless on its own—you can't say "dimension 42 represents 'negativity'" or anything that simple. Instead, the entire vector collectively represents the semantic meaning of the input text.

How embedding models are trained: An embedding model is a neural network, separate from the LLM, trained specifically to map text to vectors. It's trained on pairs of semantically similar texts (e.g., "refund policy" and "money back guarantee") and learns to put their vectors close together in space. Unlike LLMs, which generate text token-by-token, embedding models output a single fixed-size vector per input.

Why embeddings are cheaper and faster than LLMs: Embedding models are smaller and simpler than large language models. OpenAI's embedding models run in milliseconds. They cost ~$0.02 per million tokens, while LLM inference costs 10-100x more. This is why RAG systems compute embeddings at index time (embedding all your documents once, when you ingest them) rather than computing them on-the-fly for every query.

Cosine similarity: measuring distance between vectors

When you embed two pieces of text, you get two vectors. To know if they're semantically similar, you measure the distance between them. The most common metric is cosine similarity, which measures the angle between two vectors.

The intuition: Imagine two arrows pointing in slightly different directions from the origin. If they point almost the same direction (angle close to 0°), cosine similarity is close to 1 (maximum similarity). If they point in opposite directions (angle = 180°), similarity is -1 (maximum dissimilarity). If they're perpendicular (angle = 90°), similarity is 0 (neutral).

The math:

cosine_similarity(v1, v2) = (v1 · v2) / (||v1|| × ||v2||)

Where v1 · v2 is the dot product and ||v|| is the magnitude (length) of the vector.

Why cosine similarity works for embeddings: Embedding models are trained so that semantically similar texts produce vectors with small angles between them. A query about "canceling a subscription" and a document about "ending my plan" will have a high cosine similarity (say, 0.87), even though they share no exact words. A query about "order tracking" and that same cancellation document will have low similarity (say, 0.23).

Here's a real example you can run:

from sentence_transformers import SentenceTransformer
import numpy as np

# Load a pre-trained embedding model (one-time download, ~500MB)
model = SentenceTransformer('all-MiniLM-L6-v2')

# Two sentences that mean the same thing
text_a = "How do I cancel my subscription?"
text_b = "I want to end my subscription"
text_c = "What is your shipping policy?"

# Embed them
embedding_a = model.encode(text_a)  # Returns array of 384 floats
embedding_b = model.encode(text_b)
embedding_c = model.encode(text_c)

print(f"Embedding shape: {embedding_a.shape}")  # (384,)
print(f"First 10 values of embedding_a: {embedding_a[:10]}")

# Compute cosine similarity
def cosine_similarity(v1, v2):
    """Cosine similarity between two vectors."""
    return np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2))

sim_a_b = cosine_similarity(embedding_a, embedding_b)
sim_a_c = cosine_similarity(embedding_a, embedding_c)

print(f"Similarity (cancel vs end): {sim_a_b:.3f}")  # Output: ~0.95
print(f"Similarity (cancel vs shipping): {sim_a_c:.3f}")  # Output: ~0.12

What this output tells you: Even though "cancel" and "end" are different words, they embed to very similar vectors (0.95). This is exactly what makes RAG work: a user can ask "How do I end my subscription?" and the system will retrieve documents tagged with "cancel subscription" because the vectors are close.

Dense vs. sparse vectors

There are two styles of embeddings:

Dense vectors (what we've been discussing):

• Every dimension has a non-zero value
• 384-1536 dimensions typical
• Capture semantic similarity well
• Used by all modern RAG systems
• Cheaper per query (smaller vectors)

Sparse vectors (older, mostly deprecated):

• Most dimensions are zero; only a few are non-zero
• Can have thousands of dimensions
• Each non-zero dimension represents a word or concept
• Better for exact-match retrieval (if you embed the words in a document, the same words in a query have high similarity)
• Less good at paraphrasing (if the query uses synonyms, sparse vectors miss them)

For RAG, always use dense vectors. They're the standard in 2024+ because they're faster, cheaper, and better at capturing meaning across paraphrases.

Why embeddings are the foundation of RAG

RAG depends on embeddings for one core operation: retrieve documents relevant to a query. Without embeddings, you'd be limited to exact-keyword search (BM25), which fails when the query and document use different vocabulary.

Example: why keyword search fails where embeddings succeed

A knowledge base has this document:

"We offer a 30-day return window. Items must be unopened."

Four customer queries:

1. "Can I return this if I opened it?" — No exact overlap with "unopened", but semantically asking the same thing
2. "What's your refund policy?" — "refund" and "return" are synonyms, keyword search might miss it
3. "Do you have a 30-day money-back guarantee?" — "money-back" is a paraphrase of "return"
4. "Is there a timeframe for sending it back?" — "sending it back" means "return", keyword search won't find it

Keyword search (BM25): Only query 3 matches directly (has "30-day"). Queries 1, 2, and 4 miss.

Embedding search: All four queries embed to vectors close to the document's vector, because embedding models understand these are all about the same topic.

This is why embeddings moved RAG from "it only works if the query uses the exact words in documents" to "it works across paraphrases, synonyms, and different phrasings."

From embeddings to retrieval

Once you've embedded your documents and your query, retrieval is straightforward:

1. Index time: Embed all documents once, store the vectors in a database
2. Query time: Embed the user's query
3. Search: Find the K documents whose vectors are closest (highest cosine similarity)
4. Return: Pass those documents to the model

This is why vector databases exist: with a million documents, computing similarity to every single one would be slow. Vector databases use approximate nearest-neighbor (ANN) algorithms to find the closest matches in milliseconds without comparing to every vector.

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

model = SentenceTransformer('all-MiniLM-L6-v2')

# Your knowledge base (simplified)
documents = [
    "Return policy: 30 days unopened, 14 days if opened.",
    "Shipping costs: $5 or free over $50.",
    "Warranty: 1 year parts and labor.",
    "How to contact support: email [email protected] or call 1-800-HELP",
]

# Index: embed all documents
document_embeddings = [model.encode(doc) for doc in documents]

# Query
user_query = "Can I send it back if I opened the package?"
query_embedding = model.encode(user_query)

# Compute similarity to all documents
similarities = cosine_similarity([query_embedding], document_embeddings)[0]

# Find top-3
top_k = 3
top_indices = np.argsort(similarities)[-top_k:][::-1]

print("Top results for query: \"{}\"".format(user_query))
for idx in top_indices:
    print(f"  Score: {similarities[idx]:.3f} | {documents[idx]}")

# Output:
# Top results for query: "Can I send it back if I opened the package?"
#   Score: 0.897 | Return policy: 30 days unopened, 14 days if opened.
#   Score: 0.234 | Shipping costs: $5 or free over $50.
#   Score: 0.156 | Warranty: 1 year parts and labor.

The first result is almost certainly what the user was looking for. The embedding model understood that "send it back" and "opened the package" map to the return policy, even though the query used different words.

Practical considerations: embedding model choice

Different embedding models vary in quality, speed, and cost. Here's how to think about them:

| Model | Dimensions | Speed | Cost | Best for | Domain | |---|---|---|---|---|---| | OpenAI text-embedding-3-small | 512 | ~5ms | $0.02/1M | General web, questions | General | | Cohere embed-english-v3.0 | 1024 | ~10ms | $0.025/1M | Dense, semantic questions | General | | Anthropic Claude embeddings | 1024 | ~10ms | $0.10/1M | Cross-lingual, nuance | General | | all-MiniLM-L6-v2 (open-source) | 384 | ~1ms | Free | Local, development | General | | BGE-Large (open-source) | 1024 | ~3ms | Free | Strong quality/speed ratio | General |

Quick decision guide:

• Building a prototype? Use all-MiniLM-L6-v2 (open-source, local, free)
• Production, general domain? Use OpenAI or Cohere (proven, fast, cheap)
• Domain-specific (legal, medical, science)? Use a domain-specific model from Hugging Face, or fine-tune a base model
• Multi-language? Use Cohere or a multilingual open-source model
• Maximum quality? Use Cohere or Claude embeddings (slower, costlier, better at nuance)

Worked example: evaluating embedding quality

You're building a knowledge base for product documentation. You have 500 documents and want to test whether your embedding model can retrieve the right document for typical user questions.

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')

# Sample from your knowledge base
documents = [
    "How to reset your password: Go to Settings > Account > Reset Password",
    "We offer a 30-day money-back guarantee",
    "API rate limit: 10,000 requests per hour",
    # ... 497 more documents
]

# Test queries (labeled with the correct document index)
test_queries = [
    ("I forgot my password", 0),  # Should retrieve doc 0
    ("Do you have a refund policy?", 1),  # Should retrieve doc 1
    ("What's the API limit?", 2),  # Should retrieve doc 2
]

document_embeddings = [model.encode(doc) for doc in documents]

correct = 0
for query, expected_doc_idx in test_queries:
    query_embedding = model.encode(query)
    similarities = cosine_similarity([query_embedding], document_embeddings)[0]
    retrieved_doc_idx = np.argmax(similarities)  # Top-1

    if retrieved_doc_idx == expected_doc_idx:
        correct += 1
        print(f"✓ \"{query}\" → correct doc (score: {similarities[retrieved_doc_idx]:.3f})")
    else:
        print(f"✗ \"{query}\" → wrong doc (score: {similarities[retrieved_doc_idx]:.3f})")

accuracy = correct / len(test_queries)
print(f"\nTop-1 accuracy: {accuracy:.0%}")

If accuracy is below 80%, your embedding model isn't capturing the domain well. You might need to:

1. Switch to a domain-specific embedding model
2. Fine-tune the model on your domain
3. Improve your documents (ensure they actually contain the answer)
4. Increase the number of retrieved chunks (retrieve top-5 instead of top-1)

Common mistake

Assuming that higher-dimensional embeddings are always better. More dimensions capture more information, but they also:

• Cost more (larger vectors = more storage and slower search)
• Are slower to compute and compare
• Can suffer from the curse of dimensionality (in very high dimensions, all points look equidistant)

For most RAG applications, 384 or 768 dimensions is enough. 1536 dimensions is overkill unless you're dealing with very complex, long documents where extra nuance matters. The embedding model choice (which one) matters much more than the dimensionality.

Also common: treating embedding similarity as ground truth. A high cosine similarity (0.9) doesn't guarantee the document is relevant to the query—it just means the vectors are close. Always validate on your actual use cases, not just similarity scores.