Vector Databases and Embeddings for Agents

What an embedding actually is

An embedding is a list of numbers (a vector) produced by a model to represent the meaning of a piece of text, an image, or other data. The embedding is produced by an embedding model -- a neural network trained to turn text into vectors in such a way that similar texts end up with similar vectors.

Key insight: embeddings are not created by the LLM itself. They're created by a separate, smaller model trained specifically for this task (models like OpenAI's text-embedding-3-small, Anthropic's embedding models, or open-source alternatives like all-MiniLM-L6-v2). This is important because embedding models are often cheaper and faster than large LLMs; you can embed millions of documents without the cost of running inference through a large language model.

How similarity is measured

Texts with similar meaning end up with vectors that are numerically close together. The most common similarity metric is cosine similarity, which measures the angle between two vectors:

import numpy as np

# Example: two embeddings (simplified, real ones are 300-3000 dimensions)
doc_a_embedding = np.array([0.8, 0.2, 0.1])  # "book is interesting"
doc_b_embedding = np.array([0.7, 0.3, 0.15]) # "novel is engaging"
doc_c_embedding = np.array([0.1, 0.9, 0.8])  # "car is fast"

def cosine_similarity(v1, v2):
    """Compute cosine similarity between two vectors."""
    return np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2))

# Similar documents have similarity close to 1
print(f"A vs B (both about books): {cosine_similarity(doc_a_embedding, doc_b_embedding):.3f}")
# Output: ~0.975

# Dissimilar documents have lower similarity
print(f"A vs C (book vs car): {cosine_similarity(doc_a_embedding, doc_c_embedding):.3f}")
# Output: ~0.12

This is what lets a system search by meaning rather than by exact keyword match: a query about "ending a subscription" can match a document about "canceling my plan" even though they share no exact words. Both embed to vectors that are close together in embedding space.

What a vector database adds

You could, in principle, store embeddings in a plain list and compare a query against every single one:

def brute_force_search(query_embedding, all_embeddings, top_k=5):
    """Naive approach: compute similarity to every document."""
    similarities = []
    for doc_embedding in all_embeddings:
        sim = cosine_similarity(query_embedding, doc_embedding)
        similarities.append(sim)

    # Find top K
    top_indices = np.argsort(similarities)[-top_k:][::-1]
    return top_indices

This works for a few hundred documents (matching one query against 500 documents is microseconds). But for a million documents, computing cosine similarity against all million embeddings for every query becomes slow and expensive.

A vector database solves this using approximate nearest-neighbor (ANN) indexing. Instead of comparing the query against every single document, the index structure (commonly using algorithms like HNSW, IVF, or LSH) organizes embeddings in a tree-like or hash-based structure that lets you skip most of the documents and return the closest matches in milliseconds.

The tradeoff: ANN indexing trades perfect accuracy for speed. You're guaranteed to find nearby vectors, but not guaranteed to find the absolute-closest vectors. In practice, this is fine; for most agent tasks, the top-5 similar documents are "good enough," and missing one document that's 99.9% similar in favor of one that's 99.8% similar doesn't matter.

Here's what a vector database call looks like:

from pinecone import Pinecone

# Initialize and query
pc = Pinecone(api_key="your-api-key")
index = pc.Index("my-docs")

query_embedding = embedding_model.encode("what is the return policy?")
results = index.query(
    vector=query_embedding,
    top_k=5,
    include_metadata=True
)

# Returns top 5 similar documents in ~10ms, even if index has 1M+ documents
for result in results['matches']:
    print(f"Score: {result['score']}, Document: {result['metadata']['text']}")

Without vector databases, that same query on a million documents would take seconds.

Popular options, compared

Choosing between them

• No infrastructure team? → Pinecone (fully managed)
• Local development / prototyping? → Chroma (simple, embeddable)
• Need keyword + vector search? → Weaviate (hybrid capabilities)
• Already use Postgres? → pgvector (minimal new dependencies)
• Massive scale (100M+), custom ops? → Milvus (highly scalable, self-hosted)

How agents actually use this

Two common agent uses build directly on vector search:

1. Retrieval-Augmented Generation (RAG): A query is embedded and matched against a document store to pull relevant context into a prompt (covered in the next lesson). Agent asks a question, you embed it, search the vector database, and feed the results into the language model's context.

2. Long-term memory: Past interactions or facts are embedded and stored so an agent can later retrieve "have I dealt with something like this before?" rather than starting from a blank context every run. A customer service agent could remember "this customer was already issued a refund for that product last month."

Here's a practical example of an agent using vector search for memory:

def retrieve_agent_memory(query: str, memory_index):
    """Retrieve relevant past interactions for context."""
    query_embedding = embedding_model.encode(query)

    # Search past interactions
    results = memory_index.query(
        vector=query_embedding,
        top_k=3,
        filter={"agent_id": current_agent_id}  # Filter to this agent's memory
    )

    # Format for inclusion in prompt
    memory_context = "Recent relevant interactions:\n"
    for result in results['matches']:
        memory_context += f"- {result['metadata']['summary']}\n"

    return memory_context

# In agent loop:
current_query = user_input
memory = retrieve_agent_memory(current_query, memory_db)
system_prompt = f"""You are a customer service agent.

{memory}

Now handle this request: {current_query}"""

This lets an agent build context from its own history, not starting every conversation from scratch.

Vector database performance characteristics

Different vector databases trade off query speed, memory overhead, update latency, and implementation complexity. Here's how they compare at scale (illustrative performance estimates for 1M embeddings):

| Database | Query latency (p95) | Memory per embedding | Update latency | ANN accuracy | Best for | |---|---|---|---|---|---| | In-memory (FAISS) | 5-20ms | 4KB-8KB | Offline only | 98-99% | Batch/research, high speed | | Chroma (sqlite backend) | 50-200ms | 1KB-2KB (indexed) | 10-100ms | 95-98% | Development, local agents | | Pinecone (managed) | 100-300ms | Opaque | 1-5s (async) | 95-97% | Production, multi-tenant | | Weaviate (self-hosted) | 80-250ms | 2KB-4KB | 50-500ms | 94-97% | Hybrid search, control | | pgvector (Postgres) | 200-500ms | 0.5KB-1KB | 100-200ms | 92-95% | Already have Postgres | | Milvus (self-hosted) | 30-150ms | 2KB-5KB | 10-100ms | 96-98% | High-volume, custom ops |

Key observations:

• In-memory is fastest but requires reloading for updates
• Managed services add latency but hide infrastructure complexity
• Hybrid databases (Weaviate) are slower than pure vector databases but add keyword search
• Update latency varies wildly depending on whether you batch updates or insert one-at-a-time

For agents, the most important column is "Query latency": most agent loops can tolerate 100-300ms per retrieval, but not 1-2 second retrieval times. This rules out certain databases for high-frequency agents.

Seeing similarity as distance, concretely

It helps to picture embeddings as points scattered in space, where distance stands in for difference in meaning (the diagram above shows a simplified two-dimensional version -- real embeddings typically have 384-3072 dimensions, too many to draw, but the same distance logic applies).

A query like "how do I cancel my plan" embeds to a point in that space. Nearby points might be documents about:

• "ending a subscription" (high similarity)
• "downgrading your account" (moderate similarity)
• "contact support to cancel" (high similarity)

Far-away points might be:

• "resetting your password" (low similarity, different topic)
• "payment methods" (low similarity, different intent)

A vector database's job is finding the nearest points to a query's embedding, fast, out of potentially millions of candidates. This is purely a distance/geometry problem, not a language understanding problem.

A practical worked example: product recommendation agent

An e-commerce agent helps customers find products. Its flow:

1. User asks: "I'm looking for a lightweight, waterproof backpack"
2. Agent embeds the query → vector space point
3. Vector database searches the product catalog's embeddings
4. Top 5 products returned: [Backpack A (lightweight, waterproof), Backpack B (lightweight, not waterproof), Backpack C (waterproof, not lightweight), ...]
5. Agent reads the top results and ranks them by relevance to the query

The vector database finds "similar" products by embedding distance. Whether those similar products are actually relevant to the user depends on:

• Embedding model quality: A good model understands "lightweight" and "waterproof" as features; a poor one treats them as random tokens
• Product metadata completeness: If the catalog embeds only product titles ("Backpack A"), it misses feature details. If it embeds descriptions ("lightweight ripstop nylon, waterproof..."), the search is better.
• Query specificity: "Backpack" is vague and returns many results. "Lightweight waterproof hiking backpack under $100" is specific and returns more relevant results.

The vector database itself is neutral; it's a fast distance calculator. The retrieval quality depends on these factors upstream.

Edge cases and failure modes in vector retrieval

Cold-start problem: A new agent starts with an empty vector database. The first few queries will have trivial performance because there's nothing to retrieve. Some systems mitigate this with a "bootstrap" phase: pre-load a corpus of common documents before deploying the agent. But if the corpus doesn't cover the user's actual domain, performance remains poor until enough interactions are logged.

Semantic drift: A vector database trained on one domain (e.g., tech documentation) might perform poorly when queried about a different domain (e.g., HR policies). The embedding space is optimized for the training domain; out-of-domain queries can land in low-density regions of the embedding space where nearest-neighbor retrieval is unreliable. Solution: either retrain embeddings for the new domain or use domain-specific embedding models.

Dimensionality vs. search quality: Smaller embedding dimensions (e.g., 256D) are faster and cheaper to store and search. Larger dimensions (e.g., 1536D or 3072D) capture more nuance but require more storage. The tradeoff is non-linear: going from 256D to 384D improves retrieval quality by ~10%; going from 384D to 1536D improves by another ~8%. Most production systems use 384-768D as a sweet spot.

Sparsity in the corpus: If your vector database has 1M embeddings but they cluster in a few dense regions, retrieval quality suffers. Example: a customer support agent trained on 90% "billing" issues and 10% "technical" issues will retrieve mostly billing docs even for technical questions, because the embedding space is weighted toward billing. Solution: either balance the corpus or use density-aware retrieval (retrieve from multiple density regions, not just the nearest).

Here's a defensive wrapper around vector search that handles these cases:

def robust_vector_search(query, vector_db, fallback_strategy="keyword"):
    """Vector search with fallback and quality checks."""

    # Step 1: Normal vector search
    results = vector_db.search(query, top_k=5)

    # Step 2: Quality check - did we get anything relevant?
    if not results or results[0]['score'] < 0.5:
        # Low confidence: either the query is out-of-domain or corpus is sparse
        if fallback_strategy == "keyword":
            # Fall back to keyword search
            results = vector_db.keyword_search(query, top_k=5)
        elif fallback_strategy == "expansion":
            # Expand query and retry
            expanded_query = expand_query(query)  # Add synonyms
            results = vector_db.search(expanded_query, top_k=5)
        else:
            # Return empty; let caller handle no results
            return []

    # Step 3: Diversity check - are all results too similar?
    scores = [r['score'] for r in results]
    if max(scores) - min(scores) < 0.05:
        # All results are equidistant; corpus might be sparse or homogeneous
        # Return top-K but flag uncertainty
        return results, {'confidence': 'low', 'reason': 'homogeneous_corpus'}

    return results, {'confidence': 'high'}

Case study: Customer support agent with vector database retrieval failure

Scenario: An e-commerce company deployed a support agent backed by a vector database of FAQ documents. For the first month, it worked well -- agents answered 70% of questions without human escalation. Then performance degraded.

Root cause: The FAQ corpus was added once at launch. Over three months, the company launched new products, changed policies, and answered new customer question types. But the vector database wasn't updated. When customers asked about the new product ("how do I set up the new X100 device?"), the vector DB returned results for old products because the corpus was stale.

Secondary problem: The new product had similar keywords to old products ("setup" appears for all products). The embedding space wasn't sophisticated enough to distinguish "setup for X100" from "setup for old X50." Retrieval was returning wrong products.

Solution approach:

1. Establish a refresh cadence: Update the vector database weekly with new FAQ docs, policy changes, and archived support tickets (anonymized).
2. Use versioning: Keep separate indices for "current" (this month's docs) and "archive" (older docs). Search current first; if results are sparse, include archive.
3. Switch embedding model: Replace the general embedding model with a domain-specific one trained on support tickets and FAQs. The specialized model understands "what is the difference between our products X100 vs X50?" better than a general model.
4. Add metadata filtering: Before embedding search, allow filtering by product type, issue category, or date range. This reduces the search space and improves relevance.

After these changes:

• Escalation rate dropped to 15% (from 30% during degradation)
• Query latency stayed ~100ms even with more docs (because filtering reduced search space)
• Customer satisfaction on automated answers improved by 20%

Common mistake

Treating vector similarity as if it were the same thing as correctness or relevance. A vector database returns the closest matches by embedding distance, not necessarily the right ones. Problems that hurt retrieval quality:

• Wrong embedding model for your domain: A model trained on general web text might not understand domain-specific jargon. Use domain-specific embedding models when possible.
• Poor chunking strategy: If documents are chunked wrong (splitting relevant facts apart), no embedding model can retrieve what's been split apart.
• Ambiguous queries: "Apple" could mean the fruit or the company. The embedding might be in the middle of fruit/company space, retrieving neither cleanly.
• Out-of-domain documents: If your corpus includes documents very different from your typical query, similarity matching can pull irrelevant results confidently.

Retrieval quality depends on chunking strategy and embedding model choice at least as much as on which vector database you pick. The database is just the search engine; the data fed into it is what determines the quality.