Chunking Strategies: Fixed, Semantic, and Structural

Why chunking matters more than you think

Before any document can be retrieved, it has to be split into chunks small enough to:

1. Embed (embedding models can handle ~512 tokens at a time)
2. Retrieve (you want specific, relevant pieces, not entire books)
3. Fit in the context window alongside other chunks

Chunking is invisible. No one celebrates "great chunking strategy" or blames "we have a chunking problem." But chunking determines a hard ceiling on retrieval quality: no chunking strategy can retrieve what it has already split apart.

If your document says:

"Returns are allowed within 30 days. This applies to unopened items.
For opened items, we allow 14 days."

And your chunker splits it as:

Chunk 1: "Returns are allowed within 30 days. This applies to unopened items."
Chunk 2: "For opened items, we allow 14 days."

Then a query like "can I return an opened item?" will retrieve chunk 2 (high relevance). But a query like "what's your return policy?" will retrieve chunk 1, missing the "opened items" exception. Both chunks contain the answer, but neither chunk alone is complete. The split made completeness impossible.

Strategy 1: Fixed-size chunking with overlap

Fixed-size chunking splits documents every N tokens (or characters), with a configurable overlap to provide context.

Advantages:

• Simple to implement
• Predictable cost (all chunks are the same size)
• Fast to compute

Disadvantages:

• Can cut sentences or tables in half mid-token
• Produces unnatural breaks
• Misses the semantic structure of documents

Best for: Quick prototypes, when speed matters more than quality

def fixed_size_chunking(text: str, chunk_size: int = 500, overlap: int = 100) -> list[str]:
    """
    Split text into fixed-size chunks with overlap.

    Args:
        text: The document to chunk
        chunk_size: Target tokens per chunk (approximate; we count by characters, 1 token ≈ 4 chars)
        overlap: How many tokens to overlap between chunks

    Returns:
        List of chunk strings
    """

    # Convert tokens to approximate character count (1 token ≈ 4 characters)
    chunk_char_size = chunk_size * 4
    overlap_char_size = overlap * 4

    chunks = []
    start = 0

    while start < len(text):
        # Take a chunk
        end = min(start + chunk_char_size, len(text))
        chunk = text[start:end]

        chunks.append(chunk)

        # Move start forward, but leave overlap
        if end < len(text):
            start = end - overlap_char_size
        else:
            # Last chunk; we're done
            start = end

    return chunks

# Example
document = """
Our return policy is straightforward. We offer returns within 30 days for unopened items.
If the item has been opened or used, we allow returns within 14 days from the date of purchase.

Condition of the item:
- Unopened items: Full refund minus original shipping cost
- Opened but unused: Full refund minus 20% restocking fee
- Used items: 50% refund

To initiate a return, contact our support team at [email protected].
"""

chunks = fixed_size_chunking(document, chunk_size=200, overlap=50)
for i, chunk in enumerate(chunks):
    print(f"Chunk {i} ({len(chunk)} chars, ~{len(chunk)//4} tokens):\n{chunk[:80]}...\n")

When to use: Prototyping, performance testing, or when your documents are short enough that chunking doesn't matter much.

Strategy 2: Semantic chunking

Semantic chunking splits documents at points where the meaning shifts, using embedding similarity to detect boundaries. If two adjacent sentences have low similarity, there's a topic boundary.

Advantages:

• Keeps semantically related text together
• Chunk boundaries are natural (topics, paragraphs)
• Higher retrieval quality

Disadvantages:

• Slower (requires embedding every sentence)
• Chunks are variable-sized (harder to predict token usage)
• More complex to implement

Best for: When you have time to invest in better retrieval quality

Here's a practical implementation:

from sentence_transformers import SentenceTransformer
import numpy as np

def semantic_chunking(text: str, threshold: float = 0.5, min_chunk_size: int = 100) -> list[str]:
    """
    Split text where semantic meaning shifts (using embedding similarity).

    Args:
        text: Document to chunk
        threshold: Similarity threshold below which we split (0-1, lower = more splits)
        min_chunk_size: Minimum characters per chunk to avoid tiny fragments

    Returns:
        List of chunk strings
    """

    # Load embedding model
    model = SentenceTransformer('all-MiniLM-L6-v2')

    # Split into sentences (naive; a production system would use nltk or spacy)
    sentences = [s.strip() for s in text.replace(".\n", ".|").split("|") if s.strip()]

    if len(sentences) <= 1:
        return [text]

    # Embed all sentences
    embeddings = model.encode(sentences, convert_to_tensor=False)

    # Compute similarity between consecutive sentences
    chunks = []
    current_chunk = sentences[0]

    for i in range(1, len(sentences)):
        # Cosine similarity between sentence i-1 and i
        sim = np.dot(embeddings[i - 1], embeddings[i]) / (
            np.linalg.norm(embeddings[i - 1]) * np.linalg.norm(embeddings[i])
        )

        # If similarity is high, add to current chunk
        # If similarity is low, start a new chunk (topic boundary)
        if sim > threshold and len(current_chunk) < 2000:  # 2000 char limit per chunk
            current_chunk += " " + sentences[i]
        else:
            # Save current chunk and start new one
            if len(current_chunk) >= min_chunk_size:
                chunks.append(current_chunk)
            current_chunk = sentences[i]

    # Don't forget the last chunk
    if len(current_chunk) >= min_chunk_size:
        chunks.append(current_chunk)

    return chunks

# Example
document = """
Returns are allowed within 30 days for unopened items. If opened, we allow 14 days.

Shipping costs $5 or is free over $50. We ship within 2 business days.

Our warranty covers parts and labor for 1 year. Extended warranty available for $19.99.
"""

chunks = semantic_chunking(document, threshold=0.6)
for i, chunk in enumerate(chunks):
    print(f"Chunk {i}:\n{chunk[:100]}...\n")

# Output likely splits at:
# Chunk 0: "Returns..." (topic: returns)
# Chunk 1: "Shipping..." (topic: shipping; semantic boundary detected)
# Chunk 2: "Our warranty..." (topic: warranty; semantic boundary detected)

Cost-quality tradeoff: This requires embedding every sentence, which costs more upfront. But it produces better chunks, so retrieval quality improves (fewer false positives because chunks are more semantically coherent). For large corpora, the embedding cost is amortized over many queries, making it worthwhile.

Strategy 3: Structural chunking

Structural chunking respects the document's own structure: headings, paragraphs, list items, code blocks, table rows. Instead of splitting at arbitrary character boundaries, you split along the document's natural units.

Advantages:

• Most semantically meaningful (paragraphs, sections are intentional units)
• Works well for documents with explicit structure (markdown, HTML, code)
• Highest retrieval quality for structured docs

Disadvantages:

• Requires parsing document structure
• Chunks are variable-sized
• Doesn't work for unstructured prose (PDFs without structure)

Best for: Code repositories, documentation, markdown wikis, HTML pages

def structural_chunking_markdown(markdown_text: str) -> list[str]:
    """
    Split markdown by headers, keeping content under each header as a chunk.
    """

    chunks = []
    current_chunk = ""
    lines = markdown_text.split("\n")

    for line in lines:
        # Check if this is a header
        if line.startswith("#"):
            # Save previous chunk if it has content
            if current_chunk.strip():
                chunks.append(current_chunk.strip())

            # Start new chunk with the header
            current_chunk = line
        else:
            # Add to current chunk
            current_chunk += "\n" + line

    # Don't forget the last chunk
    if current_chunk.strip():
        chunks.append(current_chunk.strip())

    return chunks

def structural_chunking_code(code_text: str) -> list[str]:
    """
    Split Python code by function and class definitions.
    """

    chunks = []
    current_chunk = ""
    lines = code_text.split("\n")

    for line in lines:
        # Check if this is a function or class definition
        if line.startswith("def ") or line.startswith("class "):
            # Save previous chunk if it has content
            if current_chunk.strip():
                chunks.append(current_chunk.strip())

            # Start new chunk with the definition
            current_chunk = line
        else:
            # Add to current chunk
            current_chunk += "\n" + line

    # Don't forget the last chunk
    if current_chunk.strip():
        chunks.append(current_chunk.strip())

    return chunks

# Example: markdown
markdown = """
# Return Policy

We accept returns within 30 days.

## Unopened Items

Full refund minus shipping.

## Opened Items

14 days, 20% restocking fee.

# Shipping Policy

Free over $50, otherwise $5.
"""

chunks = structural_chunking_markdown(markdown)
for i, chunk in enumerate(chunks):
    print(f"Chunk {i}:\n{chunk[:80]}...\n")

Production note: Use a library like langchain.text_splitter.RecursiveCharacterTextSplitter (which chunks hierarchically—first by markdown structure, then by paragraphs, then by sentences if needed) rather than rolling your own. It handles edge cases better.

Comparing the three strategies

Chunk size selection heuristics

How many tokens should a chunk be? The answer depends on your use case:

• Small chunks (100-200 tokens): High precision (retrieved chunks are very relevant), but incomplete (the model has to synthesize across chunks). Good for questions with single, specific answers ("What's the API rate limit?"). Bad for holistic questions ("Explain our entire return policy").

• Medium chunks (300-600 tokens): The sweet spot for most use cases. Balances relevance and context. Most production RAG systems use this range.

• Large chunks (800-2000 tokens): Low precision (retrieve unrelated sentences), but high context (the model has the full paragraph). Good for questions that need broader context. Bad for high-volume retrieval (costs more tokens per query).

Rule of thumb: Start with 300-500 tokens. If retrieval returns "too much noise," reduce to 200-300. If retrieved chunks feel incomplete, increase to 600-800. Tune based on your actual queries and evaluation metrics, not theory.

Handled edge cases: overlap, metadata, and hierarchical chunking

Overlap: Include previous chunk's last N tokens as the start of the next chunk. This provides context and prevents important information from falling on a chunk boundary.

# E.g., chunk 1 ends with "...and 14 days for opened items."
# Chunk 2 starts with "...opened items. You must contact support..."
# The overlap ensures "opened items" appears in both chunks.

Metadata: Attach source info to each chunk: original document, section/heading, page number, date last updated.

chunk_with_metadata = {
    "text": "Return policy text...",
    "source": "policy.pdf",
    "section": "Returns",
    "page": 1,
    "date_updated": "2024-01-15"
}

Hierarchical chunking: Create chunks at multiple granularities. A query might retrieve a small chunk (specific), but the chunk also carries a reference to its parent section (broader context).

chunk_hierarchy = {
    "document": "policy.pdf",
    "section": "Return Policy",
    "subsection": "Unopened Items",
    "chunk": "Full refund minus original shipping cost",
}

Worked example: chunking a customer support knowledge base

You have 500 support articles (markdown format) averaging 3000 tokens each. You need to choose a chunking strategy and tune it for typical customer queries.

from sentence_transformers import SentenceTransformer
import numpy as np

# Load your docs
documents = [
    # Each doc is a markdown file with structure
    """
    # Billing Questions

    ## How do I update my payment method?

    Go to Settings > Billing > Payment Methods...

    ## Can I change my billing cycle?

    Yes, you can change from monthly to annual...
    """
]

# Strategy 1: Try fixed-size
print("Strategy 1: Fixed-size (500 tokens, 100 overlap)")
fixed_chunks = fixed_size_chunking(documents[0], chunk_size=500, overlap=100)
print(f"  Created {len(fixed_chunks)} chunks")
print(f"  Sample chunk: {fixed_chunks[0][:80]}...\n")

# Strategy 2: Try semantic
print("Strategy 2: Semantic (threshold=0.5)")
semantic_chunks = semantic_chunking(documents[0], threshold=0.5)
print(f"  Created {len(semantic_chunks)} chunks")
print(f"  Sample chunk: {semantic_chunks[0][:80]}...\n")

# Strategy 3: Try structural
print("Strategy 3: Structural (markdown headers)")
structural_chunks = structural_chunking_markdown(documents[0])
print(f"  Created {len(structural_chunks)} chunks")
print(f"  Sample chunk: {structural_chunks[0][:80]}...\n")

# Evaluate: run a few test queries through each strategy
test_queries = [
    "How do I update my payment method?",
    "Can I change my billing cycle?",
    "What payment methods do you accept?",
]

model = SentenceTransformer('all-MiniLM-L6-v2')

for strategy_name, chunks in [("Fixed", fixed_chunks), ("Semantic", semantic_chunks), ("Structural", structural_chunks)]:
    print(f"\n{strategy_name} Strategy:")

    correct = 0
    for query in test_queries:
        query_emb = model.encode(query)
        chunk_embs = [model.encode(chunk) for chunk in chunks]
        sims = [np.dot(query_emb, ce) / (np.linalg.norm(query_emb) * np.linalg.norm(ce))
                for ce in chunk_embs]
        top_chunk_idx = np.argmax(sims)
        top_sim = sims[top_chunk_idx]

        # Rough check: does the top chunk contain a word from the query?
        if any(word.lower() in chunks[top_chunk_idx].lower() for word in query.split()):
            correct += 1

    print(f"  Heuristic accuracy: {correct}/{len(test_queries)}")

# Result: Structural likely wins for markdown docs because it respects the doc's own structure.

Common mistake

Thinking bigger chunks are safer because they provide more context. In fact, larger chunks often degrade retrieval quality because you're mixing signal and noise. A 2000-token chunk might contain 3 answers to 3 different questions; if a query matches only one question, the other 2000 tokens are noise the model has to filter out.

Also common: not measuring chunking impact. Teams spend weeks optimizing the embedding model, then use the default chunking. But chunking is the secret lever—better chunking often beats a better embedding model.

Test on your actual queries and measure retrieval accuracy (does the top-1 retrieved chunk actually contain the answer?). Tune chunk size and strategy based on measurement, not theory.