Chunking strategies for RAG: fixed, recursive, structure-aware, semantic and more

Key points

• A chunk is both the unit you search and the unit the model reads. Those two jobs want different sizes, which is why parent-child retrieval exists.
• Recursive splitting on paragraphs, then lines, then words is a strong default; split on headings, HTML sections, functions or table rows first when the document has them.
• Overlap patches bad boundaries; it does not fix them, and it stores text twice.
• Semantic chunking costs one embedding per sentence at index time and needs a tuned threshold. It earns that on long text without headings.
• Contextual chunk headers, contextual retrieval and late chunking all solve one problem: a chunk that lost its document's context.
• Choose size and overlap with retrieval metrics on your own questions, and label answer spans rather than chunk IDs so every variant can be scored.

Why chunking decides retrieval quality

A RAG system does not search documents. It searches chunks: passages cut from documents, each embedded as one vector and returned as one unit. The chunker therefore decides what can be found and what the model gets to read. If an answer is split across two chunks, neither may rank well. If a chunk says "the limit is raised to 50 requests per minute" without saying which product or plan, it matches the wrong questions. RAG explained shows where chunking sits in the whole pipeline.

Three constraints pull on chunk size:

• Embedding input limits. Every embedding model has a maximum input, counted in its own tokens. OpenAI lists 8,192 for its text-embedding-3 models, and many open models accept far less. A chunk over the limit is rejected or truncated, depending on the API.
• One vector per chunk. The embedding summarizes the whole chunk, so a long chunk covering four topics gets a vector that blends all four and sits close to none of them.
• The prompt budget. Retrieved chunks share the context window with instructions, conversation history and the answer. Bigger chunks mean fewer fit.

A chunk also does two jobs. It is the unit matched against the query, which wants it small and focused, and the unit handed to the model, which wants enough surrounding text to be understood. Several strategies below exist to pull those two jobs apart.

Fixed-size and recursive splitting

Fixed-size chunking cuts the text every N tokens or characters and repeats the last few tokens of each chunk at the start of the next, so a sentence cut at a boundary appears whole in at least one chunk. It is simple, fast and predictable. It also ignores meaning: it cuts sentences, list items, table rows and functions in half, and overlap stores text twice and can return two near-identical chunks for one query. If you size against the embedding model's limit, count in that model's tokens, not in characters.

Recursive splitting keeps the size limit but chooses better boundaries. It tries the coarsest separator first, the paragraph break, and falls back to line breaks, then sentence ends, then spaces only for pieces that are still too long, packing small pieces together up to the limit. LangChain's RecursiveCharacterTextSplitter is a widely used implementation. Its documentation describes the effect as keeping paragraphs, then sentences, then words together as long as possible, and notes that languages written without spaces between words, such as Chinese, Japanese and Thai, need extra separators. A minimal version:

SEPARATORS = ['\n\n', '\n', '. ', ' ']  # paragraph, line, sentence, word

def split_recursive(text, max_len=1000, seps=SEPARATORS):
    if len(text) <= max_len:
        return [text] if text.strip() else []
    if not seps:  # nothing left to split on: hard cut
        return [text[i:i + max_len] for i in range(0, len(text), max_len)]
    sep, finer = seps[0], seps[1:]
    chunks, current = [], ''
    for piece in text.split(sep):
        joined = current + sep + piece if current else piece
        if len(joined) <= max_len:
            current = joined  # keep packing pieces up to the limit
            continue
        if current:
            chunks.append(current)
        if len(piece) > max_len:  # piece alone is too big: go finer
            chunks.extend(split_recursive(piece, max_len, finer))
            current = ''
        else:
            current = piece
    if current.strip():
        chunks.append(current)
    return chunks

This version measures characters; to size by tokens, replace len() with a count from the embedding model's tokenizer. It drops the separator at each cut and leaves out overlap to stay short. To add overlap, prefix each chunk with the tail of the one before it.

Structure-aware chunking: Markdown, HTML, code, tables and PDFs

Most documents already say where their boundaries are. Structure-aware chunking splits on that structure first, and uses a size-based splitter only inside sections that are still too long.

• Markdown: split on headings, and keep the heading path ("Billing > Refunds > Partial refunds") with each chunk, both as metadata and as a prefix to the text.
• HTML: remove navigation, footers and scripts, then split on headings and sectioning elements such as section and article. LangChain, for example, ships splitters for Markdown headers, HTML, JSON and source code.
• Code: split on functions, classes and methods with a parser rather than on blank lines, and keep the file path and the enclosing class or signature with each chunk.
• Tables: never split a row. For long tables, repeat the header row in every chunk, or turn each row into a sentence ("Plan: Pro. Seats: 10. Single sign-on: yes.") so it stands on its own.
• PDFs: the structure has to be recovered first. Use a layout-aware parser to get reading order, headings and tables, remove repeating page headers and footers, run OCR on scanned pages, and then chunk the result as you would Markdown.

A heading-aware Markdown splitter, reusing split_recursive() from above:

import re

def split_markdown(text, max_len=1000):
    chunks, path, lines = [], [], []

    def flush():
        body = '\n'.join(lines).strip()
        header = ' > '.join(path)
        for part in split_recursive(body, max_len - len(header) - 1):
            chunks.append({'section': header, 'text': f'{header}\n{part}'})
        lines.clear()

    for line in text.splitlines():
        m = re.match(r'(#{1,6})\s+(.+)', line)
        if m:  # a heading closes the previous section
            flush()
            level = len(m.group(1))
            path[level - 1:] = [m.group(2).strip()]
        else:
            lines.append(line)
    flush()
    return chunks

The heading path prefixed to each chunk is the simplest form of a contextual chunk header: the chunk now says where it came from, which helps both its embedding and the model reading it. A real Markdown parser also handles cases this one skips, such as # lines inside code fences.

Sentence and semantic chunking

Sentence chunking splits text into sentences and packs whole sentences into chunks up to a size limit, so no chunk starts or ends mid-sentence. It needs a sentence splitter that copes with abbreviations, decimals and the languages in your corpus.

Semantic chunking places boundaries where the topic changes. The method credited to Greg Kamradt, and implemented in LlamaIndex as SemanticSplitterNodeParser, works like this:

1. Split the text into sentences.
2. Embed each sentence, optionally together with its neighbors (the buffer_size setting), so one short sentence does not swing the result.
3. Compute the cosine distance between each pair of adjacent sentence groups.
4. Break wherever the distance is above a chosen percentile of all the distances in the document. LlamaIndex defaults to the 95th percentile; a lower percentile gives more, smaller chunks.

The costs: one embedding per sentence at index time, chunk sizes that vary widely (so you still need a maximum), and a threshold that needs tuning per corpus. On documents with clear headings, structure-aware splitting often finds similar boundaries without the extra embedding calls. Semantic chunking earns its cost on long text with no headings: transcripts, chat logs, scraped articles.

Parent-child and small-to-big retrieval

Parent-child retrieval separates the two jobs of a chunk. You index small child chunks, which match queries precisely, and each child keeps a pointer to a larger parent: its section, a wider window of text, or the whole document if it is short. At query time you search the children and give the model the parents.

• Sentence window: index single sentences, and return each one with a few sentences on either side.
• Section return: index paragraphs, and return the whole section under the same heading.
• Hierarchy: build several levels. LlamaIndex's HierarchicalNodeParser defaults to chunks of 2,048, 512 and 128 tokens, indexes the smallest, and its AutoMergingRetriever swaps retrieved children for their parent when enough children of that parent come back.

Three things to watch: deduplicate parents when several children of one parent are retrieved, cap the total size of what you return, since parents add up quickly, and update children and parents together when a document changes.

Contextual retrieval and late chunking

Both address the same problem: a chunk cut out of a document no longer carries the document's context. A chunk that says "revenue grew 3% over the previous quarter" names neither the company nor the quarter, so its vector cannot match a question that names them.

Contextual chunk headers are the cheap fix: prefix each chunk with the document title and section path before embedding it, as the Markdown splitter above does. Anthropic's contextual retrieval, published in September 2024, generalizes the idea. For each chunk, a model is given the whole document and the chunk and asked for a short context that situates the chunk within the document, usually 50 to 100 tokens. That context is prepended to the chunk before it is embedded and before it is added to a BM25 keyword index. Prompt caching keeps the indexing cost down, because the document is loaded once for all of its chunks.

Anthropic measured the share of queries where a relevant chunk was missing from the top 20, averaged across its test datasets. It reports that contextual embeddings cut that failure rate by 35%, adding contextual BM25 cut it by 49%, and adding a reranker on top cut it by 67%. The price is one model call per chunk at index time, repeated whenever a document changes.

Late chunking, from Günther and colleagues at Jina AI (2024), gets document context into every chunk vector without a generative model. Instead of embedding each chunk on its own, you run the whole document through a long-context embedding model once, keep the output vector for every token, and mean-pool the token vectors inside each chunk's span:

def late_chunk(token_vectors, spans):
    # token_vectors: one output vector per token, from a single pass
    #   of a long-context embedding model over the WHOLE document
    # spans: (start, end) token offsets of each chunk
    chunk_vectors = []
    for start, end in spans:
        rows = token_vectors[start:end]
        chunk_vectors.append([sum(col) / len(rows) for col in zip(*rows)])
    return chunk_vectors  # mean-pooled, one per chunk

Every token vector was computed with attention over the full document, so each pooled chunk vector reflects its surroundings. The authors report better retrieval than embedding chunks separately and say the method works without additional training. It needs an embedding model whose token-level outputs you can read, which usually means running the model yourself, and a document longer than the model's window still has to be processed in large pieces.

The strategies side by side

These combine. A common stack is structure-aware boundaries, recursive splitting inside long sections, a heading-path header on every chunk, and parent-child retrieval for long sections, with hybrid search on top.

How to choose chunk size and overlap, and test it

There is no correct chunk size, only one that works for your documents, your questions and your embedding model. Starting points, not rules:

• Short factual lookups (policies, FAQs, specs): a few hundred tokens, around 200 to 400.
• Explanations and narrative text: around 500 to 1,000 tokens, or parent-child retrieval with small children.
• Overlap: little or none with recursive or structure-aware splitting, since boundaries already fall between paragraphs; around 10 to 20 percent of the chunk size with fixed-size splitting.
• Stay well under the embedding model's input limit, leaving room for any header you prepend.

Then test, because intuition about chunking is often wrong:

1. Collect real questions from logs or from the people who will use the system. A hundred or so is a useful start.
2. For each question, mark the span of source text that answers it. Label spans, not chunk IDs, so the same labels score every chunking variant.
3. Index the corpus once per variant: strategy, size and overlap.
4. For each variant, measure recall@k (is a chunk containing the answer span in the top k?) and mean reciprocal rank (how high is the first one?).
5. Record the context cost as well: average tokens sent to the model per question.
6. Take the best two variants through an end-to-end evaluation of answer quality before you decide.

How to evaluate RAG covers these metrics and answer grading in depth, and how to evaluate AI agents covers building the question set. Chunk size also drives token spend; see how to reduce LLM costs.

If you want retrieval like this built into your product, see our AI agent development service: the code lives in your GitHub from day one, and AI-assisted work is reviewed by an engineer.