Why chunking dominates RAG quality

In RAG Fundamentals you built a sliding window over words. That proves the concept, but in production the chunk boundary is the single biggest lever on quality: retrieval can only ever return a chunk, so if the answer isn't cleanly inside one chunk, no retriever, reranker, or bigger model can recover it. Three things go wrong with naive chunking:

1. Splitting by words ≠ splitting by tokens. The model's context window and every embedding API bill in tokens, not words. "internationalization" is one word but several tokens; code and punctuation are token-dense. A 200-word chunk can silently be 400+ tokens.
2. Cutting mid-thought. A fixed window happily slices through the middle of a sentence, a table row, or a question/answer pair — orphaning the context that made the text meaningful.
3. The size/precision tension has no single winner. Small chunks match precisely but lack context to answer from; large chunks carry context but retrieve imprecisely. You often want both.

The rest of this topic is four techniques that fix these, each with a worked example you'll extend.

1. Chunk on a real token budget

Use the model's actual tokenizer so "300 tokens" means 300 tokens. tiktoken is OpenAI's tokenizer; every provider ships an equivalent.

import tiktoken                                  # OpenAI's tokenizer library

enc = tiktoken.get_encoding("cl100k_base")       # the encoding used by GPT-4o / text-embedding-3-*

def chunk_by_tokens(text: str, size: int = 300, overlap: int = 40) -> list[str]:
    tokens = enc.encode(text)                    # str -> list[int]: the real token ids (not words!)
    step = size - overlap                         # advance per window; overlap keeps boundary ideas whole
    chunks = []                                   # accumulator (like `const chunks = []`)
    for start in range(0, len(tokens), step):     # 0, step, 2*step, …  (a for-loop stepping by `step`)
        window = tokens[start : start + size]     # slice `size` token ids
        chunks.append(enc.decode(window))         # decode(list[int]) -> str: turn ids back into text
    return chunks

# Now "size" is a true token budget you can trust against the context window and embedding limits.
chunks = chunk_by_tokens(open("article.txt").read(), size=300, overlap=40)
len(chunks), enc.encode(chunks[0]).__len__()     # e.g. (7, 300)  -> exactly 300 tokens, guaranteed

2. Split on natural boundaries (recursive splitting)

A fixed token window still cuts mid-sentence. Recursive character splitting fixes this: try to split on the largest natural separator first (paragraph breaks), and only fall back to smaller ones (sentences, then a hard token cut) when a piece is still over budget. This is exactly what LangChain's RecursiveCharacterTextSplitter does — here it is from scratch so you can see the mechanism:

def recursive_split(text: str, max_tokens: int = 300) -> list[str]:
    # If the whole text already fits the budget, it's a single chunk — done.
    if len(enc.encode(text)) <= max_tokens:
        return [text]

    # Otherwise split on the biggest natural boundary present, in priority order:
    for sep in ["\n\n", "\n", ". ", " "]:         # paragraphs → lines → sentences → words
        if sep in text:                            # use the first separator that actually occurs
            parts = text.split(sep)                # break into smaller pieces on that boundary
            chunks = []
            for part in parts:                     # recurse: each part may still be too big
                chunks.extend(recursive_split(part + sep, max_tokens))  # re-attach sep so we don't lose it
            return _merge_small(chunks, max_tokens)  # greedily glue tiny pieces back up to the budget

    return [text]                                  # no separator left: return as-is (a hard fallback)

def _merge_small(pieces: list[str], max_tokens: int) -> list[str]:
    # Greedily pack consecutive pieces together until adding the next would exceed the budget.
    merged, buf = [], ""                           # `buf` is the chunk we're currently filling
    for p in pieces:
        candidate = buf + p                        # what buf would become if we append p
        if len(enc.encode(candidate)) <= max_tokens:
            buf = candidate                        # still fits → keep filling
        else:
            if buf:
                merged.append(buf)                 # flush the full chunk
            buf = p                                # start a new chunk with the piece that didn't fit
    if buf:
        merged.append(buf)                         # flush the last partial chunk
    return merged

The result: chunks land on paragraph/sentence boundaries whenever possible and only hard-cut as a last resort — so you almost never orphan half a sentence.

3. Attach metadata (and retrieve with a filter)

A chunk is more than its text. Store where it came from alongside it, and you unlock filtered retrieval — restrict the search to one document, one section, or the last 30 days before you even score by similarity. Represent each chunk as a small record:

from dataclasses import dataclass, field         # @dataclass auto-generates __init__ from typed fields

@dataclass                                        # like a typed struct / a TS interface with a constructor
class Chunk:
    text: str                                     # the chunk's content
    source: str                                   # which document it came from
    section: str                                  # heading/section title, if known
    position: int                                 # 0-based order within the document
    meta: dict = field(default_factory=dict)      # anything else (date, author…); default_factory=dict → fresh {} per instance

def to_chunks(text: str, source: str, section: str = "") -> list[Chunk]:
    # enumerate() gives (index, value) pairs — index becomes the chunk's position.
    return [
        Chunk(text=piece, source=source, section=section, position=i)
        for i, piece in enumerate(recursive_split(text))
    ]

# Retrieval that pre-filters by metadata, THEN ranks by similarity:
def retrieve(question: str, index: list[tuple[Chunk, list[float]]], k: int = 4, source: str | None = None):
    q = embed(question)                                            # embed the query (from Topic 1)
    pool = index if source is None else [                          # if a source filter is given…
        (c, v) for (c, v) in index if c.source == source          # …keep only chunks from that document
    ]
    scored = [(cosine_similarity(q, v), c) for (c, v) in pool]     # score the (filtered) pool
    scored.sort(key=lambda pair: pair[0], reverse=True)           # sort by score desc; key= picks the score field
    return [c for (_score, c) in scored[:k]]                       # `_score` = throwaway var; return top-k chunks

Real vector databases (pgvector, Qdrant, Azure AI Search — the next topic) do this filtering in the engine with an indexed WHERE clause, which is far faster than the Python filter above but identical in intent. Metadata is also what lets you show citations ("source: pricing.md § Plans").

4. Parent-child (hierarchical) indexing

Here's the resolution to the size/precision tension: embed small, return large. Index tiny child chunks (precise matching) but, once a child is retrieved, hand the model its bigger parent chunk (full context). This is the "small-to-big" / parent-document pattern.

# Build TWO levels: big parents, and small children that each point back to their parent.
def build_hierarchy(text: str, source: str):
    parents = recursive_split(text, max_tokens=800)   # coarse chunks — plenty of context
    parent_store, children = {}, []                    # dict: parent_id -> parent text; list of child records
    for pi, parent in enumerate(parents):
        parent_store[pi] = parent                      # remember the parent by its id (pi)
        for child in recursive_split(parent, max_tokens=150):   # split each parent into small children
            children.append({"text": child, "parent_id": pi})   # child keeps a pointer to its parent
    return parent_store, children

# Embed only the CHILDREN (small = precise match); index them with their parent pointer.
def index_children(children):
    return [(child, embed(child["text"])) for child in children]   # (child_record, child_vector) pairs

# Retrieve: match on children, then EXPAND each hit to its parent for generation.
def retrieve_parents(question, parent_store, child_index, k=3):
    q = embed(question)
    scored = [(cosine_similarity(q, v), c) for (c, v) in child_index]  # score children
    scored.sort(key=lambda p: p[0], reverse=True)
    seen, out = set(), []                                              # `set()` = a Set: dedupe parent ids
    for _score, child in scored:
        pid = child["parent_id"]
        if pid in seen:                                               # a parent may have several matching children…
            continue                                                  # …only include it once
        seen.add(pid)
        out.append(parent_store[pid])                                 # return the PARENT text, not the child
        if len(out) == k:                                            # stop once we have k distinct parents
            break
    return out

You get the best of both: the match is precise (a 150-token child about one idea), but the context handed to the model is rich (the 800-token parent around it). A common variant indexes a one-sentence summary of each parent as the child — same principle, even tighter matching.

Choosing sizes — empirically, not by vibes

There is no universal best chunk size; it depends on your documents and questions. The discipline: pick 2–3 candidate configs, run them against the golden set from Topic 1 (question → expected answer), and measure context recall (did the answer-bearing chunk get retrieved?). Rules of thumb to start from, then tune:

Content type	Starting chunk size	Why
Prose / articles / docs	300–500 tokens, 10–15% overlap	Paragraph-scale ideas fit; overlap saves boundary sentences
FAQ / Q&A / short entries	1 chunk per entry (no splitting)	Each unit is already self-contained
Code / API reference	Split on function/class boundaries	Semantic units, not token counts
Dense reference (legal, medical)	Smaller (150–250) + parent-child	Precision matters; expand to parent for context

Pitfalls

• Sizing in words or characters. Bill and budget are in tokens — size in tokens or you'll blow the context window unpredictably.
• Hard fixed windows on structured text. Tables, code, and Q&A pairs get sliced; use boundary-aware (recursive) splitting or split on the structure itself.
• Throwing away metadata. Without source/section you can't filter, can't cite, and can't debug which document a bad chunk came from.
• One chunk size to rule them all. When precision and context conflict, don't compromise the size — use parent-child instead.
• Never re-measuring. Chunking choices decay as your corpus changes; keep them tied to the golden set.

Recap

• Chunking is the highest-leverage knob in RAG: retrieval can only return a chunk, so the answer must live cleanly inside one.
• Chunk on real tokens (chunk_by_tokens) so sizes are trustworthy against context and embedding limits.
• Split on natural boundaries (recursive_split) so chunks never cut mid-sentence or mid-section.
• Attach metadata (Chunk records) to enable filtered retrieval, citations, and debugging.
• Parent-child indexing resolves the precision/context tension: embed small children, return big parents.
• Choose sizes empirically against your golden set, starting from content-type rules of thumb.
• Worked examples map 1:1 to the exercises: chunk_by_tokens (Ex 1), recursive_split (Ex 2), Chunk + filtered retrieve (Ex 3), parent-child retrieval (Ex 4).

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

text = open("article.txt").read()

# (a) naive 200-word chunks
words = text.split()
word_chunks = [" ".join(words[i:i+200]) for i in range(0, len(words), 200)]
for i, c in enumerate(word_chunks):
    print(f"word-chunk {i}: {len(enc.encode(c))} tokens")   # varies wildly — often 250–350+

# (b) exact 300-token chunks
def chunk_by_tokens(text, size=300, overlap=40):
    toks = enc.encode(text)
    step = size - overlap
    return [enc.decode(toks[i:i+size]) for i in range(0, len(toks), step)]

tok_chunks = chunk_by_tokens(text)
print("token-chunk sizes:", [len(enc.encode(c)) for c in tok_chunks])  # all == 300 (last may be short)

for c in recursive_split(text, max_tokens=300):
    print("┃", c.splitlines()[0][:70])   # first line of each chunk — note it starts on a real boundary

index = []
for path in ["pricing.md", "security.md"]:
    for ch in to_chunks(open(path).read(), source=path):
        index.append((ch, embed(ch.text)))

# Unfiltered: the pricing chunk is retrievable → the model can answer.
hits = retrieve("What does the Pro plan cost?", index, k=4)
print("unfiltered sources:", [h.source for h in hits])         # includes pricing.md

# Filtered to the wrong document: the answer-bearing chunk is excluded by metadata BEFORE scoring.
hits = retrieve("What does the Pro plan cost?", index, k=4, source="security.md")
print("filtered sources:", [h.source for h in hits])           # only security.md — answer not present

parent_store, children = build_hierarchy(open("policy.txt").read(), source="policy.txt")
child_index = index_children(children)

q = "What requires manager approval?"

# (a) flat: precise match, but the returned 150-token child may lack the antecedent it refers to.
flat = sorted(((cosine_similarity(embed(q), v), c["text"]) for c, v in child_index),
              key=lambda p: p[0], reverse=True)[:3]
print("FLAT context:\n", "\n---\n".join(t for _s, t in flat))

# (b) parent-child: same precise match, but expanded to the parent's full context.
parents = retrieve_parents(q, parent_store, child_index, k=3)
print("PARENT context:\n", "\n---\n".join(parents))