A retrieval system can only ever return a chunk. If the fact a customer needs is split across two chunks, or buried in a chunk about something else, no amount of model quality will save the answer. Chunking is where the ceiling gets set.
The two failure modes
Everything that goes wrong is a variation on one of these.
Chunks too large
A 2,000-token chunk covering all of billing has a vector that is the average of everything in it — a blurry centroid that is close to every billing question and precisely on none. It will beat a short, exactly-right chunk on some queries and then fail to contain a usable answer. It also burns context window, crowding out other candidates.
Chunks too small
A 100-token chunk retrieves sharply but arrives without the conditions attached to it. "Refunds are processed within 5 business days" is true — for annual plans, in the first 30 days, if the account is in good standing. Those qualifiers were in the paragraph you split away, and the model cannot know they exist.
Split on structure, not on character counts
Fixed-size splitting is the default in most tutorials and it is the wrong default. It cuts mid-sentence, mid-table and mid-procedure, guided by nothing but an integer.
Documentation already carries its own boundaries — headings, sections, list groups, code blocks. Split there first, and only fall back to size limits when a section is genuinely too long.
Fixed 512-token split:
...to enable the widget, open Settings and │ chunk 1
navigate to Chat widget. Then click Install │
and copy the snippet into your site's <head> │
─────────────────────────────────────────────
tag. Note that the snippet must load before │ chunk 2
any other analytics scripts, or events... │
The procedure is now unanswerable from either chunk.
Structure-aware split:
## Installing the chat widget │ chunk 1
(whole procedure, all 6 steps, intact) │
─────────────────────────────────────────────
## Script load order │ chunk 2
(the caveat, with its own heading as context) │Overlap, sparingly
A small overlap — 10 to 15 per cent — between adjacent chunks catches facts that straddle a boundary. Beyond that you are mostly paying to store and search duplicates, and near-identical chunks start competing with each other for the same slots in the results.
Carry the context down with the chunk
This is the highest-leverage change most teams have not made. Prepend each chunk with the trail that locates it, so the text stands on its own:
Raw chunk:
"Set this to at least 30 seconds on high-traffic sites."
Contextualised chunk:
"Docs › Chat widget › Performance › Polling interval
Set this to at least 30 seconds on high-traffic sites."The second version retrieves for "polling interval" and "widget performance" — queries the first could never match. It costs a handful of tokens per chunk.
Different content, different rules
| Content type | Split on | Watch for |
|---|---|---|
| Help articles | H2/H3 headings | Sections that exceed ~600 tokens |
| API reference | One chunk per endpoint | Keep params and examples together |
| FAQs | One chunk per Q&A pair | Never split a question from its answer |
| PDFs / policies | Numbered clauses | Page headers and footers leaking in |
| Changelogs | One chunk per release | Dates must stay attached to entries |
| Tables | Keep whole; add a prose restatement | Orphaned cells with no row context |
How to tell if chunking is your problem
Take twenty questions you know your docs answer. For each, look at what was retrieved — not at the final reply.
- The right chunk came back and the answer was still poor → generation or prompt problem, not chunking.
- A near-miss chunk from the correct page came back → chunks are too small, or context trails are missing.
- A chunk from a different topic came back → chunks are too large and blurry, or your index has stale content.
- Nothing relevant came back at all → a content gap. No chunking strategy invents a page you never wrote.
Debug retrieval by reading what was retrieved. Almost every team that says "the model is bad" has never once looked at the passages it was given.
Chunking is not glamorous and it never appears in a demo. It is also, reliably, where the difference between a bot people trust and a bot people close is decided.
- RAG
- Retrieval
- Architecture
