What we have learned building retrieval-based support assistants — the parts that work, the parts that quietly do not, and the documentation habits that decide which one you end up with.
The same documentation habits that make a page easy to skim make it easy to retrieve. Here is what to change, and why the fix is almost always editorial rather than technical.
Conversation counts look impressive and mean nothing. These five metrics tell you whether the thing is actually helping — and which one to fix first when it is not.
Optimising for tickets avoided quietly rewards a bot that makes itself hard to escape. There is a better target, and it is not much harder to measure.
The moment your bot gives up is the moment that decides how customers remember the whole interaction. Most teams treat it as an error path. It is a feature.
How you split documents decides what can ever be retrieved. It is the least discussed and most consequential choice in the whole pipeline.
Hallucination is not a mysterious model quirk — it is what happens when a system is designed so that answering is always cheaper than declining. Change the incentives and the behaviour changes.
You do not need translated documentation to answer in a customer’s language. You need to be deliberate about which layer does the translating — and where that quietly goes wrong.
Switching a bot on for every visitor on day one is how these projects get cancelled in week three. A staged rollout gets you the same destination with evidence at every step.
Every question a customer asks is a small report on where your product confused them. Most teams archive that and then commission a survey to learn the same thing.
