The failure pattern is consistent. A team enables an AI assistant for all traffic on a Monday, three bad answers get screenshotted by Wednesday, and by Friday it is switched off and the project has acquired a reputation. Nothing was wrong with the technology; the rollout had no stage at which problems were cheap.
Four weeks, four gates. Each week has an exit condition, and you do not move on until it is met.
Week 1 — Train it and read everything
Ingest your help centre, product docs and policy PDFs. Deliberately leave out marketing pages and anything obsolete; a smaller, cleaner corpus outperforms a large messy one every time.
Then do the unglamorous part: pull your 50 most frequent questions from last quarter's tickets, ask every one, and read the answers with the retrieved sources open beside them.
- Score each answer correct / incomplete / wrong / refused.
- For every non-correct answer, record the cause: missing page, wrong page, bad retrieval, bad phrasing.
- Fix the content problems first. They are usually more than half of the list and take an afternoon.
Week 2 — Internal only
Put it in front of your support team, not your customers. Give agents a suggest-only mode: the bot drafts, a human sends. Nothing reaches a customer unreviewed.
This week buys two things that a staging environment cannot. Real question variety — the weird, half-typed, context-free ones — and, more importantly, your support team's buy-in. A team that has spent a week correcting the bot trusts it far more than a team that had it announced to them.
Track the edit rate: what fraction of drafts get sent unchanged, lightly edited, or discarded.
Week 3 — A slice of real traffic
Now go live, but narrowly. Pick one of:
- One topic — billing questions only, everything else straight to the queue.
- One surface — the docs site, not the checkout page.
- One segment — free-tier users, or one non-critical region.
- One window — after hours, where the alternative is an empty inbox rather than a fast human.
After-hours is the best first slice for most teams: the comparison is not "bot versus agent", it is "bot versus nothing until Monday", and that is a bar it clears easily.
Read every conversation this week. All of them. It is tedious and it is the highest-value week of the four.
Week 4 — Widen, and build the loop
Open it to general traffic, with the escape hatch prominent from the first message. The rollout is the easy half of this week. The important half is the routine that keeps it working after everyone stops paying attention:
- 1Weekly: review every refused and escalated question. Cluster them. Write the top three missing pages.
- 2Weekly: sample 20 contained conversations at random and score them. Sampling catches what dashboards hide.
- 3Monthly: re-run the top-50 benchmark from week one. It is your regression test.
- 4On every docs change: confirm the index picked it up.
The bot you launch is not the bot that matters. The bot that matters is the one that exists after six months of that loop — or the one that quietly rotted because nobody owned it.
Name an owner
One person, by name, responsible for answer quality. Not a committee, not "support in general". Most bots that decay do so because the person who set it up moved on and it belonged to nobody after that. Half an hour a week from a named owner is enough; zero hours from everyone is not.
- Rollout
- Operations
- Change management
