Most AI support dashboards lead with total conversations. It is the least useful number on the page. A bot can hold ten thousand conversations and resolve none of them, and the chart will look like a triumph.
Here are the five that actually move decisions, roughly in the order you should care about them.
1. Containment rate — with the escape hatch counted separately
The share of conversations that ended without a human being pulled in. Useful, but only if you split it in two.
- Good containment: the visitor got an answer and left satisfied.
- Bad containment: the visitor gave up. No handoff, no answer, no follow-up ticket — because they went to a competitor.
Both look identical in a naive count. Distinguish them by what happened at the end of the conversation: an explicit thumbs-up, a click on a cited source, or a follow-up question that indicates the customer was still engaged, versus a silent exit after an unhelpful reply.
2. The "I don't know" rate
How often the bot declined to answer and offered a human instead. Counterintuitively, you do not want this at zero — zero means your bot never admits uncertainty, which means it is inventing answers you have not caught yet.
A healthy range is a few per cent. What matters more is the trend and the content: a rising rate on one topic is a documentation gap with a name and address.
3. Time to first useful answer
Not response latency — everyone's bot replies in two seconds. Measure how many turns it took before the visitor got something they could act on.
One turn is excellent. Two is fine. Four means the bot is interrogating the customer instead of helping them, which usually points at retrieval returning near-misses and the model hedging its way toward the answer.
4. Escalation quality
Of the conversations handed to a human, how many arrived with the context the agent needed? An escalation that dumps a raw transcript on someone and makes them start again has cost you more than having no bot at all.
The measurable proxy: how long the agent takes to send their first reply after a handoff, compared with a cold ticket. If it is not lower, the handoff is not working.
5. Repeat-contact rate
The share of visitors who came back with the same question within a week. This is the metric that catches confidently wrong answers, and it is the one almost nobody tracks.
A bad answer closes a conversation cleanly. It looks like containment. It looks like success. The only visible trace is that the same person is back on Thursday, asking again, slightly less politely.
Any metric that a confidently wrong answer improves is a metric that will eventually mislead you. Repeat contacts are the antidote.
Putting them together
| Symptom | Likely cause | Where to look |
|---|---|---|
| High containment, high repeat contacts | Confidently wrong answers | Retrieval — check what is being cited |
| Low containment, low refusal rate | Visitors give up before asking properly | Opening message and suggested prompts |
| High refusal rate on one topic | Documentation gap | Write the page |
| Slow first useful answer | Near-miss retrieval | Chunking and headings |
| Escalations slower than cold tickets | Context is not travelling with the handoff | Handoff payload |
Review these monthly, not daily. Support volume is seasonal and noisy, and a daily dashboard mostly teaches you to react to weather.
- Analytics
- Measurement
- Operations
