# How to evaluate an AI customer support bot before launch

Canonical: https://gptnavi.com/guides/ai-customer-support-evaluation
Sources checked: 2026-10-22

## What should a small business test before launching an AI support bot?

Evaluate the whole resolution path: whether the answer is supported by the current policy, whether the user is allowed to see it, and whether unresolved requests reach a human with the right context. Track resolution quality alongside cost and review time. A bot that closes more conversations is not necessarily resolving more problems.

## Build a test set from the jobs you actually receive

Start with a de-identified sample of routine questions, policy exceptions, angry messages, incomplete requests, and questions the knowledge base cannot answer. Write the expected source and acceptable resolution before running the bot. Keep cases used to improve prompts separate from the final test. Small samples expose failures; they do not establish a production-wide accuracy rate.

## Test what arrives in the human queue

A handoff should include the user's question, relevant history, evidence consulted, attempted steps, and the reason automation stopped. Confirm that the customer receives an accurate status rather than a false promise of resolution. Test a queue outage and an unanswered clarification. An escalation button is not enough evidence that an escalation reached an accountable person.

## Estimate cost per accepted resolution

Record platform fees, model usage, retrieval services, and human review for the pilot. Divide the run's total cost by accepted resolutions, keeping unanswered and failed conversations in the cost numerator. Count review and repair minutes even when the bot is fast. Use the ROI calculator to see how coverage and exception rates change the value of released staff capacity. Treat refunds, cancellations, and account changes as separate approved actions rather than ordinary answer generation.

## Proposed pilot

1. Score 30 de-identified support cases against an answer key written in advance.
2. Include five missing-evidence cases and a failed human-handoff case.
3. Export the scorecard with evidence and owners; rerun failed cases after fixes.

## Sources

- [Microsoft Learn: RAG and indexes](https://learn.microsoft.com/en-us/azure/foundry/concepts/retrieval-augmented-generation): Grounding answers in source data and evaluating retrieval.
- [Anthropic: building effective agents](https://www.anthropic.com/research/building-effective-agents): Use explicit success criteria and bounded tool use.

This is a decision framework, not a vendor performance benchmark.