Define virtual-agent success as an observable outcome for a named task. Do not equate a pleasant final message, a disconnected session, or the absence of a transfer with successful resolution. Keep unknown outcomes visible until evidence supports a stronger label.
Level AI describes quality evaluation for virtual-agent conversations. Google Cloud documents playbook conversation tests with expected responses. Use these as evaluation starting points; your operation still needs to define its own outcome evidence.
What exactly is one attempt?
Choose whether an attempt starts when a customer enters the automated channel, states an eligible task, or completes required initial checks. Publish that rule beside the metric. An eligibility definition should describe the deployment's actual scope rather than exclude difficult cases retrospectively.
A conversation can contain more than one task. For an illustrative order-support flow, a customer might ask for status and then request an address change. Track each task outcome separately while retaining the shared session identifier. Otherwise one easy answer can conceal a failed action.
Which outcome labels are needed?
| Outcome | Required evidence |
|---|---|
| Completed automatically | Requested result is confirmed by an appropriate source |
| Correct human escalation | Unfinished work reaches the agreed receiving route |
| Customer abandoned | Session ended before an outcome was established |
| Failed or incorrect | Observed result conflicts with the task definition |
| Unknown | Available evidence cannot establish the result |
Correct escalation can be a successful handling outcome while remaining outside automatic task completion. Present those measures separately. This prevents an operation from improving a headline automation rate simply by making transfer harder.
How should the business state be checked?
- Write the starting state and the permitted end state.
- Specify the system record or customer confirmation that demonstrates completion.
- Record attempted actions and identifiers needed to reconcile results.
- Distinguish a tool request being sent from its result being confirmed.
- Review a sample without relying on the agent's self-assessment.
For an information-only task, correctness may require the current source version and the customer's actual question. For an action task, a fluent explanation cannot replace a verified state change. A test should include apparent successes where the underlying action failed.
What belongs in the performance report?
Report task volume, eligible volume, outcome coverage, automatic completion, correct escalation, incorrect completion, and unknown outcomes. Break results out by task, channel, and relevant language before combining them. Include the definition version and the dates covered.
Keep a review queue for disputed outcomes and a record of label changes. If a team changes its success rule, show the old and new definitions rather than silently joining the series. Pair completion with the customer-effort investigation and failure-recovery checks to expose costly paths behind an apparently favorable result.
How should the denominator survive transfers, abandonment and unknown outcomes?
Use mutually exclusive outcomes for a fixed cohort of attempted tasks. In this invented 100-attempt example, authoritative records confirm 55 autonomous completions, 20 correct human handoffs, 10 unresolved failures, five abandonments and 10 unknown outcomes. Autonomous completion is 55/100. Correct handoffs are 20/100; they are useful routing outcomes but do not prove the customer task was completed.
If later case evidence confirms that 15 of the transferred tasks finished, report 70/100 as eventual confirmed completion for this example and retain the 55/100 autonomous figure. Do not replace one definition with the other. Show the observation window and keep the remaining transferred outcomes unresolved until evidence arrives.
| Proposed field | Purpose |
|---|---|
| Attempt and task IDs | Reconcile retries and multiple contacts without silently inflating attempts |
| Authoritative completion evidence | Distinguish business outcome from generated confirmation |
| Autonomous/human route | Keep automation performance distinct from assisted resolution |
| Window closed and uncertainty | Prevent immature or missing records from becoming successes |
Choose this outcome contract before comparing products. A containment metric can be reported separately, but a conversation ending without a human does not by itself establish a correctly completed task.
Limits
This article proposes a measurement contract rather than a benchmark or standard success threshold. Some outcomes cannot be established from interaction data alone. The required system reconciliation, customer follow-up, and observation window depend on the task and must be agreed before reporting results.