Evaluate a voice AI agent on whether it completes an authorized customer task and recovers safely when it cannot. A fluent demonstration is useful evidence of conversational behavior, but it does not establish production reliability across your callers and business systems.
Level AI documents voice virtual agents and workflow controls. Genesys describes agentic virtual agents with approved actions and contextual human handoffs. Treat those descriptions as capabilities to investigate, then use the same acceptance scenarios for both vendors.
What is the unit of evaluation?
Use a complete customer task, not a single answer. An illustrative order-status task might require finding the right order, interpreting the latest event, answering a follow-up, and handing over when the information is unavailable. Define which actions are permitted and what evidence demonstrates completion before the vendor builds the scenario.
Which scenarios belong in the test?
- Normal path: the caller provides enough information and the connected system responds.
- Conversation repair: the caller interrupts, corrects a detail, pauses, or changes intent.
- Unclear input: background noise or an ambiguous identifier requires clarification.
- System failure: the underlying service times out or returns conflicting information.
- Boundary: the caller requests an unsupported action or asks to speak with a person.
These are proposed test categories, not results from a benchmark. Include examples from the deployment's real language and call-quality mix. Retest after changing prompts, knowledge, connected tools, or business rules.
How should latency and completion be compared?
| Measure | Define before testing |
|---|---|
| Response latency | Start event, end event, and distribution across whole calls |
| Task completion | Observable business outcome rather than an agent's assertion |
| Escalation quality | Correct destination, reason, and context retained |
| Recovery | Whether a retry repeats an action or produces inconsistent state |
| Customer effort | Repetition, unnecessary turns, and abandonment |
Report failures as part of the denominator. Excluding dropped, unsupported, or transferred calls can make a narrow success rate look like broad automation. Review ordinary and difficult calls separately so a favorable average cannot hide an unacceptable failure mode.
What should a controlled rollout require?
Start with a limited task scope, a named operator, readable action logs, a working human fallback, and a way to disable the workflow. Increase scope only after examining unsuccessful attempts as carefully as successful ones. Ask for the operational ownership and ongoing maintenance responsibilities in writing.
For cross-channel needs, see voice and chat evaluation. For wider workflow boundaries, see end-to-end CX automation.
Which vendor evidence belongs in a voice-agent shortlist?
| Documented starting point | Consider when | Unverified requirement |
|---|---|---|
| Level AI: voice agents with workflow controls | Your shortlist includes customer automation beside interaction evaluation | Configured telephony path, task accuracy, latency distribution and recovery behavior |
| Genesys: approved-tool workflows and contextual human handoffs | Your proposed deployment needs those workflows in the Genesys environment | Required tool availability, complete operating costs and equivalent acceptance outcomes |
Use these conditional fits to select demonstrations, then use a common acceptance script to choose. For an illustrative appointment-change task, require the original booking lookup, permitted alternatives, customer confirmation, one authoritative update and an accurate final explanation. Include a caller changing their mind after the update, a service timeout and a request for a person.
Record correctness and effort at the whole-task level. One fluent reply cannot compensate for an incorrect booking; one slow tail case should remain visible beside average latency. Predeclare blocking failures and acceptable operating bounds with the service owner. Neither public description demonstrates a universal winner, and this guide supplies no measured performance ranking.
Limits
No latency, containment, accuracy, or return-on-investment result is measured here. Public product descriptions do not establish action availability, regional coverage, pricing, or production suitability for a particular customer. Confirm each requirement through a scoped pilot and current documentation.