Define critical QA errors separately from the overall quality score, with a specific trigger, supporting evidence and an accountable reviewer. A critical flag should explain the event requiring attention. It should not turn a broad inference or missing transcript into an unquestioned adverse finding.
Level AI describes evaluation, calibration and dispute handling in its QA product. AWS documents automatic-fail options that set the affected evaluation scope to zero. These are product-specific public descriptions, not results of our own product testing. Confirm your required configuration, evidence access and operating responsibilities in a scoped demonstration.
What would a reviewable critical-event rule say?
Use an operational event such as an unsupported completion statement, rather than a broad accusation. This synthetic rule concerns service process only. A specialist must define any regulated obligation separately.
| State | Evidence and action |
|---|---|
| Suspected | Transcript appears to say an action was completed; preserve the segment and open a case |
| Pending evidence | Authorized system record is unavailable; assign an owner and retain uncertainty |
| Confirmed operational error | Reviewer matches the agent statement to an unchanged authoritative action record |
| Rejected flag | Statement described a future step, quoted a customer, or referred to a different action |
State the action identifier and time boundary in the rule: for example, whether the task was complete at the time the agent said it was complete. A later repair must not silently rewrite that historical fact. Our recommendation is to show the original quality total beside the critical-event outcome, with any automatic-fail effect labeled separately. A suspected event is never enough to assert a legal violation or a final employment finding.
Which failures deserve separate treatment?
Start with a narrowly defined operational event that requires review regardless of the courtesy score. An illustrative example is an agent stating that a requested action was completed when the authorized record shows it was not. Whether the available evidence can establish that event is a separate question from whether the event matters.
Avoid adding an automatic fail to every undesirable behavior. If ordinary coaching opportunities all become critical, reviewers lose the distinction the flag was meant to create. Assign an owner who can explain the operational consequence and the evidence required to confirm each event.
What should the critical-event specification include?
- A stable event identifier and plain-language definition.
- The condition that makes the rule applicable.
- The observable evidence needed to confirm or reject it.
- Whether the system creates a suspected event or a reviewed finding.
- The reviewer, response window and backup owner.
- How the finding changes the score and downstream reports.
How should suspected events be handled?
Create an explicit pending-review state. Preserve the original interaction and the system’s reason for flagging it. If the decision depends on an external record, identify who may inspect that record and how its timestamp relates to the conversation. An unsupported flag should remain unconfirmed rather than becoming a pass or fail by default.
Use a second review where the consequences justify it, with a defined way to resolve disagreement. Record the final reason and any missing evidence. If the contact reveals an ongoing operational problem, route the corrective task separately from the individual evaluation so the service issue can be addressed without waiting for every reporting step.
How do you report serious events without losing context?
Show suspected, reviewed and confirmed counts separately. Keep the event rate distinct from the average quality score, and identify the population that was actually evaluated. A high overall score does not cancel a confirmed event; a low overall score does not prove one. Audit the arithmetic when automatic-fail settings change.
For consistent event definitions, see scorecard design. For denominator and weighting effects, see QA reporting calculations.
Limits
These are editorial triage recommendations. Sources were checked on 20 October 2026; no critical-event detector or automatic-fail behavior was tested. We do not establish legal noncompliance, employment consequences or acceptable error thresholds. Business owners must approve their definitions, escalation responsibilities and review requirements, and confirm which evidence the actual system can access.