Review false QA failures in a separate error queue before changing the rubric or coaching an agent. Confirm the reference outcome, classify the cause and record the repair owner. A false positive here means the system flagged a failure that the reviewed evidence does not support.
Level AI describes evaluation, calibration and dispute handling in its QA product. AWS documents AI answer justifications and transcript references. These are product-specific public descriptions, not results of our own product testing. Confirm your required configuration, evidence access and operating responsibilities in a scoped demonstration.
How can a review distinguish a false failure from a missed failure?
The following synthetic example turns an error queue into a two-direction check. It is a proposed review method, not an observed model error rate. First have reviewers resolve the intended criterion without seeing the automated label.
| Reference decision | Automated decision | Count in example | Interpretation |
|---|---|---|---|
| Failure | Failure | 12 | Supported failure |
| Pass | Failure | 3 | False failure; investigate evidence |
| Failure | Pass | 2 | Missed failure; investigate separately |
| Pass | Pass | 23 | Supported pass |
Among the 15 automated failures in this illustrative set, three are false failures: 3/15, or 20%. Among the 14 reference failures, two were missed: 2/14, about 14.3%. These are different denominators. A complaint-only queue cannot supply either population estimate. For each repair, retain a counterexample that must still fail and recheck both directions before changing scoring or coaching an agent. Do not call an explanation correct merely because the linked quote exists: check speaker, timing, negation and applicability.
Which cases should enter the queue?
Combine disputed evaluations with an independently selected sample of flagged failures. A queue consisting only of complaints may miss agents who never challenge their scores. Preserve the original model output and evidence before a reviewer edits the evaluation, so later investigation can reproduce what happened.
Record which criterion failed and why the reviewer believes that failure is unsupported. Distinguish a truly incorrect judgment from an unfortunate but correct application of a written policy. The latter may justify a policy discussion, but changing the model label alone does not fix the underlying rule.
How should the cause be classified?
| Cause | Next investigation |
|---|---|
| Missing or incorrect transcript | Compare original recording with the input available to scoring |
| Applicability mistake | Inspect the condition that made the criterion active |
| Ambiguous rubric | Find conflicting examples or unclear answer definitions |
| Unsupported interpretation | Compare the rationale with its cited evidence |
| Incorrect reference label | Have the rubric owner review the human decision |
When should you tune rather than correct one result?
Start with the smallest repair justified by the evidence. A single mislabeled reference example calls for a reference correction. Repeated misunderstanding of one phrase may call for clearer rubric instructions or a model investigation. Do not broaden a rule merely to make a troublesome example pass; test nearby counterexamples that should continue to fail.
Track the version and effective date of the repair. Re-evaluate a held-out sample and inspect whether missed failures increased while false failures decreased. Report both directions. An apparently quieter dashboard can result from a weaker detector rather than better scoring.
What belongs in the weekly error summary?
- Flagged cases reviewed and confirmed false failures, with the selection method.
- Recurring causes by criterion, channel and input condition.
- Unresolved cases awaiting evidence or rubric decisions.
- Repairs shipped, owners and follow-up results.
- Potential reference examples approved for the gold set.
Keep this summary about system behavior rather than turning review volume into a performance ranking of employees. Use gold-set maintenance for approved examples and the appeal case-file process when an individual evaluation needs a formal response.
Limits
This is an operating recommendation, not a measured estimate of vendor error rates. Product guidance was checked on 20 October 2026. We have not tested the completeness of evidence exports or the ability to retain original evaluations. A targeted error queue cannot estimate population accuracy without an appropriate sampling design; retain the sample origin beside every reported rate.