Keep purposeful human sampling after QA automation to estimate routine error, investigate difficult cases and maintain reference judgments. Use separate sample streams for those different goals. Reviewing only the worst automated scores can improve triage while telling you very little about errors in apparently good conversations.
Level AI describes evaluation, calibration and dispute handling in its QA product. AWS documents multiple reviewers evaluating the same contact during calibration. These are product-specific public descriptions, not results of our own product testing. Confirm your required configuration, evidence access and operating responsibilities in a scoped demonstration.
Which review stream answers which question?
Keep selection purpose on every reviewed record. The following allocation is a planning structure, not a prescribed sample size or an industry benchmark.
| Stream | Selection method | Permitted conclusion |
|---|---|---|
| Routine audit | Random selection from a defined eligible population | Estimate error within that population with an appropriate uncertainty analysis |
| Risk investigation | Chosen criterion, complaint, language or known failure mode | Describe the investigated cases; do not generalize the error rate |
| Change validation | Frozen holdout plus separately reported recent work | Assess a proposed release against its declared boundary |
| Agent appeal | Submitted challenge with required evidence | Resolve the individual decision and identify recurring causes |
For example, ten incorrect scores in 50 complaint cases means ten reviewed complaint cases were incorrect. It does not mean that 20% of all automated evaluations are wrong. Our recommendation is to preserve both stream and selection probability where known, and deduplicate interactions before calculating shared totals. Ask a qualified analyst to set the routine sample size using the required precision and expected prevalence; this guide deliberately supplies no universal number.
Which sample streams serve different purposes?
| Stream | Decision it supports |
|---|---|
| Routine population sample | How frequently judgments disagree in ordinary eligible work |
| Targeted risk sample | Which suspected important failures need immediate investigation |
| Disagreement sample | Why agents, reviewers and models reached different labels |
| Change-validation sample | Whether a new model or rubric changed known behavior |
Keep the selection reason with each record. A handpicked risk sample should not be merged into a population accuracy calculation without accounting for how it was chosen. Likewise, an untouched holdout used for a release check should not quietly become the next batch of tuning examples.
How should routine sampling be planned?
Define the eligible population, reporting period and unit before selecting records. Decide whether the question concerns interactions, evaluations or individual criterion judgments. Include ordinary automated passes as well as failures; missed failures can be hidden in the pass group. Ensure reviewers can inspect the original evidence rather than only the model’s explanation.
Plan representation for channels, queues and languages that matter to the decision. If a smaller group receives additional review so its problems become visible, label that deliberate allocation and avoid presenting an unadjusted combined rate as a simple random population estimate. Ask an appropriate analyst to choose a statistical design when precision matters.
What should human reviewers record?
- Selection stream and population eligibility.
- Rubric and automated evaluation version.
- Independent human label before seeing the automated conclusion where feasible.
- Evidence location and final adjudicated outcome.
- Whether the issue is a model error, reference problem or insufficient input.
- Time spent reviewing and the action created.
Keep disagreement itself distinct from confirmed model error. A reviewer can misunderstand a criterion too. Use the reference process to resolve uncertainty and preserve ambiguous cases rather than forcing a convenient answer.
When should the sampling plan change?
Revisit the plan after a material model release, a new channel, a shift in contact mix or a recurring unexplained failure. Adjust targeted investigation capacity when urgent issues grow, but preserve enough routine review to retain visibility into normal work. Record why resources moved so a later trend remains interpretable.
Use gold-set maintenance for adjudicated examples and error review for fixes. Use coverage reconciliation to check which contacts never reached scoring at all.
Limits
This guide does not prescribe a sample size, confidence interval or universal human-review percentage. Sources were checked on 20 October 2026; no sampling program or vendor accuracy was measured. A statistically meaningful population estimate requires a suitable design and enough reviewed observations. Automation coverage and sampled scoring accuracy answer different questions and should remain separate.