Build a QA gold set from independently reviewed interactions with stable labels, evidence and known limitations. Keep the final test portion separate from examples used to tune the system. The set should help identify disagreement, not merely provide easy demonstrations on which everyone already agrees.
Level AI describes evaluation, calibration and dispute handling in its QA product. AWS documents multiple reviewers evaluating the same contact during calibration. These are product-specific public descriptions, not results of our own product testing. Confirm your required configuration, evidence access and operating responsibilities in a scoped demonstration.
What would one reference case contain?
A gold-set case is a reviewed example and its decision rationale. This synthetic case illustrates a durable record without claiming that its reference answer is an unquestionable truth.
| Field | Illustrative content |
|---|---|
| Case and version | G-014, rubric R3, anonymized interaction revision 2 |
| Question | Did the agent explain the next step when a transfer was necessary? |
| Evidence | Agent turn, transfer outcome and the instruction in force at that time |
| Independent labels | Reviewer A: pass; reviewer B: insufficient evidence |
| Adjudication | Unresolved until the transfer segment is recovered; excluded from accuracy denominator |
| Split | Untouched release test; not shown during prompt tuning |
Our recommendation is to split by the underlying interaction before generating excerpts, translations or related examples. Otherwise near-duplicate material can appear in both tuning and testing. Keep a manifest of additions, removals and reference corrections, and retain earlier answers with their reasons. Report how many cases remain unresolved. A balanced collection of failures and successes can expose weaknesses, but its accuracy figure does not automatically estimate the production mixture.
How should reference interactions be selected?
Choose examples from the population the scorecard will actually evaluate. Include common requests, difficult exceptions, brief conversations and inputs with limited evidence. Select a deliberate mix rather than only the lowest automated scores. Otherwise the collection becomes a failure scrapbook instead of a useful basis for assessing everyday decisions.
Represent each relevant channel and language, and record where coverage remains thin. Keep a few deliberately similar cases with different correct outcomes: an approved action versus a proposed action, for example. Mark synthetic scenarios separately from genuine interactions so later users understand their evidential limits.
What should the reference record contain?
| Field | Why retain it? |
|---|---|
| Interaction and rubric version | Makes the reference decision reproducible |
| Criterion label and evidence location | Lets a reviewer inspect the exact basis |
| Independent reviewer labels | Preserves disagreement before adjudication |
| Final decision and reason | Explains how ambiguity was resolved |
| Split and permitted use | Distinguishes tuning examples from holdout tests |
How do labels become trustworthy enough to use?
Have reviewers work independently before discussion. Resolve disagreements with the rubric owner using the original evidence, and keep unresolved cases as unresolved. If the contact lacks the information required to answer, do not invent a definite label to complete the spreadsheet. That case can test whether the system handles insufficient evidence appropriately.
Preserve old labels when a new policy changes the correct answer. Add a new rubric version and document why the interpretation changed. A gold set should distinguish an error in the model from an error in the reference or a real change in the business requirement. Decide who may approve each kind of correction.
How should the set be maintained?
- Restrict tuning access to the development portion.
- Add newly discovered failure patterns without removing ordinary cases.
- Keep a stable historical slice and a separately identified recent slice.
- Review whether changes in queues or languages require new representation.
- Log label corrections and explain their effect on reported results.
Use false-positive review to discover candidate additions and change-control checks to reuse the set responsibly.
Limits
These recommendations do not prescribe a universally sufficient sample size or statistical confidence level. Sources were checked on 20 October 2026. No vendor model was benchmarked. Small or uneven reference sets leave meaningful uncertainty, especially for rare events. Access to real interactions, permitted retention and reviewer availability must be confirmed within your organization before assembling the collection.