Design a QA scorecard as a set of observable decisions with explicit applicability, evidence and scoring rules. Write the rubric before configuring automation. A short, interpretable scorecard is easier to improve than a large collection of overlapping questions that produces an impressive but unexplained total.
Level AI describes evaluation, calibration and dispute handling in its QA product. AWS documents conditional questions and scoring weights. These are product-specific public descriptions, not results of our own product testing. Confirm your required configuration, evidence access and operating responsibilities in a scoped demonstration.
How do you turn a vague criterion into a scoring contract?
Use a criterion that a reviewer can apply to evidence. This example is an editorial specification that must be adapted to the actual service policy.
| Specification | Illustrative rule |
|---|---|
| Criterion | The agent states the next action and who will take it |
| Trigger | A follow-up remains necessary at the end of the evaluated segment |
| Pass | Both action and owner are stated in the available evidence |
| Failure | A required action or owner is absent despite complete evidence |
| Not applicable | No follow-up is necessary under the approved task policy |
| Insufficient evidence | The final segment is missing or unintelligible |
Treat not applicable and insufficient evidence as distinct operational states even when a product maps both to an unscored option. Our recommendation is to keep the raw reason in an export or review record, test how the denominator and weights change, and verify the resulting score on a small hand-calculated fixture. Include a clear failure and a valid exception in the fixture. A criterion that cannot distinguish these states needs revision before automation.
What should each criterion specify?
Start with the decision the criterion supports. Separate whether the agent understood the issue from whether the proposed action was correct. Combining them makes a failure difficult to diagnose. Use a stable criterion identifier so a wording correction does not silently break reporting history.
| Specification field | Example design question |
|---|---|
| Applicability | When was a next-step explanation actually required? |
| Observable behavior | Did the agent state the action and its owner? |
| Evidence | Which passage or approved record supports the judgment? |
| Answer states | Pass, fail, not applicable, or insufficient evidence? |
| Effect | Does this influence a score, create a review task, or both? |
How should weights be chosen?
Treat weights as an operating choice tied to the purpose of the rubric. A convenience behavior should not erase a serious unresolved error. Conversely, avoid letting several nearly identical courtesy questions dominate the total simply because there are more of them. Test illustrative combinations on paper before a platform calculates them.
Write a few synthetic evaluations: strong service with one omission, weak service with polite language, and a contact where half the rubric does not apply. Have the intended users explain whether the resulting scores make sense. Change the rules when they contradict the stated purpose, rather than choosing weights only to preserve a familiar average.
What should happen before activation?
- Ask two reviewers to apply the written definitions independently to a small shared set.
- List disagreements caused by the wording separately from missing conversation context.
- Check that conditional questions open and close as intended.
- Preview zero, partial and not-applicable results in exported reports.
- Retain the approved rubric, examples, owner and effective date as one version.
Keep this design document separate from product configuration so a later migration can preserve the intended rules. For test examples, use a reference gold set. For changing an active rubric, use QA change control.
Limits
This article proposes scorecard design practices, not an industry-certified scoring standard. Sources were checked on 20 October 2026. We have not tested form behavior, verified package entitlements or established employment fairness. Your team must confirm export semantics and decide how scores may be used. A transcript cannot prove an action in another system unless that evidence is explicitly available.