Release QA model and rubric changes through a recorded comparison, an explicit approval and a monitored effective date. Preserve the earlier configuration so a change in scoring can be distinguished from a change in service. Treat a model update as an operational event even when the vendor manages the underlying infrastructure.
Level AI describes CX-specific models, evaluations and feedback loops. AWS documents latest, preview and specific model-version choices, with availability and end-of-life limits. These are product-specific public descriptions, not results of our own product testing. Confirm your required configuration, evidence access and operating responsibilities in a scoped demonstration.
What does a usable release decision look like?
Use this illustrative change record when deciding whether a revised QA instruction should replace the current version. The cases and counts are invented for explaining the process; they are not product results.
| Release field | Illustrative entry |
|---|---|
| Change | Rubric R7 clarifies that a customer quoting an earlier promise is not an agent making that promise |
| Comparison | Both versions score the same frozen interactions; changed judgments retain both reasons |
| Review decision | Accept corrected quoting cases only if untouched genuine-promise failures remain detected |
| Reporting | Mark the effective date and keep R6 history separate from R7 rescoring |
Our recommended release rule is to approve the changed behavior explicitly, rather than approve an improved average. If the supplier cannot replay the old model, preserve its existing outputs and compare the new judgments against reviewed reference decisions; record that this is a weaker reproducibility boundary. Keep a separate rollback owner and an explanation of what happens to evaluations generated during the release interval.
What belongs in the change record?
- Change identifier, owner and planned effective time.
- Old and proposed rubric, instructions and model identifiers where available.
- Reason for the change and expected affected criteria.
- Reference and recent test populations used in comparison.
- Acceptance decision, unresolved limitations and rollback conditions.
- Communication and reporting annotations required for affected teams.
Record configuration changes even if the vendor does not expose the full model version. Capture release notices, support statements and the identifiers that are available. An unknown version is a limitation to preserve, not a reason to describe two evaluations as technically identical.
How should the old and proposed versions be compared?
Run the same fixed interactions through both versions where the system permits it. Compare changed labels at the criterion level and inspect why they changed. A difference is not automatically a regression: the older result may have been wrong. Use reviewed reference labels and keep uncertainty when the evidence does not establish a clear answer.
Test recent representative work separately from the stable reference set. The historical set supports comparison, while recent work can expose a changed queue or customer vocabulary. Record both populations so a release cannot appear better merely because the test examples changed.
What should prevent an immediate release?
| Finding | Recommended response |
|---|---|
| New unexplained failures on an important criterion | Investigate before widening use |
| Improvement limited to examples used for tuning | Check an untouched test portion |
| Unavailable rollback or replay | Document exposure and narrow the rollout |
| Different missing-score behavior | Reconcile eligibility and reporting effects |
Separate the decision to update a rubric from the decision to re-score history. Historical rescoring can be useful, but it produces a different analytical series and may affect already reviewed evaluations. Preserve both versions and label any replacement reports clearly.
How should the change be monitored afterward?
Choose a limited observation period with a named reviewer and compare error patterns, unscored cases and appeals. Annotate dashboards at the effective time. If a rollback is necessary, document what changed back, which evaluations remain affected and how users should interpret that interval.
See reference-set maintenance for stable tests and reporting denominators before combining results across versions.
Limits
Sources were checked on 20 October 2026. We have not tested version pinning, export or rollback in either product. Vendor releases, end-of-life rules and account-specific availability can limit control. This article proposes release governance, not a guarantee of reproducibility or uninterrupted scoring; confirm actual controls and notification commitments in current documentation.