Evaluate proactive suggestions against the agent decision underway, counting distraction and missed opportunities separately from answer correctness. A perfectly accurate policy card can still be unhelpful if the customer has changed topics or the agent already completed the step. Relevance is a workflow judgment, not a substitute for grounding review.
Level AI describes contextual agent assistance. Google Cloud describes proactive query and answer suggestions. Use representative conversations to evaluate when assistance deserves attention rather than assuming that more suggestions indicate better support.
What counts as a useful suggestion?
Define the customer task and current agent decision for each reviewed moment. Mark whether a suggestion supplies necessary information, anticipates an imminent step, duplicates something already known, or distracts from a more urgent issue. Ask agents to explain their decision before showing an evaluator's label.
Keep factual accuracy in a separate column. A relevant but wrong suggestion needs a different correction from an accurate but irrelevant one. The knowledge-grounding guide addresses support and correctness; this evaluation focuses on attention and timing within the work.
Which labels should a relevance log use?
| Label | Operational meaning | Follow-up question |
|---|---|---|
| Actionable | Helps the current decision | Was the assistance needed now? |
| Premature | May help later | What event should trigger it? |
| Redundant | Repeats visible information | Can repeated cards be reduced? |
| Off-topic | Addresses another issue | Did context fail to update? |
| Missed opportunity | Needed help did not appear | Was the request recognizable? |
These labels are proposed review categories, not native features promised by a vendor. Allow more than one label when a suggestion is both late and redundant. Retain a short explanation so tuning decisions are based on observable cases.
How should you measure interruption load?
Count visible suggestions per contact, repeated cards, dismissals, and time spent shifting attention. Observe whether a new card moves existing content or covers required fields. Ask agents which prompts they ignored because they were unnecessary and which they missed because the interface was too busy.
Compare similar contact types and tenure levels. New agents may find a reminder useful when experienced agents do not. Treat that difference as a design question rather than evidence that one group is using the product incorrectly. Verify whether configuration can reflect the distinction.
How should the team tune the experience?
Change one trigger, content category, or presentation setting at a time. Preserve the prior configuration and review the same reference cases after each change. Check that suppressing noise does not remove help for uncommon but consequential situations.
Agree on an owner for the relevance log and a cadence for reviewing unresolved labels. Use the adoption diagnosis guide if low usage persists despite relevant suggestions.
How should a suggestion log distinguish relevance from correctness?
Use an illustrative damaged-order conversation. A return-policy card appears while the customer describes a late parcel; then the customer clarifies that the parcel arrived damaged. The first card may be accurate about returns yet irrelevant to both current decisions. A replacement-policy answer after the clarification can be relevant but still wrong about eligibility. Record two judgments.
| Observation | Relevance label | Separate quality check |
|---|---|---|
| Correct policy for an abandoned topic | Stale | Source may be accurate; timing failed. |
| Replacement guidance for the current issue | Potentially useful | Verify conditions and cited passage. |
| The required exception answer never appears | Missed opportunity | No answer exists to score for accuracy. |
| Guidance duplicates the step already completed | Redundant | Inspect the event that should stop repetition. |
Report useful suggestions divided by displayed suggestions and covered opportunities divided by independently identified opportunities. The first can look better if a system shows almost nothing; the second exposes omissions. Keep the opportunity-labeling rule fixed when tuning a trigger, and sample unprompted contacts so the assistant does not define its own denominator.
What are the limits of this guide?
Public sources were checked October 20, 2026. No interruption study or comparative relevance benchmark was conducted. Agent reports are valuable context but do not establish factual answer quality. Configuration options, personalization, and suppression controls require product-specific confirmation. Avoid interpreting dismissals alone as rejection; an agent may have already solved the problem or received the same information elsewhere.