The team demos a new model. It answers quickly, sounds confident and rarely fails in the meeting. A week later, an important but uncommon refund scenario starts producing the wrong advice. The answer still sounds perfectly polished.
AI Evaluation should measure the system on the tasks it is responsible for, not on how convincing a handful of examples look.
Start with the cost of failure
An internal knowledge assistant must avoid inventing policy and ideally show sources. An intake classifier needs to route the right requests and escalate ambiguous ones. An agent with write access must not perform unauthorised actions.
A single “90% quality” figure conceals those differences. A punctuation issue and a wrong bank account number should not carry equal weight. Group evaluations by real scenarios and severity of the consequences.
Build a test set that survives model changes
Collect repeatable examples covering routine questions, edge cases, ambiguity, stale documents, missing evidence and tool failures. Run them whenever prompts, models, retrieval or tools change, looking for regressions as well as improvements.
Automated checks can verify formats and known facts. Another model may serve as a judge for suitable dimensions, but model grading needs calibration against human review. Its scores are evidence, not absolute truth.
With RAG, evaluate retrieval separately
A wrong answer may originate before generation. The system might retrieve the wrong policy, select an obsolete version or miss the relevant paragraph. For RAG, measure whether relevant sources were found, whether citations support the answer and whether the final answer is accurate.
An elegant paraphrase is not proof. Important claims should be grounded in checkable information.
Evaluation continues after release
Real user requests change over time. Monitor failures, manual corrections, complaints and cost per useful completed task. Add important new failure cases to the regression set.
I would approve a model change only after comparing quality, latency, cost and risk in the actual workflows. A cheaper model may be sufficient for some tasks; for others the extra errors can erase the saving.
AI Evaluation is not asking how intelligent the model sounds. It asks whether the company can trust the system with a specific job and a known cost of getting it wrong.