Research What agent evaluations actually catch

Public benchmarks measure how capable a model is in general. They are useful for choosing a model and useless for answering the question a product team actually has: did the change we made yesterday make our feature better or worse?

That question needs an evaluation suite built from your own inputs, scored the same way every time, and run on every change. It is not glamorous work, and it is the difference between an AI feature you can improve and one you can only hope about.

Build the test set from real inputs

The best source of test cases is your own history: support tickets, documents, user requests and the outputs that were judged good or bad. Synthetic examples are useful for filling gaps, but they tend to be cleaner and more regular than what real users send.

• Start with the request types that matter most to the business, not the ones that are easiest to score.
• Include the awkward cases — the ones that caused complaints, escalations or manual fixes.
• Have the people who own the outcome write down what a correct answer looks like.
• Add a new case every time production surprises you.

Fifty well-chosen cases that run on every pull request are worth more than five thousand that run once before launch.

Score each case the right way

Different outputs need different checks. Where there is a definite answer — a category, an extracted field, a tool that should or should not be called — use exact checks. Where quality is a judgement, use a written rubric and a grader, which can be a model, provided you first compare its scores with human scores on a sample and re-check whenever the grader changes. Keep a small share of cases under regular human review, because graders drift too.

What a good suite catches

The failures an evaluation suite finds are rarely dramatic. They are the quiet regressions that nobody notices until a customer does:

• A retrieval filter that silently starts returning nothing, so the model answers from general knowledge instead of your documents.
• A prompt edit that fixes one request type and breaks two others.
• A tool that now requires a field the prompt never supplies.
• A model upgrade that is better on average and worse on the cases you care about most.
• Cost per run tripling because a context window grew.

Make it part of release

An evaluation suite only matters if it can stop a release. Run it in CI on every change that touches prompts, tools, retrieval or model choice, publish the score on the pull request, and agree a threshold below which the change does not ship.

Then watch production

Tests cover what you anticipated. Production covers everything else. Record a trace for every run — each model call, each tool call, the latency and the token count — so that when something goes wrong you can see what happened. Track cost per feature and per customer, and alert when cost or quality moves outside its normal range. When a production trace reveals a new failure, turn it into a test case. Over time the suite becomes a record of everything your feature has learned not to do.