Define a Quality Bar and the Evaluations That Enforce It

"Is it good enough to ship?" gets answered in a meeting by whoever demoed most convincingly. Then it ships, real users hit cases nobody tried, and the team learns about quality from support tickets.

An evaluation set is the alternative, and the reason to build one is not rigour for its own sake. It is that without a fixed set, you cannot tell whether a change improved the product or merely improved the three examples someone happened to try.

Derive thresholds from what errors cost

A single accuracy number is not a quality bar, because errors are not interchangeable.

Start by listing the ways the feature can be wrong and what each costs. A fabricated fact presented confidently in a medical or financial context is severe and may be unacceptable at any rate. A wrong but clearly hedged answer is much cheaper. A refusal to answer something reasonable is a mild annoyance that becomes serious in aggregate. A correct answer in the wrong format may break an integration while being substantively fine. A slow answer costs abandonment.

Then set a threshold per error type, and be honest that some of them are near-zero rather than merely low. It is entirely reasonable for a product to accept a moderate rate of unhelpful answers and essentially no rate of confident fabrication.

Two more things belong in the bar. Segment thresholds, because an aggregate that hides a group with much worse performance is a fairness problem waiting to be discovered publicly. And a comparison baseline — the current system, a human, or the previous release — since "85 percent" means nothing without knowing what it replaces.

Build the set from real failures

An evaluation set assembled by writing test cases at a desk tests what you imagined. The useful sources are:

Real user inputs, sampled across the actual distribution rather than cherry-picked. This is the backbone.

Every production failure, added the day it is found. A set that grows from real incidents is the most valuable asset in the whole practice, and it makes regressions on known problems impossible to ship twice.

Adversarial cases — prompt injection attempts, edge-case formats, ambiguous requests, and inputs designed to elicit unsafe output.

Segment coverage, deliberately including the groups, languages, and contexts that a random sample would under-represent.

Frozen golden cases that must never regress, kept separate and treated as blocking.

A few hundred well-chosen cases beats thousands of generated ones. Keep it in version control alongside the code, and treat adding a case as part of fixing a bug — a fix without a case is a regression waiting to recur.

Match the judging method to the dimension

Different quality dimensions need different judges, and using one method for everything is what makes evaluation either unreliable or unaffordable.

Deterministic checks — schema validity, required fields, forbidden content, latency — are cheap, exact, and should run on everything. Start here; a surprising share of production incidents are format violations.

Reference comparison works when there is a correct answer, though exact match is usually too strict and needs a semantic comparison.

Model-as-judge scales to subjective dimensions and needs its own validation: check the judge against human ratings on a sample before trusting it, watch for its bias toward longer and more confident answers, and re-validate when you change the judge model.

Human review remains necessary for the highest-stakes dimensions. Use it on a sample, with a written rubric and more than one rater, and measure agreement between raters — low agreement means the rubric is ambiguous rather than that the raters are careless.

Gate the release on evidence

Write down, before the release, what would stop it: golden cases must all pass; no segment may fall below its threshold; severe-error rate must not increase; latency at the ninety-fifth percentile must stay within budget; cost per task must stay within budget.

Then run the gate on every change, including prompt edits. Prompt changes are code changes with an unusually high blast radius, and the fact that they are easy to make is exactly why they need the same gate.

Two things make this stick. Keep evaluation running in production, because the input distribution shifts and providers update models underneath you. And treat a gate failure as a stop, not as a discussion — a threshold that gets waived under schedule pressure is not a threshold, and everyone learns that within one release cycle.

Practice

Build a hundred-case set for one feature: sixty from real usage, twenty from known failures, twenty adversarial and segment cases. Write the thresholds by error type with the cost reasoning next to each.

Run your current system against it and publish the results, segmented. Most teams find the aggregate looks acceptable and one segment does not, which is precisely the finding the aggregate existed to hide.

Then wire the gate into your release process and deliberately try to ship a regression. If it gets through, the gate is decorative.

Common mistakes

One accuracy number. It treats a confident fabrication and a clumsy phrasing as the same event.

Test cases written at a desk. They test what you imagined, and users do not do that.

An unvalidated model judge. It has its own biases, and length is a big one.

Waiving the gate under deadline pressure. Once, and it is no longer a gate.