AI features need regression tests too.
Evaluation suites turn subjective model behavior into something teams can discuss, monitor, and improve. They do not remove judgment, but they make judgment less anecdotal.
Without them, every prompt tweak is a leap of faith, every model upgrade is a gamble, and every stakeholder complaint becomes an argument nobody can settle with evidence.
The Problem Evals Solve
Traditional software has a convenient property: given the same input, it produces the same output. That makes testing tractable. Assert the expected value, compare, done.
AI features break this in three ways at once.
- Non-determinism. The same prompt can produce different text on consecutive calls.
- No single correct answer. Many outputs are acceptable. Many are subtly wrong in ways that are hard to express as an assertion.
- Invisible regression. A prompt change intended to improve one behaviour degrades another, and nothing crashes. The system keeps returning confident, well-formatted, worse answers.
That third one is what makes evals urgent rather than nice to have. Traditional bugs announce themselves. Quality regression in an AI feature is silent until a customer notices.
What An Eval Actually Is
An eval is a repeatable measurement of output quality against a fixed set of inputs.
Structurally it is not exotic. It is a test suite where the assertion is fuzzier and the result is a score rather than a boolean:
- A dataset of representative inputs.
- A grading method for each one.
- A threshold that determines whether the current state is acceptable.
- A record of how the score moves over time.
The discipline is the same as testing. The mechanics differ because the assertion is “is this good enough” rather than “does this equal 4”.
Build The Golden Dataset First
The dataset is the hard part and the part teams skip.
A useful eval set is not a hundred questions someone invented in an afternoon. It is drawn from reality and it covers the cases that actually break:
- Representative queries. Real questions from real users, sampled from logs once you have them and from domain experts before you do.
- Known-hard cases. Ambiguous phrasing, multi-part questions, questions whose answer spans several documents.
- Adversarial inputs. Prompt injection attempts, requests for information the user should not have, attempts to make the system speak outside its remit.
- Out-of-scope questions. Things the system should decline. The correct answer is a refusal, and systems that never refuse are systems that hallucinate.
- Permission-sensitive scenarios. The same question asked by two users with different access, where the correct answers differ.
- Retrieval failure cases. Questions where the source material genuinely does not contain the answer. The correct behaviour is “I do not know,” and this is the single most common thing production systems get wrong.
Start smaller than feels rigorous. Fifty well-chosen cases that cover real failure modes are worth more than five hundred generated variations of the same easy question.
Grow it from incidents. Every time the system produces a bad answer in production, that input becomes a permanent eval case. This is the same reflex as writing a regression test for a bug, and it compounds the same way.
Grading: Four Approaches
Different questions need different graders. Most suites use several.
Exact and structural checks. Where output is constrained — JSON shape, an enum value, a required field — assert it directly. Cheap, fast, deterministic. Use these wherever the format allows.
Reference-based similarity. Compare against a known-good answer using string or embedding similarity. Useful for factual retrieval where wording may vary but substance should not. Weak on long-form output, where two good answers can look dissimilar.
Rubric-based model grading. Use a model to grade output against explicit criteria: Does this answer the question? Is every claim supported by the retrieved context? Does it cite sources? Does it avoid speculation? Scale this carefully — model graders drift, and a grader with a vague rubric measures nothing.
Human review. Slower and unavoidable. Reserve it for a small rotating sample and for calibrating whether your automated graders agree with informed judgment. If the model grader and the human disagree, the rubric is wrong.
A practical rule: the grader must be more reliable than the thing it grades. If you cannot state the grading criteria precisely enough for two people to agree, the eval will produce numbers without meaning.
Where Evals Sit In The Pipeline
Evals belong in CI, gating the same way tests do — but the economics differ, so the placement differs.
flowchart TD
A[Change: prompt, model,<br/>retrieval, or chunking] --> B[Fast tier: structural<br/>+ reference checks]
B -->|Fail| C[Blocked]
B -->|Pass| D[Full tier: rubric grading<br/>on complete dataset]
D --> E{Above threshold?}
E -->|No| F[Blocked with<br/>failure report]
E -->|Yes| G{Regression on any<br/>previously passing case?}
G -->|Yes| H[Review required]
G -->|No| I[Merge and deploy]
I --> J[Production sampling]
J -->|Bad answer found| K[Add to dataset]
K --> ATwo tiers, because a full eval run costs real money and real time. The fast tier runs on every commit. The full suite runs before merge and on a schedule.
The loop back from production is the part that matters most. An eval suite that does not grow from live failures becomes a measurement of last quarter’s problems.
Thresholds And The Release Decision
A score is only useful attached to a decision rule.
Set thresholds per category rather than globally. An aggregate “87%” hides everything interesting. What you want to know is:
- Did the refusal rate on out-of-scope questions drop? That is a hallucination regression.
- Did any permission-sensitive case start leaking? That is a security incident, not a quality issue — it should hard-fail regardless of the aggregate.
- Did citation accuracy fall? Users will lose trust before they notice wrong answers.
- Did latency or cost per query move? Quality gains that triple cost may not be gains.
Some categories are gates and some are budgets. Permission leakage is a gate: one failure blocks release. General answer quality is a budget: you accept small movements and investigate large ones.
Track the trend, not just the current value. A suite passing at 91% that was at 95% a month ago is telling you something a threshold check will not.
What Evals Cannot Do
Worth stating, because eval suites can create false confidence quite efficiently.
They only measure what is in the dataset. A perfect score means you have not regressed on known cases. It says nothing about the failure mode you have not thought of.
They do not capture user experience. An answer can be factually correct, properly cited, and still unhelpful — too long, badly structured, or answering the question as asked rather than the question as meant.
They are a lagging indicator of corpus quality. If your source documents are contradictory or stale, evals will faithfully report that the system reproduces contradictory, stale answers.
And they cost money to run. A large suite with model grading against a frontier model is not free. Budget for it, or you will quietly stop running it.
Starting From Nothing
If there is no eval suite today, the useful first move is small.
- Collect twenty real questions from whoever will use the system. Not invented ones.
- Write down what a good answer looks like for each. Prose is fine.
- Run them. Record the outputs verbatim.
- Have a domain expert mark each output pass or fail, and note why the failures failed.
- Those failure reasons become your rubric.
That is a working eval suite in an afternoon. It will already tell you more than the team currently knows, and it establishes the reflex that matters: before and after, measured on the same inputs.
Everything else — automation, CI integration, tiering, trend dashboards — is refinement on top of that reflex.
The Bottom Line
Evals are not a research practice. They are the unit tests of AI features, and the reason to build them is the same: so that changing the system does not require hoping.
Without evals, every prompt edit is a production risk, every model upgrade is a guess, and every complaint is a debate about anecdotes.
With them, a team moves faster — not because the model got better, but because they finally have a way to know whether it did.