Evals as a deploy gate — the small harness that catches what "looks right" misses

The difference between an AI feature that ships well and one that quietly degrades comes down to a single question: how do you know the model is still doing what you wanted, the day after you change the prompt?

If the answer is "we look at the output and it seems fine," you don't have an answer. You have a guess. And the guess will be wrong about three weeks after a model upgrade silently shifts the average response in a way nobody notices until a user complaint surfaces it.

Tests don't help with this. Tests check that your code does what your code says. They can't check that the model does what the product needs, because the model is non-deterministic and your tests are deterministic.

Evals are the missing layer.

What evals actually are

An eval is a test that runs the model — or the AI feature, including its prompt, its tool definitions, its post-processing — against a fixed set of inputs and scores the outputs against an expected shape, value, or property.

Three things make them different from regular tests:

• The set of inputs is curated, not generated. A small handful of cases you've explicitly chosen because they represent something important — golden cases, regression cases, known-edge cases.
• The scoring is fuzzy, not exact. "The summary contains the article's key entities," not "the output equals this string." Sometimes scoring is exact (schema validation passes / fails), but often it's a graded judgment.
• They run on every prompt change, every model upgrade, and every release. Not just on commit. The whole point is that prompt edits and model swaps are the things you don't otherwise notice the consequences of.

The minimum viable harness

You don't need a vendor for this on day one. You need three pieces:

A list of test cases, each shaped like:

type EvalCase = {
  id: string;
  input: unknown;        // whatever the feature takes
  expected: {
    schema?: ZodSchema;  // structural correctness
    contains?: string[]; // substrings that must appear
    excludes?: string[]; // substrings that must NOT appear
    custom?: (output: unknown) => Promise<{ pass: boolean; reason: string }>;
  };
  tags: string[];        // 'golden' | 'regression' | 'edge' | etc.
};

A runner that, for each case, calls the feature end-to-end (prompt + model + post-processing), captures the output, and checks it against the expected block. Scoring per case is binary — pass or fail with a reason — and the harness aggregates the pass rate per tag.

A summary report that prints the per-tag pass rate, lists the failing cases with their reasons, and exits non-zero if the pass rate falls below a threshold you've configured.

That's it. ~200 lines of TypeScript including the runner. No vendor, no infrastructure, no UI. The artifact is a JSON report that lives next to the feature in the repo.

What I actually eval

Three categories, and they're not equally important:

1. Schema correctness. Does the output match the shape my code expects? This is the cheapest eval to write and the highest-leverage. If a prompt change causes the model to start returning a different shape, this catches it immediately. With forced tool use, schema correctness is almost free — the API enforces it — but I still eval it because I want to detect when the model starts refusing to call the tool, which is its own failure mode.

2. Property assertions. Does the output have the properties the product needs? "The summary mentions the article's main entity." "The translation preserves the proper nouns." "The extraction picks up at least one of the canonical fields." These are scored with contains, excludes, or a custom function — sometimes the custom function is just "does the output contain X," sometimes it's a small regex, sometimes it's another LLM call asking "does this summary include the main entity from this article."

3. Regression cases. Specific past failures, captured as eval cases the moment they're observed. When a user reports "the model said something weird about input X," I add input X to the eval set with an expected that asserts the weird thing doesn't happen. That eval runs forever after, and I find out the moment a future prompt change re-introduces the bug.

LLM-as-judge: when, when not

The pattern of using a separate LLM call to grade an LLM output is seductive and sometimes correct. The two cases where I do it:

• The grading criterion is genuinely fuzzy — "is this translation natural-sounding," "does this summary read like a human wrote it" — and I can write a clear rubric for the grader.
• The grading volume is small enough that the additional API cost is noise.

Cases where I avoid it:

• Whenever a deterministic check works. "Does the output contain the word X" is cheaper, faster, and more reliable than an LLM judge for the same property.
• When I haven't pinned the grader model. If my eval scores depend on a grader model that itself updates, the eval is comparing two moving targets and the results are noise.

The deploy gate pattern

Evals only earn their cost if they block bad releases. The harness runs as part of CI on every change that touches the feature — prompt files, tool schemas, post-processing code. If pass rates drop below the configured threshold for any tag, the build fails. The PR can't merge.

The threshold isn't 100%. It's whatever the current pass rate is, minus a small buffer. The harness pins the baseline at the start of each PR run and fails if the new pass rate is significantly worse. That phrasing matters: I'm not asking "are all evals passing?" — I'm asking "did this change make things worse?" The first question is too strict to be useful; the second is the actual decision the gate is making.

Build vs buy

For the first six months of an AI feature, build. The minimum harness is two hundred lines of code, owned, in your repo, with no vendor lock-in, no second pricing surface, and full visibility into what scoring actually does.

After six months, if the eval set has grown to hundreds of cases, scoring has become genuinely complex, or multiple people need to inspect runs in a UI, switch to a vendor (Braintrust, Langsmith, Phoenix). The migration is straightforward because the eval cases are just data — the runner is the part you replace.

The mistake I see most often is reaching for the vendor on day one, before the team has internalised what they actually want to score. The vendor's UI then defines the eval shape, instead of the product's needs defining it. Build first; buy when the surface justifies it.

What I skip

• Comprehensive coverage. Most AI feature evals start with twenty cases, not two thousand. Coverage compounds over time as failures get added back as regression cases. Trying to be exhaustive on day one is how the harness never ships.
• Continuous online evaluation. I run evals offline on a curated set, not on every production response. Scoring every response with an LLM judge would double the API cost; running deterministic checks on every response is rarely actionable. Online sampling is a v2 feature.
• Latency evals. Latency belongs in regular APM, not the eval harness. Mixing the two means the harness becomes about everything and useful for nothing.

The general rule

Tests answer "does my code do what I told it to." Evals answer "does the feature do what the product needs." Those are different questions, and the second one only gets answered if you build the layer that asks it explicitly.

The single biggest difference I've seen between AI features that survive in production and AI features that quietly stop working: the surviving ones have an eval harness, and the harness blocks deploys when the answer to the second question is "no, not anymore."

You don't need a fancy harness. You need a harness that runs.

---

The eval harness this post describes is part of the AI engineering surface I work on at the multi-tenant AI mirror-site platform.