AI Eval Harnesses in CI: Catch Regressions Before Users Do

Offline eval notebooks are useful until someone merges a "small prompt tweak" on Friday and Monday's users hit a quieter, worse product. If model behavior is part of the shipping surface, regressions belong in CI — next to typechecks and unit tests — not in a spreadsheet someone remembers to open after an incident.

What an eval harness is (and is not)

A harness is a fixed set of cases, a runner that calls your real feature path (prompt version, retrieval, tools, schema validation), and a scorer that produces a number you can gate on. It is not a vibes review in Slack, not a single cherry-picked demo, and not a leaderboard that only the AI team understands. If engineers cannot run it locally and see why a case failed, it will not protect the branch.

• Cases: inputs, fixtures (docs, tool stubs), and expected outcomes
• Runner: the same code path production uses, with deterministic stubs where needed
• Scorers: exact match, schema validity, rubric checks, and grounded citation rules
• Gate: a threshold that fails the PR when quality or cost slips

Put the suite on every relevant PR

Trigger evals when prompts, retrieval config, tool schemas, model IDs, or the feature code change. Do not run a full LLM suite on every CSS tweak — path filters keep CI honest and budgets sane. Cache fixtures and pin model versions so a provider rollout does not look like your regression.

Separate fast gates from deep suites

Use a fast smoke set on every PR: schema validity, refusals, and the top failure modes you already burned on. Run a deeper nightly or pre-release suite for long-tail quality, multi-turn flows, and cost comparisons across models. Fast gates protect merge velocity; deep suites protect release confidence.

Score what product owners care about

Token accuracy is not the goal. Score the contract of the feature: correct structured fields, safe refusals, grounded answers with citations, tool calls that match allowlists, and latency/cost within budget. When a score drops, the PR comment should name the failing cases — not dump a raw JSON blob nobody will read.

• Fail closed on schema and policy violations — these are bugs, not style
• Track p50/p95 latency and estimated cost per case and per suite
• Diff scores against main so reviewers see deltas, not absolute mystery numbers
• Attach failing case IDs and short diffs of expected vs actual in the CI summary

“If CI cannot tell you which case got worse and why, you do not have an eval gate — you have a flaky health check.”

Keep the harness honest

Flaky evals teach teams to ignore red builds. Stub non-determinism where you can (retrieval corpora, clocks, tool side effects), set retry budgets for provider blips, and quarantine known-flake cases instead of silently lowering the threshold. When the model vendor ships a behavior change, update the pin deliberately — do not let CI absorb the surprise as "noise."

Ownership and cost

Someone owns the suite: case quality, thresholds, and the weekly review of failures that almost shipped. Cap spend per PR and per day so evals cannot become a surprise cloud bill. At Brutanix Studios we treat AI eval harnesses like any other critical path test — versioned with the feature, gated in CI, and fed by production failures — because "we'll eyeball it" is not a release strategy.