Regression Testing AI Features in CI

Most CI gates for AI features are decorative. A small eval set runs on every pull request, the average score is compared against a frozen floor, and anything short of catastrophe passes. The dataset is not representative, the threshold was never calibrated against run-to-run variance, and the green check mark carries no information [1].

The failure is not that teams skip CI. It is that they build a gate that cannot distinguish a real regression from ordinary noise, and then either ignore it when it fires or trust it when it does not.

There is a useful way to frame the constraint: a CI eval gate needs to be cheap, fast, and statistically meaningful, and most designs achieve two of the three [1]. A 30-example suite that runs in under a minute for pennies is cheap and fast, and its variance band is wider than the regressions you are trying to catch. A two-thousand-example judge sweep on every push is statistically clean, and it gets disabled by the end of the quarter when someone reads the bill.

This guide covers how to resolve that tension. It is part of our series on evals and LLMOps, and it assumes you have already built validated evaluators through error analysis and judge validation.

Tier the suite by cost and speed

The resolution to the triangle is not one suite. It is several, running at different cadences [1].

The fast tier is where deterministic checks earn their keep. They cost nothing, run in milliseconds, and never produce a false alarm. Every failure mode that a code assertion can catch should be caught here rather than by a judge. See the grader hierarchy in the pillar.

The fast tier should also carry critical-slice invariants: small sets of cases representing behavior that must never break, such as refusing when the corpus has no answer, never leaking another tenant’s data, or always emitting valid JSON for a downstream consumer. These are binary and absolute. They block on a single failure, without statistics, because one occurrence is one too many.

Everything probabilistic belongs in the full tier, where you can afford both the tokens and the sample size to say something meaningful.

What changes should trigger a run

Any change that can move quality, which is broader than the code most teams think of as “the AI part”:

• Prompt text, including system prompts and few-shot examples
• Model, model version, provider, or region
• Sampling parameters and reasoning settings
• Retrieval configuration: chunking, embeddings, k, reranking, filters
• Tool definitions and their descriptions
• Context assembly logic and truncation rules
• The corpus itself, when documents are added, removed, or reprocessed
• The judge prompts, which need their own validation run

That last one catches people out. A judge is a piece of software with its own regression surface. Changing a judge prompt without re-validating it against human labels means every number after that change is unanchored.

Size the eval set for the regression you want to catch

This is the part that separates a real gate from theater, and it is straightforward arithmetic.

For a pass-rate metric, the standard error of the measurement is roughly the square root of p(1-p)/N, where p is your current pass rate and N is the number of cases. Research measuring noise across many evals found that paired comparisons usually show total variance close to this basic prediction, which means you can estimate your noise floor without custom analysis [2].

At a pass rate of 85%, here is what different eval set sizes can resolve:

Read that table before setting a threshold. A 30-case suite cannot reliably detect anything smaller than a 13-point collapse, so a rule like “fail if the score drops more than 5 points” on that suite will fire on noise roughly as often as on real regressions. This is exactly how engineers learn to rerun CI until it goes green.

Two ways to buy power without a proportionally bigger bill. First, pair the comparison: run the old and new versions on the same cases and analyze per-case outcomes rather than comparing two independent averages. Second, average repeated runs, since the prediction noise from regenerating answers on the same question typically exceeds the data noise from which questions you sampled, so averaging several runs per case meaningfully increases statistical power [2].

Smallest pass-rate drop a suite can resolve, by eval set size

• 30 cases 12.8 pp
• 50 cases 9.9 pp
• 100 cases 7.0 pp
• 200 cases 4.9 pp
• 500 cases 3.1 pp
• 1,000 cases 2.2 pp

Halving the noise floor costs four times the cases.

Gate on a paired comparison, not a raw threshold

A rule like “fail when the average score is below 0.80” looks precise and behaves badly, because outputs, judges, and sample composition all vary [3]. A dependable gate is a controlled paired experiment instead [3].

The shape:

1. Pin everything you can. Same cases, same order, same judge prompt and version, temperature zero for both the system under test and the judge.
2. Run both versions on the same cases and record per-case outcomes, not just totals.
3. Compare the discordant pairs. For binary pass/fail outcomes, McNemar’s test looks only at the cases where the two versions disagree, which is the right question; for small numbers of discordant pairs, use the exact binomial version rather than the chi-squared approximation [4]. A paired bootstrap confidence interval on the pass-rate difference is a good companion, since it gives you an effect size rather than only a verdict.
4. Decide against a threshold set in advance, and fail only on a significant regression [5].
5. Report the interval, not just the point estimate. A drift of -20% with an interval spanning -60% to 0% is not evidence of anything, and labeling it as such stops teams from chasing noise while still blocking on real drops [6].
6. Treat inconclusive as inconclusive, not as a pass, and log it. Otherwise a gate that never fires becomes invisible [5].

One caveat that grows with cadence. A two-sided test at the conventional 5% threshold is wrong about 5% of the time when nothing has changed, and roughly half those errors point the wrong way, so a regression-only gate fires falsely about 2.5% of the time. Run that on every pull request and you have a steady trickle of false failures, which is the alert fatigue the gate was supposed to prevent [5]. This is the strongest practical argument for keeping statistical gating on the nightly and canary tiers rather than on every push.

Always report per-category results alongside the aggregate. A net-neutral change can hide a real regression on an important slice, and inspecting which specific cases newly failed is often more informative than the overall number [7].

Establish the baseline honestly

Do not freeze a baseline on day one. Run the suite in shadow mode across normal changes first, recording false alarm rates, how often engineers rerun, time, and cost. Calibrate the judge against blinded human labels, estimate run-to-run variance, and only freeze a baseline once the measurement itself is stable [3].

Then protect it. When a product decision intentionally changes expected behavior, that is a reviewed change: the rubric, the affected cases, and the baseline all update together in a single diff, with before-and-after results attached. Updating the snapshot until CI turns green must never become the normal workflow [3]. That one habit destroys more eval suites than any technical problem.

Version the eval set alongside the code, and record which version produced every result. A score is meaningless without knowing which cases produced it.

Handle judge failures loudly

Judges are LLM calls and they fail like LLM calls: rate limits, timeouts, and malformed output. The dangerous pattern is treating an unparseable judge response as a pass.

• Parse strictly. Require a structured verdict, and treat anything unparseable as a failure rather than silently skipping the case [6].
• Keep the raw judge output. Verdicts you cannot audit are verdicts you cannot debug [6].
• Fail the run visibly on evaluator errors rather than reporting a score computed from a partial set [3].
• Report coverage alongside the score: if 12% of cases errored, the pass rate is over the 88% that completed, and the reader needs to know that.

Retry and fallback logic applies here as it does in production. See making LLM features reliable.

Keep runs fast and cheap

• Cache responses, but key the cache correctly. A cache keyed only on prompt text returns stale answers after you change a model or a parameter, so the key needs to include the model, version, and sampling settings [6].
• Parallelize across cases, respecting provider rate limits.
• Use batch APIs for nightly runs, which are typically half price and have all night to finish. See LLM API costs in production.
• Judge with the cheapest model that passed validation, which is not necessarily the one you used while establishing alignment.
• Check cost and latency in the same run. A prompt change that doubles token usage should fail CI as surely as one that drops quality, and this is the only place you will catch it before the invoice.

Make model migrations routine

This is where the investment pays off most visibly. When a provider announces a deprecation, cuts prices, or ships a new tier, the question “is this safe to switch to?” becomes a suite run and a paired comparison rather than a debate.

Run the candidate against the same cases, compare per-category, and look specifically at what newly fails rather than only at the aggregate, since a net improvement can conceal a regression on a slice that matters [7]. Watch cost and latency in the same run, because a model that is better and four times more expensive is a business decision, not an engineering one. See choosing an LLM for your product.

Common mistakes

• A tiny eval set with a threshold finer than its own noise floor.
• Comparing independent averages instead of pairing runs on the same cases.
• Reporting point estimates with no interval, then reacting to every wobble.
• Treating an inconclusive result as a pass.
• Statistical gating on every pull request, producing steady false alarms.
• Updating the baseline whenever CI goes red.
• Counting unparseable judge output as a pass.
• Reporting an aggregate that hides a collapse on one category.
• Changing a judge prompt without re-validating the judge.
• Never checking cost and latency, so token regressions ship freely.

Checklist

• Deterministic assertions and critical-slice invariants block on every pull request.
• The full judge suite runs nightly and before release, not on every push.
• The eval set is large enough to resolve the regression size you care about.
• Comparisons are paired, with per-case outcomes retained.
• Gates use a pre-set significance threshold and report effect sizes with intervals.
• Inconclusive results are logged as inconclusive.
• The baseline was frozen only after shadow-mode calibration.
• Baseline changes require a reviewed diff covering rubric, cases, and results.
• Judge errors fail the run visibly, and coverage is reported with every score.
• Cost and latency are gated alongside quality.

Keep reading

• LLM evals and LLMOps: how to know your AI feature actually works (the pillar for this series)
• Error analysis for LLM failures
• How to validate an LLM judge
• LLM observability and tracing
• Monitoring LLM quality in production

CMW AI Labs builds eval pipelines that actually gate: paired comparisons, calibrated baselines, and suites sized for the regressions that matter. If your CI passes every time and your users still find problems, talk to our team.

Sources

1. Future AGI, “CI/CD LLM Eval with GitHub Actions: 2026 Workflow,” May 2026. https://futureagi.com/blog/ci-cd-llm-eval-github-actions-2026/
2. “Measuring all the noises of LLM Evals,” arXiv:2512.21326. https://arxiv.org/abs/2512.21326
3. OneUptime, “How to Turn LLM Evaluation into a Reliable CI Regression Gate,” August 2026. https://oneuptime.com/blog/post/2026-08-31-llm-evaluation-ci-regression-gate/view
4. “Spark-LLM-Eval: A Distributed Framework for Statistically Rigorous Large Language Model Evaluation,” arXiv:2603.28769. https://arxiv.org/abs/2603.28769
5. Future AGI, “How to Calculate Statistical Significance in LLM Evals,” August 2026. https://futureagi.com/blog/statistical-significance-llm-evals/
6. EvalHive project documentation, “CI-style LLM evaluation and regression gating.” https://github.com/myb2481351924/evalhive
7. Prakash Kagitha, “Statistical Rigor in Evaluation: Confidence Intervals and Significance,” The LLM Stack. https://prakashkagitha.github.io/llm-stack-book/11-evaluation/06-statistical-rigor-eval.html