// USE CASE · REGRESSION TESTING IN CI
The prompt change that breaks checkout never merges.
The runagain-eval CLI runs your experiment in CI, applies per-scorer thresholds, and compares against a baseline with a significance test. A real regression fails the build; sampling noise does not.
THE PROBLEM
Agents ship without a test gate
Code has CI. Prompts, models and tool configs usually do not, so the change that quietly breaks a workflow sails through review and gets discovered by a customer. The pieces that behave least deterministically get the least gating.
// HOW RUNAGAIN DOES IT
runagain-eval triggers the experiment, polls for per-scorer results, applies your policy and exits non-zero on failure. It drops into GitHub Actions or any pipeline like any other check.
Set minimum averages per metric, for example faithfulness at 0.9 and correctness at 0.8, each flag repeatable, so the bar is explicit and versioned with your pipeline.
With a baseline experiment set, the build fails only on a McNemar-significant regression over shared items. Small samples and flaky items stop producing false alarms that teach people to ignore the gate.
The dataset behind the gate is captured from real traces, including past failures, and versioned. Every merge is tested against things that actually happened.
Test and evaluate run on every merge; the gate holds the bar.
FREQUENTLY ASKED
How do I run agent evals in GitHub Actions?
Add a step that calls runagain-eval with your eval worker URL, a shared secret, the experiment id and your thresholds. The CLI triggers the run, waits for results and exits non-zero when the policy fails, which fails the job.
Will flaky eval scores block my merges?
No. With a baseline set, regressions are called with McNemar's paired significance test over the items both runs share, so a drop is only blocking when it is unlikely to be sampling noise. Items whose data changed are excluded by content hash.
Can I run each test case more than once?
Yes. Experiments support N trials per item to measure non-determinism, so an occasionally flaky behaviour shows up as a rate rather than a coin flip.
BUILT ON THESE SOLUTIONS
More use cases: llm cost tracking · reduce llm costs · time-travel debugging · prompt a/b testing · hallucination detection · conversation tracing · llm-judge calibration · tool-call failure analysis · structured output validation · tracing coding agents · agent & llm alerting
Be first to run again.
Book 30 minutes and see the loop on your own agents, or write to tamas@runagain.ai.