# The prompt change that breaks checkout never merges.

The runagain-eval CLI runs your experiment in CI, applies per-scorer thresholds, and compares against a baseline with a significance test. A real regression fails the build; sampling noise does not.

## The problem: Agents ship without a test gate

Code has CI. Prompts, models and tool configs usually do not, so the change that quietly breaks a workflow sails through review and gets discovered by a customer. The pieces that behave least deterministically get the least gating.

## How RunAgain does it

1. **One CLI, wired into any CI.** runagain-eval triggers the experiment, polls for per-scorer results, applies your policy and exits non-zero on failure. It drops into GitHub Actions or any pipeline like any other check.
2. **Thresholds per scorer.** Set minimum averages per metric, for example faithfulness at 0.9 and correctness at 0.8, each flag repeatable, so the bar is explicit and versioned with your pipeline.
3. **Statistically honest gating.** With a baseline experiment set, the build fails only on a McNemar-significant regression over shared items. Small samples and flaky items stop producing false alarms that teach people to ignore the gate.
4. **Suites built from production.** The dataset behind the gate is captured from real traces, including past failures, and versioned. Every merge is tested against things that actually happened.

## FAQ

**Q: How do I run agent evals in GitHub Actions?**

A: Add a step that calls runagain-eval with your eval worker URL, a shared secret, the experiment id and your thresholds. The CLI triggers the run, waits for results and exits non-zero when the policy fails, which fails the job.

**Q: Will flaky eval scores block my merges?**

A: No. With a baseline set, regressions are called with McNemar's paired significance test over the items both runs share, so a drop is only blocking when it is unlikely to be sampling noise. Items whose data changed are excluded by content hash.

**Q: Can I run each test case more than once?**

A: Yes. Experiments support N trials per item to measure non-determinism, so an occasionally flaky behaviour shows up as a rate rather than a coin flip.

## Built on these solutions

- [Mocked environments](https://runagain.ai/solutions/mocked-environments.md)
- [Experimentation](https://runagain.ai/solutions/agent-experimentation.md)
- [Simulation & evals](https://runagain.ai/solutions/agent-simulation-evals.md)

Book a demo: https://cal.com/tamas-szuromi/30min · Get in touch: tamas@runagain.ai

---

Markdown mirror of https://runagain.ai/use-cases/agent-regression-testing-ci for agents and LLMs. Append .md to any runagain.ai page URL for its markdown twin. Overview: https://runagain.ai/llms.txt · For agents: https://runagain.ai/for-agents.md