# Stop eyeballing prompts. Start measuring them.

Run variant A and variant B over the same versioned dataset, score every output with your evals, and read the comparison with confidence intervals and a significance test. The winner is a number, not a vibe.

## The problem: Three good-looking outputs is not evidence

Most prompt changes ship after someone reads a handful of outputs and nods. Sampling noise reads as improvement, regressions hide in the cases nobody re-ran, and the same subjective argument repeats in every review.

## How RunAgain does it

1. **Variants over the same dataset.** Each experiment fixes the dataset version, prompt template and task model, and runs every item with optional repeated trials. Both variants face exactly the same cases.
2. **Scored, not skimmed.** Outputs are graded by the scorers you choose: built-in LLM judges, rubrics, assertions, text-overlap metrics or dataset comparison, each writing a score per item.
3. **A compare view with statistics.** Pass rates come with 95% Wilson intervals, and McNemar's paired test labels the change an improvement or regression only when it is unlikely to be noise. Changed items show both outputs side by side.
4. **Winner becomes the gate.** Promote the winning variant's experiment to the CI baseline, so the quality you just proved is the quality every future change is measured against.

## FAQ

**Q: How many test cases do I need to compare two prompts?**

A: Fewer than you might think, because the compare view is honest about uncertainty: Wilson intervals widen with small samples and the significance test refuses to call close races. Start with a few dozen captured production cases and grow the dataset from real traffic.

**Q: Can I A/B test models, not just prompts?**

A: Yes. The task model is a parameter of the experiment like the prompt template, so cheaper-model comparisons run exactly the same way, scored on the same items.

**Q: What stops an edited dataset from faking a regression?**

A: Items are content-hashed. If an item changed between the two runs it is excluded from the comparison and listed as not comparable, so dataset edits cannot masquerade as behaviour changes.

## Built on these solutions

- [Experimentation](https://runagain.ai/solutions/agent-experimentation.md)
- [Simulation & evals](https://runagain.ai/solutions/agent-simulation-evals.md)
- [Goal optimization](https://runagain.ai/solutions/agent-goal-optimization.md)

Book a demo: https://cal.com/tamas-szuromi/30min · Get in touch: tamas@runagain.ai

---

Markdown mirror of https://runagain.ai/use-cases/prompt-ab-testing for agents and LLMs. Append .md to any runagain.ai page URL for its markdown twin. Overview: https://runagain.ai/llms.txt · For agents: https://runagain.ai/for-agents.md