# Change the prompt. Prove it's better.

Run any prompt, model or agent config over a versioned dataset, score every output, and compare against baseline with actual statistics: confidence intervals and significance tests, so "it feels better" becomes "it is better, p < 0.05."

## The problem: Eyeballing three outputs is not an experiment

Most prompt changes ship after someone reads a handful of outputs and nods. Sampling noise reads as improvement, regressions hide in the cases nobody re-ran, and the same argument repeats every review.

## How RunAgain solves it

1. **Experiments over versioned datasets.** Pick a dataset, a prompt template and a task model; run every item with N trials to measure non-determinism. Each run item records output, latency, tokens, cost and errors, and then the scorers grade it.
2. **Statistics, not vibes.** The compare view shows pass rates with 95% Wilson intervals and runs McNemar's paired test over shared items. A change is only labeled improvement or regression when it's unlikely to be sampling noise.
3. **Honest comparisons.** Items whose data changed between runs are excluded by content hash, so an edited dataset can't masquerade as a regression. Changed items show input and both outputs side by side.
4. **From experiment to gate.** The winning config doesn't stay a chart. The same experiment runs in CI with thresholds and --fail-on-regression, so the bar you proved becomes the bar you keep.

## Related solutions

- [Simulation & evals](https://runagain.ai/solutions/agent-simulation-evals.md)
- [Mocked environments](https://runagain.ai/solutions/mocked-environments.md)
- [Goal optimization](https://runagain.ai/solutions/agent-goal-optimization.md)

Book a demo: https://cal.com/tamas-szuromi/30min · Get in touch: tamas@runagain.ai

---

Markdown mirror of https://runagain.ai/solutions/agent-experimentation for agents and LLMs. Append .md to any runagain.ai page URL for its markdown twin. Overview: https://runagain.ai/llms.txt · For agents: https://runagain.ai/for-agents.md