// USE CASE · PROMPT A/B TESTING
Stop eyeballing prompts. Start measuring them.
Run variant A and variant B over the same versioned dataset, score every output with your evals, and read the comparison with confidence intervals and a significance test. The winner is a number, not a vibe.
THE PROBLEM
Three good-looking outputs is not evidence
Most prompt changes ship after someone reads a handful of outputs and nods. Sampling noise reads as improvement, regressions hide in the cases nobody re-ran, and the same subjective argument repeats in every review.
// HOW RUNAGAIN DOES IT
Each experiment fixes the dataset version, prompt template and task model, and runs every item with optional repeated trials. Both variants face exactly the same cases.
Outputs are graded by the scorers you choose: built-in LLM judges, rubrics, assertions, text-overlap metrics or dataset comparison, each writing a score per item.
Pass rates come with 95% Wilson intervals, and McNemar's paired test labels the change an improvement or regression only when it is unlikely to be noise. Changed items show both outputs side by side.
Promote the winning variant's experiment to the CI baseline, so the quality you just proved is the quality every future change is measured against.
Fork, score, compare: the experiment stage of the loop.
Wondering exactly what you can score with? Browse the full eval catalog →
FREQUENTLY ASKED
How many test cases do I need to compare two prompts?
Fewer than you might think, because the compare view is honest about uncertainty: Wilson intervals widen with small samples and the significance test refuses to call close races. Start with a few dozen captured production cases and grow the dataset from real traffic.
Can I A/B test models, not just prompts?
Yes. The task model is a parameter of the experiment like the prompt template, so cheaper-model comparisons run exactly the same way, scored on the same items.
What stops an edited dataset from faking a regression?
Items are content-hashed. If an item changed between the two runs it is excluded from the comparison and listed as not comparable, so dataset edits cannot masquerade as behaviour changes.
BUILT ON THESE SOLUTIONS
More use cases: llm cost tracking · reduce llm costs · time-travel debugging · regression testing in ci · hallucination detection · conversation tracing · llm-judge calibration · tool-call failure analysis · structured output validation · tracing coding agents · agent & llm alerting
Be first to run again.
Book 30 minutes and see the loop on your own agents, or write to tamas@runagain.ai.