// VERIFY · EXPERIMENTATION

Change the prompt. Prove it's better.

Run any prompt, model or agent config over a versioned dataset, score every output, and compare against baseline with actual statistics: confidence intervals and significance tests, so "it feels better" becomes "it is better, p < 0.05."

THE PROBLEM

Eyeballing three outputs is not an experiment

Most prompt changes ship after someone reads a handful of outputs and nods. Sampling noise reads as improvement, regressions hide in the cases nobody re-ran, and the same argument repeats every review.

// HOW RUNAGAIN SOLVES IT

01Experiments over versioned datasets

Pick a dataset, a prompt template and a task model; run every item with N trials to measure non-determinism. Each run item records output, latency, tokens, cost and errors, and then the scorers grade it.

02Statistics, not vibes

The compare view shows pass rates with 95% Wilson intervals and runs McNemar's paired test over shared items. A change is only labeled improvement or regression when it's unlikely to be sampling noise.

03Honest comparisons

Items whose data changed between runs are excluded by content hash, so an edited dataset can't masquerade as a regression. Changed items show input and both outputs side by side.

04From experiment to gate

The winning config doesn't stay a chart. The same experiment runs in CI with thresholds and --fail-on-regression, so the bar you proved becomes the bar you keep.

01 TRACE

Every span, tool call and token, captured live.

02 EXPERIMENT

A/B prompt

Fork prompts, models and tools behind flags.

03 TEST

✓ mocks

✓ sims

Deterministic runs against mocks and sims.

04 EVALUATE

0.91

LLM-judge, rubric, assertion and dataset scoring.

05 IMPROVE

Diff regressions, fix the weak spans, ship.

06 RUN AGAIN

The loop closes; every pass raises the score.

Experiment feeds Evaluate: every fork gets a score.

COMMON USE CASES

USE CASEReduce LLM costsread →USE CASERegression testing in CIread →USE CASEPrompt A/B testingread →USE CASELLM-judge calibrationread →

Be first to run again.

Book 30 minutes and see the loop on your own agents, or write to tamas@runagain.ai.