// USE CASE · REDUCE LLM COSTS
Cut the spend without cutting the quality.
Baselines flag the runs that cost triple the norm, drift monitors catch token growth the week it starts, and the trace shows exactly which span to fix. Then experiments prove the cheaper prompt or model scores just as well before it ships.
THE PROBLEM
Optimizing blind is how quality dies
The usual cost cut is switching everything to a cheaper model and hoping. Without per-span evidence you cut the wrong thing: the real money is often one retry loop, one bloated prompt, or one agent calling a tool nine times where three would do.
// HOW RUNAGAIN DOES IT
Abnormal cost, runtime or span count is flagged automatically per agent, so the run that took a nine-tool detour gets a health score before anyone reads a bill.
Put a PSI drift monitor on total_tokens (or cost itself) per window and get alerted in Slack when usage starts growing, weeks before the invoice says so.
The trace waterfall shows where the money went inside a single run: the retry loop, the prompt that grew, the tool result that ballooned the context.
Experiments compare the cheaper model or shorter prompt against baseline on your eval scores, with confidence intervals and significance testing, so you ship the one that is cheaper and provably not worse.
One run's spans: the expensive step is visible, not inferred.
FREQUENTLY ASKED
How do I find which agent or feature costs the most?
Cost is a typed column on every trace with agent, user, version and custom metadata alongside it. Group by any of those dimensions to rank spend, then open the expensive traces to see which spans carry the cost.
Can I get alerted when costs spike?
Yes. Run baselines flag individual runs with abnormal cost per agent, and drift monitors watch token or cost distributions per window and alert on divergence. Alerts post to Slack as coalesced digests with deep links to the traces.
Will switching to a cheaper model hurt quality?
That is what experiments answer: run both configs over the same versioned dataset, score them with your evals, and compare with McNemar significance testing. A regression is only called when it is unlikely to be sampling noise.
BUILT ON THESE SOLUTIONS
More use cases: llm cost tracking · time-travel debugging · regression testing in ci · prompt a/b testing · hallucination detection · conversation tracing · llm-judge calibration · tool-call failure analysis · structured output validation · tracing coding agents · agent & llm alerting
Be first to run again.
Book 30 minutes and see the loop on your own agents, or write to tamas@runagain.ai.