# Cut the spend without cutting the quality.

Baselines flag the runs that cost triple the norm, drift monitors catch token growth the week it starts, and the trace shows exactly which span to fix. Then experiments prove the cheaper prompt or model scores just as well before it ships.

## The problem: Optimizing blind is how quality dies

The usual cost cut is switching everything to a cheaper model and hoping. Without per-span evidence you cut the wrong thing: the real money is often one retry loop, one bloated prompt, or one agent calling a tool nine times where three would do.

## How RunAgain does it

1. **Baselines flag the outliers.** Abnormal cost, runtime or span count is flagged automatically per agent, so the run that took a nine-tool detour gets a health score before anyone reads a bill.
2. **Drift monitors on tokens.** Put a PSI drift monitor on total_tokens (or cost itself) per window and get alerted in Slack when usage starts growing, weeks before the invoice says so.
3. **The expensive span, pinpointed.** The trace waterfall shows where the money went inside a single run: the retry loop, the prompt that grew, the tool result that ballooned the context.
4. **Prove the cheaper config.** Experiments compare the cheaper model or shorter prompt against baseline on your eval scores, with confidence intervals and significance testing, so you ship the one that is cheaper and provably not worse.

## FAQ

**Q: How do I find which agent or feature costs the most?**

A: Cost is a typed column on every trace with agent, user, version and custom metadata alongside it. Group by any of those dimensions to rank spend, then open the expensive traces to see which spans carry the cost.

**Q: Can I get alerted when costs spike?**

A: Yes. Run baselines flag individual runs with abnormal cost per agent, and drift monitors watch token or cost distributions per window and alert on divergence. Alerts post to Slack as coalesced digests with deep links to the traces.

**Q: Will switching to a cheaper model hurt quality?**

A: That is what experiments answer: run both configs over the same versioned dataset, score them with your evals, and compare with McNemar significance testing. A regression is only called when it is unlikely to be sampling noise.

## Built on these solutions

- [Monitoring & alerting](https://runagain.ai/solutions/agent-monitoring-alerting.md)
- [Experimentation](https://runagain.ai/solutions/agent-experimentation.md)
- [Goal optimization](https://runagain.ai/solutions/agent-goal-optimization.md)

Book a demo: https://cal.com/tamas-szuromi/30min · Get in touch: tamas@runagain.ai

---

Markdown mirror of https://runagain.ai/use-cases/reduce-llm-costs for agents and LLMs. Append .md to any runagain.ai page URL for its markdown twin. Overview: https://runagain.ai/llms.txt · For agents: https://runagain.ai/for-agents.md