# When an agent misbehaves, your phone should know first.

Failed evals, tool errors, trace errors, baseline anomalies and drift all fire as events, the moment the data lands. They reach the Slack channel you choose as one readable digest with deep links, never a flood.

## The problem: Quality incidents do not page anyone

Your infra pages you when a pod dies. Nothing pages you when the agent starts failing its faithfulness eval, burning triple tokens, or erroring on one tool. LLM issues fall in the gap between APM and hope.

## How RunAgain does it

1. **Event-driven, not polled.** Alerting is a push pipeline: a failed eval score, a tool error or an errored trace triggers as it is written, with no scheduled-job delay between the incident and the ping.
2. **Digests, not floods.** Events coalesce into one message per short window: a broken deploy is a single Tool errors (5,000) digest with the first examples and a count, and every line deep-links to the trace.
3. **Your rules, per project.** Choose what fires (failed evals, tool errors, trace errors), set the eval-failure threshold, and route each project to its own Slack channel, including private ones.
4. **The slow failures too.** Baselines flag individual runs with abnormal cost, runtime or span count, and PSI drift monitors watch metrics over windows, so gradual degradation alerts as reliably as a hard error.

## FAQ

**Q: What kinds of agent issues can trigger an alert?**

A: Hard failures (tool errors, errored traces), quality failures (eval scores below your threshold), per-run anomalies from baselines (abnormal cost, runtime or span count) and statistical drift on any metric you monitor.

**Q: Will a bad deploy spam my Slack channel?**

A: No. Events for a project coalesce into one message per short window with a count and examples, so five thousand tool errors arrive as one digest, and each example links to its trace.

**Q: Can different teams get different alerts?**

A: Alert rules are configured per project with their own destination channel, and Slack is connected once per workspace, so each team watches its own agents in its own channel.

## Built on these solutions

- [Monitoring & alerting](https://runagain.ai/solutions/agent-monitoring-alerting.md)
- [Agent observability](https://runagain.ai/solutions/agent-observability.md)
- [Continuous improvement](https://runagain.ai/solutions/continuous-improvement-loop.md)

Book a demo: https://cal.com/tamas-szuromi/30min · Get in touch: tamas@runagain.ai

---

Markdown mirror of https://runagain.ai/use-cases/agent-llm-alerting for agents and LLMs. Append .md to any runagain.ai page URL for its markdown twin. Overview: https://runagain.ai/llms.txt · For agents: https://runagain.ai/for-agents.md