// USE CASE · AGENT & LLM ALERTING
When an agent misbehaves, your phone should know first.
Failed evals, tool errors, trace errors, baseline anomalies and drift all fire as events, the moment the data lands. They reach the Slack channel you choose as one readable digest with deep links, never a flood.
THE PROBLEM
Quality incidents do not page anyone
Your infra pages you when a pod dies. Nothing pages you when the agent starts failing its faithfulness eval, burning triple tokens, or erroring on one tool. LLM issues fall in the gap between APM and hope.
// HOW RUNAGAIN DOES IT
Alerting is a push pipeline: a failed eval score, a tool error or an errored trace triggers as it is written, with no scheduled-job delay between the incident and the ping.
Events coalesce into one message per short window: a broken deploy is a single Tool errors (5,000) digest with the first examples and a count, and every line deep-links to the trace.
Choose what fires (failed evals, tool errors, trace errors), set the eval-failure threshold, and route each project to its own Slack channel, including private ones.
Baselines flag individual runs with abnormal cost, runtime or span count, and PSI drift monitors watch metrics over windows, so gradual degradation alerts as reliably as a hard error.
support-bot
last 24h
The metrics that page you, watched around the clock.
FREQUENTLY ASKED
What kinds of agent issues can trigger an alert?
Hard failures (tool errors, errored traces), quality failures (eval scores below your threshold), per-run anomalies from baselines (abnormal cost, runtime or span count) and statistical drift on any metric you monitor.
Will a bad deploy spam my Slack channel?
No. Events for a project coalesce into one message per short window with a count and examples, so five thousand tool errors arrive as one digest, and each example links to its trace.
Can different teams get different alerts?
Alert rules are configured per project with their own destination channel, and Slack is connected once per workspace, so each team watches its own agents in its own channel.
BUILT ON THESE SOLUTIONS
More use cases: llm cost tracking · reduce llm costs · time-travel debugging · regression testing in ci · prompt a/b testing · hallucination detection · conversation tracing · llm-judge calibration · tool-call failure analysis · structured output validation · tracing coding agents
Be first to run again.
Book 30 minutes and see the loop on your own agents, or write to tamas@runagain.ai.