// USE CASE · TOOL-CALL FAILURE ANALYSIS
Your agent is only as good as its worst tool call.
Trajectory scoring grades how well each run used its tools, failures are classified rather than dumped, and tool errors reach Slack the moment they land. The flaky integration stops hiding inside average latency.
THE PROBLEM
Tool failures wear disguises
A timeout becomes a retry loop, a malformed response becomes a hallucinated answer, a dangling call becomes a run that just stops. By the time the failure surfaces in the output, the tool that caused it is four steps upstream.
// HOW RUNAGAIN DOES IT
Tool spans carry the tool name, status, timing and full input and output, nested under the step that made the call. The sandbox proxy can capture even un-instrumented outbound HTTP into the same trace.
A model-free score grades each run's tool use, with recovery invalidation so an agent that hit an error and recovered is not punished like one that failed outright.
Failed runs are labeled: errored span, failed tool call, truncation, dangling tool call or silent no-output. The trace list's error views and sticks make the pattern visible across runs.
Tool errors post to your chosen Slack channel as coalesced digests, one message per window with examples and deep links, never five thousand pings for one broken deploy.
The failing tool span, flagged inside its run.
FREQUENTLY ASKED
How do I find which tool fails most often?
Tool spans carry tool_name and status as typed fields, so failure rates group by tool directly. The Errors saved view and per-run failure classification narrow it to the exact calls.
What is a dangling tool call?
A run where the model requested a tool but no result ever came back, usually a crash or truncation mid-run. RunAgain classifies these separately from tool errors because the fix is different.
Can I get alerted only for one critical tool?
Alert rules are per project and evals are scopable by agent or metadata, so you can watch the payment tool strictly while treating a flaky search API as background noise.
BUILT ON THESE SOLUTIONS
More use cases: llm cost tracking · reduce llm costs · time-travel debugging · regression testing in ci · prompt a/b testing · hallucination detection · conversation tracing · llm-judge calibration · structured output validation · tracing coding agents · agent & llm alerting
Be first to run again.
Book 30 minutes and see the loop on your own agents, or write to tamas@runagain.ai.