// USE CASE · HALLUCINATION DETECTION
Catch the confident lie before your customer does.
Built-in judges score sampled production runs for factuality and faithfulness against the context the agent actually retrieved. A grounding failure shows up as a red score on the trace and a Slack alert, not as a support ticket.
THE PROBLEM
Hallucinations do not throw exceptions
The run succeeds, the response reads beautifully, and the cited policy does not exist. Nothing in your error tracking fires because nothing errored. Hallucination is a quality failure wearing a success status code.
// HOW RUNAGAIN DOES IT
Ready-made LLM judges score factuality against a reference (subset, superset, equal or disagree) and faithfulness against the documents the run actually retrieved, with no prompt to write.
Attach the judge as an online eval, scope it to the agents where truth matters most, and sample a percentage of runs so coverage and cost stay balanced.
A failed grounding score sits inline on the trace next to the retrieval spans, so you can see what the model was given versus what it claimed, in one view.
Failed evals post to Slack as coalesced digests with deep links, and a drift monitor on the faithfulness score catches slow degradation after a model or prompt change.
A grounding failure surfacing inline as the run streams by.
Wondering exactly what you can score with? Browse the full eval catalog →
FREQUENTLY ASKED
How do you detect hallucinations without ground truth?
Faithfulness judging compares the response against the context the agent retrieved in that same run, so the reference is the run's own evidence. Where curated ground truth exists, the factuality judge scores against it as well.
Do I have to run a judge on every single request?
No. Online evals have a coverage slider and scoping, so you might judge 10% of all traffic but 100% of the agents where a wrong answer is expensive.
How trustworthy is an LLM judging another LLM?
Measurably trustworthy or not: annotation queues collect human labels on the same score names, and the judge-alignment table shows how often the judge agrees with your reviewers before you rely on it.
BUILT ON THESE SOLUTIONS
More use cases: llm cost tracking · reduce llm costs · time-travel debugging · regression testing in ci · prompt a/b testing · conversation tracing · llm-judge calibration · tool-call failure analysis · structured output validation · tracing coding agents · agent & llm alerting
Be first to run again.
Book 30 minutes and see the loop on your own agents, or write to tamas@runagain.ai.