// USE CASE · LLM-JUDGE CALIBRATION
Trust the judge because you measured it, not because it sounds right.
Annotation queues put real traces in front of humans, their labels pair against the judge's scores by name, and the alignment table tells you exactly how much to trust each judge before it gates anything.
THE PROBLEM
An uncalibrated judge is just vibes at scale
LLM-as-judge makes evaluation cheap, and cheaply wrong: a judge that praises fluent nonsense will approve it a thousand times an hour. Without measuring agreement against humans, you have automated your own blind spot.
// HOW RUNAGAIN DOES IT
Create a queue with reviewer instructions, add traces to it, and reviewers score them with a name, value and comment. Annotations land as first-class scores next to the eval scores on each trace.
Annotate using the same score name the judge writes and the two are paired per trace. The judge-alignment table then shows agreement for every score name, so calibration is a number you watch, not a hope.
When alignment is poor, tighten the judge prompt, switch the judge model or move to a weighted rubric, then re-measure against the same human labels until agreement holds.
Annotated traces double as gold datasets for experiments, so the human effort you spend calibrating also hardens your regression suites.
support-bot
last 24h
Judge agreement, tracked like any other health metric.
Wondering exactly what you can score with? Browse the full eval catalog →
FREQUENTLY ASKED
How do I know if my LLM judge is reliable?
Annotate a sample of the same traces the judge scores, using the same score name. RunAgain pairs the two per trace and reports agreement in the judge-alignment table, so reliability is measured rather than assumed.
How many human labels do I need?
Start with a few dozen representative traces per judge, weighted toward the hard cases. Alignment numbers stabilize quickly, and the queue keeps collecting as reviewers spot-check production over time.
What if the judge and humans disagree?
Treat the judge as broken, not the humans. Refine the judge prompt or model, or split the criterion into a weighted rubric with per-dimension minimums, then re-check alignment on the same labels.
BUILT ON THESE SOLUTIONS
More use cases: llm cost tracking · reduce llm costs · time-travel debugging · regression testing in ci · prompt a/b testing · hallucination detection · conversation tracing · tool-call failure analysis · structured output validation · tracing coding agents · agent & llm alerting
Be first to run again.
Book 30 minutes and see the loop on your own agents, or write to tamas@runagain.ai.