# Trust the judge because you measured it, not because it sounds right.

Annotation queues put real traces in front of humans, their labels pair against the judge's scores by name, and the alignment table tells you exactly how much to trust each judge before it gates anything.

## The problem: An uncalibrated judge is just vibes at scale

LLM-as-judge makes evaluation cheap, and cheaply wrong: a judge that praises fluent nonsense will approve it a thousand times an hour. Without measuring agreement against humans, you have automated your own blind spot.

## How RunAgain does it

1. **Human review queues.** Create a queue with reviewer instructions, add traces to it, and reviewers score them with a name, value and comment. Annotations land as first-class scores next to the eval scores on each trace.
2. **Alignment, per judge.** Annotate using the same score name the judge writes and the two are paired per trace. The judge-alignment table then shows agreement for every score name, so calibration is a number you watch, not a hope.
3. **Fix the judge, not the metric.** When alignment is poor, tighten the judge prompt, switch the judge model or move to a weighted rubric, then re-measure against the same human labels until agreement holds.
4. **Gold labels compound.** Annotated traces double as gold datasets for experiments, so the human effort you spend calibrating also hardens your regression suites.

## FAQ

**Q: How do I know if my LLM judge is reliable?**

A: Annotate a sample of the same traces the judge scores, using the same score name. RunAgain pairs the two per trace and reports agreement in the judge-alignment table, so reliability is measured rather than assumed.

**Q: How many human labels do I need?**

A: Start with a few dozen representative traces per judge, weighted toward the hard cases. Alignment numbers stabilize quickly, and the queue keeps collecting as reviewers spot-check production over time.

**Q: What if the judge and humans disagree?**

A: Treat the judge as broken, not the humans. Refine the judge prompt or model, or split the criterion into a weighted rubric with per-dimension minimums, then re-check alignment on the same labels.

## Built on these solutions

- [Simulation & evals](https://runagain.ai/solutions/agent-simulation-evals.md)
- [Continuous improvement](https://runagain.ai/solutions/continuous-improvement-loop.md)
- [Experimentation](https://runagain.ai/solutions/agent-experimentation.md)

Book a demo: https://cal.com/tamas-szuromi/30min · Get in touch: tamas@runagain.ai

---

Markdown mirror of https://runagain.ai/use-cases/llm-judge-calibration for agents and LLMs. Append .md to any runagain.ai page URL for its markdown twin. Overview: https://runagain.ai/llms.txt · For agents: https://runagain.ai/for-agents.md