// USE CASE · STRUCTURED OUTPUT VALIDATION
The JSON parsed. That does not mean it is right.
Structured-output evals run property checks on the answers your agents emit and watch the shape itself, so a missing field, a renamed key or a quietly narrowed enum raises an alert instead of a downstream exception.
THE PROBLEM
Schema drift breaks the system next door
The agent's output feeds code: a parser, a workflow, a database. When a model update renames a field or drops one, the agent looks fine and the consumer breaks. The error report comes from the wrong service.
// HOW RUNAGAIN DOES IT
The structured-output eval scores each sampled run's JSON answer against your property checks, with heuristic scorers like valid_json, json_diff and regex available for the simple cases.
Beyond pass or fail, structured_output.changed watches the output shape over time and flags when it shifts, catching the model update that reshaped your data before a parser does.
Run strict checks only on the agents and routes that emit machine-consumed output, scoped by agent or metadata, with sampling for the rest.
The same checks run as experiment scorers, so a prompt or model change that breaks the schema fails in CI instead of in production.
Structured checks scoring inline as runs stream through.
FREQUENTLY ASKED
How do I validate that my agent always returns valid JSON?
Attach a structured-output eval (or the valid_json heuristic scorer) as an online eval on the relevant agent. Every sampled run gets a score, failures alert to Slack, and the trace shows the offending output.
What is shape drift?
A change in the structure of the output rather than one bad value: fields appearing or disappearing, types changing, keys renamed. RunAgain tracks the shape over time and alerts when it diverges.
Can schema checks block a deploy?
Yes. Run them as scorers in a CI-gated experiment with runagain-eval and a threshold, and a schema-breaking change exits non-zero before merge.
BUILT ON THESE SOLUTIONS
More use cases: llm cost tracking · reduce llm costs · time-travel debugging · regression testing in ci · prompt a/b testing · hallucination detection · conversation tracing · llm-judge calibration · tool-call failure analysis · tracing coding agents · agent & llm alerting
Be first to run again.
Book 30 minutes and see the loop on your own agents, or write to tamas@runagain.ai.