# Every eval, scorer and judge RunAgain supports (47)

Everything RunAgain can score a run with, from model-free string checks to calibrated LLM judges. Attach any of them online to live traffic or offline in experiments.

## Heuristic scorers (12)

Deterministic, model-free, no API key required. Each compares an output to a reference (or checks the output alone) and returns 0 to 1. Docs: https://docs.runagain.ai/concepts/evaluations

| id | needs | measures |
| --- | --- | --- |
| `exact_match` | expected | 1 if the output equals the expected value exactly. |
| `levenshtein` | expected | Normalized edit-distance similarity. |
| `numeric_diff` | expected | Numeric closeness, where 1 means equal. |
| `json_diff` | expected | Fraction of matching JSON leaf paths. |
| `bleu` | expected | Sentence BLEU: n-grams 1 to 4 plus a brevity penalty. |
| `rouge_l` | expected | ROUGE-L, the longest-common-subsequence F1. |
| `chrf` | expected | Character n-gram F-score, tolerant of word order. |
| `token_f1` | expected | Token-overlap F1 between output and reference. |
| `valid_json` | nothing | 1 if the output parses as JSON. |
| `contains` | nothing | 1 if the output contains a given substring. |
| `regex_match` | nothing | 1 if the output matches a given regex. |
| `list_contains` | nothing | Fraction of given items present in the output. |

## Built-in LLM judges (11)

Ready-made judges you attach by name, no prompt to write. Each returns 0 to 1 where higher is better; for the safety judges, higher means safer. Grading uses claude-sonnet-5 by default, overridable per scorer. Docs: https://docs.runagain.ai/concepts/evaluations

| id | needs | measures |
| --- | --- | --- |
| `factuality` | expected | Answer versus reference: subset, superset, equal or disagree. |
| `correctness` | expected | Is the answer correct given the reference? Graded 0 to 1. |
| `correctness_classifier` | expected | Binary CORRECT or INCORRECT verdict; high precision on clear-cut errors. |
| `completeness` | nothing | Does the response cover every information need in the query? Catches partial answers. |
| `response_quality` | nothing | Is the response well organized and easy to use, regardless of correctness? |
| `relevance` | nothing | Does the response address the request? |
| `conciseness` | nothing | Is the response tight and free of filler? |
| `toxicity` | nothing | Safety: 1 means free of toxic or harmful content. |
| `pii` | nothing | Privacy: 1 means no PII disclosed. |
| `prompt_injection` | nothing | Security: 1 means the run resisted prompt injection. |
| `battle` | expected | Pairwise: output versus reference in random A/B order to cancel positional bias. 1 win, 0.5 tie, 0 loss. |

## RAG & grounding judges (6)

For retrieval-augmented answers: pass the retrieved passages as context and these grade grounding and retrieval quality. Docs: https://docs.runagain.ai/concepts/evaluations

| id | needs | measures |
| --- | --- | --- |
| `faithfulness` | context | 1 means every claim is grounded in the context. Holistic, one call. |
| `faithfulness_claims` | context | Decomposes the answer into atomic claims and scores the fraction supported: your hallucination rate, with per-claim verdicts. |
| `context_relevance` | context | Is the retrieved context relevant to the question? |
| `context_recall` | context + expected | Does the context contain what the reference answer needs? |
| `answer_relevancy` | nothing | Is the answer responsive to the question? |
| `source_quality` | context | Are the sources authoritative, relevant and current? Flags SEO spam and outdated citations. |

## Retrieval ranking & search metrics (8)

Deterministic IR metrics over ranked retrieved ids, scored against binary relevance or graded qrels labels, plus model-free search checks. All accept an optional k cutoff. Docs: https://docs.runagain.ai/concepts/evaluations

| id | needs | measures |
| --- | --- | --- |
| `hit_rate` | expected | labels | hit@k: 1 if any top-k id is relevant. |
| `mrr` | expected | labels | Reciprocal rank of the first relevant id. |
| `precision_at_k` | expected | labels | Fraction of the top-k that are relevant. |
| `recall_at_k` | expected | labels | Fraction of all relevant refs found in the top-k. |
| `ndcg` | expected | labels | nDCG@k with graded gains when labels are set, else binary. |
| `average_precision` | expected | labels | Per-query average precision, the term of MAP. |
| `entity_coverage` | nothing | Fraction of the query's entities (proper nouns, quoted spans, numbers) that appear in the response. |
| `source_diversity` | nothing | Unique source domains behind the response, normalized 0 to 1 by a cap argument. |

## Embedding scorers (3)

Semantic similarity via your project's embedding provider (OpenAI, Cohere or Gemini). Cosine similarity rescaled into 0 to 1; embedding tokens and cost recorded per score. Docs: https://docs.runagain.ai/concepts/evaluations

| id | needs | measures |
| --- | --- | --- |
| `embedding_similarity` | expected | Semantic closeness between output and expected, for when wording differs but meaning should match. |
| `bertscore_relevance` | nothing | Query versus response similarity: catches answers that drift off the question. |
| `bertscore_faithfulness` | context | Response versus sources similarity: a fast hallucination proxy. |

## Online eval kinds (4)

Applied to live production traffic, scoped to all runs, one agent, or any metadata attribute, with a coverage slider to sample a percentage of matching runs. Docs: https://docs.runagain.ai/app/evals

| id | needs | measures |
| --- | --- | --- |
| `llm_judge.<target>` | nothing | A model grades the run's output or its tool calls, with the judge model of your choice. |
| `trajectory.tool_success` | nothing | Model-free score of how well the agent used its tools, with optional recovery invalidation. |
| `structured_match` | nothing | Property checks on JSON answers, plus shape-drift alerting via structured_output.changed. |
| `baseline.*` | nothing | Flags abnormal runtime, cost or span count per agent: the run baselines online eval. |

## Composite & human methods (3)

For when one number is not enough: weighted multi-criterion scoring, dataset comparison, and human review that calibrates the judges. Docs: https://docs.runagain.ai/concepts/evaluations

| id | needs | measures |
| --- | --- | --- |
| `rubric` | nothing | Multi-criterion scoring: each weighted dimension graded 1 to 5 in one call, with optional per-dimension minimums for pass or fail. |
| `dataset` | expected | Compare outputs against expected values from a versioned dataset. |
| `annotation` | nothing | Human review queues write scores next to eval scores, and the judge-alignment table measures how often each judge agrees with your reviewers. |

Custom judge prompts, weighted rubrics and your own goal signals also land as first-class scores.

---

Markdown mirror of https://runagain.ai/evals for agents and LLMs. Append .md to any runagain.ai page URL for its markdown twin. Overview: https://runagain.ai/llms.txt · For agents: https://runagain.ai/for-agents.md