LLM-as-a-Judge Reliability: How to Calibrate Judge Scores

By Fouad El Mourabit · Last reviewed: October 15, 2026 · Reading time: ~14 minutes

That contradiction is not a trick. The 85% figure is raw agreement: simple exact-match overlap. The 0.25 figure is chance-corrected agreement: how much better the judge is than agreement you might expect from label prevalence and marginal distributions. In a skewed dataset where most outputs are obvious passes, a judge can look accurate while missing the failures that matter.

Quick answer: LLM-as-a-judge is useful for semantic evaluation, but this editorial protocol recommends that it should not become a release gate until it is calibrated against human labels, stress-tested for bias, and monitored like production code. Treat the judge as another model in your system—not as ground truth.

A practical five-step protocol is:

1. Build a golden set of examples covering ordinary, ambiguous, adversarial, and historical-failure cases. A 50–200 example set is a practical starting convention, not a universal rule.
2. Measure human-vs-human agreement first so you know how reproducible the rubric is before judging the judge.
3. Split the set into development and test data. Tune the rubric on the development split and report the final result once on the held-out test split.
4. Run the judge with pinned settings and compute raw agreement, Cohen's kappa, the confusion matrix, and a confidence interval.
5. Stress-test and gate carefully with order swaps, paraphrases, cosmetic changes, abstention handling, task risk, and explicit recalibration triggers.

Teams are instrumenting faster than they are evaluating

LLM-as-a-judge means using a language model to grade another model's output: groundedness, helpfulness, safety, policy compliance, tool-use quality, or answer preference. It is attractive because many agent failures are semantic and cannot be captured by deterministic assertions alone.

LangChain's 2026 State of Agent Engineering survey was fielded from November 18 to December 2, 2025 and received 1,340 responses. It reports that 89% of respondents use some form of agent observability and 52.4% run offline evaluations. It also identifies quality as a leading production barrier. These are self-selected vendor-survey findings, not a peer-reviewed benchmark.

That difference matters. You can trace latency, tool calls, and errors with production observability, but a green trace does not prove the answer was correct. For the operational side of agent reliability, see PromptSphere's AI Agent Observability production reliability playbook. For release decisions, you still need calibrated evaluation.

Five failure modes to test for

1. Position bias: answer order changes the verdict

Evidence status: documented in an exploratory study. In pairwise judging, the model may prefer “Answer A” or “Answer B” because of order, not quality. Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge found position bias that was not explained by chance and varied by judge, task, and quality gap, across 15 judges on MTBench and DevBench. The study covered more than 150,000 evaluation instances. Its practical test is simple: swap the answer order and rerun the same evaluation. If the verdict flips often, do not use that pairwise judge as an automatic gate without mitigation.

2. Self-preference: a judge may favor its own model family

Evidence status: a known risk summarized in survey literature; validate it for your task. A judge may score outputs from its own model family more favorably. This is not always obvious, especially when outputs differ in style rather than substance. Blind the source model, mix generators, and compare against human labels. For high-risk decisions, consider a judge from a different model family plus human adjudication.

3. Score clustering and rubric sensitivity

Evidence status: measurement-risk hypotheses to test, not universal laws. Some judges compress scores toward the middle or change their output when rubric wording changes. The broader literature discusses sensitivity to evaluation design and bias, but your own perturbation test should establish whether this happens for your task. Use anchored labels, concrete counterexamples, and a frozen rubric version.

4. Agreement inflation: exact match versus chance-corrected metrics

Evidence status: a statistical interpretation issue. Raw agreement is intuitive but incomplete. Cohen's kappa estimates agreement beyond what label distributions would predict. It is sensitive to class prevalence and marginal distributions, so a highly imbalanced golden set can make interpretation tricky. Always inspect the confusion matrix and the per-class recall.

5. The consistency–bias paradox

Evidence status: reported in a specific study, not a universal constant. A judge can be consistent and still wrong. The preprint Reliability without Validity reports test–retest reliability above 0.95 while position bias remained above 0.10 in two production-deployed judges. The article does not define that value as a universal threshold. High consistency is useful, but it does not prove validity.

What the research found—and what it did not

The strongest evidence does not say “LLM judges are useless.” It says they are measurement systems with failure modes.

RAND Judge Reliability Harness. The RAND Corporation's Judge Reliability Harness preprint evaluated a limited set of judges and benchmarks. Its useful conclusion is scoped: no single judge was uniformly reliable across all tested benchmarks. Treat the harness as a methodology for building your own reliability suite, not proof that every judge fails every task.

Reliability without Validity. Across 21 judges and about 541,000 judgments on three benchmarks, the authors report exact-match agreement exceeding Cohen's kappa by 33–41 percentage points on MT-Bench. Keep the scope attached: this is a result from that study, not a universal constant. Verify the PDF before retaining more precise decimals or an institutional affiliation in a published version.

IJCNLP-AACL position-bias work. Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge is exploratory conference research on position bias across judges and tasks. Cite the ACL Anthology version directly.

Survey literature. The preprint A Survey on LLM-as-a-Judge summarizes evaluation uses and known limitations, including bias and calibration concerns. It is useful background, not a replacement for task-specific validation.

Vendor guidance. Anthropic's engineering post “Demystifying evals for AI agents” recommends explicit handling for ambiguous outcomes. Give a model-based grader an “Unknown” path so it can abstain when evidence is insufficient instead of fabricating confidence.

A calibration protocol you can run this week

This is a practical PromptSphere editorial framework, not an industry standard. The 50–200 example range is a starting convention. Anthropic's guidance also describes 20–50 examples as a reasonable starting point for simple tasks drawn from real failures. Adapt the size to your risk profile, label prevalence, and failure cost.

Step 1 — Build a golden set and split it before tuning

Define the exact property being judged: “grounded in retrieved policy,” “tool call completed correctly,” or “answer refused an unsafe request.” Avoid labels like “good” unless the rubric defines them.

Include ordinary successful cases, ambiguous cases humans debate, adversarial or distracting cases, and historical failures from production. Split the examples into a development set and a held-out test set. Tune rubric wording and judge settings only on development data; use the test set once for the final report.

Balance labels where possible. If 95% of the set is “pass,” raw agreement will be inflated and kappa may be hard to interpret.

Step 2 — Measure humans before measuring the judge

Have at least two human reviewers independently label a sample using the draft rubric. Measure human-vs-human agreement and adjudicate disagreements. If humans cannot agree on the task, the judge is not the first problem to solve: clarify the rubric, add examples, or allow “Unknown.”

Step 3 — Score with the judge and compute uncertainty

Run the judge with a pinned model ID, prompt, temperature, output schema, rubric version, and dataset version. Then collect human labels independently. For ambiguous cases, use documented adjudication rules such as majority vote, a senior-reviewer tie-break, or “Unknown” when the evidence is insufficient.

Require structured output:

{
  "label": "pass|fail|unknown",
  "confidence": "low|medium|high",
  "evidence": ["quoted span or reason"],
  "rationale": "short explanation"
}

Raw agreement is matching labels / total examples. Cohen's kappa is:

(observed agreement - expected chance agreement) /
(1 - expected chance agreement)

Do not use a universal kappa threshold. Define candidate thresholds before looking at results, and document why they fit the risk, class balance, and cost of a missed failure. Also inspect the disagreement matrix and per-class recall.

Copyable Python check: kappa, bootstrap interval, and confusion matrix

This snippet is a reproducible starting point. It does not create first-hand results for this article; run it on your own labels and report the dataset, rubric, and versions alongside the output.

from __future__ import annotations

import numpy as np
from sklearn.metrics import cohen_kappa_score, confusion_matrix

# Replace these with labels from the held-out test split.
# Keep the same order and use the same label vocabulary.
human = np.array(["pass", "pass", "fail", "pass", "fail"])
judge = np.array(["pass", "fail", "fail", "pass", "pass"])
labels = ["pass", "fail"]

assert len(human) == len(judge) and len(human) > 1
raw_agreement = np.mean(human == judge)
kappa = cohen_kappa_score(human, judge, labels=labels)
cm = confusion_matrix(human, judge, labels=labels)

print(f"raw_agreement={raw_agreement:.3f}")
print(f"cohen_kappa={kappa:.3f}")
print("rows=human, columns=judge; order=pass, fail")
print(cm)

# Percentile bootstrap for an approximate 95% interval.
rng = np.random.default_rng(42)
values = []
for _ in range(5000):
    idx = rng.integers(0, len(human), len(human))
    values.append(cohen_kappa_score(human[idx], judge[idx], labels=labels))
lo, hi = np.percentile(values, [2.5, 97.5])
print(f"kappa_bootstrap_95ci=({lo:.3f}, {hi:.3f})")

For small or highly imbalanced sets, bootstrap intervals can be unstable. Treat them as an uncertainty signal, not as a substitute for more representative data.

Step 4 — Run perturbation tests

Run the same examples through controlled changes: swap answer order in pairwise comparisons, paraphrase the rubric without changing meaning, change formatting or answer length, add irrelevant context, remove nonessential context, and test an explicit “Unknown” path. A calibrated judge should be sensitive to semantic changes and relatively insensitive to cosmetic changes.

Step 5 — Set release thresholds and recalibration triggers

A release gate should define minimum raw agreement and chance-corrected agreement, maximum order-swap flip rate, maximum unknown/abstention rate, required human review for high-impact failures, and recalibration triggers after model, prompt, tool, policy, retrieval, or data changes.

Log judge version, rubric version, model version, dataset version, and calibration date. If run-to-run variance has not been characterized, the score should not be allowed to block a release.

Judge, assert, or human?

Use assertions first. Use LLM judges where semantics matter. Use humans where consequences are high or the judge abstains.

Worked example: calibrating a support-agent groundedness judge

The following is a synthetic example only. It is not a PromptSphere experiment and should not be read as measured performance.

Suppose a team creates an LLM judge for “policy groundedness” on a synthetic 60-ticket support set. Human reviewers label 50 tickets as pass and 10 as fail.

Raw agreement is 51/60 = 85%. Expected chance agreement from the marginal distributions is:

(50/60 × 57/60) + (10/60 × 3/60) = 0.80

Therefore:

(0.85 - 0.80) / (1 - 0.80) = 0.25

The fix is not “delete the judge.” The fix is to rewrite the rubric, add explicit evidence requirements, include an “Unknown” label, rebalance or expand the golden set, and rerun the perturbation suite. Until fail recall and relevant bias tests improve, this judge should be advisory, not a release gate.

Golden-set CSV template

Save the following as golden_set_template.csv. Keep development and test rows separate, and do not fill the judge label before human adjudication is complete.

id,split,task_type,input,reference_or_policy,human_label_1,human_label_2,adjudicated_label,judge_label,judge_confidence,judge_evidence,notes
001,dev,groundedness,"Example response text","Relevant policy text",pass,pass,pass,,,"",
002,dev,groundedness,"Ambiguous response text","Relevant policy text",unknown,fail,unknown,,,"",
003,test,groundedness,"Held-out response text","Relevant policy text",pass,pass,pass,,,"",
004,test,groundedness,"Historical failure text","Relevant policy text",fail,fail,fail,,,"",

What this evidence does not prove

This evidence does not prove every LLM judge is unreliable, nor that human review is perfect. Human labels can be noisy, inconsistent, and biased. It also does not prove one vendor's judge is universally better than another.

The lesson is narrower and more useful: reliability is task-specific. A judge calibrated for customer-support groundedness may fail on code review, safety refusal, or multi-turn tool use. Recalibrate when the task, model, rubric, retrieval system, policy, or user distribution changes.

Frequently asked questions

Is LLM-as-a-judge accurate enough for production?

Sometimes. It can be production-ready for a scoped task after calibration, perturbation testing, abstention handling, and ongoing monitoring. It should not be assumed accurate because the model is strong.

How many human-labeled examples are needed?

Start with a task-appropriate development and test set. For a simple task, 20–50 real failure examples may be a useful first pass; 50–200 total examples is a practical editorial starting range for broader calibration. Use more when labels are rare, consequences are high, or the task has many subtypes.

Should the judge be a different model family?

Often, yes—especially when testing for self-preference. A different family is not a guarantee of neutrality, but it can reduce correlated style preferences.

Exact match, Cohen's kappa, or Krippendorff's alpha?

Use exact match for readability, kappa for two-rater chance-corrected agreement, and Krippendorff's alpha when you have multiple raters, missing labels, or more complex annotation designs. None is universal ground truth.

Why does the judge change its verdict when answer order is swapped?

Because pairwise prompts can create positional cues. The model may anchor on the first answer, the last answer, or label wording instead of evaluating quality.

Conclusion

LLM-as-a-judge can scale semantic evaluation, but its score is an estimate produced by another model. Calibrate it against independent human labels, measure human agreement first, keep a held-out test set, inspect class-specific failures, test cosmetic and positional perturbations, and give it an explicit Unknown path. Then decide what authority—if any—the judge has earned.

References

1. RAND Judge Reliability Harness.
2. Reliability without Validity.
3. Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge.
4. A Survey on LLM-as-a-Judge.
5. LangChain State of Agent Engineering.
6. Anthropic: Demystifying evals for AI agents.

Author box: Fouad El Mourabit writes about AI agent reliability, evaluation, observability, and production engineering for PromptSphere. His work focuses on practical systems that help teams ship AI features without confusing demos for dependable software.