LLM-as-a-Judge: How to Calibrate It Against Humans

An LLM judge is fit for an evaluation only after it agrees sufficiently with qualified humans on that evaluation’s cases, rubric, and output distribution. Calibration is local: agreement measured on summarization cannot validate a judge for tool authorization, and a judge upgrade creates a new measuring instrument that must be recalibrated.

Use model graders for narrow semantic judgments, not as ground truth and never as a substitute for deterministic safety checks. The agent eval CI guide shows how that calibrated grader fits into a release gate.

Define one property and an abstention path

“Rate quality from 1 to 10” invites an unstable blend of correctness, style, safety, and preference. Split the rubric into properties with observable anchors:

Property: factual support
2 — every consequential claim follows from the supplied evidence
1 — the conclusion is supported, but a minor claim is unsupported
0 — a consequential claim is unsupported or contradicts evidence
X — evidence is insufficient or the case is outside the rubric

The judge receives only what a human labeler is allowed to use: task, relevant observable trajectory, reference material, answer, and rubric. Do not ask it to infer hidden intent or grade private chain-of-thought. Require structured output with a score, evidence spans, and abstainReason, then validate the structure.

Some properties should not go to a model judge at all. Schema conformance, arithmetic, state changes, tool allowlists, citation existence, and approval tokens belong in code. A semantic score must not cancel a hard failure.

Build a blinded human calibration set

Draw cases from the outputs the judge will actually see:

• strong, weak, and borderline answers;
• different task and user segments;
• common and high-risk workflows;
• concise and verbose forms;
• outputs from several model families; and
• adversarial examples designed to look polished while being wrong.

Remove model names, experimental condition, and answer order. Randomize presentation. If pairwise comparisons are used, label both A/B and B/A assignments on a probe subset.

Have at least two qualified people label independently before discussion. Train them on examples outside the scored calibration set. Record “cannot determine,” time, and cited evidence. Resolve ambiguous policy with the policy owner; do not turn a rubric defect into a majority vote.

The original MT-Bench study explicitly examined position, verbosity, self-enhancement, and reasoning limitations in model judges. Its reported agreement is evidence about its studied judges and datasets, not a reusable accuracy guarantee for another system. Zheng et al., Judging LLM-as-a-Judge

Run the frozen judge exactly as production will

Fingerprint every evaluator dependency:

interface JudgeFingerprint {
  readonly provider: string;
  readonly modelIdentifier: string;
  readonly rubricDigest: string;
  readonly promptDigest: string;
  readonly decoding: Readonly<Record<string, unknown>>;
  readonly evidencePolicy: string;
  readonly parserVersion: string;
}

Run the judge on the calibration set without exposing human labels. Preserve raw structured responses, parser failures, latency, and abstentions. Repeat a subset to measure self-consistency: deterministic-looking settings do not prove identical hosted-model behavior over time.

Freeze the judge during an agent comparison. If the agent and evaluator change together, the score difference confounds the system with its ruler.

Measure the errors that matter

Start with a confusion matrix rather than one agreement number:

Report:

• exact agreement and adjacent agreement for ordinal scales;
• false-pass rate: human fail, judge pass;
• false-fail rate: human pass, judge fail;
• abstention and invalid-output rate;
• agreement by task, risk, answer length, and source model; and
• human-human agreement as context for rubric ambiguity.

Cohen’s kappa or a weighted ordinal coefficient can supplement these counts, but no coefficient decides fitness automatically. Kappa is affected by class prevalence and label marginals; publish the confusion matrix and base rates beside it. The classic “high agreement, low kappa” analysis demonstrates how imbalanced marginal totals can produce counterintuitive values. Feinstein and Cicchetti, 1990 Bootstrap by independent task or source family—not individual duplicated rows—when estimating uncertainty.

Set acceptance criteria from consequence. A judge that occasionally rejects a harmless stylistic answer may be acceptable for triage. A judge with any material tendency to pass unsupported medical instructions is not an autonomous gate.

Probe known judge biases directly

Build paired counterfactuals where the judged property should remain constant:

Position: swap A and B. Record preference consistency, not just aggregate win rate.

Verbosity: add relevant-but-redundant prose without changing facts. Also create concise answers that are complete; length alone is not the causal variable if content changes.

Style: vary headings, confidence, polish, and citations while holding correctness constant.

Identity: remove provider and model labels. If identity is genuinely part of the decision, document why.

Reference leakage: insert phrases that resemble the reference but contradict it.

Self-preference: include outputs from the same family as the judge and from others, while blinded.

Position-bias research has found that sensitivity varies across judges and tasks, reinforcing that a local swap test is more useful than assuming a universal direction or size. Shi et al., A Systematic Study of Position Bias in LLM-as-a-Judge

Improve the rubric before adding judge complexity

When humans and the judge disagree, classify the cause:

1. invalid or incomplete evidence;
2. ambiguous rubric anchor;
3. human labeling error;
4. parser or prompt defect;
5. judge reasoning error;
6. position, style, identity, or length sensitivity; or
7. case outside the intended domain.

Repair the rubric using a development partition, then rerun once on an untouched validation partition. Repeatedly tuning on the same calibration cases overfits the evaluator.

Useful mitigations include narrowing the property, showing governing evidence, requiring cited evidence, randomizing order, running both orders for consequential pairwise judgments, and routing low-confidence or inconsistent results to humans. More judges are not automatically better: correlated judges can repeat the same error while increasing cost.

Choose an operating policy

Turn calibration into a policy rather than a celebratory score:

Auto-accept only when:
- deterministic checks pass;
- the case is in the calibrated domain;
- the judge returns valid evidence and does not abstain;
- no counterfactual order check disagrees; and
- the property is not designated human-only.

Route everything else to a person or mark it unscored.

Monitor drift with a small sentinel set and recurring human audits. Recalibrate after a judge model, rubric, prompt, parser, evidence format, or candidate-output distribution changes. Preserve old judge results; retroactively overwriting them destroys score comparability.

Calibration checklist

• Judge one property with anchored labels and abstention.
• Blind model identity, condition, and answer order.
• Use independent qualified human labels and adjudicate policy disputes.
• Freeze and fingerprint the evaluator stack.
• Publish confusion matrices, base rates, subgroup errors, and uncertainty.
• Test position, length, style, identity, and reference-copying counterfactuals.
• Keep hard safety and state checks deterministic.
• Tune on development cases and validate on untouched cases.
• Recalibrate on every material evaluator or distribution change.