LLM Judge Flips Its Verdict? Check Reruns Before Blaming Answer Order

If your LLM judge changes its winner when you swap two answers, first check whether it also changes its winner when you repeat the same order. Keep the answer text fixed, map the verdict back to the answer’s identity, and report ties and failed judgments separately. One changed verdict is a useful warning, but it does not by itself identify the cause or tell you which answer is better.

This guide is for teams comparing a candidate response with a reference response in regression evaluations. It proposes a small diagnostic record, not a benchmark of current models. All outcomes below are invented to explain the bookkeeping; no judge API was run.

Does “A won twice” mean the same answer won?

Use stable internal IDs such as reference and candidate. A and B are presentation slots, not those IDs.

• Original order: A contains reference, B contains candidate. Verdict A means reference won.
• Swapped order: A contains candidate, B contains reference. Verdict B also means reference won.
• Verdict A in both orders means different answers won.

Store the slot-to-answer mapping beside each raw verdict. Normalize to reference, candidate, tie, or invalid before comparing results. Do not classify a transport error, refusal, truncated output, or unparseable verdict as a tie.

This identity-based comparison follows the distinction in Language Model Council, Appendix B and Table 10: reversed preference labels can represent a consistent winner after the inputs are swapped. Its calibration also treats same-order invariability separately from swapped-order consistency. Paper and appendix

What should stay fixed during the check?

Our suggested diagnostic compares the judge, not a new sample of the answer generator. Freeze:

• The user question, relevant context, rubric, and both complete answer texts.
• The judge model/version, prompt template, generation settings, and output parser.
• The treatment of ties, refusals, missing results, and retries.

Repeat judgments in the original order and in the swapped order. Interleave or randomize the request schedule so that every original-order call does not precede every swapped-order call. Record request time and available model/version identifiers; a silent provider change remains a possible confound.

Do not regenerate the answers between orders. That would change both the presentation and the content being evaluated. Do not rewrite the rubric midway and pool the results as one condition.

Judging the Judges explicitly examines repetition stability before position consistency. That supports measuring the two separately rather than attributing every disagreement to order. Its reported rates describe its own judges, datasets and settings, not your production workload. Section 2.2

A five-case record that avoids a misleading flip rate

Here is a fictional example with two planned judgments per order, already mapped to stable answer identities. Two repeats make the bookkeeping visible; they are not a recommended statistical sample size.

1. Case 1: original reference, reference; swapped reference, reference. Observed winner is stable across these calls.
2. Case 2: original reference, reference; swapped candidate, candidate. Repeats are stable within each order, but the preferred identity changes across orders. Investigate an order-associated effect.
3. Case 3: original reference, candidate; swapped reference, candidate. There is already same-order instability. A single selected pair of calls would hide that fact.
4. Case 4: original tie, tie; swapped tie, tie. Stable ties are not candidate wins, losses, or malformed responses.
5. Case 5: original invalid, reference; swapped candidate, candidate. The original-order repeat is incomplete. Keep the failed judgment visible rather than quietly dropping the case.

The ledger contains 20 planned judgments: 19 valid, including 4 ties and 15 decisive verdicts; 1 invalid. Those counts describe invented records, not a measured model. A rate restricted to decisive, complete pairs would answer a different question from one covering every planned case.

Original workflow. Repetition and order checks describe consistency; they do not establish correctness.

For a real report, include the number of distinct cases as well as calls. Repeating one case many times does not create many independent examples of your workload. Define the aggregation and uncertainty method before looking for a winning model; do not subtract a rerun-disagreement percentage from a swap-disagreement percentage and call the remainder a causal bias estimate.

Should I force a winner or use majority vote?

Choose the output contract for the decision you need. If your rubric permits equally acceptable answers, specify how a tie is represented and scored. If it requires a forced preference, report that design choice rather than describing the result as evidence that one answer is meaningfully better.

A majority vote can summarize repeated judgments, but it does not supply a missing correctness reference. Nor should a tie or a failed call be silently converted into support for the incumbent. Keep raw outcomes so a changed aggregation rule can be audited.

The Language Model Council appendix tests different tie options and temperatures, then chooses a particular setup for its own study. It also reports imperfect invariability for some models at temperature zero. That is a reason to measure your setup, not to copy its configuration as a universal fix. Appendix B

When can the score support a release decision?

Consistency is one check. The next is whether the judge applies the intended rubric to representative cases. Anthropic’s evaluation guidance recommends human calibration of model graders and using deterministic checks where they fit the task. Official guide, January 9, 2026

Our practical release record would separate three decisions:

• Harness repair: mapping, parsing, truncation, or missing-result handling is wrong. Repair it before interpreting a quality score. A valid output schema alone does not establish a valid judgment; see structured-output validation.
• Evaluator review: repeat or order sensitivity could change the deployment choice. Inspect the affected cases and rubric with a human reviewer; do not promote a candidate on an unexplained aggregate gain.
• Quality comparison: use a held-out, human-reviewed calibration set and task-appropriate checks, then report candidate performance and uncertainty on the intended workload. Agreement within a judge is not the same as agreement with that reference.

Predefine what disagreement would change the decision and what requires escalation. There is no universal acceptable flip percentage here. For responses that never completed, diagnose generation and transport failures separately rather than treating missing judgments as evidence of answer quality.