LLM-as-a-judge: calibrate it before you trust it

Around 100 to 200 carefully labelled, representative examples is a workable starting point for most tasks, and the emphasis is on representative and careful rather than on the raw number. The examples should span the real range of outputs you will ask the judge to score, including the close calls and the failure modes you care about, because a judge that agrees with you on easy cases but diverges on the hard ones is not calibrated where it matters. Label them against the same explicit rubric the judge will use, run the judge over the set, and measure agreement with a statistic like Cohen's kappa, aiming for roughly 0.6 or higher before you rely on it at scale. If agreement is low, inspect the disagreements: they usually reveal an ambiguous rubric, a biased judge prompt, or inconsistent human labels, and fixing the rubric often helps more than adding examples. Re-check periodically, because a model upgrade can quietly move the judge's behaviour and silently invalidate last quarter's calibration.

Three are well enough documented to design against by default. Position bias is the tendency to favour the option shown first or last regardless of content, and you counter it by randomising option order on every comparison and, for important gates, averaging the score across both orderings. Verbosity bias is the tendency to rate longer answers higher just for being longer, which you counter by scoring against an explicit rubric and penalising or normalising for padding so length earns nothing on its own. Self-enhancement is the tendency to prefer outputs written in the judge's own style, which is especially dangerous when a model grades a sibling from the same family; the defence is cross-family judging, letting a different model family score the one under test. Underlying all three is the same fix: replace vague rate-it-out-of-ten prompts with explicit criteria and worked examples, because the more you leave to the judge's taste, the more its biases fill the gap.

It can, and for low-stakes, high-volume scoring it is often perfectly fine, but self-enhancement bias makes it risky for any comparison that gates a decision. The problem is that a model tends to prefer outputs written in its own style, so when it judges a sibling from the same family it can reward familiarity rather than quality, quietly tilting the result. For important gates the safer choice is cross-family judging: have a different model family score the one under test, which blunts the bias because the judge has no stylistic stake in the output. If you must use the same model — for cost or simplicity — at minimum validate that same-model judging still agrees with human labels on your calibration set, and watch the agreement over time. A practical compromise is to use the same model for cheap, frequent screening and bring in a different family plus human spot-checks for the high-stakes comparisons that actually decide a release.

It depends on the question you are answering, and most mature setups use both for different jobs. Pointwise scoring grades a single output against a rubric on an absolute scale, which is what you want when you need a threshold — is this answer good enough to ship — and when you are tracking a metric over time on a fixed set. Pairwise scoring shows the judge two outputs and asks which is better, which is more reliable when you are comparing two models, two prompts or two versions, because relative judgements are easier and more consistent for a model to make than absolute ones. The trade-off is that pairwise is noisier to turn into a single number and more exposed to position bias, so you must randomise order, while pointwise depends heavily on a well-anchored rubric to stay stable. A common pattern is pairwise for choosing between candidates during development and pointwise for the production gate that has to output a pass-or-fail against a bar. Pick the mode that matches the decision, and calibrate whichever you choose.