Inter-annotator agreement on subjective tasks: Reading disagreement as data

Most evaluation infrastructure assumes a single right answer. Unit tests pass or fail. Retrieval hits or misses. Exact-match accuracy compares an output against a key. 

This holds until you’re scoring subjective quality, like whether a response is helpful, whether its tone fits the situation, or whether a safety call holds up across cultures. On those axes, no ground truth sits in a lookup table, and the standard machinery starts returning misleading numbers.

Two failures tend to arrive together. Static golden datasets get gamed as models train against them, so a system can look strong offline while degrading for real users. LLM-as-judge setups, the usual way to buy back scale, are unstable on the very subjective axes that matter most, vary widely by judge model, and favor outputs that resemble their own (EMNLP 2025). 

Teams that fall back on human evaluation to recover the signal then hit the problem this piece is about. On subjective tasks, humans disagree, and the field has no settled way to interpret that disagreement. 

The stakes here aren’t academic. The fastest-shipping teams run tight evaluation loops that continuously feed production failures back into their eval sets, and a loop calibrated on mislabeled human judgment compounds error at the speed it was built to move.

Why kappa breaks on subjective tasks

The agreement metrics most teams reach for, Cohen's kappa for two raters, Fleiss' kappa for several, Krippendorff's alpha for mixed designs, carry a hidden assumption. They score agreement against the premise that each item has one correct label, and they discount every departure from consensus as error (Bassi et al., 2025). 

For objective work, the premise is fine. Two annotators marking whether a string contains a phone number should agree; if they don't, one is wrong.

Subjective tasks break the premise at the root. Ask a representative group whether a chatbot reply is appropriately cautious, or whether a summary keeps the point that mattered, and a spread of answers is the expected result of a well-run study, because the question admits more than one defensible response. A chance-corrected coefficient can’t see that. It reads the spread as low reliability, which looks like a data-quality problem. 

So, a single low kappa is ambiguous. It can mean the rubric is vague and the annotators are guessing. It can also mean the item genuinely divides careful readers. The coefficient gives you the same number either way, and that number points you toward the wrong fix as often as the right one.

Subjectivity-aware alternatives exist. The cross-rater reliability framework, xRR, scores how consistently annotators reproduce each other’s labeling behavior instead of measuring distance from a single gold label, which suits pluralistic tasks where several answers are legitimate (Bassi et al., 2025). The question then shifts from whether raters matched a key to whether their judgment patterns align.

Disagreement as data

Barbara Plank's 2022 position paper named the underlying issue ‘human label variation.’ It argued that treating it as noise discards information the field needs (Plank, 2022). 

The argument runs through the whole pipeline. Variation driven by subjectivity, ambiguity, and multiple plausible readings shows up in the training data, in how models learn, and in how they’re scored, so labeling it measurement error distorts all three stages at once. The idea has older roots in Lora Aroyo and Chris Welty's work on crowd truth, which shows how assuming a single annotation truth misleads the people building datasets (Aroyo and Welty, 2015).

For an eval team, it’s useful to separate two things that both surface as low agreement: 

• Noise, which can come from annotators reading a loose rubric in their own ways, rushing, misreading the task, or drifting in their standards over a long session. 
• The real differences in how reasonable people judge the same item. 

The first is a defect you remove. The second is a property of the task you measure and keep, because it carries information a point estimate buries. For example, a safety policy may land differently across groups, or a reply one user finds helpful may read as condescending to another.

Ellie Pavlick and Tom Kwiatkowski demonstrated this in natural language inference, where annotator disagreements proved systematic and reproducible rather than random error (Pavlick and Kwiatkowski, 2019). A reproducible label distribution is a stable object. A model can be trained toward it and scored against it, putting Plank's modeling half of the argument to work.

How the field measures it now

Research on subjective annotation has moved toward scoring systems against the distribution of human judgments. The clearest reference point is the Learning With Disagreements shared task, now in its third edition, which ran datasets covering paraphrase identification, irony, sarcasm, and natural language inference (LeWiDi-2025, also on the ACL Anthology). 

It grades systems two ways: 

• Soft-label evaluation asks a model to predict the full label distribution a population would produce, then measures how close that prediction is to the human distribution. 
• Perspectivist evaluation asks a model to reproduce individual annotators' judgments, keeping separate viewpoints intact instead of averaging them into a single label.

The organizers also moved away from cross-entropy as the default distance measure after it was shown to perform poorly when comparing label distributions, and tested alternatives such as Manhattan and Euclidean distance. For a team choosing a metric for its own setup, a 2026 survey, Counting on Consensus, sorts agreement measures by task type and examines the effects of label imbalance and missing data, and reports practices like confidence intervals and direct analysis of where raters split (James et al., 2026). 

That reporting point is the one most pipelines skip. A kappa or alpha value with no interval around it and no view of which items or which raters drove the disagreement gives you very little you can act on.

The two paradigms answer different questions, and most teams want both: 

• A soft-label model for when you care how a whole population would react, for example ahead of a broad release. 
• A perspectivist model for when specific viewpoints must stay legible, for example, when a safety review needs to know whether a particular group would object to a response. 

Collapsing the two back into one aggregate recreates the problem the whole approach was built to avoid.

Sorting real variation from error

Before you treat a disagreement as signal worth keeping, rule out the cheaper explanations, because most low agreement in practice traces back to fixable causes. 

Vague rubrics are the common one. When instructions say "rate overall quality" with no anchored criteria, each annotator gives a private definition and the resulting spread is noise. Calibration sessions catch this early: annotators label a shared set, then talk through the cases they split on until the criteria are concrete. 

Gold-standard items seeded through the task and overlap protocols, where a fixed share of items is labeled by more than one person so pairwise agreement can be tracked per annotator, do the same monitoring job during production. A working range for the overlap is fifteen to twenty percent of items, raised for smaller datasets, with round-robin pairing so every annotator pair appears and weak raters become visible. Thin instructions produce noisy labels, regardless of annotator caliber, which is why task design deserves as much attention as annotator selection.

Annotator quality is a separate axis, and it doesn’t collapse into domain credentials. A strong clinician can be a weak annotator because rubric comprehension, consistency across hundreds of items, and the task's structure shape the label. Compensation sits inside the same analysis. Fatigue degrades labels, and rushed, underpaid work feeds the noise that agreement metrics later report as unreliability. The HUMAINE study, for instance, paid participants at a fair recommended rate as part of its quality design (Nora Petrova, Andrew Gordon, 2026).

What remains once the noise is controlled is what's worth studying. Google's DICES dataset shows why (Aroyo et al., 2023). Built for conversational AI safety, it uses high replication per item and fine-grained rater demographics, and stores votes as distributions across groups so patterned differences stay visible. 

It draws on around 350 raters and more than 2.5 million individual ratings, sized so the demographic breakdowns carry statistical weight. Safety judgments split along demographic lines in ways a majority vote flattens, and a diversity-aware read of the data recovers concerns minority groups raised that aggregation would have dropped. The disagreement was doing work. Averaging it away threw out a real safety signal.

What changes in your eval design

A few choices follow from all of this. The metric should match the task. A chance-corrected coefficient is reasonable for objective labeling with a defensible key. Subjective work is better served by distribution-based and perspectivist metrics, reported with confidence intervals and a breakdown of where raters diverged. 

Point scores should never travel alone, since a single preference number hides the structure that decides whether a model is ready to ship. Two systems can tie on average and diverge sharply on who prefers which, and that split is often what you care about. 

Reporting the distribution makes it visible. A model preferred sixty percent of the time overall might earn that margin by being strongly favored in one age band and mildly disliked in another. This is a very different product decision from one where a model is preferred sixty percent evenly across the board.

Disagreement becomes interpretable only when you know who is disagreeing, which puts the rater pool's makeup at the center of the design. A pool skewed toward one demographic or one level of expertise yields a confident-looking score built on a narrow slice of judgment, and you have no way to tell whether a low agreement figure reflects a bad rubric or a population that genuinely divides.

Pairwise comparisons, where a rater picks between two outputs, tend to produce higher agreement than absolute Likert ratings. Choosing between two concrete options is easier and steadier than placing a single output on an abstract scale. When the goal is to compare model versions, paired judgments on the same items cut noise and reduce the sample size needed to separate them. HUMAINE runs on comparative assessment for exactly this reason.

This is the foundation of our work. HUMAINE is a human-preference evaluation framework that ran more than 23,000 participants against frontier models, recruited through demographic stratification and adjusted to census proportions, and scored models across several dimensions (HUMAINE paper, live leaderboard). Preferences varied significantly across demographics, with age the single largest source of disagreement among the models tested. A leaderboard reporting one collapsed average would have hidden that entirely. 

The question HUMAINE is built to answer (which model is best for whom and on which dimension) is what taking disagreement seriously looks like in a working eval.

The target that matters

Pushing agreement upward made sense when annotation meant writing down an already settled fact. Subjective evaluation is a different job. 

The right target is a faithful, well-characterized picture of how real people respond, with the noise cleaned out and the genuine variation kept and marked for what it is. Optimizing for the highest possible kappa works against that, because it rewards suppressing variation the task legitimately contains. Tightening a rubric until every rater agrees on a genuinely contested question does not improve the data. It has trained the disagreement out of view.

That picture is only as good as the people producing it. Representative, profiled, fairly paid evaluators are what make a disagreement legible instead of confusing, and assembling that pool is the part most eval stacks underinvest in.

Prolific gives frontier AI teams the human evaluation infrastructure they need, with verified evaluators, reproducible methodology, and rapid results. Learn more.