Data Labelling: Agreement, Guidelines and Noisy Labels

Every model trained on examples learns from labels: this email is spam, this X-ray shows a fracture, this comment is toxic. Those labels are written by people, quickly, often hundreds an hour, and people disagree, get tired and take shortcuts. A model can only be as good as its labels, and you can only measure it as well as your labels allow. Here you’ll see how to tell whether labellers really agree, and what different kinds of labelling mistakes do to a model and to the score you think it has.

Labels are measurements#

A label is the answer attached to an example, the thing a model learns to predict. The people who add them are labellers or annotators, and the instructions they follow are the labelling guidelines. It helps to think of a label as a measurement rather than a fact: like any measurement it has error, and the first job is to find out how much.

The standard way is to have two or more people label the same examples without seeing each other’s answers, and compare. If they rarely agree, the task is unclear, the guidelines are vague, or someone isn’t trying, and no model can learn a rule the labellers themselves don’t share.

Agreement, and agreement by chance#

When B reads every comment, the two agree on 94.0% of them. When B reads none and simply clicks “fine”, they still agree on 89.5%. The headline number barely moves, from 94% to 90%, while the labels go from useful to worthless. The fix is to subtract what chance would give. Cohen’s kappa (Jacob Cohen, 1960) works out how often the two would agree if each labelled at random at their own rates (79.6% here, because both say “fine” so often), and reports how much of the remaining distance to perfect agreement they cover: (94.0 − 79.6) ÷ (100 − 79.6) = 0.71. Kappa is 1 for perfect agreement and 0 for no better than chance. As B gets lazier it falls through 0.51 at half effort to 0.00. Report agreement beyond chance, such as kappa, never raw agreement alone.

What counts as good? A rule of thumb from 1977 calls 0.61–0.80 “substantial” and above 0.80 “almost perfect”, but it’s arbitrary: a task like spotting sarcasm may never get past 0.6, and that’s a fact about the task worth knowing. With more than two labellers, use Fleiss’ kappa or Krippendorff’s alpha, which also copes with missing labels.

Guidelines that make labels consistent#

Most disagreement comes from unclear instructions rather than careless people. Good guidelines:

• Define each label in a sentence, then show clear examples of each and, more useful still, the near misses: “rude but not toxic”, “toxic even though it’s polite”.
• Decide the edge cases in advance: sarcasm, quotes of someone else’s abuse, slurs used about oneself. Every disagreement you find later becomes a new line here.
• Allow “unsure” rather than forcing a guess, and send those examples to an expert.

Then run a pilot: two or three people label the same 50–100 examples, you measure kappa, read every disagreement together, fix the guidelines and repeat until agreement stops improving. In full production, keep doing two things. Mix in gold questions, examples with an agreed answer, to check each labeller over time. And have a share of examples labelled twice, so you keep measuring agreement and catch drift, such as a labeller who speeds up at the end of a shift.

What wrong labels do to a model#

Some mistakes will get through. What matters is what kind they are. A random slip goes either way: a toxic comment marked fine as often as a fine one marked toxic. A systematic mistake always goes the same way, like a rushed labeller who lets mild toxicity through.

With random slips, even a quarter of the labels wrong barely moves the learned line: the mistakes pull equally both ways and cancel out. The model stays about 95% right. But scored against test labels with the same slips, it seems to fall to 71.5%, because the test is now marking it wrong for disagreeing with wrong labels. With systematic mistakes it’s the other way round. At 20% the line moves up, the model catches only 38% of toxic comments and is really 83.8% right, yet the measured score says 96.3%, better than with clean labels. Random label errors mostly hide how good your model is; systematic ones teach it the error and hide that too.

Two cautions. A simple model like this one, with plenty of data, shrugs off random noise better than a large neural network, which can memorise individual wrong labels; there, random noise costs more. And the cure for the measurement problem is the same in both cases: a test set labelled more carefully than the rest, by several people or an expert, with disagreements settled by discussion. It’s usually the best-spent labelling money in a project.

Labelling with LLMs#

Large language models are now often used as labellers: they’re fast, cheap and never get tired. Treat one like any new labeller. Have people label a sample, measure the model’s agreement with them using kappa, and read the disagreements before trusting it. Watch especially for systematic mistakes: a model that always calls sarcasm friendly, or leans towards whichever category was listed first, makes exactly the kind of error the figure above shows is hardest to see. A good pattern is to let the model label everything and send people the examples where it’s least sure.

In code#

Measuring agreement on a labelled sample and pulling out the disagreements to discuss:

import pandas as pd
from sklearn.metrics import cohen_kappa_score

# the same 12 comments, labelled by two people working separately
df = pd.DataFrame({
    "comment": [
        "Great point, thanks for sharing", "You clearly have no idea what you're talking about",
        "Worst take I've read all week", "Can someone explain the second chart?",
        "People like you ruin every thread", "lol ok", "Nobody asked, mate",
        "Interesting, but the sample is tiny", "Get lost", "I disagree, and here's why",
        "Wow, what a genius idea /s", "Thanks, fixed it",
    ],
    "a": ["fine", "toxic", "toxic", "fine", "toxic", "fine", "toxic", "fine", "toxic", "fine", "fine", "fine"],
    "b": ["fine", "toxic", "fine", "fine", "toxic", "fine", "fine", "fine", "toxic", "fine", "toxic", "fine"],
})

print(f"agreement {(df.a == df.b).mean():.0%}, kappa {cohen_kappa_score(df.a, df.b):.2f}")
print(df[df.a != df.b])  # the disagreements: send them to a third person, then fix the guidelines
# agreement 75%, kappa 0.47
#                           comment      a      b
# 2   Worst take I've read all week  toxic   fine
# 6              Nobody asked, mate  toxic   fine
# 10     Wow, what a genius idea /s   fine  toxic

The three disagreements are exactly the edge cases a guideline should settle: a harsh opinion, a dismissive reply, and sarcasm. Decide each one, write the decision down with the example, and label the next batch.