Data Labelling for Vision: Annotation Quality Sets Your Ceiling
No architecture change fixes a model that was trained on ambiguous labels. In computer vision, the annotation guidelines and the auditing process quietly decide how good your best possible model can ever be.
The ceiling nobody budgets for
When a computer vision model plateaus, the instinct is to reach for a bigger backbone, more augmentation, or a longer training schedule. I understand the instinct because architecture changes are legible: you can point to a paper, cite a parameter count, and justify the decision in a meeting. Auditing your labels is none of those things. It is slow, unglamorous, and often reveals that the problem was never the model at all.
Here is the uncomfortable version of the idea. A model trained on a dataset with genuinely ambiguous or inconsistently applied labels cannot exceed the accuracy of those labels, no matter how it is trained. If two competent annotators disagree on where a box should go around an object fifteen percent of the time, then fifteen percent of your evaluation set is effectively unresolved ground truth. Your model is being scored against a moving target, and no amount of tuning closes that gap because the gap is not in the model.
This matters more in vision than people expect, because visual tasks feel objective. A cat is a cat, a pedestrian is a pedestrian, surely the labels write themselves. In practice, edge cases dominate: partially occluded objects, motion blur, objects that straddle a class boundary, or crowded scenes where it is unclear whether two overlapping instances should be one box or two. These edge cases are exactly where annotation guidelines either hold up or fall apart, and they are disproportionately represented in the hard examples your model needs to learn from.
A worked example: bounding boxes on a busy street scene
Take a detection task: identifying vehicles in dashcam footage. Suppose the guideline says draw a tight box around each visible vehicle. Sounds unambiguous. Now put three annotators in front of the same frame showing a van partially hidden behind a bus, with only the wing mirror and a sliver of the body visible.
Annotator one draws a box around the visible sliver only, following the guideline literally. Annotator two estimates where the full van would be and draws a box around the inferred extent, reasoning that downstream tracking needs the true object size. Annotator three decides the visible fragment is too small to count as a detectable instance and skips it entirely. All three are behaving reasonably. None of them is wrong given a loosely written guideline. But now you have three different ground truths for the same pixel region, and whichever one lands in your training set becomes what the model is taught to reproduce.
Scale this across a dataset of fifty thousand frames and the effect compounds. If occluded objects appear in roughly one frame in eight, and the three-way disagreement pattern above holds for even half of those cases, you have introduced systematic noise into a meaningful fraction of your hardest and most valuable training examples. Your model will learn an inconsistent policy for occlusion because it was shown an inconsistent policy for occlusion. Evaluation metrics will look noisy in exactly the same way, and a validation set built from the same guideline will not reveal the problem because it inherits the same ambiguity.
The fix is not more data. It is a sharper guideline: for instance, specify a minimum visible fraction threshold for a box to count, define explicitly whether boxes should reflect visible extent or estimated full extent, and give annotators worked examples of borderline cases with the correct answer marked. Then measure inter-annotator agreement on a held-out sample before scaling up labelling, using a metric like intersection-over-union agreement or Cohen's kappa for classification-style labels. If agreement is weak, no downstream modelling choice will rescue the dataset.
Why this is easy to miss and expensive to fix late
Annotation quality problems hide well because they masquerade as model problems. A confusion matrix showing the model mixing up two visually similar classes looks like a representation learning issue. Sometimes it is. But it is worth checking, before reaching for a new loss function, whether the two classes were themselves inconsistently labelled by different annotators or annotation batches. I have found it useful to treat unexpected confusion patterns as a prompt to re-examine the labels first, not last.
There is also a leakage-adjacent trap here that deserves attention: guideline drift over time. Large labelling efforts happen in batches, often across weeks or months, sometimes with turnover in the annotation team. If the guideline is clarified partway through, perhaps because someone finally asks how to handle occluded vehicles, then early-batch labels and late-batch labels are not drawn from the same labelling policy even though they sit in the same dataset. If your train and test splits are not stratified by labelling batch, you can end up training on one policy and testing on a blend that includes a different one, which quietly inflates or deflates your reported numbers depending on which split absorbed which batch. This is a leakage problem in disguise: information about labelling era correlates with the target, and an unaware split lets that correlation bleed across train and test.
The practical remedy is process, not cleverness. Write the guideline with edge cases spelled out before labelling begins, not after disputes arise. Sample a subset for double annotation early and compute agreement before committing budget to the full dataset. Keep a record of guideline versions and, where feasible, tag examples with which version produced them so you can audit for drift later. Treat annotator disagreement not as noise to be averaged away but as a signal pointing directly at where your guideline is underspecified.
The practical takeaway
Before tuning a model further, ask a blunter question: if you handed the same images to a second annotation team following the same instructions, how often would they draw the same boxes and assign the same classes? If you cannot answer that with a number, you do not yet know your ceiling. Spend the first slice of any vision project's budget on a small, deliberately adversarial batch of double-annotated edge cases, measure agreement honestly, and only then decide whether the bottleneck is the model or the label. In my experience it is the label far more often than the literature likes to admit.
