Sequence Labelling Metrics: Why Token Accuracy Lies
Token-level accuracy looks reassuring on named entity recognition and tagging tasks, but it systematically underweights the errors that actually matter. Here is why, with a worked example.
The seduction of a high number
Sequence labelling covers tasks like named entity recognition, part-of-speech tagging, and slot filling, where a model assigns a label to every token in a sentence. The obvious way to score such a model is token-level accuracy: count how many of the individual token predictions match the gold labels, divide by the total number of tokens. It is easy to compute, easy to explain, and almost always produces a flattering number.
That flattering number is the problem. In most tagging schemes, the overwhelming majority of tokens carry the label 'O', meaning no entity at all. A sentence like 'Book me a flight to Manchester next Tuesday' has eight tokens and perhaps two entities, so six tokens are trivially 'O' and easy to get right. A model that never predicts an entity at all, that simply outputs 'O' for everything, can already score well above ninety percent token accuracy on many corpora, purely because negatives dominate. Accuracy rewards you for agreeing on the easy majority class and barely penalises you for missing the rare, useful one.
This is not a niche statistical curiosity; it is the single most common way I see sequence labelling results misreported in coursework and in papers that should know better. A model gets described as achieving ninety-six percent accuracy, and the reader assumes it is nearly perfect, when in fact it might be missing or mangling a third of the actual entities. The gap between the reported number and the practical usefulness of the system is exactly the gap that accuracy is structurally blind to.
A worked example with realistic numbers
Suppose we have a test set of one hundred sentences, roughly twelve tokens each, so twelve hundred tokens in total. Say there are one hundred and fifty true entity mentions, most of them single tokens under a simple scheme, so fifteen hundred tokens are 'O' and one hundred and fifty carry a genuine entity label such as PERSON or LOCATION, giving one thousand and fifty non-entity tokens against one hundred and fifty entity tokens; adjust as needed, the ratio is the point, not the exact count.
Now imagine a mediocre model that correctly labels ninety-eight percent of the 'O' tokens as 'O', but only gets sixty of the one hundred and fifty entity tokens exactly right, either mislabelling the entity type or missing it entirely. Correct 'O' predictions: about 1029. Correct entity predictions: 60. Total correct: 1089 out of 1200, giving a token accuracy of just over ninety percent. That sounds like a solid, deployable model.
But look at what actually happened to the entities, which are the whole point of the task. The model got sixty out of one hundred and fifty entity mentions right, a recall of forty percent. If we also account for spans it partially got right, splitting a two-token entity into one correct and one wrong token, or predicting an entity where none exists, a proper entity-level evaluation using exact span and type matching, the kind used in standard conll-style scoring, would likely report an F1 somewhere in the fifties or sixties. That is the difference between a system a business would call 'basically working' and one it would call 'not ready'. Token accuracy told us ninety percent; the metric that reflects what the system is for told us barely half.
The mechanism behind this gap is simple once you see it: accuracy is computed per token, but usefulness is computed per entity, and an entity's tokens must all be correct, with the right boundaries and the right type, for it to count as a hit. A single misplaced boundary token turns a whole entity into an error, yet it barely dents an accuracy score built from thousands of easy 'O' tokens.
What to measure instead, and why
The standard fix is entity-level, or span-level, precision, recall, and F1, computed with exact match on both the boundaries and the type of each predicted span against the gold spans. This is harsher than token accuracy by design, because it insists that getting an entity 'mostly right' still counts as wrong if the boundary or type is off. That harshness is exactly what makes it informative: it mirrors how a downstream system, or a human reviewer, would judge the output. Nobody using an information extraction pipeline cares that seven of an entity's eight characters were tagged correctly; they care whether the entity was extracted, correctly typed, and usable.
It is also worth reporting a confusion breakdown by entity type rather than a single aggregate number. A model can have a perfectly respectable overall entity F1 while being nearly useless for one specific type, say dates, because that class is rare and structurally different from person or organisation names. Aggregating hides this the same way accuracy hides the entity-versus-O imbalance; the fix is the same instinct applied one level down, look at where the imbalance and the errors actually live rather than trusting a single summary statistic.
None of this means token accuracy is useless; it can be a fine sanity check during early debugging, or useful when comparing models on tasks where every token genuinely matters, such as certain part-of-speech tagging setups without a dominant 'O' class. The point is to know what a metric is structurally capable of hiding before you trust it as the headline number, and for anything with a majority negative class, that means treating token accuracy as, at best, a secondary diagnostic.
The practical takeaway: whenever you evaluate a sequence labelling model, report entity-level precision, recall, and F1 with exact span and type matching as your primary metric, break it down by entity type, and only quote token accuracy alongside it, clearly labelled, never in its place. If someone hands you a token accuracy figure with no entity-level breakdown, ask for one before believing the model works.