Multi-Label vs Multi-Class: Metrics Quietly Misapplied
A lot of reported accuracy and F1 scores are computed for the wrong problem entirely. Here is how to tell, and what to do instead.
Two problems that look similar and are not
Multi-class classification assigns exactly one label from a set of mutually exclusive options: a photo is a cat, a dog, or a bird, never two at once. Multi-label classification assigns any number of labels from a set, including zero or all of them: a news article might be tagged politics and economics and foreign-affairs simultaneously. The mathematics of evaluation for these two settings diverges quickly, but the vocabulary does not. People say accuracy, precision, recall and F1 in both contexts, and the reader assumes the same meaning is intended. It usually is not.
I have seen this most often in text classification and medical coding tasks, where each document can carry several correct tags. A team reports ninety-two percent accuracy and everyone nods, but accuracy for multi-label problems is not the fraction of correct predictions in the multi-class sense at all. It is either subset accuracy, which demands every single label be exactly right, or some looser per-label average, and the two numbers can differ by tens of percentage points on the same predictions. Nobody misled anyone on purpose. The metric name just travelled across a boundary where its definition quietly changed.
The practical danger is that these numbers get compared across papers, dashboards and stakeholder reports as if they are the same currency. A multi-label model reported at ninety percent subset accuracy sounds worse than a multi-class model at ninety-five percent accuracy, until you realise the multi-label task is strictly harder because it demands an exact match on a combinatorial output space. Without knowing which flavour of accuracy is in play, that comparison is meaningless, yet it gets made in meetings constantly.
A worked example with real numbers
Suppose we have a document tagging system with five possible tags, and a test set of one thousand documents, each carrying between one and three true tags. Consider a single document with true tags {politics, economics}. The model predicts {politics, sport}. In multi-class evaluation this comparison would not even arise, because multi-class assumes one true label and one predicted label. Forcing this into a multi-class frame by picking the top prediction only would score this as correct if politics happened to be ranked first, discarding the fact that economics was missed and sport was wrongly added.
Now score it properly as multi-label. Subset accuracy: the predicted set does not exactly equal the true set, so this document scores zero, full stop, no partial credit. Hamming-based accuracy, which looks at each of the five possible tags independently as its own binary decision, tells a different story: politics correct (true positive), economics wrong (false negative), sport wrong (false positive), and the two remaining tags correctly predicted as absent (true negatives). That is three correct out of five binary decisions, so sixty percent under this view. Per-example precision here is one correct prediction out of two predicted, fifty percent, and per-example recall is one correct out of two true tags, also fifty percent.
Three numbers, zero percent, sixty percent and fifty percent, all legitimately describing the same single prediction. None of them is wrong. They answer different questions: did we get the whole set exactly right, how good are our individual tag decisions across the whole label space, and how good is our set overlap ignoring the easy true negatives. Averaged across a thousand documents these three views routinely produce headline numbers that differ by twenty or thirty points, and a report that states just one number, unlabelled, is not really reporting anything a reader can act on.
Why averaging choice matters just as much
Even once you have settled on set-based precision, recall and F1 for a multi-label problem, there is a second, quieter decision: how to average across labels. Micro-averaging pools every true positive, false positive and false negative across all labels and all examples before computing precision and recall, which means frequent labels dominate the result. Macro-averaging computes precision and recall separately for each label and then takes the unweighted mean across labels, which gives a rare label exactly the same voice as a common one.
Imagine our five tags again, but now politics appears in six hundred of the thousand documents while a niche tag like agriculture appears in only ten. Suppose the model is excellent on politics, ninety-five percent F1, but essentially guesses on agriculture, twenty percent F1, because there is so little training signal for it. Micro-averaged F1 will sit close to the politics number, because politics contributes the overwhelming majority of the true positives and false positives being pooled; the headline might read ninety-one percent. Macro-averaged F1 treats agriculture as equally important as politics in the average, so the same model might score sixty-two percent macro F1. Both numbers are computed correctly from the same confusion counts. They tell a stakeholder wildly different stories about whether the model is fit for purpose, and the right choice depends entirely on whether rare tags matter to the business problem or whether overall throughput on common tags is what counts.
This is exactly the kind of detail that gets lost when a metric is copied from a multi-class habit. In multi-class work, accuracy is often the default because classes are frequently reasonably balanced or the cost of imbalance is at least visible in a single confusion matrix. In multi-label work, silently defaulting to micro-averaged F1 because it looks like the highest number, without checking whether rare labels are the ones that actually matter clinically, legally or commercially, is a very easy way to ship a model that fails exactly where it needed to succeed.
What to actually do about it
First, name the metric precisely in any report or paper, including the averaging scheme: micro-F1, macro-F1, subset accuracy, or Hamming score, not just F1 or accuracy on their own. This single habit prevents most of the confusion I have described. Second, report more than one view when the problem is multi-label: subset accuracy tells you how often the whole prediction is exactly usable, macro-F1 tells you how well rare labels are served, and micro-F1 tells you overall tagging quality weighted by frequency. Together they triangulate a much more honest picture than any single number.
Third, check your label split before trusting any of these numbers at all. Leakage across a train and test split is even easier to introduce in multi-label settings, because stratifying by a single label is not enough when documents carry combinations of tags; a document with a rare label combination can end up entirely in training or entirely in test by chance, distorting exactly the macro-averaged rare-label scores you are relying on to catch failure. Finally, when reading someone else's reported number, ask which of these definitions they used before comparing it to anything else. A quietly misapplied metric is rarely a lie, but it is very often a misunderstanding waiting to be repeated by the next person who reads the report.
