Class Imbalance Beyond Binary: Skewed Multi-Class Problems

Why binary intuitions break down

Most tutorials on class imbalance use a fraud detection or disease screening example: two classes, one rare, one common. You learn about oversampling, class weights, and precision-recall curves, and it all feels tidy. The trouble starts when you move to a real multi-class problem, say ten categories of manufacturing defect, or twenty species of plant disease, where the class frequencies form a long, uneven tail rather than a clean 95-5 split.

With two classes, there is one minority to worry about and one majority to correct for. With many classes, you can have several minorities of wildly different sizes, classes that are easily confused with each other but not with the rest, and classes that are rare but trivially easy to spot once you see one. A single scalar like accuracy or even a single precision-recall curve cannot represent that structure. The moment you have more than two classes, imbalance stops being a single number problem and becomes a per-class diagnostic problem.

This matters because the standard fixes, oversampling the minority, undersampling the majority, adjusting a decision threshold, were designed for a world with exactly one boundary to shift. In a multi-class setting there are many boundaries, and moving one to help a rare class can quietly damage another class you were not even monitoring. I have seen this happen in practice: an intervention aimed at improving recall on the smallest class pushed several medium-sized classes into worse confusion with each other, and the headline accuracy barely moved, hiding the damage completely.

A worked example with real numbers

Suppose you are classifying support tickets into six categories: billing, technical, account access, feature request, bug report, and other. Say the training set has 6,000 billing tickets, 4,500 technical, 2,000 account access, 900 feature request, 400 bug report, and 200 other. That is a ratio of thirty to one between the largest and smallest class, and the middle classes are themselves unevenly spaced.

If you train a plain classifier and report accuracy, you might see 88 percent, which sounds respectable. But suppose the confusion matrix shows that of the 200 'other' tickets, only 40 are correctly labelled, with the rest scattered mostly into billing and technical. Recall for 'other' is 20 percent. Because 'other' makes up only 1.4 percent of the data, this catastrophic failure barely dents overall accuracy, yet if 'other' tickets are the ones that need urgent manual triage, this is precisely the failure that matters most to the business.

Now apply a common fix: oversample the minority classes so every class appears roughly equally during training. Recall on 'other' might jump to 65 percent, which looks like a win. But check technical and billing, the two largest classes: their precision may have dropped, because the model now more readily predicts rarer classes and steals some genuine technical tickets away from their correct label. Overall accuracy could actually fall even though you have fixed the original problem, because you have created a new, smaller one elsewhere. Without per-class metrics before and after the change, you would not see this trade-off at all; you would only see one aggregate number move and assume progress.

Metrics and methods that actually respect the structure

The first fix is reporting, not modelling. Report precision, recall, and F1 for every class individually, not just a single macro or weighted average. Macro F1 treats every class equally regardless of size, which is useful for spotting neglected minorities, but it can also overstate the importance of a class with only twenty test examples where a handful of mistakes swing the score wildly. Weighted F1 respects class frequency but can then hide small-class failures again, which is the exact problem you started with. Report both, alongside the raw confusion matrix, and look at where the errors actually go, not just how many there are.

Second, be deliberate about where imbalance correction happens. Class weighting in the loss function is often gentler than resampling because it does not distort the data distribution or duplicate examples; it simply tells the optimiser to penalise mistakes on rare classes more heavily. For genuinely tiny classes, a small amount of oversampling combined with careful data augmentation, where sensible for the domain, tends to be more stable than pure duplication, which risks overfitting to a handful of repeated examples.

Third, watch for classes that are imbalanced with respect to each other rather than the whole dataset. Two classes might each be moderately sized but almost always confused with one another, a much more common failure mode than one class simply being rare overall. This is a pairwise confusion problem, not a frequency problem, and no amount of resampling by overall class count will fix it. You need to inspect the confusion matrix directly and, if two classes are systematically swapped, consider whether your labelling scheme genuinely separates them or whether a hierarchical approach, first predicting a coarse group then a fine-grained label within it, would suit the data better.

Finally, be strict about how you split data for evaluation. Stratified splitting, which preserves each class's proportion across train, validation, and test sets, is essential in multi-class imbalance; a random split can easily leave your smallest classes with only one or two test examples, making their reported metrics almost meaningless and unstable across runs. If a class has fewer than about thirty examples in total, treat any single-run metric for it with real scepticism and consider cross-validation specifically to stabilise that estimate.

The practical takeaway

Multi-class imbalance is not a scaled-up version of binary imbalance; it is a different problem with its own failure modes, chiefly hidden per-class collapse and pairwise confusion that aggregate metrics conceal. Before reaching for oversampling or class weights, build the full confusion matrix and per-class precision and recall table first, on a properly stratified split. Only once you can see exactly which classes are failing and why should you choose a correction, and even then, re-check the same full table afterwards rather than trusting a single summary number to tell you whether things actually improved.