Choosing a Metric for Imbalanced Text Classification

Why accuracy quietly fails you

Imagine a spam filter dataset where 95 out of every 100 emails are legitimate and 5 are spam. A model that simply predicts 'not spam' for everything will score 95% accuracy without ever learning a single thing about spam. That number looks impressive on a slide, but it is measuring the wrong thing: it rewards the model for exploiting the base rate of the classes rather than for actually discriminating between them.

This is the core problem with accuracy under class imbalance. It treats every correct prediction as equally valuable, but in most real text classification tasks, from detecting abusive comments to flagging fraudulent reviews, the minority class is usually the one you actually care about. A metric that can be gamed by ignoring that class is not a metric worth reporting, no matter how standard it feels to compute.

I have seen this trip up otherwise careful work: a model is tuned, accuracy climbs, everyone is pleased, and only later does someone notice the confusion matrix shows the minority class recall sitting near zero. The fix is not a better model at that point, it is a better question asked earlier: what does 'good' mean for this task, and does accuracy actually capture it?

Precision, recall, and the trade-off they expose

Precision and recall force you to be specific about the kind of mistake you care about. Precision asks: of everything the model flagged as spam, how much actually was spam? Recall asks: of everything that actually was spam, how much did the model catch? These are often in tension, and the tension is exactly what accuracy hides.

Take a concrete case. Suppose a review-flagging model is run over 1,000 reviews, 50 of which are genuinely fraudulent. The model flags 80 reviews as fraudulent, and of those, 40 are correct. It also misses 10 fraudulent reviews entirely. Precision here is 40 out of 80, which is 50%. Recall is 40 out of 50, which is 80%. Accuracy, by contrast, would be calculated over all 1,000 reviews and would land somewhere above 96%, because the 910 correctly identified legitimate reviews dominate the arithmetic. The accuracy figure tells you almost nothing about how the model handles the class you actually built it to catch.

The F1 score, the harmonic mean of precision and recall, is a useful single number when you need one, because it punishes a model that sacrifices one for the other too aggressively. In the example above, F1 works out to about 61.5%. That number is far less flattering than accuracy, and that is precisely its value: it reflects the thing you are actually trying to measure.

Where it gets more interesting is multi-class text problems, such as classifying support tickets into a dozen categories with very uneven frequencies. Here you choose between macro-averaged and weighted-averaged F1. Macro F1 treats every class equally regardless of how many examples it has, so a rare category contributes as much to the score as a common one. Weighted F1 scales each class's contribution by its frequency, which can mask poor performance on rare classes behind strong performance on common ones. If your rare categories matter, and in support ticket routing they often matter more because they signal unusual or urgent issues, macro F1 is the honest choice even though it will usually produce a lower, less comfortable number.

Beyond a single threshold

Most text classifiers do not output a hard label directly, they output a probability or score that gets thresholded into a decision. Precision, recall, and F1 as computed above all depend on wherever that threshold happens to sit, usually 0.5 by default, which is an arbitrary choice with no particular justification on imbalanced data.

This is why metrics like the area under the precision-recall curve are worth reporting alongside, or instead of, a single threshold-dependent number. The PR curve shows you precision and recall across every possible threshold, which matters a great deal when imbalance is severe, because the ROC curve, a more commonly taught alternative, can look deceptively strong even when the minority class is barely being distinguished. ROC curves plot true positive rate against false positive rate, and when negatives vastly outnumber positives, the false positive rate stays small even with a fair number of false positives in absolute terms, flattering the curve. PR curves do not have this blind spot because they are built from precision, which is directly sensitive to how many of the flagged cases are wrong.

In practice, I would also recommend looking at the confusion matrix itself rather than relying purely on summary statistics. A matrix immediately reveals which classes are being confused with which others, something no single number, however well chosen, can tell you. If a support ticket classifier is confusing 'billing' with 'account access' far more than any aggregate metric suggests, that is a labelling or feature problem worth investigating directly, not something to average away.

A practical way to decide

Start by asking what kind of error is more costly in your actual application, not in the abstract. If missing a positive case is expensive, such as failing to catch genuine abuse, prioritise recall and report it alongside precision so the trade-off is visible. If false alarms are expensive, such as a model that keeps flagging legitimate transactions and eroding user trust, precision deserves more weight.

Then pick an aggregate metric that matches that priority: F1 when you need a balanced single number, F-beta with beta above one when recall matters more, macro F1 for multi-class problems where rare categories count, and always a precision-recall curve rather than a single threshold when you are still deciding how conservative the model should be. Report accuracy too if you like, it is not useless, but never let it stand alone on imbalanced data, and never let a leaderboard number substitute for actually inspecting the confusion matrix.