NLP Techniques Before Deep Learning: A Practical Refresher

Why bother with the old stuff

It is tempting, when everyone around you is fine-tuning transformers, to treat pre-2015 NLP as a museum exhibit. I think that is a mistake, and not out of nostalgia. Classic techniques such as bag-of-words, TF-IDF, n-gram language models and simple linear classifiers are cheap, fast, interpretable and, crucially, excellent diagnostic tools. They tell you things about your data that a large neural model will happily hide from you until it fails in production.

Consider a sentiment classification task on product reviews. If a logistic regression trained on TF-IDF features gets 89 percent accuracy, and your fine-tuned transformer gets 91 percent, that two-point gap needs explaining. Is the transformer actually learning deeper semantics, or is the dataset simply easy, with obvious lexical cues like 'terrible' and 'excellent' doing most of the work? A strong classical baseline forces that question onto the table early, before you have spent a week on GPU time and written a paragraph in your report claiming a breakthrough that is really just noise.

There is also a leakage argument. Classical pipelines are simple enough that you can inspect every feature by hand. If your vocabulary suddenly contains a token that looks suspiciously like a document ID, or a rare word that appears in exactly one class because of how the data was scraped, a quick look at your TF-IDF matrix will often reveal it. Deep models with subword tokenisers and millions of parameters make the same leakage much harder to spot, because the signal gets absorbed into weights you cannot read.

The core toolkit, briefly

Bag-of-words represents a document as a vector of word counts, ignoring order entirely. It sounds crude, and it is, but for many classification tasks word order contributes far less than people assume. TF-IDF improves on raw counts by weighting terms that are frequent in a document but rare across the corpus more heavily, which downweights words like 'the' and 'good' that appear everywhere and upweights genuinely distinguishing terms.

Worked example: imagine a corpus of one thousand restaurant reviews. The word 'delicious' appears in forty reviews, mostly positive. The word 'the' appears in nearly all one thousand. Raw counts would treat a document containing 'the' three times as similarly weighted to one containing 'delicious' once, which is unhelpful. TF-IDF for 'delicious' in a document where it appears twice, out of two hundred total words, with document frequency forty out of one thousand, gives a term frequency of 0.01 multiplied by an inverse document frequency of roughly log(1000/40), around 3.2, yielding a meaningful score of about 0.032. The word 'the', appearing say ten times in the same document, has a much larger term frequency but an inverse document frequency close to zero, since it appears in almost every document, so its TF-IDF score collapses towards zero. The representation naturally foregrounds the informative word and suppresses the filler.

N-gram models extend this by capturing short sequences, bigrams and trigrams, which recover some local word order. 'Not good' and 'good' point in opposite sentiment directions, and a pure bag-of-words unigram model conflates them because it only sees 'good' once negation is stripped out during tokenisation choices. Adding bigrams as features lets a linear model learn that 'not good' pushes towards negative, without needing any notion of syntax or attention.

Naive Bayes, despite its almost comically strong independence assumption, remains a genuinely useful baseline, particularly on small or moderately sized text datasets. It estimates the probability of a class given a document by multiplying the probability of each word given that class, assuming words are conditionally independent given the label. That assumption is false in almost every real sentence, yet the classifier still performs respectably, because classification only needs the correct ranking of class probabilities, not an accurate joint distribution of language. Logistic regression over the same TF-IDF features usually edges it out slightly by learning weighted combinations rather than assuming independence, but Naive Bayes trains in seconds and gives you a sanity check almost for free.

Where these methods still earn their place

The most practical use case today is the baseline. Any serious evaluation of a new NLP model should report what a TF-IDF plus linear classifier achieves on the same train, validation and test split. If your split has any leakage, duplicate documents across train and test, or near-duplicate templated text, the classical baseline will often score suspiciously high, which is your signal to go back and audit the split before trusting anything built on top of it.

Classical methods are also valuable for feature-level interpretability. With a linear model over TF-IDF features you can read off the coefficients directly and say precisely which words push a prediction towards which class. This matters in domains such as clinical text or legal document triage, where a model that cannot explain itself in terms a domain expert recognises is difficult to deploy responsibly, regardless of its raw accuracy. A logistic regression coefficient table is not a substitute for proper explainability work, but it is a starting point that costs almost nothing to produce.

There is a resource argument too. On genuinely small datasets, say a few hundred labelled examples, a fine-tuned transformer can overfit badly or simply have too little signal to learn from, while a well-regularised linear model over n-gram features can still generalise reasonably. I have seen cases where the honest answer to 'should we use a transformer here' is no, not because transformers are bad, but because the dataset is too small and too narrow for their capacity to help rather than hurt.

The practical takeaway

None of this is an argument against deep learning for NLP, which has earned its dominance on tasks involving long-range context, generation and genuine semantic reasoning. The argument is narrower and more useful day to day: before you trust a complex model's number, build the boring baseline first. A TF-IDF vectoriser and a logistic regression take minutes to train, cost almost nothing in compute, and give you a defensible floor to measure everything else against.

If your fancy model cannot clearly beat that floor by a margin that survives a proper significance check on a leakage-aware split, you have not learned that deep learning works on your problem. You have learned that your problem is easy, or that something in your evaluation is broken. Either way, the classical toolkit told you first, and told you cheaply.