Named Entity Recognition: Three Approaches Compared

Why the choice of approach still matters

Named entity recognition sounds like a solved problem until you try to deploy it on text that does not look like a benchmark. The task is simple to state: find spans in a sentence that refer to people, organisations, locations, dates, or whatever category set you care about, and label them correctly. What is not simple is that the three dominant families of solutions, rule-based, statistical, and neural, make very different assumptions about what signal is reliable, and those assumptions show up as very different failure modes once you leave clean training-like text.

I think this comparison is worth doing carefully because the temptation in 2024 is to reach straight for a transformer and assume it dominates on every axis. It often does on accuracy for well-resourced languages and domains. It does not always win on cost, latency, interpretability, or robustness to a narrow, well-defined entity set. A sensible engineer picks the tool that matches the constraints of the problem, not the tool with the most recent paper attached to it.

Rule-based systems: transparent but brittle

A rule-based NER system is built from hand-written patterns: gazetteers of known names, regular expressions for dates and phone numbers, and heuristics like capitalisation or trigger words such as 'Mr' or 'Inc'. Consider a simple rule for detecting company names: match a capitalised sequence of one to three words immediately followed by a suffix from a fixed list such as 'Ltd', 'Inc', or 'GmbH'. On a set of formal business documents, a rule like this can achieve genuinely high precision, sometimes above ninety-five percent, because the pattern is specific and the domain is narrow.

The weakness appears the moment coverage matters. That same rule will miss 'Deutsche Bank' with no suffix, misfire on 'Ford' the common surname versus 'Ford' the car company, and say nothing useful about a name it has never seen in a gazetteer, such as a newly founded startup. Recall tends to be the casualty: precision stays respectable while recall drops sharply outside the patterns you anticipated. Maintenance is the other hidden cost. Every new entity type or new domain means someone sits down and writes more rules, and those rules interact with each other in ways that become hard to reason about once the rule set grows past a few hundred lines.

Where rule-based systems genuinely earn their place is in narrow, high-stakes, well-specified domains: extracting drug dosages from a fixed prescription format, or pulling case numbers from a specific legal document template. When the format is stable and the cost of a false positive is high, a transparent rule that a domain expert can read and audit is often preferable to a model whose decision boundary nobody can fully explain.

Statistical sequence models: the middle ground

Statistical approaches, most commonly Hidden Markov Models and Conditional Random Fields, treat NER as a sequence labelling problem using the BIO tagging scheme, where each token gets a label such as B-PER, I-PER, or O for outside any entity. A CRF learns weights over hand-engineered features: the word itself, its part-of-speech tag, whether it is capitalised, whether it appears in a gazetteer, and the labels of neighbouring tokens. Crucially, it models the dependencies between adjacent labels directly, which matters because an I-PER tag almost never follows an O tag in a well-formed sequence, and the model can learn that constraint explicitly rather than hoping a classifier infers it token by token.

The practical gain over pure rules is graceful handling of ambiguity through learned probabilities rather than brittle if-then logic. Imagine a CRF trained on newswire text seeing the token 'Washington'. Context features, such as the preceding word being 'President' or the following word being 'D.C.', shift the probability towards PER or LOC respectively, without anyone writing an explicit rule for that specific case. On a moderately sized labelled corpus, a well-tuned CRF with good features can reach competitive performance, often within a few points of a neural model on the same data, while training in minutes on a laptop rather than hours on a GPU.

The catch is feature engineering. Getting a CRF to perform well requires deciding which context window to use, which gazetteers to build, and how to encode morphology for rare or unseen words. That effort does not transfer cleanly between domains; a feature set tuned for financial news needs substantial rework for clinical notes. Statistical models also struggle with genuinely novel surface forms that share no features with anything seen in training, since they cannot lean on the kind of distributed semantic similarity a neural embedding provides.

Neural approaches and the honest trade-offs

Neural NER, whether a BiLSTM with a CRF layer on top or a fine-tuned transformer, replaces hand-built features with learned representations. A pretrained transformer arrives already knowing that 'Aspirin' and 'Ibuprofen' sit in similar semantic space to other drug names it saw during pretraining, even if the exact token never appeared in your labelled data. This is the main practical advantage: far better generalisation to unseen entity mentions, which directly addresses the recall problem that plagued the rule-based approach.

The worked comparison is useful here. Suppose all three systems are evaluated on a test set containing an entity type unseen in training, like a newly coined product name. The rule-based system misses it entirely unless someone adds a specific pattern. The CRF partially catches it if the surrounding context features are strong enough, capitalisation, position, neighbouring words, but confidence is often low. The fine-tuned transformer frequently gets it right, because the surrounding sentence structure and subword patterns resemble entities it has implicitly learned to recognise as a class, not just as memorised strings.

None of this is free. Transformer-based NER needs labelled data to fine-tune, computational resources to train and serve, and it is considerably harder to debug when it makes a confident, wrong prediction with no obvious reason attached. It also inherits whatever biases sit in its pretraining data, which can quietly skew entity recognition towards names and formats common in the training corpus and away from underrepresented naming conventions. And on small, narrow, template-like datasets, the accuracy gap over a well-tuned CRF can be small enough that the added complexity is not worth it.

My practical takeaway is to treat this as a decision problem rather than a leaderboard. If the domain is narrow and stable, start with rules and measure precision and recall honestly on a held-out set that reflects real deployment text, not just your development examples. If you have moderate labelled data and need something interpretable and cheap to run, a CRF with good features is a strong, underrated baseline that many neural comparisons skip. Reach for a neural model, ideally a pretrained transformer fine-tuned on your labelled data, when generalisation to novel entities matters more than latency or interpretability, and always validate on a split that genuinely resembles production, since leakage between similar documents in training and test sets is the fastest way to overstate how well any of these three approaches will actually perform.