Tokenization Choices and Why They Change Your Results

Why this is not a footnote

When people write up an NLP experiment, tokenization usually gets a single throwaway sentence: 'we used a standard tokenizer'. I understand the impulse. Tokenization feels like plumbing, something you set once and forget. But the split you choose determines your vocabulary, your sequence lengths, how rare words are represented, and ultimately what patterns your model is even capable of learning. Two teams can train the same architecture on the same corpus and get meaningfully different results purely because one used word level splitting and the other used a subword scheme.

The reason this matters more than people admit is that tokenization decisions are rarely tuned as hyperparameters. They are baked in at the start of the pipeline, often inherited from whatever library defaults ship with a pretrained model, and then everything downstream, embeddings, attention patterns, evaluation metrics, is built on top of that choice. If the foundation is wrong for your data, you are compensating for it everywhere else without realising it.

A worked comparison: word, subword, character

Take a concrete case: a dataset of product reviews containing a fair number of misspellings, brand names, and compound words like 'waterproof-ish' or 'overpriced-but-worth-it'. Suppose the vocabulary of unique words across the corpus is around 40,000 terms, but roughly 15% of those words appear fewer than five times.

With a pure word level tokenizer, every one of those rare words becomes either its own embedding, learned from very few examples, or gets mapped to an out-of-vocabulary token and effectively erased. If 'overpriced-but-worth-it' appears three times in your training set, the model has almost nothing to learn from and no way to relate it to the word 'overpriced', which appears constantly. You lose the compositional signal entirely.

Switch to a subword scheme, something like byte pair encoding trained with a vocabulary of 30,000 merges, and that same compound term gets broken into pieces such as 'over', 'priced', '-but-', 'worth', '-it'. Each of those fragments appears far more often across the corpus, so each gets a well-trained embedding, and the model can compose meaning from parts it has actually seen enough times to learn something stable. The out-of-vocabulary problem essentially disappears, because any string can be built from a small set of subword units or, worst case, individual characters.

Character level tokenization goes further still: no unknown tokens ever, but now your sequences are five to seven times longer for the same sentence, your model has to learn spelling and word boundaries from scratch, and you pay for that in training time and in how far back the model can effectively attend. On the review dataset, a character level model might match the subword model's accuracy eventually, but it typically needs more data and more epochs to get there, and in a project with a fixed compute budget, that gap is not academic. It is the difference between a result you can ship and one you cannot.

Where this quietly breaks evaluation

The subtler danger is that tokenization choices interact with evaluation in ways that are easy to miss. If you compute perplexity or any length normalised metric, changing the tokenizer changes what a 'token' means, so numbers from two different tokenization schemes are not comparable even if they are labelled with the same metric name. I have seen comparisons in papers and blog posts that quietly stack a subword model's perplexity against a word level baseline as though they were on equal footing. They are not, because the denominator itself has shifted.

Sequence length effects also bleed into leakage-adjacent problems. Suppose you truncate inputs at 128 tokens for a classification task. Under word level tokenization, 128 tokens might cover a full customer review. Under a subword tokenizer with more fragments per word, that same review might get truncated halfway through the sentence that actually contains the sentiment-bearing clause. Your model's apparent weakness on long reviews might have nothing to do with modelling capacity and everything to do with the tokenizer silently eating your signal before the model ever sees it.

There is also a domain mismatch trap. A subword vocabulary trained on general web text will fragment domain-specific vocabulary, drug names in clinical text, function names in code, ticker symbols in financial text, into awkward pieces that do not correspond to any meaningful unit. This inflates sequence length, dilutes attention, and can make a domain-specific model look worse than it is, purely because the tokenizer was never adapted to the domain. Retraining or extending the vocabulary on in-domain text is often a larger single improvement than switching architectures.

What to actually do about it

Treat your tokenizer as part of the experimental design, not a fixed preprocessing step, and report it explicitly: vocabulary size, algorithm, and whether it was trained on your data or inherited from a pretrained checkpoint. When comparing models, keep the tokenizer identical unless the tokenizer itself is the variable under test; otherwise you are running a confounded experiment and drawing conclusions about architecture that might really be conclusions about vocabulary.

Before committing to a scheme, look at your own data's token length distribution and out-of-vocabulary rate under a couple of candidate tokenizers. This takes an afternoon and tells you immediately whether truncation will bite, whether rare terms will be shredded into meaningless fragments, and whether a pretrained tokenizer actually fits your domain or needs retraining. That small diagnostic step has saved me from chasing phantom modelling problems that were really tokenization artefacts, and it is the cheapest form of rigour available in an NLP pipeline.