How Pretraining Data Shapes What a Transformer Can Do

The architecture is not the interesting part

When people discuss what a transformer can or cannot do, the conversation usually drifts towards attention heads, layer counts, or positional encodings. I understand the appeal; architecture is concrete, diagrammable, and satisfying to reason about. But in my experience, the more decisive factor is duller and less discussed: what the model actually saw during pretraining. Two models with identical architecture and parameter counts can behave completely differently depending on the corpus they were trained on, and that difference is often larger than anything a change in layer depth would produce.

This matters because it reframes how we should evaluate and trust a model. If a transformer struggles with a task, the first question should not be 'is the architecture wrong', it should be 'did the pretraining distribution contain enough signal for this task to be learnable in the first place'. A model cannot represent a pattern it has never been exposed to in any form, no matter how many parameters it has to play with.

Think of pretraining as building a map of the world from whatever routes happen to be walked most often. Densely travelled roads become wide and well marked; rarely used paths barely register. The model's internal representations end up shaped the same way, richly detailed where the data was rich, thin and unreliable where it was sparse, regardless of how important that thin region might be to a downstream user.

A worked example: arithmetic versus prose

Consider a plain text corpus scraped broadly from the internet. Suppose, for illustration, that fluent English prose makes up the overwhelming majority of tokens, while multi-digit arithmetic expressions appear only occasionally, embedded in forum posts or textbooks. A transformer trained on this mixture will become excellent at predicting the next word in a sentence about, say, gardening or history, because it has seen millions of structurally similar examples. Its internal representations for grammar, common phrasing, and topic coherence become dense and well calibrated.

Now ask the same model to add two five-digit numbers. Even though addition is a simple, well-defined operation, the model has seen relatively few worked examples of long-form arithmetic during pretraining, and even fewer that show the intermediate carrying steps explicitly. The result is a model that can often get small sums right through pattern memorisation, but starts failing as digit count grows, not because addition is conceptually hard, but because the training distribution never forced it to learn a general, position-independent procedure. The failure looks like a reasoning gap, but the root cause is a data gap.

This distinction matters enormously for evaluation. If I test a model on five-digit addition and it fails, I should not conclude that transformers 'cannot do maths'. I should ask whether the pretraining mixture contained enough structured numerical reasoning, with intermediate steps, for the relevant circuitry to form. Later models that deliberately upweight code, step-by-step maths solutions, or synthetic arithmetic in their pretraining mixture tend to show markedly better arithmetic behaviour, and that improvement tracks the data change far more tightly than it tracks any single architectural tweak.

Why this creates blind spots that fine-tuning cannot fix

A common assumption is that fine-tuning can patch whatever pretraining missed. In practice this is only partly true, and the limits matter. Fine-tuning is very effective at reshaping how a model expresses knowledge it already has, adjusting tone, format, or task framing. It is far less effective at installing a capability that has no representational foothold at all. If a concept was essentially absent from pretraining, a modest fine-tuning set is unlikely to build the necessary internal structure from scratch; there simply were not enough exposures during the phase where the bulk of the model's representations were formed.

Here is a concrete framing. Suppose a model is fine-tuned on a few thousand examples of a specialised task, such as parsing a niche legal document format, and the pretraining corpus contained almost nothing resembling that format or its vocabulary. The fine-tuned model may learn surface patterns, such as expected section headers, but it will generalise poorly to documents phrased slightly differently, because it never built a deep representation of the domain, only a thin overlay on top of unrelated pretraining knowledge. Contrast this with fine-tuning a model on a task that is well represented in pretraining, such as summarising news articles, where even a small fine-tuning set can unlock strong performance because the underlying competence was already there, waiting to be steered.

This has a direct, practical consequence for anyone evaluating or deploying these models: before blaming an architecture or a fine-tuning recipe for poor performance, it is worth asking what evidence exists that the target skill was ever present in the pretraining mixture at meaningful scale. This also explains why benchmark performance can be misleading if the benchmark's style overlaps with common pretraining text; a model can look competent by pattern-matching surface form rather than by possessing the underlying skill, which is exactly the kind of gap that a leakage-aware, out-of-distribution evaluation set is designed to expose.

What this means in practice

If you are choosing or building a model for a specific task, treat the pretraining data composition as a first-class design variable, not an afterthought hidden behind a benchmark score. Where possible, look for documentation of what domains, languages, and formats were represented, and in what proportions. A model that scores well on a general benchmark can still be nearly blind to your specific domain if that domain was a rounding error in the training mixture.

When evaluating a model's failures, resist the instinct to reach for architectural explanations first. Ask instead whether the failure pattern looks like a genuine reasoning limit or a data coverage gap, since the two require completely different fixes: one needs a different model design, the other needs different data, whether through targeted fine-tuning, retrieval augmentation, or simply choosing a base model trained on a more relevant mixture. Getting this diagnosis right saves a great deal of wasted effort chasing architectural tweaks for a problem that data was always going to solve more directly.