Fine-Tuning a Small Transformer: What Actually Changes

Why fine-tuning deserves more than a shrug

It is easy to treat fine-tuning as a formality: load a pretrained checkpoint, bolt on a classification head, run a training loop for a few epochs, and call it done. I used to think of it that way too, until I watched a small transformer's validation accuracy swing wildly between runs that differed only in learning rate. That experience taught me that fine-tuning is not a rubber stamp on a pretrained model; it is a second, smaller training problem with its own failure modes, and understanding what actually moves inside the network is the difference between a model that generalises and one that quietly memorises.

Consider a small transformer, something in the range of a few million parameters, pretrained on a large general corpus and now being adapted to classify short pieces of text into five categories. Before any fine-tuning, its token embeddings and attention layers already encode useful structure: which words tend to appear together, roughly how sentences are organised, some notion of semantic similarity. Fine-tuning does not teach the model language from scratch. It nudges an already competent representation towards the specific decision boundary your task needs. That framing matters because it tells you how gently you should intervene.

What is actually changing, layer by layer

Picture the model as a stack of twelve transformer blocks sitting beneath a freshly initialised linear classification head. At the start of fine-tuning, the head's weights are random, so its gradients are large and somewhat noisy; it has everything to learn. The deeper blocks, closer to the input, hold general-purpose features and tend to need only small adjustments. The blocks nearer the output, which feed directly into the task-specific head, usually need to shift more, because they are the ones translating general representations into something the new head can use.

If you fine-tune everything with a single learning rate, say 2e-5, a common and often sensible default for a small transformer, you are implicitly assuming that the random head and the well-trained early layers should move at the same pace. They should not. A head starting from scratch benefits from a higher effective learning rate, since it has far to travel, while early layers risk being dragged away from useful structure if the learning rate is too high. This is one reason practitioners often use a warm-up phase, slowly increasing the learning rate over the first few hundred steps, and sometimes a layer-wise decay scheme where deeper layers get smaller updates than shallower ones.

Here is a concrete illustration. Suppose you fine-tune for three epochs on four thousand labelled examples with a batch size of sixteen, giving roughly 750 optimisation steps. With a flat learning rate of 5e-5 and no warm-up, I would expect the loss to spike in the first fifty steps as the random head sends large, poorly calibrated gradients backward through the whole network, perturbing early layers before the head has learned anything coherent. With a short warm-up over the first fifty to one hundred steps, the head gets a chance to settle into a sensible region before full-strength gradients start reshaping the rest of the model. The difference on a held-out validation set might be a few percentage points of accuracy, which sounds small until you remember that is often the entire gap between two competing approaches in a paper.

Freezing, forgetting, and the leakage trap

A second lever is deciding how much of the network to freeze. Freezing the bottom six of twelve layers and only training the top six plus the head is a reasonable compromise when your fine-tuning dataset is small, say under a few thousand examples. The intuition is that general linguistic structure, learned from a far larger pretraining corpus, is more reliable than anything a few thousand task-specific examples can teach, so you protect it from being overwritten. I have found this especially useful when the fine-tuning set is noisy or narrow in topic; full fine-tuning on such data can cause what is often called catastrophic forgetting, where the model becomes excellent at the new task but loses general competence it might need if the test distribution drifts even slightly from the training distribution.

This connects directly to evaluation, which is where I think most fine-tuning projects quietly go wrong. If your train and validation splits are drawn from the same small pool of documents, perhaps the same handful of sources or authors, a fully fine-tuned model can achieve suspiciously high validation accuracy by latching onto superficial patterns, specific phrasing, formatting quirks, author style, rather than the underlying task. A partially frozen model, forced to rely more on its general pretrained representations, is often more honest about its true performance, because it cannot overfit as aggressively to those idiosyncrasies. The lesson is not that freezing is always correct; it is that your evaluation protocol needs to be leakage-aware regardless of how much of the network you train, with splits built around natural groupings like document source or time period rather than random shuffling.

A useful sanity check, which costs almost nothing, is to compare your fine-tuned model against a frozen-embedding baseline: take the pretrained model exactly as it is, extract its representations without updating any weights, and train only a simple linear classifier on top. If that baseline already achieves, say, 78 percent accuracy and your fully fine-tuned model reaches 81 percent, you have learned something real but modest. If the baseline reaches 78 percent and fine-tuning jumps to 97 percent, I would be suspicious before celebratory, because large gaps like that often point to leakage, label imbalance exploited by the head, or a validation set that is too similar to training data rather than a genuine improvement in understanding.

A practical checklist

None of this requires exotic tooling. Before fine-tuning a small transformer, I now routinely do four things: build a frozen-embedding baseline first so I have a reference point; use a short learning rate warm-up even for a handful of epochs; consider freezing lower layers when the fine-tuning dataset is small relative to the pretraining corpus; and construct validation splits around meaningful groupings rather than random rows, so that performance gains reflect the task rather than the dataset's accidental structure.

The broader point is that fine-tuning is not a single homogeneous operation applied uniformly across a network. Different layers are at different stages of readiness, the new head starts from nothing while the body starts from a great deal, and your evaluation setup determines whether the numbers you get back mean what you think they mean. Treat fine-tuning with the same scepticism and care you would apply to training from scratch, and the results become far more trustworthy, even if, on paper, they look a little less dramatic.