Transfer Learning: Freeze, Fine-Tune, or LoRA?

Why this choice matters more than it looks

Almost every applied deep learning project now starts with a pretrained model rather than a blank slate. The question is never whether to use transfer learning, it is how much of the pretrained model you should let move. Freeze the backbone and train only a new head, fine-tune the whole network, or insert a small set of trainable low-rank adapters through LoRA. Each choice trades off compute, data requirements, and risk of overfitting differently, and getting it wrong wastes far more time than the decision itself takes to make.

I have seen people default to full fine-tuning because it is the most familiar recipe, then wonder why their model with three hundred labelled examples has memorised the training set. I have also seen people freeze everything out of caution and leave meaningful accuracy on the table because the pretrained features were a reasonable but imperfect match for the new domain. The right answer depends on how much labelled data you have, how different your task is from the pretraining distribution, and how much compute and storage you can spend. None of that is exotic, but it is easy to skip the reasoning and just copy whatever recipe appeared in the most recent tutorial.

Freezing: cheap, safe, and sometimes just enough

Freezing the pretrained backbone and training only a new head on top is the lightest option. You run the frozen network once to extract features, then train a small classifier, often logistic regression or a shallow layer, on those fixed representations. The number of trainable parameters can be tiny, sometimes a few thousand, so the method works with remarkably little labelled data and almost no risk of catastrophic overfitting.

Suppose you have a pretrained image encoder trained on a broad, general-purpose dataset, and you want to classify a few hundred photos of manufacturing defects into three categories. With only four hundred labelled images, full fine-tuning of a network with tens of millions of parameters is asking for trouble: the model can fit noise long before it learns anything generalisable, and your validation curve will confirm it by diverging early. Freezing the backbone and training a small head on extracted features is often the more sensible starting point here, because it respects how little signal you actually have to work with.

The limitation is equally clear. If your new domain is visually or structurally distant from what the backbone saw during pretraining, frozen features may simply not encode the distinctions you need. A backbone trained on natural photographs may struggle to separate subtle textures in satellite imagery or medical scans, no matter how good the head is, because the relevant information was never preserved in the frozen representation. Freezing is a strong baseline precisely because it is cheap to try, and a leakage-aware validation split will tell you quickly whether it is enough.

Full fine-tuning: powerful, hungry, and easy to overdo

Full fine-tuning updates every parameter in the pretrained network, usually with a small learning rate so as not to destroy the useful structure already learned. When you have enough labelled data and a task that genuinely benefits from adjusting low-level features, this tends to outperform both frozen extraction and lightweight adapters, because nothing in the network is held back from adapting.

Return to the defect classification example, but now imagine you have forty thousand labelled images instead of four hundred, with clear class imbalance and a domain shift that a frozen backbone visibly struggles with. With that much data, full fine-tuning has enough signal to safely reshape the earlier layers without collapsing into memorisation, and the accuracy gain over a frozen-head baseline can be substantial, often several percentage points on a held-out test set drawn from a genuinely separate batch of production images rather than a random split of the same shoot.

The cost is real, though. You need enough compute to backpropagate through the entire network for many steps, enough memory to store gradients and optimiser states for every parameter, and enough storage if you plan to keep a separately fine-tuned copy of the model for each downstream task. If you are supporting ten different clients or ten different tasks from one base model, ten full copies of a large network is not a trivial storage or maintenance burden, and it makes reproducibility harder because now there are ten sets of weights, each with its own training run, learning rate schedule, and checkpoint to track.

LoRA: a practical middle ground, not a magic trick

Low-Rank Adaptation freezes the original pretrained weights entirely and instead learns small low-rank matrices that are added to specific layers, typically the attention projections in a transformer. Only these small matrices are trained, so the number of trainable parameters can be a tiny fraction, often under one percent, of the full model, while the frozen base weights remain untouched and shareable across tasks.

Consider a language model with a few hundred million parameters that you want to adapt to a specific customer support domain using perhaps five thousand labelled examples. Full fine-tuning at that data size risks overfitting and definitely requires storing a full new copy of the model. LoRA instead trains adapter matrices that might total a few million parameters, small enough to store per task as a lightweight file measured in megabytes rather than gigabytes, while the shared frozen backbone stays fixed and reusable. In practice this gives results close to full fine-tuning on many tasks, though not always identical, and the gap depends heavily on how much the task requires reshaping the model's earlier representations rather than just its output behaviour.

LoRA is not free of judgement calls either. You still choose the rank of the adapter matrices, which layers to attach them to, and a learning rate, and each of these affects the outcome. Too low a rank and the adapters cannot capture the needed adjustment; too high and you approach the cost and overfitting risk of full fine-tuning without necessarily gaining anything. It is a genuinely useful middle ground for teams serving many tasks from one base model, but it is a technique to validate against a proper held-out set, not a default you apply blindly because it is currently fashionable.

Choosing without guessing

My practical rule is to start cheap and escalate only when the evidence demands it. Try frozen feature extraction first, because it costs almost nothing and tells you quickly whether the pretrained representations are already close to sufficient. If that clearly underperforms and you have enough labelled data and compute, and only need one or a few adapted models, full fine-tuning is worth the cost. If you need to support many tasks from a shared backbone, or your labelled data is moderate rather than abundant, LoRA is usually the sensible middle path.

Whatever you choose, evaluate on a split that genuinely reflects deployment conditions, not a random shuffle that leaks information from near-duplicate examples across train and test. The method matters less than knowing, with an honest held-out evaluation, whether it actually worked.