Fine-Tuning versus Prompt Design: A Worked Comparison

Why this comparison matters

Every time I start a small text classification task, the same question shows up before any modelling begins: do I fine-tune a model on labelled examples, or do I write a careful prompt and lean on a general-purpose model's existing knowledge? The honest answer is that it depends on the task, the data budget, and how much you trust your evaluation, but that answer is unsatisfying without numbers attached to it. So I want to walk through a worked comparison on a small, concrete task rather than argue in the abstract.

The task I will use is short customer feedback snippets, each one to three sentences long, labelled as one of three categories: complaint, praise, or neutral query. This is deliberately unglamorous. It is exactly the kind of task that shows up in real products, and it is small enough that the difference between fine-tuning and prompting is not settled by sheer data volume alone. Suppose we have six hundred labelled examples total: four hundred for training, one hundred for validation, and one hundred held out as a genuinely untouched test set.

That test set matters more than people admit. If you peek at it while tuning prompts, or while choosing which fine-tuning checkpoint to keep, you have quietly turned it into a second validation set, and your final number is optimistic. I treat the hundred held-out examples as sealed: no prompt wording decisions, no hyperparameter choices, and no early stopping criteria are ever chosen by looking at test performance. Everything is decided on the validation split first.

Setting up a fair fight

For fine-tuning, I take a small pretrained encoder and add a classification head, training on the four hundred labelled examples for a handful of epochs, watching validation accuracy to decide when to stop. This is a cheap, ordinary setup: nothing exotic, just a sensible learning rate and early stopping based on validation loss rather than a fixed epoch count.

For prompt design, I do not just write one prompt and call it done. I treat prompt development the same way I would treat model selection: several candidate prompts, each evaluated on the validation set, with the best one carried forward to the sealed test set. My candidates range from a bare instruction ('classify this feedback as complaint, praise, or neutral') to a version with three worked examples embedded directly in the prompt, one per class, chosen to be typical rather than cherry-picked for difficulty.

This last point deserves emphasis: it is tempting to pick your few-shot examples because they happen to make the model perform well on cases you have already seen fail. That is a subtle form of leakage into your prompt design process, functionally similar to tuning hyperparameters on the test set. I chose the three examples before looking at any validation results, based only on them being clear, representative instances of each class.

With that discipline in place, suppose the fine-tuned model reaches eighty-nine per cent accuracy on the held-out test set, and the best prompt, the one with three worked examples, reaches eighty-four per cent. The bare-instruction prompt, for comparison, lands at seventy-six per cent. These are the kinds of numbers I would actually expect on a task like this: fine-tuning wins, few-shot prompting helps a lot over zero-shot, but there is still a real gap.

Where the gap actually comes from

An eighty-nine versus eighty-four split sounds decisive, but it is worth asking what that five-point gap is made of before drawing conclusions. Looking at the confusion matrix for the prompted model, most of the errors are neutral queries being classified as mild complaints. That makes sense: a neutral query like 'how do I change my delivery address' shares surface vocabulary with complaints about delivery, and without seeing many labelled neutral examples, the model's prior leans towards treating anything logistics-related as negative.

The fine-tuned model, having seen roughly one hundred and thirty neutral examples during training, has learned this specific boundary directly from the label distribution of this dataset. The prompted model has not seen this dataset's particular quirks at all; it is relying on general language understanding plus three examples. That is precisely the trade-off: fine-tuning buys you dataset-specific calibration, prompting buys you a model that never needed labelled training data in the first place.

This matters enormously for practical decisions. If four hundred labelled examples took two analysts a week to produce and label consistently, and the task is going to run for years, that five-point accuracy gain is probably worth the fixed cost. If the task is a one-off classification you need results for by tomorrow afternoon, or if the label categories are likely to shift in a month, the eighty-four per cent prompt-based result deployed today is worth more than a theoretically superior model deployed next week.

There is also a maintenance dimension that raw accuracy numbers hide. The fine-tuned model needs retraining if the label definitions change, if new categories appear, or if the underlying data distribution drifts. The prompt-based approach can often absorb small changes just by editing the instruction text and swapping one worked example, no retraining pipeline required. That flexibility has real value that a single accuracy figure does not capture.

A practical takeaway

The comparison I would encourage is not fine-tuning against prompting in the abstract, but fine-tuning against the best prompt you can honestly justify, evaluated with the same discipline you would apply to any other model: a validation set for decisions, a sealed test set for the final number, and prompt examples chosen before you see how well they perform. Skipping that discipline on either side makes the comparison meaningless, because you cannot tell whether a win came from genuine capability or from quietly optimising against your own scorecard.

On this small task, fine-tuning bought roughly five points of accuracy in exchange for four hundred labelled examples and a training pipeline to maintain. Whether that trade is worth it depends entirely on how long the task will live, how often its definition will change, and how expensive labelling actually was. There is no universal answer, but there is a universal method: measure both fairly, on the same sealed data, before deciding which cost is worth paying.