Prompt Engineering versus Fine-Tuning: Choosing the Lever

Why this choice matters more than it looks

Every time I need a language model to do something specific, be it extracting fields from invoices, classifying support tickets, or writing in a particular house style, I face the same fork in the road. I can describe the task better through the prompt, or I can change the model's weights through fine-tuning. These are not interchangeable options with the same effect dialled up or down. They are different levers attached to different parts of the machine, and pulling the wrong one wastes time, money, and sometimes introduces risks that are hard to spot until the system is in front of real users.

The temptation is to treat this as a maturity ladder: start with prompting, and once you are serious, graduate to fine-tuning. I think that framing is wrong more often than it is right. Fine-tuning is not a more advanced version of prompting; it solves a different kind of problem. Prompting adjusts what the model is asked to do at inference time, using its existing knowledge and behaviour. Fine-tuning adjusts the model itself, baking a pattern into its parameters so it no longer needs to be told every time.

Getting this choice wrong has concrete costs. I have seen effort poured into fine-tuning when a better-structured prompt with a couple of well-chosen examples would have solved the problem in an afternoon. I have also seen prompts grow into unwieldy walls of instructions and edge-case patches, trying to compensate for a behaviour that really needed to be trained in. Neither situation is a technical failure so much as a diagnostic one: the practitioner picked a lever before understanding what kind of problem they had.

What each lever actually changes

Prompt engineering works within the model's existing capabilities. You are choosing how to phrase the task, what context to supply, which examples to show, and how to structure the output format, so that the model's pretrained knowledge gets routed toward the behaviour you want. It costs no training compute, requires no labelled dataset beyond a handful of examples, and can be changed in seconds. Its ceiling, however, is set by what the model already knows and can express through instructions alone.

Fine-tuning changes the model's parameters using a training set of input-output pairs. It is the right tool when the task requires a consistent, narrow behaviour that is expensive to specify in words every time, or when you need the model to internalise patterns that are not easily explained, such as a very particular tone, a domain-specific taxonomy, or a mapping between inputs and outputs that has many exceptions. It costs data curation time, training compute, and ongoing maintenance, since every change to the task may mean retraining.

Consider a concrete case: a company wants a model to tag customer emails into twelve internal categories that map loosely, but not exactly, onto common-sense categories. With prompting, you write out the twelve definitions, give two or three examples per category, and ask the model to pick one. On a sample of two hundred held-out emails, this might get eighty-four per cent accuracy, with most errors clustering around three categories whose definitions overlap in ways hard to explain in a sentence. You try refining the definitions, adding more examples, restructuring the prompt: accuracy edges up to eighty-seven per cent, but with diminishing returns and a prompt that is now nine hundred words long.

At that point, fine-tuning becomes attractive, not because it is inherently superior, but because the residual errors are exactly the kind of nuanced, hard-to-verbalise pattern that training on a few thousand labelled examples handles well. After fine-tuning on three thousand labelled emails with a leakage-aware train and test split, accuracy might reach ninety-four per cent, and crucially, the prompt at inference time shrinks back down to a short instruction, because the category boundaries are now learned rather than explained. The lesson is not that fine-tuning wins outright; it is that the two levers were solving different parts of the same problem, and the second was only worth pulling once the first had been pushed close to its ceiling and the remaining errors had a specific character.

How I decide, and how I evaluate the decision

My first question is always whether the task can be fully specified in language. If the rule can be written down and the model demonstrably follows it when given clear instructions and a few examples, prompting is very likely sufficient, and fine-tuning would just be an expensive way of encoding something you could have written in the system prompt. My second question is about volume and stability: if the task will run millions of times a day, a shorter fine-tuned inference path can be cheaper and faster than a long, example-laden prompt, even if the accuracy gain is modest.

My third question is about the shape of the errors. Errors from an under-specified prompt tend to look inconsistent across similar inputs and improve noticeably as you add examples or restructure instructions. Errors that persist despite well-crafted prompts, and cluster around genuinely ambiguous or nuanced cases, are a signal that the model needs to learn a pattern rather than be told one. I try to resist fine-tuning as a reflex fix for what is actually a prompt design problem, because I have wasted time doing exactly that.

Whichever lever I use, the evaluation has to be honest. A held-out test set must never overlap with anything used to write prompt examples or to fine-tune, and I always check for near-duplicate leakage, not just exact matches, since paraphrased examples slipping between train and test sets will quietly inflate scores. I also insist on a strong baseline: a simple, well-written prompt as the floor that any fine-tuned model must clearly beat, and a plain rule-based or nearest-neighbour classifier where relevant, because it is embarrassingly common to fine-tune a model that performs no better than a baseline nobody bothered to build.

The practical takeaway is this: treat prompting as the default, low-cost probe into what the model can already do, and treat fine-tuning as the deliberate, higher-cost step you take once you know precisely which residual behaviour needs to be learned rather than described. Choosing the lever is really a diagnostic act, and the diagnosis should come from a careful look at error patterns under a leakage-free evaluation, not from habit, budget, or which technique happens to be fashionable that month.