Fine-Tuning vs Few-Shot Prompting on Small Datasets

Why this choice is harder than it looks

When someone hands me a labelled dataset with three hundred examples and asks whether to fine-tune a model or just write a good prompt with a handful of demonstrations, my honest answer is that the dataset size alone does not settle it. What settles it is how the task's decision boundary behaves, how stable the label definitions are, and how you plan to evaluate the result. Too often the comparison gets reduced to a vague intuition that fine-tuning is for 'more data' and few-shot prompting is for 'less data', which is not wrong exactly, but it hides the mechanics that actually matter.

Fine-tuning adjusts the model's internal weights using your labelled examples as gradient signal. Few-shot prompting keeps the weights frozen and instead relies on a handful of examples placed in the context window to steer the model's behaviour at inference time. These are fundamentally different mechanisms for injecting task-specific knowledge, and they fail in different ways when the labelled set is small. Understanding those failure modes is more useful than any rule of thumb about sample counts.

I want to work through a concrete case: classifying customer support tickets into three categories, billing, technical, and account access, using two hundred labelled tickets. This is a realistic size for a small internal project, and it is exactly the regime where the fine-tuning versus few-shot question becomes genuinely contested rather than obvious.

What fine-tuning actually buys you, and what it costs

With two hundred examples split, say, one hundred and forty for training, thirty for validation, and thirty for a held-out test set, fine-tuning a smaller pretrained model can work well if the categories are cleanly separable by vocabulary and structure. The model learns weight adjustments specific to your label distribution, and at inference time it needs no examples at all, just the raw ticket text. This means lower latency and lower per-call cost in production, since you are not paying to process demonstration examples in every prompt.

The cost is that one hundred and forty training examples is a thin signal. If your three categories are imbalanced, say ninety billing tickets, thirty technical, and twenty account access, the model may become very good at billing and mediocre at account access simply because it saw fewer examples and fewer edge cases. Fine-tuning also risks overfitting to incidental patterns: if most of your technical tickets happen to mention a specific product name that rarely appears in billing tickets, the model may learn that shortcut rather than the underlying distinction you care about. This only becomes visible if your test set contains tickets where that shortcut fails, which is exactly why a careless train and test split, one drawn from the same time period or the same small pool of customers, can flatter a fine-tuned model with a leakage-inflated accuracy score.

There is also a practical overhead people underrate: fine-tuning requires infrastructure, checkpointing, and a decision about how many epochs to run before you overfit the small training set into memorising rather than generalising. With one hundred and forty examples, three or four epochs might already be enough to start memorising surface patterns, and you will only catch this if your validation set is genuinely held out and reasonably representative.

What few-shot prompting actually buys you, and what it costs

Few-shot prompting sidesteps the overfitting risk in a specific sense: you are not adjusting millions or billions of parameters based on one hundred and forty examples, so there is no weight-level memorisation of your particular training set. Instead, you select perhaps six to twelve representative examples, two or three per category, and place them directly in the prompt alongside the new ticket to be classified. The underlying pretrained model brings general language understanding, and the demonstrations narrow that general capability toward your specific category definitions.

This approach tends to be more robust when your labelled set is genuinely tiny, under a hundred examples, because there simply is not enough signal to responsibly update weights without a serious risk of collapsing onto noise. It is also faster to iterate on: changing your few-shot examples or your instructions takes seconds, whereas retraining a fine-tuned model takes a training run and a fresh evaluation cycle.

The cost is subtler and easy to miss. Few-shot performance is sensitive to which examples you choose and the order you present them in, so a comparison between 'fine-tuned model' and 'few-shot prompt' is not comparing two fixed things, it is comparing a fixed thing against a family of prompts whose performance can shift by several percentage points depending on demonstration selection. If you report the best few-shot configuration you found after trying a dozen variations, you have quietly introduced a form of leakage: you tuned your method on the same test set you are using to report results. The honest version of a few-shot evaluation selects demonstrations using only the training pool, then reports test performance once, exactly as you would with a fine-tuned model's held-out test accuracy.

The practical takeaway

On my two hundred ticket example, I would expect few-shot prompting to be the more forgiving starting point precisely because the training pool is small and imbalanced; it lets a capable pretrained model do most of the work while your examples merely calibrate category boundaries. I would expect fine-tuning to overtake it only if you can grow the labelled set meaningfully, into the low thousands, where weight updates have enough signal to learn distinctions that demonstrations alone cannot convey, and where per-call latency or cost genuinely matters at scale.

The comparison only means anything if both approaches are evaluated on the same untouched test split, with demonstration selection and hyperparameter choices made strictly on training or validation data. Skipping that discipline is how teams end up with a headline number that quietly cannot be reproduced once the underlying data shifts even slightly. Choose based on data volume and deployment constraints, but trust the comparison only if the evaluation was leakage-aware from the start.