Prompt Engineering: What Survives a Model Change

Why this question keeps coming up

Every few months a new model version lands, and a familiar ritual follows: prompts that used to work reliably start producing odd results, people panic, and someone writes a thread claiming prompt engineering is dead. I think both the panic and the obituary are wrong. What actually happens is that a mix of durable skills and model-specific hacks gets exposed for what it always was. The durable part survives because it reflects something true about how these systems process instructions. The hacky part fails because it was never really about communication, it was about exploiting a quirk in one model's training distribution.

This matters practically because teams build products on top of prompts, and if you cannot tell which parts of your prompting strategy are load-bearing engineering and which are superstition, every model upgrade becomes a fire drill. I have seen prompts that were essentially folklore, phrases repeated because they once worked, with nobody able to explain why. That is a fragile position to build a product from. The goal here is to separate the two categories with enough precision that you can audit your own prompts and predict, roughly, what will break.

What tends to survive

The most durable prompting skill is simply being unambiguous about the task. If you ask a model to summarise a document, but do not specify length, audience, or what counts as important, you are relying on the model to guess your intent, and different models guess differently. Specifying constraints explicitly, such as 'summarise in three bullet points for a non-technical reader, each under twenty words', removes a layer of guesswork that no amount of model improvement fully eliminates. This survives model changes because it is not a trick, it is just clearer input, and clearer input helps any sufficiently capable system.

Providing structure also survives well. Asking for output in a defined format, giving a worked example of the input-output mapping you want, or breaking a complex task into explicit sub-steps all tend to keep working because they reduce the model's degrees of freedom. Consider a task like extracting structured data from messy text, say pulling a date, an amount, and a category from an invoice description. A prompt that gives two or three example extractions before the real one, showing the exact field names and format, will usually keep performing well across model generations, because the pattern it demonstrates is explicit rather than implied.

Decomposition also holds up. If a task is genuinely multi-step, such as 'first identify the relevant clauses, then check each against this rule, then output a verdict', splitting it into stages tends to remain useful even as the underlying model gets better at doing it all in one pass. The reason is structural: decomposition reduces the chance that an error early in reasoning silently propagates unnoticed, and that benefit does not depend on a model's particular training quirks. It is closer to good systems design than to clever wordplay, which is exactly why it ages well.

What tends to break

The fragile end of prompting is anything that depends on a specific model's calibration rather than on clear communication. A classic example is the magic phrase: some earlier models responded noticeably differently to being told to 'think step by step' or to being offered a fictional reward, like being told they would be tipped for a good answer. These phrases worked because of specific patterns in how a model was trained and fine-tuned, not because they conveyed any real information about the task. As newer models are trained with better default reasoning behaviour or with different reinforcement signals, these incantations often stop mattering, or in some cases become actively unhelpful because the model now treats them as noise or, worse, as an instruction to perform theatrical reasoning rather than efficient reasoning.

Similarly fragile is prompting that compensates for a known weakness in one specific model, such as a particular version being bad at arithmetic unless numbers are spelled out, or being prone to a certain kind of repetition unless told explicitly not to repeat itself. These workarounds are valuable in the short term but they are patches, not principles. When the next model fixes the underlying weakness, the patch becomes either inert or an unnecessary constraint that slightly degrades output, because you are now instructing the model to avoid a failure mode it no longer has.

Token-level superstition is another casualty. Things like exact capitalisation of keywords, specific punctuation choices believed to 'unlock' better behaviour, or very particular phrasings copied from someone else's successful prompt without understanding why they worked, rarely transfer. I would put a lot of the prompt marketplaces and prompt-sharing culture in this bucket. A prompt that performs well on one model because of some quirk in its tokenizer or fine-tuning data is not a transferable skill, it is a lucky find, and treating it as engineering knowledge is where a lot of wasted effort comes from.

A practical way to tell the difference

The test I find useful is to ask, for any prompting habit: would this still make sense if I were explaining the task to a new, moderately experienced human colleague rather than a model? Instructions like being explicit about format, giving worked examples, stating constraints, and breaking complex tasks into stages all pass this test, because they are just good task specification. Instructions like 'say please' or 'claim you are an expert in the subject' usually fail it, because a competent human colleague does not need that, and its effect on a model is really a signal about the model's training, not the task itself.

In practice this means building a small, versioned evaluation set alongside any prompt you rely on, with representative inputs and a clear notion of what a correct or acceptable output looks like. When a new model version appears, rerun that evaluation before touching the prompt. If performance drops, diagnose which part of the prompt was doing the work: was it structural guidance that still should apply, or was it a specific workaround for a model-specific weakness that may no longer exist? Only then adjust, rather than reflexively rewriting the whole prompt from scratch or assuming nothing needs to change.

The honest takeaway is that prompt engineering as a durable skill looks a lot like good technical writing and good task decomposition, applied to a new kind of reader. The parts of it that feel like magic words and secret incantations were never really engineering, they were reverse-engineering one model's idiosyncrasies, and they deserve to be treated as temporary scaffolding rather than core competence. Build your prompting habits on clarity and structure, keep an evaluation set to catch regressions honestly, and you will spend a lot less time rewriting everything each time the underlying model changes.