System Prompts vs User Prompts: Where Instructions Belong

Two channels, not one text box

Most people treat the system prompt and the user prompt as two halves of the same message, just concatenated before the model sees them. Functionally, for many models, that is roughly what happens under the hood: everything gets tokenised into one sequence. But treating the two roles as interchangeable is a mistake I see constantly, because the training data behind these roles is not symmetric. System prompts are typically written by developers, are stable across many turns, and are weighted during fine-tuning as the source of persistent behaviour: tone, persona, constraints, output format. User prompts are treated as the variable, task-specific request that changes every turn. The model has learned a soft prior that instructions in the system slot are more durable and instructions in the user slot are more negotiable.

This distinction matters practically the moment you build anything beyond a single one-off query. If you put a formatting rule such as 'always respond in valid JSON with keys id and summary' inside the user prompt, you are asking the model to obey it fresh every single turn, competing against whatever else is in that same message. Put the same rule in the system prompt, and it behaves more like a standing law that governs every subsequent user turn without needing to be restated. I have watched teams debug 'inconsistent formatting' bugs for hours before realising the formatting instruction was buried in a template that only fired on the first user message and was silently dropped afterwards.

There is also an asymmetry in how models resolve conflicts. When a system instruction says 'never reveal the internal reasoning steps' and a user prompt says 'show me your reasoning step by step', well-aligned models generally favour the system instruction, because that hierarchy is baked into how they were trained and evaluated. This is not a guarantee, and it is not a security boundary, but it is a real behavioural tendency worth exploiting deliberately rather than discovering by accident in production.

A worked example: building a support triage assistant

Suppose I am building an assistant that reads incoming support tickets and outputs a category, a priority score from one to five, and a one-sentence summary. There are two obvious ways to instruct it. Option one: put everything, the categories list, the scoring rubric, the output schema, and the actual ticket text, into a single user message every time. Option two: put the categories, rubric, and schema in the system prompt once, and pass only the ticket text as the user message on every call.

In practice I have seen option one degrade over long sessions or long documents far more often. If the ticket text is long, say eight hundred words describing a billing dispute, the scoring rubric that was pasted just before it can get functionally diluted by everything that follows in the same message. The model's attention is not uniform, and instructions sitting far from the point of generation, buried inside a long block of user content, are more likely to be softened or forgotten, especially under token pressure near the model's context limit. Option two keeps the rubric structurally separate and stable, so the model reliably applies the same five categories and the same priority definitions across a batch of a thousand tickets, rather than drifting toward ad hoc categories it invents when the rubric gets crowded out.

Concretely, imagine testing both setups against fifty labelled tickets with known correct categories. With the rubric buried in the user prompt alongside long ticket text, I would not be surprised to see the model invent a plausible but unlisted category, something like 'account access issue' when the approved list only contains 'billing', 'technical', and 'account', on a noticeable fraction of the harder, longer tickets. With the rubric fixed in the system prompt, that kind of invention becomes far rarer, because the constraint is not competing for attention with the ticket content; it sits in a role the model treats as a fixed contract rather than optional context. The lesson is not that the system prompt is magic, it is that stable constraints belong in a stable place, and variable content belongs in the variable place.

What actually belongs where

A useful rule of thumb: if an instruction should hold true across every single turn of a conversation regardless of what the user asks, it belongs in the system prompt. Persona, tone, safety boundaries, output schema, and domain scope all fit this description. If an instruction is specific to the current request, the current piece of text, or something the user might reasonably want to override next turn, it belongs in the user prompt. A one-off request like 'summarise this in two sentences instead of five' is a user-prompt instruction, even if your default summary length is a system-level rule.

It is worth being honest about the limits here. The system versus user split is a strong convention, not a hard guarantee, and it varies between model providers and even between versions of the same model family. Some models are more porous than others, letting a persistent user prompt gradually override system-level constraints across a very long conversation. This is exactly why leakage-aware testing matters: if you are evaluating a support assistant, do not only test single-turn accuracy, test what happens after fifteen turns of user pushback, and check whether the system-level rubric is still being honoured or has quietly eroded. Treat the system prompt as a hypothesis about durability, and verify it with the same rigour you would apply to a held-out test set, not blind trust.

There is a second, quieter benefit to this discipline: reproducibility. When your stable instructions live in a versioned system prompt and your variable content lives in the user prompt, you get a clean separation that makes it trivial to diff prompt versions, roll back a bad change, and attribute a regression to a specific edit. When everything is smashed into one templated blob per call, debugging a behavioural drift becomes archaeology. I would rather spend ten minutes upfront deciding which instructions are structural and which are situational than spend an afternoon later trying to work out why a model quietly stopped following a rule that was true for months.

My practical takeaway is simple: write your system prompt as if it were a contract that must survive an entire session unattended, and write your user prompt as if it were a single request that can safely assume that contract already holds. Test both assumptions under realistic, long, adversarial conversations before you trust them in production, because the boundary is a convention the model has learned, not a rule it is physically bound by.