Positional Encodings: Why Transformers Need Them at All

The problem hiding in plain sight

When people meet self-attention for the first time, they usually focus on the query, key and value matrices and skim past a much stranger fact: attention, on its own, has no idea what order the tokens came in. Every token looks at every other token through dot products of learned projections. If you shuffled the input sequence and shuffled the output the same way, the attention weights would come out identical. The mechanism is permutation-equivariant: it treats the sentence as a set, not a sequence.

This is easy to prove to yourself with a small example. Take the sentence 'the cat chased the dog' and compare it with 'the dog chased the cat'. Both contain exactly the same multiset of tokens. A pure self-attention layer with no other information computes, for each token, a weighted sum of value vectors based on similarity between its query and every key. Swap the positions of 'cat' and 'dog' in the input and you get the same set of query-key pairs, just relabelled. The model cannot distinguish subject from object, because nothing in the maths refers to position at all, only to content.

Contrast this with a recurrent network, which processes tokens one at a time and carries a hidden state forward, so order is baked into the computation by construction. Convolutional networks also encode locality directly, because a filter only looks at neighbouring positions. Self-attention deliberately throws away that structural bias in exchange for the ability to connect any two tokens directly, regardless of distance. That is precisely what makes it powerful for long-range dependencies, but it also means order has to be reintroduced some other way, or the model is doing grammar and meaning with a bag of words.

What goes wrong without position information

It helps to be concrete about the failure. Consider a toy task: given a short sequence of numbers, output the one that appears third. Without positional information, a transformer can learn statistical shortcuts from training data (perhaps the third number is often larger, or has some spurious pattern), but it cannot reliably learn the actual rule 'pick whatever sits in slot three', because slot three is a positional concept and the model has no representation of slots at all. Give it the same multiset in a different order and, absent position, its prediction should not change, even though the correct answer does.

Natural language shows the same issue in a softer form. 'The bank approved the loan after the manager reviewed it' and a scrambled version of the same tokens are wildly different in meaning, and even in grammaticality. Word order carries tense, negation scope, subject-object roles, and much of the compositional structure that makes language mean anything. A model that cannot see order is stuck relying on co-occurrence statistics between words, which is closer to a bag-of-words classifier than to a language model. Early results after self-attention was introduced without positional signals showed exactly this: reasonable performance on tasks tolerant of scrambled order, and poor performance wherever order-sensitive structure mattered, such as syntax-heavy tasks or anything requiring counting or relative position.

The practical consequence for anyone building or evaluating sequence models is that positional information is not a minor implementation detail, it is load-bearing. If you ever ablate a transformer by removing its positional encodings and performance barely drops on your evaluation set, that is a strong signal your task does not actually require sequence order, which is often a sign the benchmark is weaker than it looks, or that shortcuts are available. I would treat that ablation as a basic sanity check before trusting any claimed result on an order-sensitive task.

How position gets put back in

The original fix was refreshingly simple: add a vector to each token embedding that encodes its position, before the first attention layer sees it. The classic version uses sine and cosine waves of different frequencies, so position 5 and position 6 get distinct, smoothly varying vectors, and the relative offset between any two positions is recoverable from the geometry of these vectors. The elegant part is that this lets the model infer relative distance (token 12 is 3 steps after token 9) using linear combinations of the encodings, without ever being told distance explicitly.

Learned positional embeddings are the alternative: treat each position index as a lookup into a trainable table, exactly like a word embedding but keyed by slot number instead of vocabulary. This is simpler to implement and often works about as well in practice, but it struggles to generalise to sequence lengths longer than anything seen during training, because position 5000 has no learned vector if the model only ever saw sequences up to length 2000. Sinusoidal encodings, by contrast, are defined by a formula and can technically be evaluated at any position, though extrapolation quality still degrades in practice.

More recent approaches, such as rotary position embeddings, take a different route: rather than adding a positional vector to the input, they rotate the query and key vectors by an angle that depends on position, directly inside the attention computation. This bakes relative position into the dot product itself, so the attention score naturally depends on how far apart two tokens are, not just their content. It has become popular in large language models partly because it tends to generalise better to longer sequences and integrates cleanly with efficient attention implementations. Whatever the mechanism, the goal is the same: give the model a reliable signal for where each token sits, since attention itself will never supply that on its own.

The practical takeaway

If you are building or reviewing a transformer-based system, it is worth asking two questions explicitly. First, does my task actually depend on order, and if so, have I checked that removing or corrupting positional encodings degrades performance the way it should? A model that is indifferent to shuffled input on an order-sensitive task is a red flag, not a feature. Second, does my choice of positional encoding match my deployment constraints, particularly around sequence length: learned embeddings are fine if you never exceed training length, but sinusoidal or rotary schemes are the safer default if you expect longer contexts at inference time than you trained on.

The deeper lesson generalises beyond transformers: whenever you adopt an architecture that is more permissive than the data actually is, structure has to be reintroduced somewhere, either in the architecture, the input representation, or the loss. Self-attention gained flexibility by discarding order, and positional encodings are the price of buying that flexibility back. Understanding that trade explicitly, rather than treating positional encodings as boilerplate, makes it much easier to diagnose why a model fails on order-sensitive tasks and to choose the right fix rather than a fashionable one.