The Transformer Architecture, Explained Without the Maths

Why this architecture ate the field

Before transformers, the dominant approach to sequence data, sentences, time series, protein chains, was recurrence. A recurrent network read a sequence one token at a time, carrying a running summary forward. This is intuitive but slow: you cannot process token ten until you have processed token nine, and by the time you reach token fifty, the summary has often forgotten details from token two. Long-range dependencies leaked away.

The transformer, introduced in a 2017 paper on machine translation, threw out recurrence entirely. Instead of reading a sequence step by step, it looks at the whole sequence at once and asks a simple question for every token: which other tokens in this sequence matter for understanding me? That question, asked and answered in parallel for every position, is called attention, and it is the entire trick. Everything else in the architecture is scaffolding to make that trick work at scale.

I think the reason this matters for anyone building or evaluating models today is not nostalgia for how we got here. It is that almost every large language model, vision transformer, and speech model in production right now is some variant of this same core idea. If you understand attention properly, you understand the shared skeleton underneath a huge fraction of modern machine learning, and you stop treating these models as black boxes that just happen to work.

Attention, without the equations

Here is the intuition I find most useful. Imagine you are reading the sentence: 'The trophy did not fit in the suitcase because it was too small.' To understand what 'it' refers to, your brain does not process the sentence left to right and forget everything before the word 'it'. Instead, it glances back across the whole sentence and weighs each earlier word by how relevant it is to resolving 'it'. 'Suitcase' gets a lot of weight. 'Trophy' gets some. 'Because' gets almost none. That weighted glancing-back is attention.

Mechanically, each word is turned into three different representations, conventionally called a query, a key, and a value. Think of the query as the question the word is asking ('what resolves me?'), the key as the advertisement each other word puts out ('here is what I offer'), and the value as the actual content that gets passed along if there is a match. The word 'it' compares its query against every other word's key, gets a compatibility score for each, turns those scores into weights that sum to one, and then builds its new representation as a weighted blend of everyone's values, weighted by those scores. High compatibility with 'suitcase' means 'suitcase' dominates the blend.

Crucially, this happens for every word simultaneously, not sequentially. Every token asks its question against every other token in one parallel operation. That is the property that recurrent networks lacked and that made transformers so much faster to train on modern hardware: the whole computation is a handful of matrix multiplications that a GPU chews through in parallel, rather than a chain of steps that must happen in order.

The architecture does not do this once. It stacks many layers of attention on top of each other, and within each layer it runs several attention computations side by side, called heads, each free to specialise in a different kind of relationship. One head might learn to track grammatical subject-verb agreement, another might track coreference like our trophy and suitcase example, another might pick up on topical similarity. Nobody tells the heads what to specialise in; it emerges from training on the objective, usually predicting the next word or filling in masked words.

The parts people forget: position and depth

There is a problem with the pure attention idea as described so far: it has no sense of order. If attention just computes weighted averages based on content similarity, the sentence 'the dog bit the man' and 'the man bit the dog' would look identical to it, because it is the same set of words being compared against each other regardless of sequence. Word order carries meaning, obviously, so the architecture needs to inject it artificially.

This is done through positional encoding: a fixed pattern of numbers added to each word's representation that varies smoothly with position, so that position one, two, three and so on each get a distinguishable signature. The original design used sine and cosine waves of different frequencies for this, chosen mainly because they let the model infer relative distances between positions cleanly, though many later variants use learned position embeddings or other schemes instead. The point is not the specific mathematical choice; it is that without some explicit positional signal, attention is a bag-of-words operation, and with it, order becomes recoverable.

The other underrated piece is depth combined with a plain feed-forward step after each attention block. Attention lets tokens gather information from each other, but the network also needs capacity to transform that gathered information into something more abstract, which is what the feed-forward layers do, applied independently to each token's representation. Stack this pattern, attention then feed-forward, a dozen or more times, and each layer builds a progressively more abstract representation: early layers might capture local syntax, later layers might capture something closer to meaning or discourse structure. This is analogous to how early layers in an image classifier detect edges while later layers detect whole objects.

Residual connections, where each layer adds its output to its input rather than replacing it, and normalisation steps that keep the numbers well-behaved, are the unglamorous plumbing that makes it possible to stack dozens of these layers without training collapsing. They rarely get discussed in popular explanations, but in practice, removing them makes deep transformers untrainable, which tells you they are doing real work rather than being decorative.

What this means in practice

If you take one idea away from this, take the query, key, value framing. It explains why transformers handle long-range dependencies so much better than recurrent networks: a word at position five hundred can attend directly to a word at position one, in a single step, with no information having to survive a long relay of intermediate summaries. It also explains the main practical cost of the architecture: comparing every token against every other token means the computation grows quickly as sequences get longer, which is precisely why so much recent engineering effort has gone into making attention cheaper for long documents.

For anyone evaluating a transformer-based system rather than just building one, this intuition is genuinely useful. When a model fails on a long document, ask whether the failure looks like an attention problem, information that should have been retrievable getting drowned out or truncated, versus a knowledge problem, the model simply never having seen the relevant fact. Those are different failure modes with different fixes, and conflating them leads to wasted effort. Understanding the mechanism, even at this non-mathematical level, turns debugging from guesswork into something closer to reasoning about a system you actually understand.