Attention Is All You Need

You can't build modern sequence models without starting here. Swapping recurrence and convolution for attention was one of those moves that looks obvious in hindsight and quietly reorganized everything after it. And we'll come right out and say it: where a strong sequence prior matters, attention is close to the ceiling straight out of random init. That's just true — we're not here to explain it away.

But the title's a dare, and we take it as one. Because for the things we actually build, attention is all you need doesn't quite hold — in two directions.

At scale, it's not enough. Dense attention is quadratic. Let the context run long and the attention term stops being a line item and becomes most of the bill — more than the whole rest of the model, doubling every time the sequence does. “Attention” as a single idea doesn't fix that; compute, memory, and communication are three different walls that need three different tools. What actually holds up under load isn't attention-the-brand — it's the part underneath: one numerically honest, associative softmax over pieces of the key space. Get that single operator right and the whole zoo of variants stops being rival architectures and turns into settings on the same engine. The paper named the mechanism. We think it undersold the primitive.

In the systems we ship, it's not special either. When you're building for the real world — tight latency, hard guarantees, things that have to work every single time — a Transformer is one tool on the bench, no more fundamental than the ones sitting next to it. Plenty of jobs reward a different reach entirely. The frontier we care about isn't a faster attention kernel; it's systems that predict, act, and stay provably inside their limits in real time. Attention still earns a spot in that toolbox — we keep it, we test it without flattery, and we prove it behaves instead of hoping it does — but it earns that spot as one option among many, not as the thing the whole machine hangs on.

So this paper sits on our shelf on purpose, and not as ours. It's as much what we build against as what we build on: the wall our efficiency work tears down, and a default our other work is a deliberate answer to. Read straight, “attention is all you need” is a claim we don't buy. Read one level down — as the idea that a single operator for deciding what to look at shows up nearly everywhere something has to choose where to point — it's more right than its authors could've known. Both of those are why it's here.