Research
LongNet: Scaling Transformers to 1,000,000,000 Tokens
This is third-party work — not authored by or affiliated with DarcStar Technologies.
By Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, Furu Wei
Abstract
Scaling sequence length has become a critical demand in the era of large language models. However, existing methods struggle with either computational complexity or model expressivity, rendering the maximum sequence length restricted. To address this issue, we introduce LongNet, a Transformer variant that can scale sequence length to more than 1 billion tokens, without sacrificing the performance on shorter sequences. Specifically, we propose dilated attention, which expands the attentive field exponentially as the distance grows. LongNet has significant advantages: 1) it has a linear computation complexity and a logarithm dependency between any two tokens in a sequence; 2) it can be served as a distributed trainer for extremely long sequences; 3) its dilated attention is a drop-in replacement for standard attention, which can be seamlessly integrated with the existing Transformer-based optimization. Experiments results demonstrate that LongNet yields strong performance on both long-sequence modeling and general language tasks. Our work opens up new possibilities for modeling very long sequences, e.g., treating a whole corpus or even the entire Internet as a sequence.
DarcStar commentary
Our perspective on this work and how it informs our research — not part of the original paper.
We didn't just read this one — we built it. LongNet's dilated attention is the mechanism behind a good chunk of our own long-context work, so we're speaking from the guts of the thing, not the abstract.
Give the paper its due: it takes the quadratic wall head-on and gets a Transformer to linear compute with only a logarithmic dependency between any two tokens — reach across a billion-token sequence in a bounded number of hops, as a drop-in for standard attention. That log dependency is the load-bearing idea, and it's a real one. A pattern where everything can reach everything in a few hops keeps the full expressive range of dense attention on the table; the long-range ceiling isn't structurally forced. That connectivity skeleton is what we took from this paper and kept.
Here's where we push. Dilation decides what to look at with fixed, geometric strides — before it has seen a single token. The reach is guaranteed; the selection is blind. Widen the stride and sooner or later you step right over the one far token that mattered, because nothing in the pattern knows it's there. So the skeleton is the keeper; the fixed spacing isn't. The move we favor is to keep a guaranteed-reach skeleton as the safety net and let the content pick the rest — selection that looks at the data instead of a stride table.
And building it taught us the part the paper makes look easy. The hard bit of dilated attention isn't the striding — it's combining the softmax over the scattered pieces without corrupting the shared denominator. Get that wrong, as a surprising number of implementations quietly do, and the long-range math is broken while the loss curve still looks fine.
So this sits on our shelf as external work we've actually worn the tread off of — not ours, but close to the bone. The ambition in its last line — treat a whole corpus, even the entire Internet, as one sequence — is exactly the right size of goal. We just think you get there by keeping the guaranteed reach, making the selection smart enough to earn its skips, and proving the softmax underneath is honest before you trust a word of the result.