FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

Abstract

Attention, as a core layer of the ubiquitous Transformer architecture, is the bottleneck for large language models and long-context applications. FlashAttention elaborated an approach to speed up attention on GPUs through minimizing memory reads/writes. However, it has yet to take advantage of new capabilities present in recent hardware, with FlashAttention-2 achieving only 35% utilization on the H100 GPU. We develop three main techniques to speed up attention on Hopper GPUs: exploiting asynchrony of the Tensor Cores and TMA to (1) overlap overall computation and data movement via warp-specialization and (2) interleave block-wise matmul and softmax operations, and (3) block quantization and incoherent processing that leverages hardware support for FP8 low-precision. We demonstrate that our method, FlashAttention-3, achieves speedup on H100 GPUs by 1.5-2.0× with FP16 reaching up to 740 TFLOPs/s (75% utilization), and with FP8 reaching close to 1.2 PFLOPs/s. We validate that FP8 FlashAttention-3 achieves 2.6× lower numerical error than a baseline FP8 attention.

DarcStar commentary

Our perspective on this work and how it informs our research — not part of the original paper.

FlashAttention-3 is what the line looks like once the algorithm is settled and the only room left is the silicon. The wins here aren't a better formula — they're asynchrony and precision: overlap the data movement with the compute using the newest hardware's async engines, interleave the matmul and the softmax so neither waits, and drop to FP8 where the numerics allow. It pushes attention close to the theoretical ceiling of the chip.

Two things in it match how we think. First, this is hardware co-design, not algorithm design — and at the extreme, co-designing with the hardware is the move that actually flips the table; everything above it is rearranging furniture on the same memory system. Second, and quieter: the paper doesn't just claim FP8 is faster, it bounds the error — a couple times lower than a naive low-precision baseline. That's the right discipline. Going faster in low precision only counts if you can say, with a number, how much accuracy you spent — measured, not hoped.

So this is the current ceiling of the prerequisite corner — the fast, exact, hardware-aware kernel the rest of the stack assumes underneath it. Like its predecessors, it's work we build on and port, not rebuild. When the frontier of a problem moves into the memory hierarchy and the number format, that's usually a sign the layer above it is mature.