Research
Papers & preprints
Original papers and technical reports from our work on GIDE — alongside the foundational third-party research it builds on.
What these topics mean
- Distributed Training
- Sharding computation and memory across many devices to scale beyond a single accelerator.
- Efficient Attention
- Methods that cut attention's compute and memory cost without changing its output.
- GPU Kernels & Systems
- Hardware-aware implementations that map attention efficiently onto GPU memory and compute.
- Long-Context
- Extending models to very long sequences — hundreds of thousands to billions of tokens.
- Low-Precision & Quantization
- Trading numerical precision for speed and memory while bounding the accuracy cost.
- Safety & Control
- Methods that keep a learning system provably within specified safety limits while it acts.
- Self-Supervised Learning
- Learning useful representations from unlabeled data by predicting parts of the input from other parts.
- State-Space Models
- Linear-time sequence models that replace attention with a recurrent state, scaling to long sequences without the quadratic cost.
- Transformer Architecture
- The attention-based neural network architecture that underlies modern sequence models.
- World Models
- Models that learn how a system evolves in a latent representation space, for prediction and planning rather than generation.
Topic
GPU Kernels & Systems
Hardware-aware implementations that map attention efficiently onto GPU memory and compute.
Showing 4 of 18 papers.
Foundational reading
Notable third-party research we build on.
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Preprint Third-party DarcStar commentary arXiv·February 16, 2025By Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, Wangding Zeng
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Long-context modeling is crucial for next-generation language models, yet the high computational cost of standard attention mechanisms poses significant computational challenges. Sparse attention offers a promising direction for improving efficiency while maintaining model capabilities. We present NSA, a Natively trainable Sparse Attention mechanism that integrates algorithmic innovations with hardware-aligned optimizations…
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
Published Third-party DarcStar commentary NeurIPS·July 11, 2024By Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Attention, as a core layer of the ubiquitous Transformer architecture, is the bottleneck for large language models and long-context applications. FlashAttention elaborated an approach to speed up attention on GPUs through minimizing memory reads/writes. However, it has yet to take advantage of new capabilities present in recent hardware, with FlashAttention-2…
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Published Third-party DarcStar commentary ICLR·May 7, 2024By Tri Dao
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Scaling Transformers to longer sequence lengths has been a major problem in the last several years, promising to improve performance in language modeling and high-resolution image understanding, as well as to unlock new applications in code, audio, and video generation. The attention layer is the main bottleneck in scaling to…
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Published Third-party DarcStar commentary NeurIPS·May 27, 2022By Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Transformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention are quadratic in sequence length. Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup. We argue that…