Research
Papers & preprints
Original papers and technical reports from our work on GIDE — alongside the foundational third-party research it builds on.
What these topics mean
- Distributed Training
- Sharding computation and memory across many devices to scale beyond a single accelerator.
- Efficient Attention
- Methods that cut attention's compute and memory cost without changing its output.
- GPU Kernels & Systems
- Hardware-aware implementations that map attention efficiently onto GPU memory and compute.
- Long-Context
- Extending models to very long sequences — hundreds of thousands to billions of tokens.
- Low-Precision & Quantization
- Trading numerical precision for speed and memory while bounding the accuracy cost.
- Safety & Control
- Methods that keep a learning system provably within specified safety limits while it acts.
- Self-Supervised Learning
- Learning useful representations from unlabeled data by predicting parts of the input from other parts.
- State-Space Models
- Linear-time sequence models that replace attention with a recurrent state, scaling to long sequences without the quadratic cost.
- Transformer Architecture
- The attention-based neural network architecture that underlies modern sequence models.
- World Models
- Models that learn how a system evolves in a latent representation space, for prediction and planning rather than generation.
Topic
Efficient Attention
Methods that cut attention's compute and memory cost without changing its output.
Showing 7 of 18 papers.
Foundational reading
Notable third-party research we build on.
MoBA: Mixture of Block Attention for Long-Context LLMs
Preprint Third-party DarcStar commentary arXiv·February 18, 2025By Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Neo Y. Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, Jiezhong Qiu
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Scaling the effective context length is essential for advancing large language models (LLMs) toward artificial general intelligence (AGI). However, the quadratic increase in computational complexity inherent in traditional attention mechanisms presents a prohibitive overhead. Existing approaches either impose strongly biased structures, such as sink or window attention which are task-specific,…
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Preprint Third-party DarcStar commentary arXiv·February 16, 2025By Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, Wangding Zeng
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Long-context modeling is crucial for next-generation language models, yet the high computational cost of standard attention mechanisms poses significant computational challenges. Sparse attention offers a promising direction for improving efficiency while maintaining model capabilities. We present NSA, a Natively trainable Sparse Attention mechanism that integrates algorithmic innovations with hardware-aligned optimizations…
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
Published Third-party DarcStar commentary NeurIPS·July 11, 2024By Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Attention, as a core layer of the ubiquitous Transformer architecture, is the bottleneck for large language models and long-context applications. FlashAttention elaborated an approach to speed up attention on GPUs through minimizing memory reads/writes. However, it has yet to take advantage of new capabilities present in recent hardware, with FlashAttention-2…
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
Preprint Third-party DarcStar commentary arXiv·May 7, 2024By DeepSeek-AI
This is third-party work — not authored by or affiliated with DarcStar Technologies.
We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees…
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Published Third-party DarcStar commentary ICLR·May 7, 2024By Tri Dao
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Scaling Transformers to longer sequence lengths has been a major problem in the last several years, promising to improve performance in language modeling and high-resolution image understanding, as well as to unlock new applications in code, audio, and video generation. The attention layer is the main bottleneck in scaling to…
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Preprint Third-party DarcStar commentary arXiv·July 5, 2023By Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, Furu Wei
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Scaling sequence length has become a critical demand in the era of large language models. However, existing methods struggle with either computational complexity or model expressivity, rendering the maximum sequence length restricted. To address this issue, we introduce LongNet, a Transformer variant that can scale sequence length to more than…
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Published Third-party DarcStar commentary NeurIPS·May 27, 2022By Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Transformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention are quadratic in sequence length. Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup. We argue that…