Research
Papers & preprints
Original papers and technical reports from our work on GIDE — alongside the foundational third-party research it builds on.
What these topics mean
- Distributed Training
- Sharding computation and memory across many devices to scale beyond a single accelerator.
- Efficient Attention
- Methods that cut attention's compute and memory cost without changing its output.
- GPU Kernels & Systems
- Hardware-aware implementations that map attention efficiently onto GPU memory and compute.
- Long-Context
- Extending models to very long sequences — hundreds of thousands to billions of tokens.
- Low-Precision & Quantization
- Trading numerical precision for speed and memory while bounding the accuracy cost.
- Safety & Control
- Methods that keep a learning system provably within specified safety limits while it acts.
- Self-Supervised Learning
- Learning useful representations from unlabeled data by predicting parts of the input from other parts.
- State-Space Models
- Linear-time sequence models that replace attention with a recurrent state, scaling to long sequences without the quadratic cost.
- Transformer Architecture
- The attention-based neural network architecture that underlies modern sequence models.
- World Models
- Models that learn how a system evolves in a latent representation space, for prediction and planning rather than generation.
Topic
Long-Context
Extending models to very long sequences — hundreds of thousands to billions of tokens.
Showing 8 of 18 papers.
Foundational reading
Notable third-party research we build on.
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
Preprint Third-party DarcStar commentary arXiv·April 28, 2025By Amir Zandieh, Majid Daliri, Majid Hadian, Vahab Mirrokni
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Vector quantization, a problem rooted in Shannon's source coding theory, aims to quantize high-dimensional Euclidean vectors while minimizing distortion in their geometric structure. We propose TurboQuant to address both mean-squared error (MSE) and inner product distortion, overcoming limitations of existing methods that fail to achieve optimal distortion rates. Our data-oblivious…
MoBA: Mixture of Block Attention for Long-Context LLMs
Preprint Third-party DarcStar commentary arXiv·February 18, 2025By Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Neo Y. Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, Jiezhong Qiu
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Scaling the effective context length is essential for advancing large language models (LLMs) toward artificial general intelligence (AGI). However, the quadratic increase in computational complexity inherent in traditional attention mechanisms presents a prohibitive overhead. Existing approaches either impose strongly biased structures, such as sink or window attention which are task-specific,…
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Preprint Third-party DarcStar commentary arXiv·February 16, 2025By Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, Wangding Zeng
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Long-context modeling is crucial for next-generation language models, yet the high computational cost of standard attention mechanisms poses significant computational challenges. Sparse attention offers a promising direction for improving efficiency while maintaining model capabilities. We present NSA, a Natively trainable Sparse Attention mechanism that integrates algorithmic innovations with hardware-aligned optimizations…
QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead
Preprint Third-party DarcStar commentary arXiv·June 5, 2024By Amir Zandieh, Majid Daliri, Insu Han
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Serving LLMs requires substantial memory due to the storage requirements of Key-Value (KV) embeddings in the KV cache, which grows with sequence length. An effective approach to compress KV cache is quantization. However, traditional quantization methods face significant memory overhead due to the need to store quantization constants (at least…
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
Preprint Third-party DarcStar commentary arXiv·May 7, 2024By DeepSeek-AI
This is third-party work — not authored by or affiliated with DarcStar Technologies.
We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees…
Ring Attention with Blockwise Transformers for Near-Infinite Context
Published Third-party DarcStar commentary ICLR·May 7, 2024By Hao Liu, Matei Zaharia, Pieter Abbeel
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Transformers have emerged as the architecture of choice for many state-of-the-art AI models, showcasing exceptional performance across a wide range of AI applications. However, the memory demands imposed by Transformers limit their ability to handle long sequences, thereby posing challenges in utilizing videos, actions, and other long-form sequences and modalities…
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Preprint Third-party DarcStar commentary arXiv·December 1, 2023By Albert Gu, Tri Dao
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational…
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Preprint Third-party DarcStar commentary arXiv·July 5, 2023By Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, Furu Wei
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Scaling sequence length has become a critical demand in the era of large language models. However, existing methods struggle with either computational complexity or model expressivity, rendering the maximum sequence length restricted. To address this issue, we introduce LongNet, a Transformer variant that can scale sequence length to more than…