Research
Papers & preprints
Original papers and technical reports from our work on GIDE — alongside the foundational third-party research it builds on.
What these topics mean
- Distributed Training
- Sharding computation and memory across many devices to scale beyond a single accelerator.
- Efficient Attention
- Methods that cut attention's compute and memory cost without changing its output.
- GPU Kernels & Systems
- Hardware-aware implementations that map attention efficiently onto GPU memory and compute.
- Long-Context
- Extending models to very long sequences — hundreds of thousands to billions of tokens.
- Low-Precision & Quantization
- Trading numerical precision for speed and memory while bounding the accuracy cost.
- Safety & Control
- Methods that keep a learning system provably within specified safety limits while it acts.
- Self-Supervised Learning
- Learning useful representations from unlabeled data by predicting parts of the input from other parts.
- State-Space Models
- Linear-time sequence models that replace attention with a recurrent state, scaling to long sequences without the quadratic cost.
- Transformer Architecture
- The attention-based neural network architecture that underlies modern sequence models.
- World Models
- Models that learn how a system evolves in a latent representation space, for prediction and planning rather than generation.
Topic
Low-Precision & Quantization
Trading numerical precision for speed and memory while bounding the accuracy cost.
Showing 3 of 18 papers.
Foundational reading
Notable third-party research we build on.
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
Preprint Third-party DarcStar commentary arXiv·April 28, 2025By Amir Zandieh, Majid Daliri, Majid Hadian, Vahab Mirrokni
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Vector quantization, a problem rooted in Shannon's source coding theory, aims to quantize high-dimensional Euclidean vectors while minimizing distortion in their geometric structure. We propose TurboQuant to address both mean-squared error (MSE) and inner product distortion, overcoming limitations of existing methods that fail to achieve optimal distortion rates. Our data-oblivious…
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
Published Third-party DarcStar commentary NeurIPS·July 11, 2024By Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Attention, as a core layer of the ubiquitous Transformer architecture, is the bottleneck for large language models and long-context applications. FlashAttention elaborated an approach to speed up attention on GPUs through minimizing memory reads/writes. However, it has yet to take advantage of new capabilities present in recent hardware, with FlashAttention-2…
QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead
Preprint Third-party DarcStar commentary arXiv·June 5, 2024By Amir Zandieh, Majid Daliri, Insu Han
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Serving LLMs requires substantial memory due to the storage requirements of Key-Value (KV) embeddings in the KV cache, which grows with sequence length. An effective approach to compress KV cache is quantization. However, traditional quantization methods face significant memory overhead due to the need to store quantization constants (at least…