Research
Papers & preprints
Original papers and technical reports from our work on GIDE — alongside the foundational third-party research it builds on.
What these topics mean
- Distributed Training
- Sharding computation and memory across many devices to scale beyond a single accelerator.
- Efficient Attention
- Methods that cut attention's compute and memory cost without changing its output.
- GPU Kernels & Systems
- Hardware-aware implementations that map attention efficiently onto GPU memory and compute.
- Long-Context
- Extending models to very long sequences — hundreds of thousands to billions of tokens.
- Low-Precision & Quantization
- Trading numerical precision for speed and memory while bounding the accuracy cost.
- Safety & Control
- Methods that keep a learning system provably within specified safety limits while it acts.
- Self-Supervised Learning
- Learning useful representations from unlabeled data by predicting parts of the input from other parts.
- State-Space Models
- Linear-time sequence models that replace attention with a recurrent state, scaling to long sequences without the quadratic cost.
- Transformer Architecture
- The attention-based neural network architecture that underlies modern sequence models.
- World Models
- Models that learn how a system evolves in a latent representation space, for prediction and planning rather than generation.
Topic
State-Space Models
Linear-time sequence models that replace attention with a recurrent state, scaling to long sequences without the quadratic cost.
Showing 1 of 18 papers.
Foundational reading
Notable third-party research we build on.
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Preprint Third-party DarcStar commentary arXiv·December 1, 2023By Albert Gu, Tri Dao
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational…