Research
Papers & preprints
Original papers and technical reports from our work on GIDE — alongside the foundational third-party research it builds on.
What these topics mean
- Distributed Training
- Sharding computation and memory across many devices to scale beyond a single accelerator.
- Efficient Attention
- Methods that cut attention's compute and memory cost without changing its output.
- GPU Kernels & Systems
- Hardware-aware implementations that map attention efficiently onto GPU memory and compute.
- Long-Context
- Extending models to very long sequences — hundreds of thousands to billions of tokens.
- Low-Precision & Quantization
- Trading numerical precision for speed and memory while bounding the accuracy cost.
- Safety & Control
- Methods that keep a learning system provably within specified safety limits while it acts.
- Self-Supervised Learning
- Learning useful representations from unlabeled data by predicting parts of the input from other parts.
- State-Space Models
- Linear-time sequence models that replace attention with a recurrent state, scaling to long sequences without the quadratic cost.
- Transformer Architecture
- The attention-based neural network architecture that underlies modern sequence models.
- World Models
- Models that learn how a system evolves in a latent representation space, for prediction and planning rather than generation.
Topic
Distributed Training
Sharding computation and memory across many devices to scale beyond a single accelerator.
Showing 1 of 18 papers.
Foundational reading
Notable third-party research we build on.
Ring Attention with Blockwise Transformers for Near-Infinite Context
Published Third-party DarcStar commentary ICLR·May 7, 2024By Hao Liu, Matei Zaharia, Pieter Abbeel
This is third-party work — not authored by or affiliated with DarcStar Technologies.
Transformers have emerged as the architecture of choice for many state-of-the-art AI models, showcasing exceptional performance across a wide range of AI applications. However, the memory demands imposed by Transformers limit their ability to handle long sequences, thereby posing challenges in utilizing videos, actions, and other long-form sequences and modalities…