Papers & preprints

The Intelligence Ratchet: A Theoretical Framework for Self-Stabilizing Artificial Superintelligence

By Michael Harris

Current paradigms in Artificial Intelligence face a seemingly intractable trade-off between Capability and Safety. Systems capable of recursive self-improvement (the "Singularity") risk unbounded instability, while systems with provable safety guarantees ("Bounded Optimality") are mathematically prevented from generating novel solutions. This paper introduces the General Interaction Dynamics Engine (GIDE), a theoretical framework…

V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

By Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xiaodong Ma, Sarath Chandar, Franziska Meier, Yann LeCun, Michael Rabbat, Nicolas Ballas

This is third-party work — not authored by or affiliated with DarcStar Technologies.

A major challenge for modern AI is to learn to understand the world and learn to act largely by observation. This paper explores a self-supervised approach that combines internet-scale video data with a small amount of interaction data (robot trajectories), to develop models capable of understanding, predicting, and planning in…

TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate

By Amir Zandieh, Majid Daliri, Majid Hadian, Vahab Mirrokni

This is third-party work — not authored by or affiliated with DarcStar Technologies.

Vector quantization, a problem rooted in Shannon's source coding theory, aims to quantize high-dimensional Euclidean vectors while minimizing distortion in their geometric structure. We propose TurboQuant to address both mean-squared error (MSE) and inner product distortion, overcoming limitations of existing methods that fail to achieve optimal distortion rates. Our data-oblivious…

MoBA: Mixture of Block Attention for Long-Context LLMs

By Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Neo Y. Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, Jiezhong Qiu

This is third-party work — not authored by or affiliated with DarcStar Technologies.

Scaling the effective context length is essential for advancing large language models (LLMs) toward artificial general intelligence (AGI). However, the quadratic increase in computational complexity inherent in traditional attention mechanisms presents a prohibitive overhead. Existing approaches either impose strongly biased structures, such as sink or window attention which are task-specific,…

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention

By Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, Wangding Zeng

This is third-party work — not authored by or affiliated with DarcStar Technologies.

Long-context modeling is crucial for next-generation language models, yet the high computational cost of standard attention mechanisms poses significant computational challenges. Sparse attention offers a promising direction for improving efficiency while maintaining model capabilities. We present NSA, a Natively trainable Sparse Attention mechanism that integrates algorithmic innovations with hardware-aligned optimizations…

FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

By Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao

This is third-party work — not authored by or affiliated with DarcStar Technologies.

Attention, as a core layer of the ubiquitous Transformer architecture, is the bottleneck for large language models and long-context applications. FlashAttention elaborated an approach to speed up attention on GPUs through minimizing memory reads/writes. However, it has yet to take advantage of new capabilities present in recent hardware, with FlashAttention-2…

QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead

By Amir Zandieh, Majid Daliri, Insu Han

This is third-party work — not authored by or affiliated with DarcStar Technologies.

Serving LLMs requires substantial memory due to the storage requirements of Key-Value (KV) embeddings in the KV cache, which grows with sequence length. An effective approach to compress KV cache is quantization. However, traditional quantization methods face significant memory overhead due to the need to store quantization constants (at least…

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

By DeepSeek-AI

This is third-party work — not authored by or affiliated with DarcStar Technologies.

We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees…

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

By Tri Dao

This is third-party work — not authored by or affiliated with DarcStar Technologies.

Scaling Transformers to longer sequence lengths has been a major problem in the last several years, promising to improve performance in language modeling and high-resolution image understanding, as well as to unlock new applications in code, audio, and video generation. The attention layer is the main bottleneck in scaling to…

Ring Attention with Blockwise Transformers for Near-Infinite Context

By Hao Liu, Matei Zaharia, Pieter Abbeel

This is third-party work — not authored by or affiliated with DarcStar Technologies.

Transformers have emerged as the architecture of choice for many state-of-the-art AI models, showcasing exceptional performance across a wide range of AI applications. However, the memory demands imposed by Transformers limit their ability to handle long sequences, thereby posing challenges in utilizing videos, actions, and other long-form sequences and modalities…

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

By Albert Gu, Tri Dao

This is third-party work — not authored by or affiliated with DarcStar Technologies.

Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational…

LongNet: Scaling Transformers to 1,000,000,000 Tokens

By Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, Furu Wei

This is third-party work — not authored by or affiliated with DarcStar Technologies.

Scaling sequence length has become a critical demand in the era of large language models. However, existing methods struggle with either computational complexity or model expressivity, rendering the maximum sequence length restricted. To address this issue, we introduce LongNet, a Transformer variant that can scale sequence length to more than…

Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture

By Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, Nicolas Ballas

This is third-party work — not authored by or affiliated with DarcStar Technologies.

This paper demonstrates an approach for learning highly semantic image representations without relying on hand-crafted data-augmentations. We introduce the Image-based Joint-Embedding Predictive Architecture (I-JEPA), a non-generative approach for self-supervised learning from images. The idea behind I-JEPA is simple: from a single context block, predict the representations of various target blocks…

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

By Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré

This is third-party work — not authored by or affiliated with DarcStar Technologies.

Transformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention are quadratic in sequence length. Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup. We argue that…

VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

By Adrien Bardes, Jean Ponce, Yann LeCun

This is third-party work — not authored by or affiliated with DarcStar Technologies.

Recent self-supervised methods for image representation learning are based on maximizing the agreement between embedding vectors from different views of the same image. A trivial solution is obtained when the encoder outputs constant vectors. This collapse problem is often avoided through implicit biases in the learning architecture, that often lack…

Control Barrier Functions: Theory and Applications

By Aaron D. Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, Paulo Tabuada

This is third-party work — not authored by or affiliated with DarcStar Technologies.

This paper provides an introduction and overview of recent work on control barrier functions and their use to verify and enforce safety properties in the context of (optimization based) safety-critical controllers. We survey the main technical results and discuss applications to several domains including robotic systems.

Safe Reinforcement Learning via Shielding

By Mohammed Alshiekh, Roderick Bloem, Ruediger Ehlers, Bettina Könighofer, Scott Niekum, Ufuk Topcu

This is third-party work — not authored by or affiliated with DarcStar Technologies.

Reinforcement learning algorithms discover policies that maximize reward, but do not necessarily guarantee safety during learning or execution phases. We introduce a new approach to learn optimal policies while enforcing properties expressed in temporal logic. To this end, given the temporal logic specification that is to be obeyed by the…

Attention Is All You Need

By Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin

This is third-party work — not authored by or affiliated with DarcStar Technologies.

Introduces the Transformer, a sequence-transduction architecture based solely on attention mechanisms, dispensing with recurrence and convolutions.