Research
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
This is third-party work — not authored by or affiliated with DarcStar Technologies.
By Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xiaodong Ma, Sarath Chandar, Franziska Meier, Yann LeCun, Michael Rabbat, Nicolas Ballas
Abstract
A major challenge for modern AI is to learn to understand the world and learn to act largely by observation. This paper explores a self-supervised approach that combines internet-scale video data with a small amount of interaction data (robot trajectories), to develop models capable of understanding, predicting, and planning in the physical world. We first pre-train an action-free joint-embedding-predictive architecture, V-JEPA 2, on a video and image dataset comprising over 1 million hours of internet video. V-JEPA 2 achieves strong performance on motion understanding and human action anticipation tasks, and is able to be aligned with a large language model to yield state-of-the-art performance on multiple video question-answering tasks. By further training with a small amount of robot interaction data, we obtain a latent action-conditioned world model, V-JEPA 2-AC, that enables zero-shot robot planning to pick and place objects in new environments.
DarcStar commentary
Our perspective on this work and how it informs our research — not part of the original paper.
If I-JEPA is the idea, V-JEPA 2 is the idea aimed at the thing we actually care about: acting. It scales latent-space predictive learning to video and then does the move that matters — uses the learned world model for planning, closing the loop from passive observation to choosing what to do next, with only a little interaction data on top of internet-scale video.
This is the shape of world model we find most convincing: learn the dynamics of the world by watching it, in a representation space rather than in pixels, and then plan against that model instead of against reality directly. Understanding, prediction, and control as one stack, learned mostly by observation, is a serious answer to how an agent gets competent without a million expensive trials in the real world.
Our standing caution applies here more than anywhere: a model that predicts well in its own latent space and plans well in a benchmark has not yet proven it will plan well when the world diverges from its training distribution — and the gap between “predicts the next latent state” and “is safe to hand the controls” is precisely the gap our interest in provable safety filters exists to close. We read V-JEPA 2 as the capability half of a system whose other half has to be the guarantee that its plans stay inside known-safe limits.