Research
Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
This is third-party work — not authored by or affiliated with DarcStar Technologies.
By Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, Nicolas Ballas
Abstract
This paper demonstrates an approach for learning highly semantic image representations without relying on hand-crafted data-augmentations. We introduce the Image-based Joint-Embedding Predictive Architecture (I-JEPA), a non-generative approach for self-supervised learning from images. The idea behind I-JEPA is simple: from a single context block, predict the representations of various target blocks in the same image. A core design choice to guide I-JEPA towards producing semantic representations is the masking strategy; specifically, it is crucial to (a) sample target blocks with sufficiently large scale (semantic), and to (b) use a sufficiently informative (spatially distributed) context block. Empirically, when combined with Vision Transformers, we find I-JEPA to be highly scalable. For instance, we train a ViT-Huge/14 on ImageNet using 16 A100 GPUs in under 72 hours to achieve strong downstream performance across a wide range of tasks, from linear classification to object counting and depth prediction.
DarcStar commentary
Our perspective on this work and how it informs our research — not part of the original paper.
I-JEPA points at something we think is under-appreciated: the most useful thing a model can learn to do is predict in representation space, not pixel space. Instead of reconstructing every detail of a masked image — burning capacity on texture nobody needs — I-JEPA predicts the representation of the missing parts from what it can see. It learns what a thing is without wasting itself on exactly how it looks.
That non-generative framing is the part we care about. A system that acts in the world doesn't need to hallucinate the future in full fidelity; it needs a compact, predictive model of how its situation will evolve — a world model it can plan against. Predicting in latent space is how you get that without paying the tax of generation, and it sidesteps the trap of a model that looks impressive because it draws well while understanding little.
The honest caution is the one every joint-embedding method carries: predict-your-own-representation invites collapse — the degenerate solution where everything maps to the same point and the loss looks great. I-JEPA handles it with architecture; others handle it with explicit regularization. Which cure you pick, and whether the learned space is actually useful for downstream acting rather than merely non-collapsed, is the empirical question that matters — and it has to be measured on the task, not inferred from the pretraining loss.