VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

Abstract

Recent self-supervised methods for image representation learning are based on maximizing the agreement between embedding vectors from different views of the same image. A trivial solution is obtained when the encoder outputs constant vectors. This collapse problem is often avoided through implicit biases in the learning architecture, that often lack a clear justification or interpretation. In this paper, we introduce VICReg (Variance-Invariance-Covariance Regularization), a method that explicitly avoids the collapse problem with a simple regularization term on the variance of the embeddings along each dimension individually. VICReg combines the variance term with a decorrelation mechanism based on redundancy reduction and covariance regularization, and achieves results on par with the state of the art on several downstream tasks. In addition, we show that incorporating our new variance term into other methods helps stabilize the training and leads to performance improvements.

DarcStar commentary

Our perspective on this work and how it informs our research — not part of the original paper.

VICReg is a small idea that solves a real problem cleanly, which is exactly the kind of thing we like to point at. Any method that learns by making two views of the same input agree has a trapdoor: the trivial solution where the encoder outputs a constant and “agreement” is perfect and meaningless. VICReg closes the trapdoor with three explicit terms — keep the variance up so representations don't collapse, keep the covariance down so dimensions don't become redundant, and keep the two views invariant — no architectural tricks, no stop-gradients, just a loss that says what you want.

We appreciate it for its legibility. A lot of self-supervised learning avoids collapse through implicit biases that are hard to reason about; VICReg makes the anti-collapse mechanism an explicit, inspectable term you can point to and analyze. Naming your invariants in the objective, rather than hoping the architecture enforces them, is the disposition we bring to our own work.

It's a component, not a system — a regularizer other methods slot in — and that's precisely its value: the collapse problem is a tax that anyone building joint-embedding or world-model representations has to pay, and VICReg is one of the cleanest ways to pay it.