01 · Demand capacity
What the output asks for
qreg rises as the Demand Regularizer is satisfied; qexc measures current sample variation at Z. Both are read with stop-gradient.
Self-supervised representation learning · JEPA · World models
1State Key Laboratory of CAD&CG, Zhejiang University2RoboParty Lab3Alibaba Group4AIR, Tsinghua University5InSpatio
† Corresponding authors
Overview · 3 min
Abstract
In Joint-Embedding Predictive Architectures (JEPAs), a well-constrained projector output does not by itself prevent representation collapse. We find that, from initialization, the encoder representation is dominated by a component shared by different inputs. A batch-normalized projector removes this component, so regularization after the projector cannot correct it, and the differences between representations shrink until they collapse. We propose SADL (Supply And Demand Loss), which keeps the original training objective and constrains the encoder directly when needed. SADL compares the representation differences that the projector output demands with the differences the encoder can retain and pass on to the projector, and adds a constraint on the encoder only when supply falls short of demand. It works with different Demand Regularizers and introduces no auxiliary networks, additional trainable parameters, or architectural changes. In single-task and jointly trained four-task LeWorldModel (LeWM) and INTACT experiments, SADL lowers the collapse rate over ten random seeds from 50% to 0%. After one epoch of training, encoder effective rank is 3.4–6.4× that of the corresponding baseline without SADL, and latent prediction error falls to 1/20–1/303 of it. Without additional tuning, SADL stays robust as its loss weight spans orders of magnitude, and with several Demand Regularizers INTACT exceeds 95% mean search-free control success. On four image datasets, SADL also consistently improves accuracy and stability across random seeds.
01 · Any Demand Regularizer
SADL does not assume a target geometry. With eight Demand Regularizers, including non-Gaussian and density-free ones, INTACT exceeds 92% search-free success after one epoch; seven of eight reach 95.03–95.81%. Drag to rotate the actual learned endpoints.




02 · The blind spot
At initialization the encoder representation H is almost entirely one vector shared by every input: sample variation carries about 0.01% of its energy. A batch-normalized projector subtracts the batch mean, so moving all of H along that shared direction leaves Z unchanged. The Demand Regularizer and the task loss, both computed from Z, get zero gradient there.
Try it · illustration
03 · Method
SADL keeps the task loss and the Demand Regularizer unchanged. It reads a detached demand capacity q from Z, measures the supply capacity k of H, and adds a Supply Regularizer on the encoder only when k < q.
01 · Demand capacity
qreg rises as the Demand Regularizer is satisfied; qexc measures current sample variation at Z. Both are read with stop-gradient.
02 · Supply capacity
kexc scores the energy and spread of sample differences in H; ktr measures how much reaches Z through regularized canonical correlations.
03 · One-sided Supply Regularizer
The penalty grows with the supply deficit and is exactly zero once supply meets demand, so SADL stays out of the way of a healthy encoder.
Training objective
ℒ = ℒtask + λD D(Z) + λS S(H, sg Z)
04 · World-model control
LeWM and INTACT trained jointly on PushT, Cube, Reacher and TwoRoom. Within each pair the attention backend, update budget and evaluation are matched; the baseline keeps the task loss and SIGReg, with no SADL term.
Collapse rate over ten random seeds, single-task and four-task LeWM and INTACT.
LeWM encoder effective rank after one epoch.
INTACT + SADL at epoch 2, with either MATH or Flash attention (baseline 70.0 / 66.1).
05 · Rescue
Starting from fixed near-collapsed checkpoints, training resumes either with the original Demand Regularizer only or with SADL added at its default weight. INTACT recovers from 18.9% to 68.3% control success within one more epoch. The checkpoints SADL rescues at its default weight λS = .01 are shown below.
INTACT seed 42, epoch 1 → 2. Replicated on a second machine: 23.1 → 69.1.
INTACT seed 3072 (24.4 → 53.9).
LeWM seed 21, epoch 2 → 4 (25.3 → 46.9).
06 · Beyond control
On ImageNet-LT, Galaxy10, CIFAR-10 and Imagenette, under matched budgets and Demand Regularizers, SADL improves early learning on all four datasets. Imagenette gains 8.91 points at epoch 20; the gap narrows to 0.44 points by epoch 800.
fewer updates to reach the baseline's final accuracy on Imagenette (23.2k → 17.8k; median of 3 seeds).
fewer updates to reach 90% of the baseline's final accuracy on ImageNet-LT (34.8k → 31.2k).
Input
VISReg
+ SADL07 · Mainstream SSL
We train VICReg, BYOL, MoCo v3 and SimCLR on ImageNet-100 with ViT-S/16, raise the learning rate four-fold and remove warmup, a recipe where training becomes unstable, and add SADL with no other change.
VICReg online accuracy, 52.0 → 60.4; kNN +7.8.
BYOL online accuracy, 48.6 → 53.5; linear probe +3.4.
Share of VICReg encoder energy carried by sample variation at the end of training.
Citation
@article{sun2026sadl,
title = {{SADL}: On the Demand-Supply Geometry of Representation Collapse},
author = {Sun, Junhan and Feng, Yigui and Liu, Mingzhe and Xu, Haoyan and
Ye, Lei and Han, Ziqi and Bai, Hanru and Zhang, Yanzhao and
Zhao, Hao and Zhang, Guofeng},
journal = {arXiv preprint arXiv:2611.21596},
year = {2026}
}