Self-supervised representation learning · JEPA · World models

SADL: On the Demand-Supply Geometry of Representation Collapse

  • No architecture changes
  • No extra trainable parameters
  • One hyperparameter
  • High robustness
  • High stability
  • High success rate
  • Junhan Sun1
  • Yigui Feng2
  • Mingzhe Liu2
  • Haoyan Xu2
  • Lei Ye2
  • Ziqi Han2
  • Hanru Bai3
  • Yanzhao Zhang3
  • Hao Zhao2,4,†
  • Guofeng Zhang1,5,†

1State Key Laboratory of CAD&CG, Zhejiang University2RoboParty Lab3Alibaba Group4AIR, Tsinghua University5InSpatio

† Corresponding authors

50%→0%
collapse rate over ten random seeds (LeWM and INTACT)
3.4–6.4×
encoder effective rank after one epoch, vs. the same model without SADL
1/20–1/303
latent prediction error after one epoch, relative to baseline
>95%
INTACT search-free control success with several Demand Regularizers
0
auxiliary networks, extra parameters, or architecture changes

Overview · 3 min

The idea in one film.

Abstract

In Joint-Embedding Predictive Architectures (JEPAs), a well-constrained projector output does not by itself prevent representation collapse. We find that, from initialization, the encoder representation is dominated by a component shared by different inputs. A batch-normalized projector removes this component, so regularization after the projector cannot correct it, and the differences between representations shrink until they collapse. We propose SADL (Supply And Demand Loss), which keeps the original training objective and constrains the encoder directly when needed. SADL compares the representation differences that the projector output demands with the differences the encoder can retain and pass on to the projector, and adds a constraint on the encoder only when supply falls short of demand. It works with different Demand Regularizers and introduces no auxiliary networks, additional trainable parameters, or architectural changes. In single-task and jointly trained four-task LeWorldModel (LeWM) and INTACT experiments, SADL lowers the collapse rate over ten random seeds from 50% to 0%. After one epoch of training, encoder effective rank is 3.4–6.4× that of the corresponding baseline without SADL, and latent prediction error falls to 1/20–1/303 of it. Without additional tuning, SADL stays robust as its loss weight spans orders of magnitude, and with several Demand Regularizers INTACT exceeds 95% mean search-free control success. On four image datasets, SADL also consistently improves accuracy and stability across random seeds.

01 · Any Demand Regularizer

Gaussian or not, the encoder keeps up with what is demanded.

SADL does not assume a target geometry. With eight Demand Regularizers, including non-Gaussian and density-free ones, INTACT exceeds 92% search-free success after one epoch; seven of eight reach 95.03–95.81%. Drag to rotate the actual learned endpoints.

SIGReg
Epoch-1 control success 95.58%
PushTCubeReacherTwoRoom
Endpoint latents shaped as the letter J
Endpoint latents shaped as the letter E
Endpoint latents shaped as the letter P
Endpoint latents shaped as the letter A

02 · The blind spot

A well-regularized output cannot see what the encoder has lost.

At initialization the encoder representation H is almost entirely one vector shared by every input: sample variation carries about 0.01% of its energy. A batch-normalized projector subtracts the batch mean, so moving all of H along that shared direction leaves Z unchanged. The Demand Regularizer and the task loss, both computed from Z, get zero gradient there.

Try it · illustration

Shared shift added to every row of H0.0
Sample variation in H0.30
0.000change in Z from the shift
2%energy in sample variation of H

Fraction of energy in sample variation · after one epoch

Encoder H · baseline
3.3%
Encoder H · + SADL
98.6%
Endpoint Z · baseline
35.5%
Endpoint Z · + SADL
99.0%

03 · Method

Read the demand at the output. Repair the supply at the encoder.

SADL keeps the task loss and the Demand Regularizer unchanged. It reads a detached demand capacity q from Z, measures the supply capacity k of H, and adds a Supply Regularizer on the encoder only when k < q.

x EncoderEθ H Projector PφLinear · BatchNorm · … Z Demand RegularizerλD D(Z) demand q · stop-gradient Supply RegularizerλS S(H, sg Z) acts on H only if k < q supply k from Hendpoint

01 · Demand capacity

What the output asks for

qreg rises as the Demand Regularizer is satisfied; qexc measures current sample variation at Z. Both are read with stop-gradient.

q=qregqexc

02 · Supply capacity

What the encoder can give

kexc scores the energy and spread of sample differences in H; ktr measures how much reaches Z through regularized canonical correlations.

k = min{kexc, ktr}

03 · One-sided Supply Regularizer

Act only on the shortfall

The penalty grows with the supply deficit and is exactly zero once supply meets demand, so SADL stays out of the way of a healthy encoder.

S = [log(q+ε) − log(k+ε)]+2

Training objective

ℒ = ℒtask + λD D(Z) + λS S(H, sg Z)

No extra trainable parametersNo auxiliary networksOriginal architecture and inferenceWorks with any Demand Regularizer
Demand q0.65
Supply k0.30

04 · World-model control

One epoch with SADL beats two epochs without it.

LeWM and INTACT trained jointly on PushT, Cube, Reacher and TwoRoom. Within each pair the attention backend, update budget and evaluation are matched; the baseline keeps the task loss and SIGReg, with no SADL term.

Epoch 1 · every Demand Regularizer lifts together (%)

Demand Regularizer only+ SADLmean ± SD, 3 training seeds
Show as table

Four-task mean control success (%)

SIGReg onlySIGReg + SADL
Show as table
50% → 0%

Collapse rate over ten random seeds, single-task and four-task LeWM and INTACT.

14.5 → 52.7

LeWM encoder effective rank after one epoch.

96.3%

INTACT + SADL at epoch 2, with either MATH or Flash attention (baseline 70.0 / 66.1).

05 · Rescue

SADL can also reopen an encoder that has already collapsed.

Starting from fixed near-collapsed checkpoints, training resumes either with the original Demand Regularizer only or with SADL added at its default weight. INTACT recovers from 18.9% to 68.3% control success within one more epoch. The checkpoints SADL rescues at its default weight λS = .01 are shown below.

Control success after continuation (%)

Continue without SADLContinue with SADL
Show as table
18.9 → 68.3

INTACT seed 42, epoch 1 → 2. Replicated on a second machine: 23.1 → 69.1.

+29.5 pts

INTACT seed 3072 (24.4 → 53.9).

+21.7 pts

LeWM seed 21, epoch 2 → 4 (25.3 → 46.9).

06 · Beyond control

The same loss speeds up image representation learning.

On ImageNet-LT, Galaxy10, CIFAR-10 and Imagenette, under matched budgets and Demand Regularizers, SADL improves early learning on all four datasets. Imagenette gains 8.91 points at epoch 20; the gap narrows to 0.44 points by epoch 800.

Classification accuracy (%)

Baseline+ SADL

SADL minus its paired baseline (points)

23%

fewer updates to reach the baseline's final accuracy on Imagenette (23.2k → 17.8k; median of 3 seeds).

10%

fewer updates to reach 90% of the baseline's final accuracy on ImageNet-LT (34.8k → 31.2k).

Input imageInput
Attention map, VISRegVISReg
Attention map, SADL + VISReg+ SADL

07 · Mainstream SSL

SADL stabilizes aggressive training in mainstream SSL.

We train VICReg, BYOL, MoCo v3 and SimCLR on ImageNet-100 with ViT-S/16, raise the learning rate four-fold and remove warmup, a recipe where training becomes unstable, and add SADL with no other change.

Online linear accuracy (%)

Method only+ SADLImageNet-100 · ViT-S/16 · 100 epochs · lr ×4, no warmup
Show as table
+8.4 pts

VICReg online accuracy, 52.0 → 60.4; kNN +7.8.

+4.9 pts

BYOL online accuracy, 48.6 → 53.5; linear probe +3.4.

0.38 → 0.99

Share of VICReg encoder energy carried by sample variation at the end of training.

Citation

BibTeX

@article{sun2026sadl, title = {{SADL}: On the Demand-Supply Geometry of Representation Collapse}, author = {Sun, Junhan and Feng, Yigui and Liu, Mingzhe and Xu, Haoyan and Ye, Lei and Han, Ziqi and Bai, Hanru and Zhang, Yanzhao and Zhao, Hao and Zhang, Guofeng}, journal = {arXiv preprint arXiv:2611.21596}, year = {2026} }