How can we relight a moving person with precise control over light direction and relative intensity?
DynOLAT predicts how the same performance responds to individual light directions, then combines those responses using weights from the target environment.
Accurate rendering depends on both the recovered person and the approximation of light transport.
Recovering the person.
Images do not uniquely determine geometry or the materials of skin, hair, and clothing. Dedicated capture and per-person optimization make scaling to new people and performances costly.
Computing the appearance.
Scattering and self-shadowing depend on that scene model and its numerical approximation. Soler et al. (2022) analyzed finite-basis approximations of light transport. They show that, in general, increasing the basis size cannot make the worst-case error vanish across all lighting conditions.
Reflection, scattering, and self-occlusion shape appearance through geometry, materials, and visibility.
Generative priors offer a way to learn appearance responses without explicitly recovering every physical property.
02Supervision
Why can realistic results still miss the target light?
Visual realism alone does not ensure that an output follows the specified illumination.
Most generative relighting methods directly learn the final relit image or video, using mixed illumination—such as an environment map or a background image—as the condition.
Related work: learning the relit result directly
IC-Light. The target lighting is expressed through an environment or background condition, and the model generates the relit image directly. Its in-the-wild branch constructs pairs using appearance degradation or augmentation and estimated target lighting. Training also uses rendered and real Light Stage data, with light-transport consistency. The constructed pairs help scale training, while their lighting supervision remains approximate.
DiffusionRenderer. Its forward renderer directly generates video conditioned on G-buffers and an environment map. To bridge the synthetic-to-real gap, it adds auto-labeled real videos and a real-data LoRA. Its training therefore combines exact synthetic labels with estimated geometry, materials, and lighting from real footage.
Direction must match.
Shadows and highlights can drift away from the target direction, even in a convincing generated video.
Relative strength must match.
Output brightness should follow the relative intensities in the environment map; a plausible overall appearance does not establish this control.
Capturing real relighting pairs is difficult, and even accurately measured pairs under mixed illumination leave the contributions of individual light directions entangled.
Each output combines many lighting directions. Its appearance alone does not reveal their separate contributions.
Can we supervise each direction’s response separately?
Why independent observations matter: a two-light example
Fix the person, pose, and camera, and consider one pixel in one color channel with a linear response. Let a and b be its unknown responses to the left and right lights.
Two unknowns → two independent observations
Photo 1: 80% left + 20% right → brightness 12. Photo 2: 20% left + 80% right → brightness 18.
The responses are a = 10 and b = 20: 0.8 × 10 + 0.2 × 20 = 12, and 0.2 × 10 + 0.8 × 20 = 18. These are illustrative linear brightness values.
Doubling the first setup gives brightness 24, but no independent equation. A different mixing ratio adds information; a brighter copy does not.
For N independent directions, recovering N unknown responses without additional assumptions needs at least N independent equations. An h × w environment map has h × w directional bins, with one unknown response per bin at each pixel.
One-Light-At-a-Time (OLAT) capture measures one direction at a time. These lighting patterns are orthogonal because they share no illuminated directions. With nonnegative intensities, overlapping directions cannot cancel to make two patterns orthogonal; a complete orthogonal design over the directional bins therefore leads naturally to OLAT.
Independent mixed-light patterns can also supply the equations needed to solve for all N unknown directional responses, but OLAT’s orthogonal patterns isolate each response directly. This substantially simplifies the learning task: the model learns a single direction’s effect without first disentangling overlapping lights.
03OLAT
How does OLAT make lighting control explicit?
Separate directional response prediction from the weights that set relative lighting strength.
Learn one direction at a time.
One-Light-At-a-Time (OLAT) capture records a single light’s full response, including shadows and indirect scattering. Aligned flat-lit / OLAT pairs adapt a pretrained generative prior to predict that response from a flat-lit reference and a target direction.
Assemble the target illumination.
Under linear light transport, separate lights’ contributions add. We combine the decoded responses in linear color space: the predicted basis specifies the direction, and its environment-derived weight sets the relative strength per color channel.
Related work: measured lighting responses
Paul Debevec’s Light Stage measured directional facial reflectance; HumanOLAT provides multi-view full-body OLAT captures; Meta’s Relightable Gaussian Codec Avatars learns relightable avatars from controlled lighting capture. These works use different capture patterns and representations, while demonstrating the value of measured appearance.
Relit frame ≈ Σ(direction weight × OLAT response)
The same pose under three single lights, weighted to compose a new environment.
For a moving person, these responses must track the entire performance.
OLAT prediction and lighting composition
Adapt in latent space; compose in linear image space
pθ(ZOLAT | Zflat, ωk) It(x; L) ≈ ∑kwk(L) Bt,k(x)
Zflat and ZOLAT encode paired sequences; pθ denotes the adapted conditional generation distribution for direction ωk. Bt,k(x) is the decoded response at frame t, pixel x, and direction k. L is the target environment; wk(L) integrates its linear HDR radiance over the region assigned to that direction.
The model learns directional responses in latent space and combines them in linear image space after decoding. The supervision design supplies isolated targets.
Training samples individual lighting directions on a sphere; weighted composition supplies their combinations under new environments. Paper §3.6 ↗
How the project started
DifFRelight inspired us: its subject-specific diffusion model demonstrated realistic facial performance relighting from flat-lit and OLAT captures. We wanted to extend this direction to full bodies, motion, and unseen people using a pretrained video model.
We initially collected conventional static flat-lit / OLAT pairs. Early experiments showed that their pose coverage was insufficient even for a fixed subject, motivating dynamic capture.
04Motion
Why must we capture and learn through motion?
A few static poses cannot cover the range of motion in a performance. Learning changing light responses calls for dynamic pose coverage and sequence-level prediction.
Capture changing poses.
Full-body motion changes clothing, surface orientation, and self-occlusion throughout the performance.
Build aligned supervision.
We alternate flat-lit and OLAT frames, then use optical flow to align them. The model learns missing directional responses from these partial observations across motion.
Learn sequences while retaining detail.
A video model predicts a directional response for the whole sequence. Latent Bridge Matching starts from the input video’s latents, preserving the person’s identity, textures, and motion as the lighting changes. To further improve detail preservation, we jointly train with static images for sharper supervision.
Static capture holds one pose across a lighting sweep. Dynamic capture alternates Flat-lit and OLAT during motion, then aligns the observations into video pairs.Capture design, long sequences, and consistency
Learn a response across an entire sequence
B̂1:T,k = Fθ(Vflat1:T, ωk)
Fθ predicts an OLAT basis sequence B̂1:T,k for direction ωk, conditioned on the flat-lit sequence Vflat1:T. T is the sequence length.
Capture. We switch the illumination between flat-lit and OLAT while the person moves. Using two spatially distant OLAT directions within each sequence helps disentangle lighting changes from motion. Optical flow between flat-lit frames aligns the intervening OLAT observations into paired video supervision. From these sampled observations, the model learns to predict full directional response sequences. Paper §4 ↗
Long sequences. Independently generated video chunks can flicker at their boundaries. Recurrent bridge context reuses previously generated OLAT frames to constrain overlapping windows, carrying appearance and lighting context beyond the training window. Paper §3.5 ↗
Consistency. Latent Bridge Matching keeps predicted OLAT videos closely aligned with the input performance. A 4D Gaussian Splatting reconstruction built from independently relit, synchronized multi-view videos further illustrates the model’s implicit consistency across views. Supplement §F ↗
Learn directional responses through motion. Compose them for explicit lighting control.
What comes next?
Future work can build on this foundation with more efficient OLAT basis prediction, explicit consistency across directions and views, finer illumination capture, and dynamic lighting benchmarks. The dataset can also support neural rendering and controllable generative video editing. Paper §6 ↗
Some interesting candidate titles for this paper
Neural Light Stage. A proposal-oriented name: we bake the capabilities of a Light Stage into a generative model through fine-tuning.
LoongFlight (飞龙在天). “Learning OLAT Video Prediction for Generalizable Dynamic Full-body Relighting.” The highlighted letters spell the name.
MOONPHASE. “Modeling OLAT Video Prediction for Photoreal Full-body Avatar Sequential Relighting.” The name evokes the Moon under changing directions of sunlight—a natural analogy for OLAT observations.
