Estimating Physically-Based Material Representations from Diffusion-Generated Lighting Variations

Recovering physically-based material properties (albedo, roughness, metallic) from a single photograph is a classic ill-posed problem: any dark pixel could be a dark surface or a poorly-lit one. Prior diffusion-based approaches try to regress these material maps directly, but since real ground-truth PBR data doesn’t exist, those models are trained only on synthetic interiors — and they generalize poorly to real photos.

We flip the approach: instead of asking a diffusion model to predict materials, we use it to generate new observations. From a single input image, a large image-editing model (Flux.2 Klein or GPT-Image-2) produces several relit versions of the same scene, alongside a diffusion-based normal map estimate.

These generated images become a synthetic multi-illumination dataset, similar to lab-captured datasets like MIT’s Multi-Illumination collection — except it only needs one input photo.

With this dataset in hand, a classical optimization takes over. A differentiable Cook-Torrance renderer, driven by an L-BFGS optimizer, jointly fits per-pixel albedo/roughness/metallic and per-image spherical-harmonic lighting to reconstruct all the generated views. Because the SH lighting is truncated at band ℓ≤3, both the diffuse and specular BRDF terms admit closed-form solutions — no numerical integration needed at render time, which keeps the optimization tractable.

The naive version of this optimization bakes shading artifacts into the albedo map. Four regularization priors fix this: metallic sparsity, SAM2-guided per-object cohesion, spatial smoothness, and a monochromatic-light constraint. This costs a little reconstruction accuracy but produces much more physically plausible material maps.

On self-rendered 3D-Front scenes with known ground truth, the method achieves near-zero reconstruction error and low relighting error when using its own renderer; errors grow under a bigger model gap (Blender Cycles, or the higher-detail NFINITE dataset). On real photos, where no ground truth exists, qualitative comparisons show the full pipeline producing more natural relit images than direct diffusion baselines like Marigold and RGB→X — attributed to leveraging a foundation model’s knowledge of real-world lighting rather than only synthetic training data.

This page provides a high-level overview of the project. For detailed information about the dataset, training setup, experimental evaluation, results, and future work, please refer to the full report.