VideoMatGen: PBR Materials through Joint Generative Modeling

We present a method for generating physically-based materials for 3D shapes based on a video diffusion transformer architecture. Our method is conditioned on input geometry and a text description, and jointly models multiple material properties (base color, roughness, metallicity, height map) to form physically plausible materials. We further introduce a custom variational auto-encoder which encodes multiple material · Authors: Jon Hasselgren, Zheng Zeng, Milos Hasan, Jacob Munkberg · Published 2026-03-17 · arXiv:2603.16566

NVIDIA

Paper

Published: 2026-03-17
Authors: 4
arXiv: 2603.16566
Category: cs.CV

Abstract

arXiv:2603.16566

Jon Hasselgren, Zheng Zeng, Milos Hasan, Jacob Munkberg

We present a method for generating physically-based materials for 3D shapes based on a video diffusion transformer architecture. Our method is conditioned on input geometry and a text description, and jointly models multiple material properties (base color, roughness, metallicity, height map) to form physically plausible materials. We further introduce a custom variational auto-encoder which encodes multiple material modalities into a compact latent space, which enables joint generation of multiple modalities without increasing the number of tokens. Our pipeline generates high-quality materials for 3D shapes given a text prompt, compatible with common content creation tools.

Abstract

Related papers