English | 简体中文

LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequences

Ao Liang     Youquan Liu     Yu Yang     Dongyue Lu     Linfeng Li Lingdong Kong     Huaici Zhao     Wei Tsang Ooi

<p align="center">
<a href="https://arxiv.org/abs/2508.03692" target='_blank'>
<img src="https://img.shields.io/badge/Paper-%F0%9F%93%96-darkred">
</a>&nbsp;
<a href="https://lidarcrafter.github.io/" target='_blank'>
<img src="https://img.shields.io/badge/Project-%F0%9F%94%97-blue">
</a>&nbsp;
<a href="https://huggingface.co/datasets/Pi3DET/data" target='_blank'>
<img src="https://img.shields.io/badge/Dataset-%F0%9F%94%97-green">
</a>&nbsp;
<a href="" target='_blank'>
<img src="https://visitor-badge.laobi.icu/badge?page_id=lidarcrafter.toolkit">
</a>
</p>

In this work, we introduce LiDARCrafter, a unified framework for 4D LiDAR generation and editing. We contribute:

• The first 4D generative world model dedicated to LiDAR data, with superior controllability and spatiotemporal consistency.
• We introduce a tri-branch 4D layout conditioned pipeline that turns language into an editable 4D layout and uses it to guide temporally stable LiDAR synthesis.
• We propose a comprehensive evaluation suite for LiDAR sequence generation, encompassing scene-level, object-level, and sequence-level metrics.
• We demonstrate best single-frame and sequence-level LiDAR point cloud generation performance on nuScenes, with improved foreground quality over existing methods.

:books: Citation

If you find this work helpful for your research, please kindly consider citing our paper:

@inproceedings{liang2026lidarcrafter,
    title     = {{LiDARCrafter}: Dynamic {4D} World Modeling from {LiDAR} Sequences},
    author    = {Ao Liang and Youquan Liu and Yu Yang and Dongyue Lu and Linfeng Li and Lingdong Kong and Huaici Zhao and Wei Tsang Ooi},
    booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)},
    volume    = {40},
    year      = {2026},
}

Updates

• [11.2025] - LiDARCrafter has been accepted to AAAI 2026 for Oral Presentation. :tada:
• [10.2025] - We will soon start organizing the code. All pretrained weights for evaluation can be found at Hugging Face.
• [08.2025] - The technical report of LiDARCrafter is available on arXiv.

Outline

• Updates
• Outline
• :gear: Installation
• :hotsprings: Data Preparation
• :rocket: Getting Started
  • Evaluation
• :wrench: Generation Framework
  • Overall Framework
  • 4D Layout Generation
  • Single-Frame Generation
• :snake: Model Zoo
• :memo: TODO List
• License
• Acknowledgements

:gear: Installation

Please configure your environment according to the version information in environment.yml.

:hotsprings: Data Preparation

• Create dataset: same as DrivingDiffusion

  ln -s ${ROOT_DATA_PATH} ./data/nuscenes

Run bash scripts/create_data.sh for generate:

• info with track and state

• Updated pkl with scene graph

• CLIP feature of scene graph

The file-tree of data is like:

data
├── clips
│   └── nuscenes
│       ├── obj_text_feat.pkl
│       ├── train
│       └── val
├── infos
│   ├── needed_5_framed_token.pkl
│   ├── nuscenes_dbinfos_10sweeps_withvelo.pkl
│   ├── nuscenes_infos_10sweeps_train.pkl
│   ├── nuscenes_infos_10sweeps_val.pkl
│   ├── nuscenes_infos_lidargen_train.pkl
│   ├── nuscenes_infos_lidargen_val.pkl
│   ├── nuscenes_infos_train.pkl
│   ├── nuscenes_infos_val.pkl
│   ├── nuscenes_object_classification_train.pkl
│   └── nuscenes_object_classification_val.pkl
└── nuscenes

:rocket: Getting Started

Evaluation

• Train classification model
  • python train/train_classification_pointmlp.py
• Train uncertainty model
  • python train/train_uncertainty_glenet.py

For each generated 1w model

• Extract foreground samples
  • python evaluation/extract_foreground_samples.py --model ori

:wrench: Generation Framework

Overall Framework

4D Layout Generation

Single-Frame Generation

:snake: Model Zoo

To be updated.

:memo: TODO List

• [x] Initial release. 🚀
• [x] Release the training code.
• [x] Release the inference code.
• [x] Release the evaluation code.
• [ ] Refine the Readme.md

License

This work is under the <a rel="license" href="https://www.apache.org/licenses/LICENSE-2.0">Apache License Version 2.0</a>, while some specific implementations in this codebase might be under other licenses. Kindly refer to LICENSE.md for a more careful check, if you are using our code for commercial matters.

Acknowledgements

This work is developed based on the MMDetection3D codebase.

<img src="https://github.com/open-mmlab/mmdetection3d/blob/main/resources/mmdet3d-logo.png" width="31%"/><br>
MMDetection3D is an open-source toolbox based on PyTorch, towards the next-generation platform for general 3D perception. It is a part of the OpenMMLab project developed by MMLab.

Part of the benchmarked models are from the OpenPCDet and 3DTrans projects.

Related Projects