N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Models

Yuxin Wang1,2, Lei Ke2, Boqiang Zhang2, Tianyuan Qu2,3, Hanxun Yu2,4, Zhenpeng Huang2,5, Meng Yu2, Dan Xu1✉️, Dong Yu2 1HKUST, 2Tencent AI Lab, 3CUHK, 4ZJU, 5NJU

</p>

https://github.com/user-attachments/assets/077b6a9c-d1b3-4ebe-a9f1-e652b654b7d1

Overview

N3D-VLM is a unified vision-language model for native 3D grounding and 3D spatial reasoning. By incorporating native 3D grounding, our model enables precise spatial reasoning, allowing users to query object relationships, distances, and attributes directly within complex 3D environments.

Updates

2025/12/19: We released this repo with the pre-trained model and inference code.

Installation

git clone --recursive https://github.com/W-Ted/N3D-VLM.git
cd N3D-VLM

conda env create -n n3d_vlm python=3.11 -y
conda activate n3d_vlm
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt

Pre-trained model

We provide the pre-trained model here.

Inference

We provide three examples for inference of N3D-VLM. You could check the source files in data directory, where *.jpg are the source images and *.npz are the monocular point clouds obtained by using MoGe2.

# inference 
python demo.py

Demo 1

https://github.com/user-attachments/assets/e86306f2-152d-4337-a8d2-d165a26ce305

Demo 2

https://github.com/user-attachments/assets/1bc0ee64-7a15-4592-941d-1037a26fb108

Demo 3

https://github.com/user-attachments/assets/ba7ece12-4288-411d-9964-c676b78c6d5c

After running the code above, the inference results will be saved in the outputs directory, including generated answers in *.json format, and 3D grounding results in *.rrd format.
The rrd files can be visualized by using Rerun:

rerun outputs/demo1.rrd

If you want to do the 3D Detection only, please check the example as below.

# inference 
python detection.py
# visualization
rerun outputs/test1.rrd

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Citation

@article{wang2025n3d,
    title={N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Models},
    author={Wang, Yuxin and Ke, Lei and Zhang, Boqiang and Qu, Tianyuan and Yu, Hanxun and Huang, Zhenpeng and Yu, Meng and Xu, Dan and Yu, Dong},
    journal={arXiv preprint arXiv:2512.16561},
    year={2025}
}

README