Map-Det3D — What's That AI

# Map-Det3D: Metric Feed-Forward 3D Reconstruction Prior for Multi-view 3D Object Detection from Streaming Inputs

Map-Det3D: Metric Feed-Forward 3D Reconstruction Prior for Multi-view 3D Object Detection from Streaming Inputs \
Yung-Hsu Yang, Luigi Piccinelli, Samuel Rota Bulò, Sunghwan Hong, Denis Rozumny, Johannes Schönberger, Zuria Bauer, Hermann Blum, Peter Kontschieder, and Marc Pollefeys \
ECCV 2026,
Paper at arXiv 2608.12179

News and ToDo

[x] 17.08.2026: Release code and models.
[x] 18.06.2026: Map-Det3D is accepted at ECCV 2026!

Getting Started

Try our HuggingFace Demo with your own data directly!

Installation

We support Python 3.11+ and PyTorch 2.8.0+.
Please install the correct PyTorch version according to your own hardware settings.

conda create -n mapdet3d python=3.11 -y

conda activate mapdet3d

# Install PyTorch
pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu126

# Install CUDA ops
pip install git+https://github.com/SysCV/vis4d_cuda_ops.git --no-build-isolation --no-cache-dir

# Install Map-Det3D
pip install -v -e .

Model

We host our model on HuggingFace and provide the demo.py as the example.

import torch

from mapdet3d.model.mapdet3d import MapDet3D
from mapdet3d.op.mapdet3d.head import RoI2Det

device = "cuda" if torch.cuda.is_available() else "cpu"

# TF32
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True

torch.set_float32_matmul_precision("highest")

# Init model
model = MapDet3D.from_pretrained("RoyYang0714/Map-Det3D").to(device)

# (Optional) Enable tracking -> the output 3D bounding boxes will be in world coordinate if enable tracking
model.track_whole_scene = True
model.roi2det = RoI2Det(nms=True, score_threshold=0.25, iou_threshold=0.5)

# Inference
model.eval()

with torch.no_grad():

with torch.autocast("cuda", enabled=True, dtype=torch.bfloat16):
    predictions: MapDet3DOut = model(
        images=[image],
        intrinsics=[intrinsics],
        extrinsics=[extrinsics],
        frame_ids=[frame_id],
    )

The model weight can also be download as map-det3d-ca1m.pt.

Data

CA-1M

We use CA-1M as the training and in-domain testing sets.

Download the train.txt and val.txt from here, and put them under data/CA1M
Use the provided script to download the data and unzip them.

python scripts/ca1m/download.py --split train
python scripts/ca1m/download.py --split val

It will download the full CA1M train and val data under data/CA1M.

Convert the dataset for training and testing.

python scripts/ca1m/convert.py --split train
python scripts/ca1m/convert.py --split val

It will parse the dataset and save the cached files and HDF5 under data/ca1m.

(Optional) Mesh file.

The mesh is for the visualization purpose. You can download CA-1M mesh from BoxFusion or just generate them with open3d.

The final data structure should be like this:

REPO_ROOT
├── data
│   ├── CA1M
│   │   ├── train.txt
│   │   ├── val.txt
│   │   ├── train
│   │   │   ├── $SEQ_NAME
│   │   │   ├── ...
│   │   └── val
│   └── ca1m
│       ├── cache
│       ├── mesh
│       │   ├── $SEQ_NAME
│       │   │   └── mesh.ply
│       │   ├── ...
│       ├── train
│       │   ├── $SEQ_NAME.hdf5
│       │   ├── ...
│       └── val
├── ...

ScanNet

We follow BoxFusion and use ScanNet as the out-of-domain testing sets.

Download ScanNet and use the script to extract the images, depth, intrinsics, and poses according to the val.txt:

REPO_ROOT
├── data
│   ├── scannet
│   │   ├── data
│   │   │   └── $SEQ_NAME
│   │   │   │   ├── frames
│   │   │   │   │   ├── colors
│   │   │   │   │   ├── ...
│   │   │   │   ├── ...
│   │   └── meta_data
├── ...

Extract annotations.

python scripts/scannet/batch_load_scannet_data.py
python scripts/scannet/batch_load_scannet_data.py --scannet200

Convert the data for testing.

python scripts/scannet/convert.py
python scripts/scannet/convert.py --scannet200

The final data structure should be like this:

REPO_ROOT
├── data
│   ├── scannet
│   │   ├── cache
│   │   ├── data
│   │   ├── meta_data
│   │   ├── scannet_instance_data
│   │   └── scannet200_instance_data
├── ...

Training

# 2 nodes and 8 gpus each node 
mapdet3d fit --config mapdet3d/zoo/mapdet3d/mapdet3d_ca1m.py --gpus 8 --nodes 2

The output will be dumped under ./work_dir/${experiment_name}/${version}.

You can also enable wandb logging by adding --wandb.

Testing

# CA1M
mapdet3d test --config mapdet3d/zoo/mapdet3d/mapdet3d_ca1m.py --gpus 1 --ckpt https://huggingface.co/RoyYang0714/map-det3d-ca1m/resolve/main/map-det3d-ca1m.pt

# ScanNet200
mapdet3d test --config mapdet3d/zoo/mapdet3d/mapdet3d_scannet200.py --gpus 1 --ckpt https://huggingface.co/RoyYang0714/map-det3d-ca1m/resolve/main/map-det3d-ca1m.pt

# Tracking
mapdet3d test --config mapdet3d/zoo/mapdet3d/mapdet3d_track_scannet.py --gpus 1 --ckpt https://huggingface.co/RoyYang0714/map-det3d-ca1m/resolve/main/map-det3d-ca1m.pt

We provide the rerun visualization.
Enable it with --vis flag, and the .rrd file will be saved under ./work_dir/${experiment_name}/${version}/rerun_vis folder.

Citation

If you find our work useful in your research, please consider citing our publications:

@misc{yang2026mapdet3dmetricfeedforward3d,
      title={Map-Det3D: Metric Feed-Forward 3D Reconstruction Prior for Multi-view 3D Object Detection from Streaming Inputs}, 
      author={Yung-Hsu Yang and Luigi Piccinelli and Samuel Rota Bulò and Sunghwan Hong and Denis Rozumny and Johannes Schönberger and Zuria Bauer and Hermann Blum and Peter Kontschieder and Marc Pollefeys},
      year={2026},
      eprint={2608.12179},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2608.12179}, 
}

Acknowledgements

This project builds upon Vis4D, 3D-MOOD, BoxFusion, and MapAnything.
We thank the authors of these projects for making their code available.

README