ParticleSfM

Paper | Video | Project Page

Code release for our ECCV 2022 paper "ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild." by Wang Zhao, Shaohui Liu, Hengkai Guo, Wenping Wang and Yong-Jin Liu.

[Introduction] ParticleSfM is an offline structure-from-motion system for videos (image sequences). Inspired by Particle video, our method connects pairwise optical flows and optimizes dense point trajectories as long-range video correpondences, which are used in a customized global structure-from-motion framework with similarity averaging and global bundle adjustment. In particular, for dynamic scenes, the acquired dense point trajectories can be fed into a specially designed trajectory-based motion segmentation module to select static point tracks, enabling the system to produce reliable camera trajectories on in-the-wild sequences with complex foreground motion.

Contact Wang Zhao ([email protected]), Shaohui Liu ([email protected]) and Hengkai Guo ([email protected]) for questions, comments and reporting bugs.

If you are interested in potential collaboration or internship at ByteDance, please feel free to contact Hengkai Guo ([email protected]).

Update by 2025.02.05

We support GLOMAP in our pipeline, which achieves more accurate results on 13 sequences of the Sintel dataset:

Test it by simply changing the sfm_type to global_glomap:

python run_particlesfm.py --image_dir /path/to/the/image/folder/ \
                          --output_dir /path/to/output/workspace/ \
                          --sfm_type global_glomap  # "global_theia" for the paper version

Installation

1. Install dependencies:

• Ceres >= 2.0.0

For using gcolmap (Theia) as in the original ParticleSfM paper:

• COLMAP point trajectory -> motion seg -> sfm`. The final results will be saved inside the image data folder with COLMAP output format.

  If you have the prior information that the scene to be reconstructed is fully static, you can skip the motion segmentation module with --assume_static. Conversely, if you only want to run the motion segmentation, attach --skip_sfm to the command.

  To speed up

  • Use "--skip_path_consistency" to skip the path consistency optimization of point trajectories
  • Try higher down-sampling ratio for optimizing point trajectories: e.g. "--sample_ratio 4"

3. Visualize the outputs using COLMAP GUI (Download the COLMAP Binary and import the data sequence directory) or just your customized visualizer.

Evaluation

MPI Sintel dataset

1. Download the Sintel dataset. You also need to download the groundtruth camera motion data and the generated motion mask to evaluate the pose and motion segmentation.

2. Prepare the sequences:

   python scripts/prepare_sintel.py --src_dir /path/to/your/sintel/training/final/ \
                                 --tgt_dir /path/to/the/data/root/dir/want/to/save/

3. Run ParticleSfM reconstructions:

   python run_particlesfm.py --root_dir /path/to/the/data/root/dir/

4. To evaluate the camera poses:

   python ./evaluation_evo/eval_sintel.py --input_dir /path/to/the/data/root/dir/ \
                                       --gt_dir /path/to/the/sintel/training/data/camdata_left/ \
                                       --dataset sintel

   This will output a txt file with detailed error metrics. Also, the camera trajectories are plotted and saved inside each data sequence folder.

5. To evaluate the motion segmentation:

   python ./motion_seg/eval_traj_iou.py --root_dir /path/to/the/data/root/dir/ \
                                     --gt_dir /path/to/the/sintel/rigidity/

ScanNet dataset

1. Download the test split of ScanNet dataset, extract the data from .sens data using the official script.

2. Prepare the sequences:

   python scripts/prepare_scannet.py --src_dir /path/to/your/scannet/test/scans_test/ \ 
                                  --tgt_dir /path/to/the/data/root/dir/want/to/save/

   We use the first 20 sequences of test split and perform downsampling with stride 3, resize the image to 640x480.

3. Run ParticleSfM reconstructions:

   python run_particlesfm.py --root_dir /path/to/the/data/root/dir/ \
                          --flow_check_thres 3.0 --assume_static

4. To evaluate the camera poses:

   python ./evaluation_evo/eval_scannet.py --input_dir /path/to/the/data/root/dir/ \
                                        --gt_dir /path/to/the/scannet/test/scans_test/ \
                                        --dataset scannet

This will output a txt file with detailed error metrics. Also, the camera trajectories are plotted and saved inside each data sequence folder.

Training

1. Download the Flyingthings3D dataset from the official website. We need the RGB images (finalpass) and optical flow data.

2. Download the generated binary motion labels from here or GoogleDrive, and unpack this archive into the root directory of the FlyingThings3D dataset. We thank the authors of MPNet for kindly sharing it.

3. Prepare the training data:

   python ./scripts/prepare_flyingthings3d.py --src_dir /path/to/your/flyingthings3d/data/root/

4. To launch the training, configure your config file inside ./motion_seg/configs/ and then run:

   cd ./motion_seg/
   python train_seq.py ./configs/your-config-file
   cd ..

Applications

• Motion Control for Video Generation: MotionCtrl, CamCo
• Motion Evaluation for Video Generation: AnimateAnything, AC3D
• Kinematic Control Annotation: EgoVid-5M

Citation

@inproceedings{zhao2022particlesfm,
      author    = {Zhao, Wang and Liu, Shaohui and Guo, Hengkai and Wang, Wenping and Liu, Yong-Jin},
      title     = {ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild},
      booktitle = {European conference on computer vision (ECCV)},
      year      = {2022}
  }

More related projects

• DynaSLAM. Bescos et al. DynaSLAM: Tracking, Mapping and Inpainting in Dynamic Scenes. IROS 2018.
• TrianFlow. Zhao et al. Towards Better Generalization: Joint Depth-Pose Learning without PoseNet. CVPR 2020.
• VOLDOR. Min et al. VOLDOR-SLAM: For the times when feature-based or direct methods are not good enough. ICRA 2021.
• DROID-SLAM. Teed et al. DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras. NeurIPS 2021.

Acknowledgements

This project could not be possible without the great open-source works from COLMAP, Theia, hloc, RAFT, MiDaS and OANet. We sincerely thank them all.