QUEEN: QUantized Efficient ENcoding of Dynamic Gaussians for Streaming Free-viewpoint Videos

QUEEN (QUantized Efficient ENcoding) is a novel framework for efficient, streamable free-viewpoint video (FVV) representation using dynamic 3D Gaussians. QUEEN enables high-quality dynamic scene capture, drastically reduces model size (to just 0.7 MB per frame), trains in under 5 seconds per frame, and achieves real-time rendering at ~350 FPS.

This repository contains the official implementation for QUEEN, as introduced in the NeurIPS 2024 paper:

QUEEN: QUantized Efficient ENcoding for Streaming Free-viewpoint Videos
Sharath Girish,
*Tianye Li**,
*Amrita Mazumdar**,
Abhinav Shrivastava,
David Luebke,
Shalini De Mello
NeurIPS 2024

For business inquiries, please visit our website and submit the form: NVIDIA Research Licensing.

Links

• Project Page
• arXiv/Preprint
• QUEEN GTC 2025 Demo Homepage

🔥 News

• 2025/06: Initial code release!
• 2025/03: We showed a live demo of QUEEN in VR and light field displays at GTC San Jose 2025!
• 2024/12: QUEEN was published at NeurIPS 2024.

Contents

• 🔥 News
• Contents
• 🔧 Environment Setup
• Data Preparation
• 💻 Training
• 🎥 Rendering and Evaluation
  • Rendering from Dense 3DGS
  • Rendering from Compressed
• 📋 Pre-trained Models
• 🎓 Citations
• 🙏 Acknowledgements

Code Setup

0. Dependencies

We have only tested on Linux environments with CUDA 11.8+ compatible systems.

1. Clone and Setup Repo

Our software has some submodules, please clone the repo recursively.

git clone --recurse-submodules [email protected]:NVlabs/queen.git queen
cd queen
# set up some relevant directories
mkdir data
mkdir logs
mkdir output

2. Environment Setup

We suggest using the Dockerfile to reproduce the environment. We also provide a conda environment, b

Dockerfile

Please use the Dockerfile for training within a working container.

Conda Environment

# Create the conda environment
conda create -n queen python=3.11
conda activate queen

# Install the package and its dependencies
pip install -e .

# Install CUDA-dependent submodules (must be installed with --no-build-isolation)
pip install --no-build-isolation ./submodules/simple-knn
pip install --no-build-isolation ./submodules/diff-gaussian-rasterization
pip install --no-build-isolation ./submodules/gaussian-rasterization-grad

# Apply timm package patch (required for compatibility)
python scripts/patch_timm.py

Alternative: Manual Installation

If you prefer to install dependencies manually without using setup.py:

# Create and activate conda environment
conda create -n queen python=3.11
conda activate queen

# Install dependencies from requirements.txt
pip install -r requirements.txt

# Install CUDA-dependent submodules
pip install --no-build-isolation ./submodules/simple-knn
pip install --no-build-isolation ./submodules/diff-gaussian-rasterization
pip install --no-build-isolation ./submodules/gaussian-rasterization-grad

# Apply timm package patch
python scripts/patch_timm.py

About the timm Package Patch

This repository uses timm==0.6.13 which requires a patched version of maxxvit.py to fix a compatibility issue. See this GitHub issue for details.

The patched file is included in this repository (maxxvit.py), and the scripts/patch_timm.py script automatically:

• Locates your timm installation
• Creates a backup of the original file (.py.backup)
• Copies the patched version to the correct location

This patching step is already included in both the Dockerfile and the Conda installation instructions above.

3. Download weights for MiDaS

For MiDaS, please download their pretrained weights dpt_beit_large_512.pt from their official repo. We tested with the V3.1 release.

wget -P MiDaS/weights https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_beit_large_512.pt

Data Preparation

We assume datasets are organized as follows:

DyNeRF

| --- data
|   | [dataset_directory]
│     | [scene_name] 
│         | cam01
|            | images
|             | ---0000.png
│             | --- 0001.png
│             | --- ...
│         | cam02
|            | images
│             | --- 0000.png
│             | --- 0001.png
│             | --- ...
│         | ...
│         | sparse_
│             | --- cameras.bin
│             | --- images.bin
│             | --- ...
│         | points3D_downsample2.ply
│         | poses_bounds.npy

To generate the points3D_downsample2.ply , please use the multipleviewprogress.sh script from 4DGaussians.

Google Immersive

| --- data
|   | [dataset_directory]
│     | [scene_name] 
│         | camera_0001
|            | images_scaled_2
|             | ---0000.png
│             | --- 0001.png
│             | --- ...
│         | camera_0001
|            | images_scaled_2
│            | --- 0000.png
│             | --- 0001.png
│             | --- ...
│         | ...
│         | colmap
|            | sparse
|                | 0  
|             | --- points3d.bin
│             | --- cameras.bin
│             | --- images.bin

If you have a dataset of multi-view videos, please follow the steps to produce the required data:

1. organize the videos into folders of images for each camera view
2. follow the instructions in the gaussian-splatting codebase to create the COLMAP camera data at the sparse_ directory

For more information on how datasets are loaded, please see scene/dataset_readers.py.

💻 Training

You can train a scene by running:

python train.py --config [config_path] -s [source_path] -m [output_name]

For example:

python train.py --config configs/dynerf.yaml -s data/dynerf/coffee_martini -m ./output/coffee_martini_trained

The training script builds on the original 3DGS training script, and, as such, shares many of the same command line arguments. We add new arguments to control compression hyperparameters and 3DGS training hyperparameters for the initial frame and residual frames.

Please see specific configuration files in configs for examples, and arguments/__init__.py for the full list of arguments.

Useful Command Line Arguments for train.py

--source_path / -s

Path to the source directory containing a COLMAP or Synthetic NeRF data set.

--model_path / -m

Path where the trained model should be stored (output/ by default).

--white_background / -w

Add this flag to use white background instead of black (default), e.g., for evaluation of NeRF Synthetic dataset.

--sh_degree

Order of spherical harmonics to be used (no larger than 3). 3 by default.

--max_frames

Maximum number of frames to process, 300 by default.

--log_images

Flag to save rendered images during training.

--log_ply

Flag to save point cloud in PLY format during training.

--log_compressed

Flag to save compressed model during training.

--save_format

Format to save the model in, ply by default.

--use_xyz_legacy

Flag to use the _xyz implementation to reproduce the paper results. Note that using the legacy implementation with --log_compressed is unsupported.

🎥 Rendering and Evaluation

Rendering from Dense 3DGS

python render.py -s  -m  # Generate renderings
python metrics.py -m  # Compute error metrics on renderings

We also provide a script to render spiral viewpoints.

python render_fvv.py  -s  -m  # Generate renderings

Example usage:

# Train coffee-martini scene
python train.py --config configs/dynerf.yaml --log_images --log_ply -s data/dynerf/coffee_martini -m ./output/coffee_martini_trained 

# Compute metrics
python metrics_video.py -m ./output/coffee_martini_trained 

# Render static camera viewpoints and spiral
python render.py -s data/dynerf/coffee_martini -m ./output/coffee_martini_trained
python render_fvv.py --config configs/dynerf.yaml  -s data/dynerf/coffee_martini -m ./output/coffee_martini_trained

Rendering from Compressed

After training with --log_compressed to save compressed scenes, we can decode and render frames.

python render_fvv_compressed.py --config configs/dynerf.yaml [any additional config flags] -s  -m  # Generate renderings

If you provided any additional command-line arguments during training to adjust the training configuration, you may need to add them to properly generate the renderings, as well.

📋 Pre-trained Models

We provide pre-trained models for the N3DV dataset. The provided models are subject to the Creative Commons — Attribution-NonCommercial-ShareAlike 4.0 International — CC BY-NC-SA 4.0 License terms. The download links for these models are provided in the table below, or on the Models documentation. Please use the "render from compressed" instructions to render from the compressed pkl's.

Note: we provide two versions of our models -- "NeurIPS24" and "Compressed." The "NeurIPS24" version includes only the dense .ply files corresponding to the training runs reported in the paper, while the "Compressed" version contains both dense .ply files and compressed .pkl representations. We identified and corrected a bug before code release that affected the rendering from the compressed .pkl files. This fix has resulted in a slight change in the results (we include a training flag, --use_xyz_legacy to train with or without the fix).

NeurIPS24 -- Paper Models (download)

Compressed -- Corrected Compressed Models (download)

🎓 Citation

If you find QUEEN useful in your research, please cite:

@inproceedings{
girish2024queen,
  title={{QUEEN}: {QU}antized Efficient {EN}coding for Streaming Free-viewpoint Videos},
  author={Sharath Girish and Tianye Li and Amrita Mazumdar and Abhinav Shrivastava and David Luebke and Shalini De Mello},
  booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
  year={2024},
  url={https://research.nvidia.com/labs/amri/projects/queen/}
}

🙏 Acknowledgements

QUEEN builds upon the original gaussian-splatting codebase, and uses the pretrained MiDaS model for depth estimation. We thank the authors for their contributions.