Gaussian Splatting Monitor

Pre-built Viewer for Windows

This repository builds upon the "3D Gaussian Splatting for Real-Time Radiance Field Rendering" project by adding new features that enhance its utility for research purposes. Originally, Gaussian splatting excells in producing high-quality renderings but is constrained to only rendering RGB images and backpropagating gradients based on RGB loss. This limitation hindered the potential for investigating the volumetric analysis of the Gaussian Splatting (GS) model and the development of novel loss functions. In contrast, models derived from Neural Radiance Fields (NeRF) leverage their fully connected MLP architectures to offer greater versatility in processing various input and output features, as well as in crafting loss functions. Inspired by these advancements, this codebase supports additional differentiable outputs, aiming to spur innovative research ideas.

Applications

Welcome to use GS Monitor for your GS-based projects and let us enrich the README application gallery.👏

The repository supports

• AtomGS: Atomizing Gaussian Splatting for High-Fidelity Radiance Field

AtomGS.mp4

• 2D Gaussian Splatting for Geometrically Accurate Radiance Fields

2DGS.Viewer.mp4

• Feature 3DGS 🪄: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields

Feature.3DGS.mp4

New Features

The latest updates enhance the existing Gaussian Splatting (GS) codebase while maintaining compatibility:

Expandable Viewer

The viewer is now customizable, allowing for visualization of various render items. Ensure that the rendered image adheres to the formats (1, h, w) or (3, h, w), with single-channel images automatically converting to the turbo colormap. To configure the render items, modify the render_items list in arguments/__init__.py:

self.render_items = ['RGB', 'Alpha', 'Depth', 'Normal', 'Curvature', 'Edge']

Then implement the calculation for each item in the render_net_image() function located in utils/image_utils.py.

Metrics Viewer

View metrics directly within the viewer, eliminating the need to switch between the viewer and the terminal for RGB effects and loss metrics. Configure the metrics dictionary as follows in train.py or view.py:

metrics_dict = {
    "iteration": iteration,
    "number of gaussians": gaussians.get_xyz.shape[0],
    "loss": loss,
    # Add more metrics as needed
}

Implementation of Five Additional Features

We have added five new features: 'Alpha', 'Depth', 'Normal', 'Curvature', and 'Edge'. These are designed to demonstrate the capabilities of the GS Monitor and are applicable across all GS models.

Setup

This codebase builds upon the original GS repository and maintains compatibility with it. Therefore, if you want to set up the repository smoothly or you face some errors, we strongly advise you to explore the video tutorial, review the issues, and check the FAQ section. This may help you identify if your concern is a known issue and, ideally, lead you to a solution.

Installation

git clone https://github.com/RongLiu-Leo/Gaussian-Splatting-Monitor.git
cd Gaussian-Splatting-Monitor
conda env create --file environment.yml
conda activate gs_monitor

Please note that this process assumes that you have CUDA SDK 11 installed, not 12.

Interactive Viewers

Remote Viewer and Gaussian Viewer are integrated into one Viewer and it is driven by train.py or view.py. We provide pre-built binaries for Windows here for an efficient setup. If your OS is Ubuntu 24.04, you need to compile the viewer locally:

# Dependencies
sudo apt install -y libglew-dev libassimp-dev libboost-all-dev libgtk-3-dev libopencv-dev libglfw3-dev libavdevice-dev libavcodec-dev libeigen3-dev libxxf86vm-dev
# download and unpack https://github.com/RenderKit/embree/releases/download/v3.13.5/embree-3.13.5.x86_64.linux.tar.gz
# Project setup
cd SIBR_viewers
cmake -Bbuild . -DCMAKE_BUILD_TYPE=Release -Dembree_DIR:PATH=/path/to/embree-3.13.5.x86_64.linux/lib/cmake/embree-3.13.5/ # add -G Ninja to build faster
cmake --build build -j24 --target install

How to use

Firstly open the viewer,

<path to downloaded/compiled viewer>/bin/SIBR_remoteGaussian_app_rwdi.exe

and then

# Monitor the training process
python train.py -s <path to COLMAP or NeRF Synthetic dataset> 
# View the trained model
python view.py -s <path to COLMAP or NeRF Synthetic dataset> -m <path to trained model> 

Acknowledgments

This repository owes its foundation to the original GS repository and incorporates CUDArasterater code from diff-gaussian-rasterization. We are grateful to the original authors for their open-source codebase contributions.

Citation

If you find our code or paper helps, please consider giving us a star or citing:

@misc{liu2024atomgs,
    title={AtomGS: Atomizing Gaussian Splatting for High-Fidelity Radiance Field}, 
    author={Rong Liu and Rui Xu and Yue Hu and Meida Chen and Andrew Feng},
    year={2024},
    eprint={2405.12369},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://rongliu-leo.github.io/AtomGS/}
}

License

This project is licensed under the Gaussian-Splatting License - see the LICENSE file for details.