Installation

Prerequisites and build configurations

Quick install (one command)

Installs Rust, ffmpeg, COLMAP, and builds rgsplat:

Linux / macOS

bash <(curl -fsSL https://raw.githubusercontent.com/mfaeezshabbir/rgsplat/main/scripts/install.sh)

Windows (PowerShell as Admin)

powershell -ExecutionPolicy Bypass -File https://raw.githubusercontent.com/mfaeezshabbir/rgsplat/main/scripts/install.ps1

Or clone and run locally:

git clone https://github.com/mfaeezshabbir/rgsplat.git
cd rgsplat
bash scripts/install.sh          # Linux / macOS
powershell -File scripts/install.ps1   # Windows

System requirements

Build variants

# CPU-only (default, no GPU driver needed)
    cargo build --release
    
    # With GPU support (wgpu / WGSL compute shaders)
    cargo build --release --features gpu
    
    # With ffmpeg bindings (optional, alternative to subprocess approach)
    cargo build --release --features video
    
    # Full build: GPU + video + Python FFI
    cargo build --release --features full
    
    # Python wheel (requires maturin)
    maturin build --release --features python

Add as library dependency

[dependencies]
    rgsplat = { git = "https://github.com/mfaeezshabbir/rgsplat", features = ["gpu"] }

Quick start

From video to trained Gaussians in one call

CLI

# Basic run (CPU mode)
    rgsplat --input my_scene.mp4 --output ./output
    
    # GPU mode, 2fps extraction, 15k iterations
    rgsplat --input my_scene.mp4 --output ./output --gpu --fps 2 --iters 15000
    
    # White background, checkpoint every 1k iters, skip SfM
    rgsplat --input ./frames --output ./out --bg 1,1,1 --save-interval 1000 --skip-sfm
    
    # View all flags
    rgsplat --help

All CLI flags

Rust API

use rgsplat::pipeline::{Pipeline, PipelineConfig};
    
    #[tokio::main]
    async fn main() -> anyhow::Result<()> {
        let cfg = PipelineConfig::from_video(
            "scene.mp4",
            "./output",
        );
    
        let pipeline = Pipeline::new(cfg).await?;
        let output_ply = pipeline.run().await?;
    
        println!("Done: {:?}", output_ply);
        Ok(())
    }

Customise config

let mut cfg = PipelineConfig::from_video("scene.mp4", "./out");
    cfg.extract_fps = 3.0;          // frames per second to extract
    cfg.use_gpu = true;              // enable GPU if available
    cfg.train.num_iterations = 15_000;
    cfg.train.max_gaussians = 1_000_000;
    cfg.bg_color = [1.0, 1.0, 1.0]; // white background
    cfg.save_interval = 2000;        // checkpoint every 2k iters

Expected output layout

output/
    ├── frames/          # Extracted PNG frames
    │   ├── frame_000001.png
    │   └── ...
    ├── colmap/          # COLMAP workspace
    │   ├── colmap.db
    │   └── sparse/0/
    │       ├── cameras.bin
    │       ├── images.bin
    │       └── points3D.bin
    ├── ckpt_005000.ply  # Periodic checkpoints
    ├── ckpt_010000.ply
    └── output.ply       # Final trained splat

Pipeline stages

What happens inside Pipeline::run()

Stage 1 — Frame extraction

Calls ffmpeg as an async subprocess. Non-blocking: the Tokio runtime handles other tasks while ffmpeg runs.

sfm::extract_frames(&video_path, &frames_dir, fps).await?

Stage 2 — Structure from Motion (COLMAP)

Runs three COLMAP passes in sequence, all as async subprocesses:

Stage 3 — Gaussian initialisation

Parses COLMAP binary files (points3D.bin, cameras.bin) and converts each SfM point into a Gaussian3D:

Stage 4 — Training loop

30 000 iterations (default). Each iteration:

Pick random camera → Render (GPU or CPU) → Compute L1+D-SSIM loss
    → Backward pass (per-Gaussian gradients) → Adam step → [Densify/Prune every 100 iters]

Key events during training:

Stage 5 — Export

Saves a PLY point cloud where each Gaussian is a point with custom properties (position, rotation, scale, opacity, 48 SH coefficients). Compatible with standard Gaussian Splatting viewers.

Module map

Every source file and what it owns

Memory model

How data is stored and moved through the pipeline

Gaussian3D memory layout

Gaussian3D is #[repr(C)] with explicit field ordering to minimise padding and allow zero-copy GPU uploads via bytemuck::Pod.

Allocation strategy

GPU buffer flow

Vec<Gaussian3D>
      → bytemuck::cast_slice (zero-copy view)
      → wgpu STORAGE buffer (COPY_DST | COPY_SRC)
      → compute shader: project pass
      → SplatInfo2D STORAGE buffer
      → radix sort (depth keys)
      → compute shader: rasterize pass
      → texture_storage_2d (rgba8unorm)
      → readback buffer (MAP_READ)
      → Vec<u8> on CPU

Concurrency model

How the pipeline stays non-blocking end-to-end

Thread roles

Key async boundaries

// This blocks the thread for ~minutes — must be spawn_blocking
    let (gaussians, cameras) = tokio::task::spawn_blocking(move || {
        load_sfm_output(&workspace)
    }).await??;
    
    // Render runs in blocking pool, GPU future awaits in async context
    let rendered = task::spawn_blocking(move || {
        cpu::rasterizer::rasterize(&gaussians, &camera, bg)
    }).await?;
    
    // Checkpoint saves don't block the training loop
    task::spawn(async move {
        task::spawn_blocking(move || save_ply(&g, &path)).await
    });

Tile rasterizer zero-contention design

The screen is divided into 16×16 tiles. Each tile writes to a disjoint range of the output pixel buffer. Rayon iterates over tiles in parallel with no mutexes needed — a single unsafe pointer write is used with a safety invariant that's trivially verified (each pixel belongs to exactly one tile).

Gaussian primitives

The mathematical object at the core of 3DGS

The 3D Gaussian function

Each scene primitive is a 3D Gaussian density function centred at μ with covariance Σ:

Covariance factorisation

Σ is parameterised as the product of a rotation matrix R and a diagonal scale matrix S, ensuring Σ is always positive semi-definite:

In code, R is stored as a unit quaternion and S as log-scale (so optimisation can take unconstrained steps):

// Decode scale: always positive
    s = [log_scale[0].exp(), log_scale[1].exp(), log_scale[2].exp()]
    
    // Decode opacity: always in (0, 1)
    α = sigmoid(opacity_logit) = 1 / (1 + exp(−opacity_logit))

Why logit/log parameterisation?

Adam takes unconstrained gradient steps. If we stored opacity directly as a float in (0,1), a gradient step could push it outside the valid range. Storing the logit (inverse sigmoid) means any real number is valid, and we decode to (0,1) at evaluation time. Same for scale — storing log-scale means any real number decodes to a positive scale via exp.

Trainable parameters per Gaussian

Covariance projection (EWA)

Mapping 3D Gaussian ellipsoids to 2D screen splats

The EWA splatting formula

Based on Zwicker et al. 2001. The 2D screen covariance is:

where W is the view rotation (upper-left 3×3 of the view matrix) and J is the affine approximation Jacobian of the perspective projection at the Gaussian centre:

Conic representation

For rasterization, we store the inverse 2×2 covariance (the conic) to avoid a matrix inverse at every pixel:

Gaussian evaluation at pixel (dx, dy) from the centre:

Low-pass anti-aliasing filter

A 0.3-pixel constant is added to the diagonal of Σ′₂D before inversion. This prevents Gaussian splats from becoming sub-pixel (infinitely sharp) and causing aliasing:

sigma_2d[(0,0)] += 0.3;
    sigma_2d[(1,1)] += 0.3;

Screen-space AABB

The tight bounding box of the Gaussian splat (n_sigma = 3, covering 99.7% of density) is computed from the eigenvalues of the 2×2 covariance:

mid = (a + c) / 2
    disc = sqrt(((a−c)/2)² + b²)
    λ₁ = mid + disc  (largest eigenvalue)
    λ₂ = mid − disc
    r = n_sigma × sqrt(max(λ₁, λ₂))

Spherical harmonics

View-dependent colour via degree-3 real SH basis

What SH gives us

Each Gaussian can emit different colour depending on the viewing direction — mimicking specular highlights, subsurface scattering, and non-Lambertian surfaces. The colour for view direction d is:

Coefficient layout

48 floats per Gaussian: 16 SH coefficients × 3 RGB channels, stored channel-major:

sh_coeffs[0..16]  = Red channel (l=0 to l=3)
    sh_coeffs[16..32] = Green channel
    sh_coeffs[32..48] = Blue channel

Degree 0 (DC term)

The degree-0 term gives a view-independent base colour:

Constants

Learning rate

Higher-degree SH coefficients use a 20× smaller learning rate than the DC term, preventing high-frequency colour noise from appearing early in training:

lr_sh_dc   = 0.0025
    lr_sh_rest = lr_sh_dc / 20.0 = 0.000125

Camera & projection

Pinhole camera model, NDC mapping, frustum culling

Pinhole camera model

Perspective projection

World point P → screen pixel (sx, sy):

NDC projection matrix

OpenCV-convention pinhole camera to OpenGL NDC (for wgpu):

Frustum culling

Six planes extracted from the MVP matrix (Gribb-Hartmann method). Sphere test using Gaussian position and bounding radius keeps only visible Gaussians. Reduces GPU/CPU work proportionally to scene complexity.

Gaussian3D

src/math/mod.rs · #[repr(C)] · impl Pod + Zeroable

Fields

Methods

// Constructor: identity rotation, small scale, low opacity
    fn new(position: [f32; 3]) -> Gaussian3D
    
    // Decode opacity from logit space → (0, 1)
    fn opacity(&self) -> f32
    
    // Decode scale from log space → (0, ∞)
    fn scale(&self) -> [f32; 3]
    
    // Decode rotation quaternion as nalgebra UnitQuaternion
    fn rotation_quat(&self) -> UnitQuaternion<f32>
    
    // Build 3×3 world-space covariance Σ = R·S·Sᵀ·Rᵀ
    fn covariance_3d(&self) -> Matrix3<f32>
    
    // Project 3D covariance to 2D screen space
    fn projected_covariance_2d(&self, view: &Matrix4<f32>, focal: (f32,f32)) -> Matrix3<f32>
    
    // Get base (view-independent) colour from DC SH coefficient
    fn base_color(&self) -> [f32; 3]

Camera

src/math/mod.rs · Pinhole camera model

Fields

Methods

// World-to-camera matrix (inverse of c2w)
    fn view_matrix(&self) -> Matrix4<f32>
    
    // Perspective projection matrix for NDC mapping
    fn projection_matrix(&self, near: f32, far: f32) -> Matrix4<f32>
    
    // (fx, fy) focal length pair
    fn focal(&self) -> (f32, f32)

CPU rasterizer

src/cpu/rasterizer.rs · Tile-based, Rayon-parallel CPU

rasterize()

pub fn rasterize(
        gaussians: &[Gaussian3D],
        camera: &Camera,
        bg_color: [f32; 3],
    ) -> Framebuffer

Main entry point. Must be called from tokio::task::spawn_blocking — it blocks the calling thread.

Algorithm

Framebuffer

pub struct Framebuffer {
        pub width: usize,
        pub height: usize,
        pub data: Vec<f32>,  // RGBA linear, row-major
    }
    
    // Convert linear → sRGB u8 for PNG export
    fn to_u8_srgb(&self) -> Vec<u8>

Constants

GPU context

src/gpu/context.rs · wgpu device management GPU

GpuContext::new()

pub async fn new() -> Result<GpuContext>

Requests a high-performance adapter (discrete GPU preferred). Logs the adapter name and backend. Returns error if no compatible GPU is found — callers should fall back to CPU.

Buffer helpers

// Upload Vec data → STORAGE | COPY_DST | COPY_SRC buffer
    fn upload_buffer<T: Pod>(&self, data: &[T], label: &str) -> Buffer
    
    // Allocate empty output buffer
    fn output_buffer(&self, size_bytes: u64, label: &str) -> Buffer
    
    // Staging buffer for GPU→CPU readback
    fn readback_buffer(&self, size_bytes: u64) -> Buffer
    
    // Submit command encoder and poll until complete
    async fn submit_and_wait(&self, encoder: CommandEncoder)
    
    // Read GPU buffer back to CPU Vec
    async fn readback<T: Pod + Clone>(&self, buf: &Buffer, count: usize) -> Vec<T>

gpu_available()

pub async fn gpu_available() -> bool

Safe probe — does not panic if no GPU exists. Use this before calling GpuContext::new() if you want to suppress the error log on headless machines.

Optimizer

src/pipeline/optimizer.rs · Adam + densification

adam_step()

pub fn adam_step(
        gaussians: &mut [Gaussian3D],
        grads: &[GaussianGrad],
        state: &mut AdamState,
        cfg: &TrainConfig,
    )

Updates all Gaussian parameters using the Adam algorithm with bias correction. Quaternions are renormalised after every step. Parallelised over Gaussians with Rayon.

AdamState

pub struct AdamState {
        pub m1: Vec<f32>,          // first moment, shape [N × 59]
        pub m2: Vec<f32>,          // second moment
        pub grad_accum: Vec<f32>,  // accumulated screen gradient norm
        pub grad_denom: Vec<u32>,  // count for average
        pub t: usize,              // iteration (for bias correction)
    }

densify_and_prune()

pub fn densify_and_prune(
        gaussians: &mut Vec<Gaussian3D>,
        state: &mut AdamState,
        cfg: &TrainConfig,
        scene_extent: f32,
    )

GaussianGrad

pub struct GaussianGrad {
        pub d_position:      [f32; 3],
        pub d_rotation:      [f32; 4],
        pub d_log_scale:     [f32; 3],
        pub d_opacity_logit: f32,
        pub d_sh:            [f32; 48],
        pub screen_grad_norm: f32,   // for densification threshold
    }

SfM pipeline

src/pipeline/sfm.rs · Frame extraction, COLMAP, point cloud parsing

extract_frames()

pub async fn extract_frames(
        video_path: &Path,
        output_dir: &Path,
        fps: f32,
    ) -> Result<Vec<PathBuf>>

Spawns ffmpeg as a tokio::process::Command. Non-blocking.

run_colmap()

pub async fn run_colmap(
        image_dir: &Path,
        workspace_dir: &Path,
        use_gpu: bool,
    ) -> Result<()>

Runs three COLMAP commands in sequence. Each is awaited independently — if one fails, the pipeline surfaces the error immediately.

parse_points3d_bin()

pub fn parse_points3d_bin(data: &[u8]) -> Result<Vec<SfMPoint>>

Parses COLMAP's binary points3D.bin format. Track data (image observations) is skipped. Returns SfMPoint with xyz, RGB, reprojection error, and track length.

gaussians_from_sfm()

pub fn gaussians_from_sfm(points: &[SfMPoint]) -> Vec<Gaussian3D>

Converts SfM point cloud to initial Gaussians. Computes per-point initial scale from K=3 nearest-neighbour distances using a spatial sort for O(n log n) complexity.

splat.wgsl

shaders/splat.wgsl · WGSL compute shader · Two-pass GPU pipeline GPU

Entry points

Bindings — project pass (group 0)

Bindings — rasterize pass (group 1)

Depth key encoding

Camera-space depth (positive float) is encoded as a sortable u32 by bit-flipping. This allows a standard ascending radix sort to produce back-to-front order:

// In WGSL
    depth_keys[idx] = ~bitcast<u32>(tz);  // flip all bits → back-to-front order

TrainConfig

All hyper-parameters with defaults from the 3DGS paper

Cargo features

Build-time feature flags

Release profile

GPU vs CPU guide

Choosing the right compute backend for your environment

Decision matrix

Automatic fallback

let pipeline = Pipeline::new(cfg).await?;
    // If GPU init fails, Pipeline logs a warning and continues with CPU.
    // No code change needed.

Manual override

let available = gpu::gpu_available().await;
    cfg.use_gpu = available && std::env::var("GSPLAT_NO_GPU").is_err();

Performance expectations

Python bindings

Using the pipeline from Python via PyO3 / maturin

Build the wheel

pip install maturin
    maturin build --release --features python
    pip install target/wheels/rgsplat-*.whl

Usage

import rgsplat as gs
    import asyncio
    
    async def main():
        cfg = gs.PipelineConfig("scene.mp4", "./output")
        cfg.use_gpu = True
        cfg.train.num_iterations = 15000
    
        pipeline = await gs.Pipeline.new(cfg)
        output_path = await pipeline.run()
        print(f"Done: {output_path}")
    
    asyncio.run(main())

Load and inspect Gaussians

gaussians = gs.load_ply("output/output.ply")
    print(f"Count: {len(gaussians)}")
    print(f"First position: {gaussians[0].position}")
    print(f"First opacity: {gaussians[0].opacity():.3f}")

Performance tuning

Tips for faster training and lower memory use

Reduce scene size

Speed up COLMAP

# Restrict to sequential matching for large frame sets
    rgsplat --input scene.mp4 --output out --colmap-matcher sequential
    
    # Use GPU SIFT (3-5x faster than CPU SIFT)
    rgsplat --input scene.mp4 --output out --gpu

Profile with Tokio Console

RUSTFLAGS="--cfg tokio_unstable" cargo build --release --features full
    RUST_LOG=debug ./target/release/rgsplat --input scene.mp4 --output out

Install tokio-console to visualise async task execution and identify blocking tasks that should be moved to spawn_blocking.

Memory usage estimate

// Rough formula:
    // RAM = N × (240B Gaussian + 2 × 59 × 4B Adam state)
    // For 1M Gaussians: 1M × (240 + 472) ≈ 680MB RAM
    // GPU VRAM: same for buffers + framebuffer (W×H×16B)