3DGS Implementation Details

Based on the 3DGS paper [Kerbl et al. 2023]¹, the gsplat doc , and the math supplement [Ye and Kanazawa 2023]²

Representation

Each splat is represented as a 3D Gaussian blob, parameterized as mean (position) \(\mu \in \mathbb R^3\), covariance (isotropic shape) \(\Sigma\in\mathbb R^{3\times 3}\), color \(c\in\mathbb R^3\), and opacity \(o\in\mathbb R\).

The 3D covariance \(\Sigma\) can be further parameterized as per-axis scale \(s\in\mathbb R^3\) and rotation in quaternion \(q\in\mathbb R^4 = (x,y,z,w)\). The quaternion is then converted to rotation matrix \(R\in SO(3)\) as

the 3D covariance \(\Sigma\) is given by

The rendering camera is described by the extrinsic \(R_{c2w}, t_{c2w}\) in the world coordinate space and intrinsic \(f_x, f_y\) (focal length), \(h, w\) (height, width), \(f, n\) (far and near clipping distance).

Following the rasterization pipeline, we transform each splat into the camera space through the world to camera transformation

and then perspective projection

to transform the camera space into normalized clip space (NDC space).

Gaussian projection

For the projection, the position \(\mu\) is transformed into NDC space through the homogeneous transformation

then into pixel coordinates

the covariance \(\Sigma\) is projected using the transformation from EWA Splatting paper [Zwicker et al. 2002]³

and the projection is

Sorting

A naive sorting will sort all splats by the view-space depth \(t_z\). Instead, notice that the size of splats is often few pixels and the frame is thus divided into \(16\times 16\) pixels tiles. We find all overlapping tiles for each splat in the frame space (from 2D position \(t'\) and 2D covariance \(\Sigma'\), we can find the bounding box that bounds \(99\%\) confidence ellipse). Note that some splats may overlap more than one tile, thus we need to duplicate them.

To utilize parallel sorting, assign each tiled splat a key, where first 32 bits encode the tile ID (uint32) and the lower 32 bits encode the depth (float32), and we do a parallel radix_sort on all splats. Then, finds the start and end indices of each tile by simply compare the tile ID with the neighbors.

In general the pseudo-code is

def create_splat_keys(g_frame: ProjectedGaussians, tiled_grid):
    # the number of tiles each splat overlaps, the last index is 
    # for exclusive scan
    n_tiles_overlapped = zeros_arr(len(g_frame) + 1)

    @parallel
    for idx, splat in enumerate(g_frame):
        # integer bbox, for all tiles that overlaps the gaussian
        bbox = find_bbox(splat, tiled_grid)
        n_tiles_overlapped[idx] = bbox.area()

    tiled_splat_key_start = parallel_exclusive_scan(n_tiles_overlapped)

    tiled_splat_keys = empty_arr(tiled_splat_key_start[-1])

    @parallel
    for i, splat in enumerate(g_frame):
        bbox = find_bbox(splat, tiled_grid)
        for j, tile_id in enumerate(bbox):
            key = get_key(tile_id, splat.depth)
            tiled_splat_keys[tiled_splat_key_start[i] + j] = key
    return tiled_splat_keys

def rasterize(cam: Camera, g: Gaussians):
    # cull gaussians outside of the camera frustum
    g_cull = cull_gaussian(g, cams) 
    # project gaussians into 2D frame space
    g_frame = project_gaussians(g, cam)
    # create tiles from the frame height and width
    tiled_grid = create_tiles(cam.w, cam.h)

    # create tiled splat keys
    tiled_splat_keys = create_splat_keys(g_frame, tiled_grid)

    # sort the keys
    sorted_keys = parallel_radix_sort(tiled_splat_keys)
    # find the start and end for each tile
    start_indices, end_indices = parallel_find_edges(sorted_keys)

    @parallel
    for tile in tiled_grid:
        start, end = start_indices[tile], end_indices[tile]
        @parallel
        for pixel in tile:
            pixel = rasterize(sorted_keys[start:end+1])

Composition

Following the classic rendering equation, we need to blend the colors from front to back as

where \(a\), or the conic opacity is computed from opacity parameter \(o\) multiples 2D Gaussian distribution

where \(\mathbf u = (u,v)\) is the pixel center.