Fig 1.: SplatAD is the first method capable of realistic camera and lidar rendering using 3D Gaussian Splatting. It enables real-time, high-quality rendering, and can reach competitive performance within minutes of training.

---

Abstract

Ensuring the safety of autonomous robots, such as self-driving vehicles, requires extensive testing across diverse driving scenarios. Simulation is a key ingredient for conducting such testing in a cost-effective and scalable way. Neural rendering methods have gained popularity, as they can build simulation environments from collected logs in a data-driven manner. However, existing neural radiance field (NeRF) methods for sensor-realistic rendering of camera and lidar data suffer from low rendering speeds, limiting their applicability for large-scale testing. While 3D Gaussian Splatting (3DGS) enables real-time rendering, current methods are limited to camera data and are unable to render lidar data essential for autonomous driving. To address these limitations, we propose SplatAD, the first 3DGS-based method for realistic, real-time rendering of dynamic scenes for both camera and lidar data. SplatAD accurately models key sensor-specific phenomena such as rolling shutter effects, lidar intensity, and lidar ray dropouts, using purpose-built algorithms to optimize rendering efficiency. Evaluation across three autonomous driving datasets demonstrates that SplatAD achieves state-of-the-art rendering quality with up to +2 PSNR for NVS and +3 PSNR for reconstruction while increasing rendering speed over NeRF-based methods by an order of magnitude. Code to be released upon publication.

---

Videos

Image and lidar rendering

SplatAD can render both images and lidar point clouds. The videos demonstrate rendered images, bird’s-eye views of the rendered point cloud, and point clouds overlaid on images, with points colored based on range and intensity.

Note: Videos have been compressed to reduce file size.

Sequence browser

Note: Videos have been compressed to reduce file size.

Trajectory editing

Note: Videos have been compressed to reduce file size.

---

Method

SplatAD can render both sensor-realistic lidar and camera data from the same 3D Gaussian representation. For lidar rendering, we:

1. Project the 3D Gaussians to spherical coordinates.
2. Intersect Gaussians with non-equdistant tiles, matching to the lidar beam distribution.
3. Rasterize depth and features for a non-linear grid of points, with rolling shutter compensation.
4. Decode features to lidar intensity and lidar ray drop probability.

For efficiency, we implement these operations using custom CUDA kernels.

For the camera rendering, we modify the standard 3DGS pipeline:

• Rasterize RGB and features, which are decoded to view-dependent colors using a small CNN.
• Apply rolling shutter compensation.

---

Comparisons

Fig. 2: SplatAD vs NeuRAD NVS rendering on the PandaSet dataset.

Fig. 3: SplatAD vs Street Gaussians NVS rendering on the Argoverse2 dataset.

Fig. 4: SplatAD vs NeuRAD NVS rendering on the Argoverse2 dataset.

Fig. 5: SplatAD vs Street Gaussians NVS rendering on the nuScenes dataset.

Fig 6.: SplatAD learns both sharp and accurate geometries, in constrast to NeuRADs oversmoothed results, and Steet Gaussians cutting through objects due to erroneous line-of-sight.

---

BibTeX

@article{hess2024splatad,
  title        = {SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Driving},
  author       = {Hess, Georg and Lindstr{\"o}m, Carl and Fatemi, Maryam and Petersson, Christoffer and Svensson, Lennart},
  journal      = {arXiv preprint arXiv:2411.16816},
  year         = {2024}
}