Online 3D reconstruction

ReCoSplat: Online Feed-Forward Gaussian Splattingvia Render-and-Compare

Freeman Cheng Botao Ye Xueting Li Junqi You Fangneng Zhan Ming-Hsuan Yang

Real-time online feed-forward Gaussian Splatting from posed or unposed image streams via render-and-compare conditioning and KV-cache compression.

Causal processing Each update uses only current and past observations.

Anytime rendering An explicit Gaussian scene is available after every update.

Posed or unposed Camera poses and intrinsics can be provided or estimated.

Abstract

Online novel view synthesis requires a model to reconstruct a scene causally from a stream of observations while keeping it renderable at every moment. We present ReCoSplat, an online feed-forward Gaussian Splatting model supporting both posed and unposed inputs, with or without camera intrinsics. While assembling local Gaussians with camera poses scales better than canonical-space prediction, stable training requires ground-truth poses, creating a distribution mismatch when predicted poses are used at inference. To address this, we introduce a Render-and-Compare module that renders the accumulated scene from the viewpoint of the incoming observation and compares the render with the observation to produce a stable conditioning signal. For long sequences, a hybrid KV-cache compression strategy combines early-layer truncation with chunk-level selective retention, reducing the KV cache by more than 90% for streams of 100 or more frames.

Method

Reconstruct, compare, and update

Each incoming chunk updates both the compressed history and the explicit Gaussian scene.

ReCoSplat pipeline showing the causal backbone, pose prediction, Render-and-Compare module, local Gaussian prediction, and scene update.

Estimate the camera

The pose head predicts camera parameters when they are not provided.

Render and compare

The accumulated scene is rendered at the assembly pose and paired with the incoming observation.

Assemble the scene

New local Gaussians are transformed into world coordinates and merged into the scene.

Quantitative results

Novel view synthesis on DL3DV

Table 1 evaluates 32, 64, 128, and 256 input views sampled from the first 180, 240, 300, and up to 360 frames of each scene, respectively.

PSNR and SSIM are higher-is-better; LPIPS is lower-is-better; offline methods access all input views at once and are excluded from the online ranking; N/A† indicates that a method exceeds 48 GB at its training resolution.

Video results

One model, three input settings

Explore novel-view renderings as the input stream grows from 32 to 256 views.

Videos are muted and loop automatically while visible; select a video to pause or resume it.

Online efficiency

Designed for long image streams

Hybrid KV-cache compression keeps the update cost bounded while preserving an explicit, renderable scene.

Input throughput as the stream grows and peak memory versus number of input views.

45.1 FPS

Average input throughput across a 256-view stream.

41.1 FPS

Input throughput at the end of a 256-view stream.

<12 GiB

Average peak memory at 600 input views.

Runtime and memory are profiled at 224 × 224 resolution with chunk size 8 on an RTX 6000 Ada GPU.

Citation

Cite ReCoSplat

@article{cheng2026recosplat,
  title   = {ReCoSplat: Online Feed-Forward Gaussian Splatting via Render-and-Compare},
  author  = {Cheng, Freeman and Ye, Botao and Li, Xueting and You, Junqi and Zhan, Fangneng and Yang, Ming-Hsuan},
  journal = {arXiv preprint arXiv:2603.09968},
  year    = {2026}
}