Online Neural Space Time Memory for Dynamic Novel View Synthesis

Baback Elmieh1,2, Lynn Tsai2, Zeman Li2, Srinivas Kaza2, Tiancheng Sun2, Gabor Csapo2, Ali Behrouz2, Yuan Deng2, Stephen Lombardi2, Steven M. Seitz2, Xuan Luo2
1University of Washington 2Google
TL;DR: Real-time, minute-long memory for dynamic NVS from multi-view videos: Remembers the past and reconstructs occluded regions. Beats SOTA like LaCT-NVS and LVSM.

Abstract

Online novel view synthesis from multi-view streaming videos faces a fundamental trade-off: maintaining a persistent, long-horizon memory to reconstruct temporarily occluded regions while operating under strict real-time constraints. While Test-Time Training (TTT) offers a powerful memory mechanism, standard models mandate gradient-based memory updates at every frame to adapt to the changing motion in dynamic scenes. The computational cost of heavy memory updates precludes real-time application and can lead to instability over long contexts. Given that memory updates are more demanding than memory application and video content is largely redundant, we propose to decouple the frequencies of these two processes. Our approach performs periodic memory updates while applying the memory on a per-frame basis, using cross-view attention to manage deformations between the prior memory state and the current frame. To lock in the historical context, we introduce two critical mechanisms: an auxiliary Memory Loss that forces persistent internalization of the scene, and a Memory Caching strategy that regularizes active weights against catastrophic drift. Our method demonstrates real-time, state-of-the-art performance on scenes with dynamic human motion as well as minute-scale online memorization.

NSTM overview: input stereo video, periodic memorization, and per-frame synthesis over time.

Neural Space-Time Memory (NSTM) synthesizes novel views in amortized real-time from 30 FPS multi-view videos, while updating its memory at 1 FPS. Middle: NSTM captures the back logo as the subject rotates (see memory readout), faithfully recalling it later from memory given only frontal views. Right: While full self-attention in typical feed-forward models (e.g., LVSM) scales quadratically with history sequence length, NSTM scales with 𝒪(1) complexity, achieving amortized real-time inference at 256×256 resolution on one H100 GPU.

1-Min Results

Minute-long memory recall.

Select a scene, then click models in the comparison strip to inspect them in detail below. Check the memory box to see our memory readout visuals.

NSTM 256×256
NSTM 512×512
Token-Mem
LVSM
LACT-NVS
0:00 / 0:00

360° Demos

360° Novel-View-Synthesis with memory recall.

Select a scene, then click models in the comparison strip to inspect them in detail below. Check the memory box to see our memory readout visuals.

NSTM 256×256
NSTM 512×512
Token-Mem
LVSM
LACT-NVS
0:00 / 0:00

Method Overview

NSTM is the first online framework for dynamic novel-view synthesis from multi-view videos to sustain minute-long memory at amortized real-time speed. The two diagrams below show how NSTM achieves this by (1) decoupling periodic, heavy memory updates from continuous per-frame synthesis, and (2) training with an alternating memorization and synthesis regime plus an auxiliary memory loss that internalizes persistent scene context. Our method also contributes Memory Caching and an L2 inner loss formulation that are key to our minute-long stability and described in our paper.

NSTM architecture with memorization and synthesis steps and layer details

Online Inference Pipeline and NSTM Layer Design. Left: During inference, we run the memorization step periodically while performing the synthesis step per-frame, enabling amortized real-time speed. Right: The NSTM Layer consists of cross-view attention, a fast-weight block and a feed-forward layer. We also incorporate a learned scale that allows the network to balance the contribution of memory and attention to each layer's output.

NSTM training regime with alternating memory and synthesis supervision

Training Scheme. Left: Our alternating training scheme explicitly enforces memorization. In the Memory Supervision step, isolated memory tokens (target camera rays) perform strict self-attention (sharing weights with the cross-view attention layer) without attending to input views. Once the memory is updated by the inputs, these isolated tokens query the memory to reconstruct the target view. This forces a pure memory readout, ensuring context internalization. During Synthesis Supervision, we perform self-attention across input and target tokens to capture ongoing motion, and then query the previously updated memory state. Right: Visualizations of the inputs, targets, and memory in both steps. Crucially, the historical memory state often exhibits a different pose from the current observation during synthesis. Our cross-view attention addresses this deformation, successfully fusing past appearance with current motion.

Acknowledgements

We would like to thank Michael Broxton and Ryan Overbeck for their encouragement and support of the project at Google. We also thank John Flynn and Osman Ulusoy for their valuable discussions and feedback on the manuscript, and Tianyuan Zhang and Bowei Chen for early clarifying discussions on LaCT. Finally, we extend our appreciation to Xiaoguang Han, Hongjie Liao, and Yihao Zhi for providing full-length MVHumanNet++ videos for our experiments.

BibTeX

@misc{elmieh2026nstm,
      title={Online Neural Space Time Memory for Dynamic Novel View Synthesis},
      author={Baback Elmieh and Lynn Tsai and Zeman Li and Srinivas Kaza and Tiancheng Sun and Gabor Csapo and Ali Behrouz and Yuan Deng and Stephen Lombardi and Steven M. Seitz and Xuan Luo},
      year={2026},
      eprint={2607.15271},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.15271},
}