Spatial-TTT builds 3D memory from video streams via test-time training

Spatial-TTT stores information from a video stream by updating a small subset of model parameters as frames arrive. In the research shared by Tencent, these adjustable parameters, called fast weights, act as memory for spatial evidence accumulated over time. The design combines that memory with attention over a recent window of video. Project explanation.

Its update mechanism uses 3D spatiotemporal convolution so the memory captures relationships across neighboring views and moments, and training uses dense scene descriptions covering global context, objects and counts, and spatial relations. The intended result is a model that can answer questions about a scene using observations spread across a long video. Here, 3D memory means information encoded in model parameters, not an exportable 3D asset. Method overview.