Mamba3R: Fixed-Capacity Selective State Memory for Long-Sequence 3D Reconstruction

📌 Highlights

Mamba3R presents a compact adaptation path for streaming 3D reconstruction: a trainable selective state-memory module improves accumulated geometry while the pretrained recurrent backbone and task heads remain frozen. The method is most compelling when viewed as a systems design, because its added memory is fixed-capacity and its measured cost remains nearly independent of traversal length.

• 1.28 million adapter parameters are trained while inherited reconstruction weights remain frozen.
• The evaluated adapter uses fixed-capacity auxiliary memory aligned with recurrent state slots and K=64 memory channels.
• Under reset100, Mamba3R reports lower CD, ATE, and AbsRel than TTT3R on ScanNet and KITTI-360.
• On the robot platform, the front-end averages 11.734 FPS at 320×240 and the complete coupled system averages 9.849 FPS.
• The main caveat is that robot results use offline replay and periodic resets rather than a demonstrated live end-to-end deployment.

🎯 Introduction

The paper addresses streaming 3D reconstruction, where a moving camera must commit depth, pose, and point-map predictions while accumulated errors progressively misregister later observations. Attention-cached streaming models retain historical keys and values but grow with sequence length, whereas recurrent models such as CUT3R and TTT3R keep bounded state at the cost of restricting how historical information is represented and updated. The paper-level objective is therefore to improve long-stream geometry without abandoning the fixed-budget behavior of recurrent inference.

Mamba3R adds a trainable state-memory adapter beside a frozen recurrent predictor. The design preserves the inherited encoder, decoder, pose memory, confidence-based state update, and geometric heads, while providing a separate route for learned adaptation. Its evaluation covers fixed-length public benchmarks, longer KITTI-360 prefixes, robot-recorded indoor and outdoor traversals, inference cost, matched mixer and memory ablations, and substitution into fixed loop-closing back-ends.

🔬 Methodology

The recurrent interface maintains a scene state S_{t-1}, a native pose memory, and an auxiliary activation memory M_{t-1} aligned to the inherited recurrent slots. The auxiliary memory is initialized to zero, has fixed capacity, and uses K=64 channels in the evaluated configuration. The adapter receives the concatenated state and memory, then splits the mixer output into state-side features that correct the current frame and memory-side features that are written for the next frame. This makes the added memory persistent across frames without turning it into a frame cache.

The mixer is a selective state-space block operating along the fixed recurrent slot order. Its input-dependent transition uses a learned continuous-time matrix and per-slot step sizes, while normalization, gating, and output projection complete the block. The paper uses a single forward SS1D scan because the recurrent slots do not correspond to a two-dimensional patch grid. The resulting per-frame scan cost is O(LD_sN), and the auxiliary memory occupies O(LK) elements for fixed slot count L.

Feedback is controlled in two stages. First, the adapter response is evaluated with zero auxiliary memory and with the carried memory; their difference isolates the memory-induced component, which is multiplied by alpha=0.5 before being added to the candidate state. Second, the inherited confidence-gated commit controls how much of the corrected state replaces the previous recurrent state. Memory writes are masked residual updates, and history-only decay is applied after the first 32 effective writes; resets clear the scene state, pose state, auxiliary memory, and write counter.

📊 Experiments

The public benchmark evaluation uses five ScanNet sequences and three KITTI-360 sequences at 500 frames under matched reset100, with CD and ATE measured in meters and AbsRel computed after one sequence-wide depth scale. The comparison includes CUT3R, TTT3R, TTSA3R, Point3R, Spann3R, STream3R, ZipMap-W1, StreamVGGT, and Mamba3R. Mamba3R records 0.1032 CD, 0.4589 ATE, and 0.04666 AbsRel on ScanNet, and 1.712 CD, 7.788 ATE, and 0.1881 AbsRel on KITTI-360. The paper reports relative improvements over TTT3R of 9.6%, 7.3%, and 17.7% on ScanNet CD, ATE, and AbsRel, increasing to 25.1%, 11.7%, and 23.9% on KITTI-360.

Long-stream evaluation averages KITTI-360 test_2 and test_3 prefixes at 1,000, 2,000, and nominally 3,000 frames. Mamba3R has the lowest ATE at every tier: 25.54 m, 94.88 m, and 100.6 m, compared with 32.81 m, 105.6 m, and 114.4 m for TTT3R and 57.23 m, 125.0 m, and 127.4 m for reset-matched CUT3R. The same evaluation also reports that continuous no-reset operation is substantially worse in the tested KITTI-360 conditions, so the result supports bounded recurrent operation with periodic bridging rather than unrestricted memory retention.

The efficiency measurements distinguish server-side inference cost from onboard replay. On H200, Mamba3R reaches 14.61 FPS at 500 frames, 14.96 FPS at 1,000 frames, and 14.76 FPS at 2,000 frames, with allocated memory of 5.404, 4.633, and 4.633 GiB. On the RTX 4060 platform, the front-end reaches 11.734 FPS at 320×240 and the complete coupled system reaches 9.849 FPS. The robot cohort contains ten indoor and three outdoor traversals; the paper reports 27.9% lower indoor mean ATE and 37.5% lower observed-surface CD than TTT3R, with lower ATE, CD, and higher NC on all ten indoor sequences against both recurrent baselines. Outdoor results are reported with trajectory error only because the RGB-D sensor's dependable range does not support comparable depth and surface evaluation.

The ScanNet ablations support several mechanisms but not an exhaustive causal proof. Carrying the added memory improves CD from 0.1021 to 0.1009, ATE from 0.4790 to 0.4465, and AbsRel from 0.04838 to 0.04653. Selective SS1D is better than the tested pointwise MLP, cross-slot convolution, and lightweight attention configurations. Full-strength feedback raises CD to 0.1796 and ATE to 0.6575, while the no-decay continuous no-reset setting raises ATE from 0.6277 to 0.9952 in the reported intervention.

fig:method

Caption: Overview of . A frozen encoder and recurrent decoder are augmented by the state--memory adapter: it mixes the auxiliary memory with the state carried into the call, its read residual corrects the current frame's decoder feature before the heads run, and its memory half supplies the write read by the next frame (Section~sec:method; execution order in supplementary Algorithm~S1).

Why It Matters: This figure shows the central integration point: the adapter augments the recurrent state while leaving the inherited decoder and prediction heads intact.

fig:clouds500

Caption: 500-frame point clouds for the reference (GT) and every evaluated method on ScanNet and KITTI-360; stays registered where the cached and spatial-memory comparators fragment.

Why It Matters: The qualitative comparison illustrates whether improved trajectory consistency translates into more coherent accumulated point clouds.

fig:robot_forward

Caption: Front-end trajectories on I2, I3, and I8: one full-sequence Sim(3) per method, no ICP or segment-wise fit.

Why It Matters: These trajectories expose accumulated drift and repeated-pass registration behavior on robot-recorded sequences.

fig:geom_detail

Caption: Local map detail in two fixed metric crops (top-down slab views): a revisited wall on I2 (top) and ScanNet scene0499\_00 (bottom); each row is one axis-aligned box in the shared reference frame, so position inside a panel is comparable across panels. Crop CDs are in the supplementary material.

Why It Matters: Fixed metric crops provide a localized view of surface placement and map detail beyond whole-trajectory scores.

🔮 Conclusion

Mamba3R demonstrates that a frozen recurrent reconstruction model can gain a trainable adaptation path through a fixed-capacity auxiliary memory and selective state-space mixer. The reported improvements cover accumulated surface geometry, camera trajectory, and depth on public data and robot-recorded traversals, while the measured memory footprint remains effectively constant as sequence length increases.

The evidence supports the combination of carried memory, selective mixing, attenuated memory feedback, and history decay under the paper's reset100 protocol. It does not establish unrestricted reset-free stability or live end-to-end operation: the front-end is above the 10 Hz input rate in replay, while the complete coupled system is slightly below it and exhibits long-tail latency during loop optimization.

🛠️ Future Research Improvements

A direct next step is a streaming back-end designed for frame-by-frame monocular operation rather than offline whole-sequence optimization. The paper identifies the gap between 11.734 FPS front-end replay and 9.849 FPS complete-system throughput, together with p95 delay during loop optimization, as an engineering target for sensor-rate mapping with bounded output latency.

The method would also benefit from stronger stability and generalization studies: reset-free sequences, learned or adaptive reset policies, longer traversals, live synchronized sensor input, more diverse motion and lighting, and experiments that feed optimized map corrections back into the compact recurrent state. These directions are motivated by the paper's own observation that the coupled state-memory recurrence can still amplify perturbations and that current gains are concentrated under tested reset and decay settings.

🏭 Potential Industry Use Scenarios

The method is a plausible front-end for mobile mapping, facility inspection, assistive robotics, and service robots that need coherent RGB-based geometry while sharing an embedded GPU with localization, planning, or control. Its fixed-capacity state is particularly relevant to long traversals where a historical attention cache or spatial memory would exceed the available budget.

A practical deployment would pair Mamba3R with calibrated RGB or RGB-D sensing, a bounded loop-closing back-end, and explicit monitoring of synchronization, reset boundaries, memory usage, and tail latency. The robot experiments support recorded-traversal feasibility on an RTX 4060, but live deployment and immediate pose-feedback guarantees remain to be validated.

💬 Critical Analysis

The strongest aspect of the work is the alignment between mechanism and systems constraint. The adapter changes only 1.28 million parameters, preserves the inherited prediction interface, and has ablations that separately test memory continuation, mixer choice, feedback strength, and decay. The public-benchmark and robot results also extend beyond a single metric: Mamba3R improves trajectory and accumulated surface quality while retaining competitive depth behavior and bounded inference cost.

The main interpretive caution is protocol dependence. Periodic reset100 and predicted-camera bridging are central to the reported long-stream results, while the continuous no-reset intervention is less favorable. The loop-closing study further shows that a better front-end does not guarantee a better final surface map: sparse or poorly distributed loop constraints can improve ATE while displacing already-correct surfaces. Reproduction should therefore treat front-end geometry, back-end optimization, reference construction, and latency as separate evaluation axes rather than collapsing them into one deployment claim.