μVLA: On Recurrent Memory for Partially Observable Manipulation in VLA Models

Attention overlay videos

Top row shows all overlays at once: MIKASA-Robo memory → vision, MIKASA-Robo action → vision, LIBERO memory → vision, LIBERO action → vision.

Abstract

Vision-language-action (VLA) models predict chunks of future actions from the current observation, an assumption that fails under partial observability, where decisions depend on information no longer visible. Existing memory-augmented VLAs simultaneously introduce recurrence, retrieval, compression modules, auxiliary objectives, hierarchical memory, or task-specific architectural changes, so the contribution of recurrence itself remains entangled with surrounding machinery.

We present a controlled isolation study of recurrence in a strong pretrained VLA backbone. Our formulation augments the transformer with a small set of learnable memory tokens carried across timesteps and updated through self-attention, trained end to end with truncated backpropagation through time, with no auxiliary losses and no architectural changes. We instantiate this as μVLA, a family of OpenVLA-OFT variants parameterized by memory width m, TBPTT length K, and the memory update rule (cross-step gradients or a detached EMA), so that recurrence is the only varying factor.

On MIKASA-Robo, μVLA improves average success rate on five training tasks from 0.42 to 0.84 at the strongest setting and reaches 0.23 on held-out tasks with the same memory structure versus 0.07 for the memoryless baseline. On tasks requiring different memory structure, performance remains near baseline. On LIBERO, the strongest recurrent variant achieves 96.2% average success, indicating no regression under full observability.

Method

Action / memory attention to vision

Successful rollouts with attention projected onto visual patches. By default the attention maps are aggregated as mean over layers and heads. Per-layer mode keeps the same rollout but replaces either lower row independently with mean-over-heads attention from a selected transformer layer.

plain rollout

L31

L0

mean

mean

memory → vision, aggregated over layers and heads

L31

L0

mean

mean

action → vision, aggregated over layers and heads

LIBERO attention rollouts

Aggregated attention rollouts and frame-by-frame overlays for successful LIBERO episodes. Choose a suite, task, and episode; attention maps are mean over layers and heads.

Task 00, episode 00: plain rollout

memory → vision, aggregated over layers and heads

action → vision, aggregated over layers and heads

Results

Per-environment success rate on MIKASA-Robo (23 environments)

100 deterministic episodes per environment; mean reported. OpenVLA-OFT† uses the episodic dataloader. The (+1st obs.) column appends the first observation at every timestep and acts as an oracle-style upper bound for first-frame cue tasks. μVLA columns use m=64 unless marked m=1.

Memoryless vs. memory-augmented VLAs on MIKASA-Robo

Success rates are reported per task. CronusVLA and MemoryVLA results are from the MemoryVLA paper; memory-augmented models are shaded in the paper.

Performance comparison on LIBERO (Franka robot)

Success rates (%) are reported for the four LIBERO suites and their average. * denotes memory-augmented VLA models.

Memory representation dynamics

Per-token L2 norms, cosine-distance-to-previous, and rollout for a successful episode at K=2, m=64. Cue-recall tasks show a single dominant write event followed by a secondary bump aligned with the cue-removal window.

Per-token L2 norms

1 − cos(M_t, M_t−1)

Per-token norm heatmap

Per-token L2 norms

Cosine distance

Per-token norm heatmap

Per-token L2 norms

Cosine distance

Per-token norm heatmap

Per-token L2 norms

Cosine distance

Per-token norm heatmap

Per-token L2 norms

Cosine distance

Per-token norm heatmap

Causal memory intervention

Success rate at 100 episodes under baseline (clean memory), noise (M_t replaced with i.i.d. Gaussian noise before every forward), and freeze_first (memory locked to M₁). The noise condition drops SR sharply on every cell with a non-trivial baseline, confirming that the recurrent channel is functionally read at inference.

Chunk-length sweep

For each carrier (K=8, EMA, K=2) we report SR at chunk lengths {1, 2, 4, 8, 16} with the memory channel active (solid) and zeroed at inference (dashed). The with-memory minus no-memory gap is largest at chunk=1 on cue-recall tasks; long-chunk inference partially bypasses the recurrent channel.

RememberColor5 RememberShapeAndColor3x3 ShellGamePush TakeItBack InterceptMedium

Generalization probes (RememberColor5)

(a) Both task phases fixed to N∈{3, 5, 10, 20, 50, 100} steps; training covers N∈{1,…,5} per phase. (b) 1–5 in-distribution colors replaced with the OOD palette {Pink, Orange, Purple, Brown, White}. K=2 dominates in-distribution and remains most robust to color swap, but degrades fastest with phase length; K=8 is flat-but-low; EMA collapses with phase length.

Phase-length sweep Color-swap sweep