Robot learning · Multi-view video editing

ERMV: Action-Aligned 4D Robot-Trajectory Editing with State-Conditioned Epipolar Motion Kernels

Chang Nie1,2 Tianchen Deng1,2 Guangming Wang3 Zhe Liu1,2 Hesheng Wang1,2
1Shanghai Jiao Tong University 2Shanghai Key Laboratory of Navigation and Location Based Services 3University of Cambridge
Project page · Preprint
Paper arXiv Code
ERMV turns an observed synchronized multi-view robot trajectory into an edited trajectory using sparse context selection, epipolar exchange, and autonomous feedback.
ERMV propagates an edited visual condition through an observed 4D robot trajectory while retaining synchronized views and the recorded state–action structure.

Overview

Abstract

Robot policies learn from multi-view image trajectories paired with actions, yet collecting demonstrations across diverse visual conditions remains expensive. Visual edits that preserve robot and object geometry, motion, and action labels can expand demonstration datasets without additional collection.

We introduce Editing Robot Multi-View trajectories (ERMV), a diffusion editor that jointly processes the observed 4D trajectory as a time–view grid and conditions explicitly on recorded state and action. A Sparse Spatio-Temporal selector allocates compact long-range context, Epipolar Motion-Aware Attention aligns synchronized views through state- and action-conditioned supports, and an autonomous feedback cascade prevents task-critical errors from propagating through autoregressive memory.

Experiments span simulation, real-world recordings, physical Franka Panda deployment, sim-to-real transfer, and five policy families. The results show that visual diversity is most useful when it remains tied to the demonstrated interaction.

Approach

Editing around the action, not around a plausible video

ERMV organizes diffusion around the structure already present in a robot demonstration: time, synchronized camera views, robot state, and action.

01

Sparse long-range context

SST retains informative observations across long trajectories under a compact attention budget, helping recover appearance and identity after motion or occlusion.

02

Motion-aware view exchange

EMA-Attn couples synchronized cameras through localized epipolar supports modulated by the recorded motion, while preserving self evidence when another view is unreliable.

03

Autonomous feedback

Trajectory diagnostics guide acceptance, regeneration, or local correction before an edited segment becomes context for subsequent windows—without human intervention.

Detailed ERMV pipeline with visual anchor conditioning, robotic and camera state-action injection, sparse spatio-temporal context, epipolar motion-aware attention, and autonomous feedback.
The visual anchor specifies the desired appearance; the source trajectory and state–action stream constrain how that edit evolves across time and views.

Evaluation

Trajectory-consistent edits improve downstream robustness

ERMV is evaluated as both a video editor and a data engine: edited trajectories should look different, preserve the demonstrated interaction, and make policies more robust to visual shift.

5 policy families

Consistent downstream evaluation across conventional policies, VLA models, and a world–action model.

+11.6–17.8 percentage points

Success improvement under visual shifts over source-only training across the five policy families.

Sim + real closed-loop validation

Evaluation includes physical cup-to-plate placement and sim-to-real deployment, beyond visual metrics alone.

Simulation editing

Synchronized variation across four cameras

The top row shows the source observations and the bottom row shows the corresponding ERMV edit. The appearance changes across all views while robot motion and scene correspondence remain aligned.

Source and ERMV outputs are synchronized at each timestep across head, front, left, and right cameras.

Qualitative comparisons

Strong video priors are not action supervision

The tested image-to-video systems receive the edited first frame and a text description. They can synthesize visually compelling clips, but their interfaces do not receive the recorded robot actions or companion camera streams available to ERMV.

Grok Imagine Video 1.5Edited first frame + text
Seedance 2.0Edited first frame + text
Veo 3.1Edited first frame + text
Kling AI 3.0Edited first frame + text

Physical deployment

Franka Panda cup-to-plate placement

ACT is trained either without augmentation or with ERMV-augmented trajectories, then evaluated on the same cup-to-plate objective in the seen setup and an unseen cluttered setup.

Seen setupACT · no augmentation
Seen setupACT · ERMV augmentation
Unseen cluttered setupACT · no augmentation
Unseen cluttered setupACT · ERMV augmentation

Reference

Citation

@article{nie2025ermv,
  title   = {ERMV: Action-Aligned 4D Robot-Trajectory Editing with
             State-Conditioned Epipolar Motion Kernels},
  author  = {Nie, Chang and Deng, Tianchen and Wang, Guangming and
             Liu, Zhe and Wang, Hesheng},
  journal = {arXiv preprint arXiv:2507.17462},
  year    = {2025}
}