Robotics
Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration
Xinghao Zhu, Zixi Liu, Shalin Jain et al. (+18 more)
Dexterous
robot manipulation can benefit from the abundance of human demonstrations, but transferring such demonstrations to
robot policies remains challenging. We present Contact Wrench Guidance from Human Demonstration in Robotic Dexterous Manipulation (CHORD), a framework for long-horizon manipulation of rigid and articulated objects with reinforcement learning. The key idea is object-centric contact wrench space guidance: we represent human and
robot motions by the forces and torques they can induce on the object, enabling similarity to be measured by the induced instantaneous motions. This guidance makes reinforcement learning more scalable for contact-rich dexterous manipulation. We further introduce a large-scale simulation
benchmark with 4,739 bimanual dexterous manipulation tasks, constructed from motion-capture
datasets and reconstructed in-house
videos. Evaluated on 1,831
benchmark tasks, CHORD achieves an average success rate of 82.12%, demonstrating strong scalability. CHORD also generalizes to whole-body manipulation from hand-only and third-person demonstrations, achieving a 90.77% success rate, and the learned policies transfer to the real world in both open-loop and closed-loop settings.
Read the full abstract ▾
All authors Xinghao Zhu, Zixi Liu, Shalin Jain, Chenran Li, Milad Noori, Michael Andres Lin, Huihua Zhao, John Welsh, Mrinal Verghese, Wei Liu, Tingwu Wang, Xingye Da, Zhengyi Luo, Vishal Kulkarni, Naema Bhatti, Yuke Zhu, Linxi Fan, Bowen Wen, Danfei Xu, Soha Pouya, Yan Chang
Robotics
Joint Discovery of Object and Action Symbols through Effect Prediction for Robotic Manipulation Planning
Burcu Kilic, Berke Kartal, Fatih Dogangun et al. (+2 more)
To perform complex manipulation planning, autonomous
robots are required to abstract continuous, high-dimensional sensorimotor interactions into discrete object and action representations. Earlier work either categorized objects based on visual appearances, which fails to distinguish objects that appear similar but behave differently, or based on effects under interaction, but was limited to predefined actions. To address these limitations, we propose a model that jointly discovers high-level manipulation primitives and object categories through a binary bottleneck layer, trained to predict multi-modal outcomes, including object motion, contact, and force feedback, from random interaction data. Building on these discovered binary representations, we leverage a discrete planning method that uses intermediate steps in the predicted effect trajectory to enable partial action executions for precise low-level control. Additionally, we evaluate our framework's generalization capabilities on novel objects by assigning object categories through comparing a small number of interaction effects with the predicted effects of learned object symbols, enabling few-shot generalization based on behavior rather than visual similarity. We conduct experiments on tabletop repositioning and stacking tasks, and confirm that our effect-driven planning approach outperforms both a state-of-the-art method and a visual-based alternative in planning precision across seen and novel objects.
Read the full abstract ▾
All authors Burcu Kilic, Berke Kartal, Fatih Dogangun, Erhan Oztop, Emre Ugur
Robotics
A Unified Benchmark for RCM-Constrained Visual Servoing: Modeling-Controller Interaction and Robustness Analysis in Laparoscopic Robots
Jing Zhang and Mengtang Li
In
robot-assisted laparoscopic minimally invasive surgery (MIS), accurate enforcement of the remote center of motion (RCM) constraint is critical for safe and stable automatic field-of-view (FoV) adjustment. Although control-based RCM strategies are widely adopted due to their flexibility and cost-effectiveness, systematic comparison of different RCM formulations and image-based visual servoing (IBVS) frameworks remains challenging due to the lack of a unified and reproducible
benchmark. This paper presents an open-source simulation framework integrating three representative RCM modeling approaches and six IBVS-based control architectures within a unified velocity-level formulation, enabling controlled and consistent evaluation. Through structured case studies, the framework reveals key structural sensitivities arising from modeling and controller interactions, including the impact of tangent-plane definition, constraint dimensionality, open- versus closed-loop enforcement, and robustness near kinematic singularities. All resources are released and demostrations are provided in the supplementary
video, providing a reproducible foundation for RCM-constrained visual servoing research.
Read the full abstract ▾
Robotics
Memory-Native Non-Terrestrial Networks for Embodied Intelligence
Chengyang Li, Yikun Wang, Jiahui He et al. (+6 more)
Non-terrestrial networks (NTN) provide ubiquitous connectivity for
embodied intelligence (EI), enabling
robots in the wilderness to leverage cloud resources or report critical information to remote centers. However, the synergy is nontrivial due to the highly dynamic, resource-constrained,
topology-varying, and task-oriented environment. Existing memoryless NTN protocols become inefficient, since the decisions are driven by local channel conditions and instantaneous service demands. To address these limitations, this paper proposes the
memory-native NTN (Mem-NTN) paradigm that leverages long-horizon contexts for
memory-augmented system optimization. To realize this paradigm shift, we establish a dual-
memory architecture that distinguishes between physical
memory representing the state of the world and digital
memory encoding historical network experience. We develop
memory acquisition, compression, valuation, update, and utilization mechanisms that facilitate cross-layer,
memory-native decision-making, spanning from the physical and access layers up to the network and application layers. Experiments in satellite
embodied question answering (SEQA) demonstrate that the proposed Mem-NTN consistently outperforms conventional stateless NTN and terrestrial approaches.
Read the full abstract ▾
All authors Chengyang Li, Yikun Wang, Jiahui He, Yujie Wan, Shuai Wang, Yuan Wu, Yik-Chung Wu, Chengzhong Xu, Huseyin Arslan
Robotics
RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis
Minh-Loi Nguyen, Nghiem T. Diep, Hung K. Nguyen et al. (+13 more)
Robot world models now generate synthetic manipulation
videos, but evaluating them is hard: realistic-looking outputs often violate physical laws, temporal consistency or task logic, while conventional metrics and monolithic
Vision-
Language Model (VLM) judges return scores that do not say what failed. We present RoboGaze, a training-free multi-
agent VLM evaluator that grounds the task and scene, routes each subtask to dimension-specific specialists and verifies their findings with a critic, producing temporally localized diagnostic reports that state what failed, when, why and how severely, under a schema of 6 dimensions and 30 failure types whose annotation reliability and coverage we measure. To
benchmark it, we introduce RoboGazeBench, 914 human-annotated clips from five generators, conditioned on simulated and real-world data and running from 4.9 to 148.9 s. Across eight open-source and proprietary VLM backbones, RoboGaze outperforms every baseline; even a published agentic detector given the same schema and a slightly larger inference budget trails it by 10.1-12.3 description-F1 points, a margin that blinded human raters confirm. On the strongest backbone it closes 83% of the gap between chain-of-thought prompting and human agreement, and its critic curbs the "cry-wolf" false positives of standard VLMs, lifting mean clean-clip accuracy from 17% to 85%. The advantage is largest on held-out generators and long clips, and carries over to selecting generated demonstrations (80% vs. 71% precision).
Read the full abstract ▾
All authors Minh-Loi Nguyen, Nghiem T. Diep, Hung K. Nguyen, Minh Le, Doanh L. Thien, Tuan A. Tran, Hoang H. Tran, Dung D. Le, Vu N. Duong, Khoa D. Doan, Ngan Le, Daniel Sonntag, An T. Le, Vien Anh Ngo, Duy M. H. Nguyen, Tran V. Nhiem
Robotics
A Query-Driven Communication-Efficient Digital Twins Design for Autonomous Driving
Nuocheng Yang, Longyu Zhou, Sihua Wang et al. (+2 more)
Digital twins (DTs) have become a potential technology to perform risk-free simulation of physical entities for deterministic and high-reliability services in diverse scenarios such as autonomous driving and low-altitude economy. In the autonomous driving scenario, traditional DT methods that rely solely on vehicle's real-time state synchronization, however, might lead to unacceptable computing and communication consumption for construction of high-fidelity DT with redundant data. To address this issue, we first propose a query-driven DT architecture to enable the DT to actively request the desired environment data from vehicles based on its simulation result. Then, we formulate an optimization problem whose goal is to minimize autonomous driving position error while accounting for DT fidelity and communication constraints. We also design a cross-time-step progressive query mechanism to further improve communication efficiency. The simulation results show that our proposed method achieves a 24% reduction in planning position error compared to traditional methods, while reducing communication overhead by 40%.
Read the full abstract ▾
All authors Nuocheng Yang, Longyu Zhou, Sihua Wang, Changchuan Yin, Tony Q. S. Quek
Robotics · Cross-listed from cs.AI
Critique of Agent Model
Eric Xing, Mingkai Deng and Jinyu Hou
What is an
agent? What constitutes agency? With the rise of Large
Language Model (LLM) systems marketed as “coding
agents”, “AI co-scientists”, and other “agentic" tools that promise to drive up productivity, and at the same time, “existential" concerns such as AI escaping human control with destructive power under a speculative “machine agency" against humans, it has become essential to clarify where automation ends and agency begins, both for building capable systems and for understanding whether and what to fear. Drawing on Descartes' grounding of agency in independent thought, and on portrayals of autonomous beings in science fiction, we survey the current landscape of AI
agents, and analyze
agent architectures along five dimensions: goal, identity, decision-making, self-regulation, and learning. Specifically, we argue that genuine agency requires these structures to be internalized within the system itself rather than assembled through external scaffolding. This distinction between agentic systems, whose competence resides in engineered workflows, and agentive systems, whose capabilities (including social interaction) arise endogenously, defines the boundary between systems designed for prescribed tasks, and those capable of operating in the open world with true autonomy. Building on this analysis, we propose the Goal-Identity-Configurator (GIC) architecture for a general-purpose
agent model, combining hierarchical goal decomposition, identity
evolution, simulative
reasoning grounded in a separately trained world model, learned self-regulation, and self-directed learning from both real and simulated experience. Furthermore, we share insight on the auditability, controllability, and safety of agentive systems that possess greater autonomy and “agency", but remain under human oversight.
Read the full abstract ▾
Robotics
Topological Online Learning for Displacement-based Formation Control
Saksham Sharma, Shubhankar Gupta, Sumant A Gunagi et al. (+1 more)
This paper addresses the problem of robust formation control by introducing Topological Online Learning for Displacement-based (TOLD) formation control, a real-time edge-level adaptation framework. Unlike conventional node-level robust controllers that regulate individual
robot inputs without modifying the interaction
topology, TOLD updates the interaction
topology weights online to directly minimize formation distortion. Two strategies are proposed under the TOLD formation control framework: Online Gradient Flow (OGF) with unconstrained weights and Online Exponential Gradient Flow (OExpGF) with non-negative convex weights. Theoretical analysis establishes that, for single-integrator
agents over directed graphs, OExpGF guarantees asymptotic consensus, while OGF ensures bounded formation distortion. Simulations with twelve
robots under intermittent disturbances show 1.2%-33.14% median cumulative Root Mean Distortion Error reduction when augmenting TOLD with node-level controllers. Hardware experiments with Crazyflie 2.0 quadrotors demonstrate over 62% (OGF) and 31.4% (OExpGF) reduction in median formation distortion compared to fixed-weight consensus.
Read the full abstract ▾
All authors Saksham Sharma, Shubhankar Gupta, Sumant A Gunagi, Suresh Sundaram
Robotics
Enforcing Human-like Kinematics in Dexterous Piano Playing via Adversarial Posture Regularization
Bin Qiu, Yanming Shao, Guanyu Cai et al. (+1 more)
Reinforcement learning can train bimanual dexterous hands to play piano in physics simulation with high note accuracy, but for high-DoF dexterous hands, relying solely on task rewards or IK inversion often leads to unnatural postures and joint overextension. We propose
Adversarial Posture Regularization (APR). It avoids expensive, song-aligned expert demonstration data and instead uses a small amount of casual human playing data. By matching the distribution of the posture of the policy with the human prior through an
adversarial objective, APR encourages more human-like hand shapes. Meanwhile, we collect and release unstructured hand motion data of piano playing using a consumer-grade Meta Quest 3, and retarget the key motion information to the Shadow Hand. Finally, we achieve significantly better performance than prior methods on all three human-likeness metrics (cPSI, BSE, and FAC) as well as in visual quality. Project repository: https://github.com/APRProject/APRPianist.
Read the full abstract ▾
All authors Bin Qiu, Yanming Shao, Guanyu Cai, Yao Mu
Robotics · Cross-listed from cs.MA
Decentralized Coordination of Autonomous Traffic Through Advanced Air Mobility Corridors
Jasmine Jerry Aloor and Hamsa Balakrishnan
The use of dedicated corridors for Advanced Air Mobility (AAM) traffic is one of the most commonly proposed pathways to integrating them into existing airspace operations. Most prior research has focused on the design of networks of AAM corridors and conflict resolution for aircraft within corridors. It is also generally believed that while attractive from an implementation perspective, corridor-based operations may be inefficient, especially in the absence of centralized traffic management. In this paper, we show that contrary to this belief, it is possible for autonomous aircraft to learn to self-organize into corridor flows in decentralized settings. We illustrate our approach using scenarios in which fixed-wing aircraft need to safely and efficiently traverse (1) a single corridor with metering after the exit, (2) a sequence of two consecutive corridors, and (3) a corridor that splits into two. We find that in decentralized settings with only local information, the aircraft are able to conform to the corridor boundaries more than 94% of the time and reach their goal in a relatively efficient manner. Furthermore, tactical interventions to handle violations of the separation minimum are needed only infrequently in low- and medium-density settings. However, such tactical interventions become more frequently necessary only when traffic density is high.
Read the full abstract ▾
Robotics
Engineering Reliable Autonomous Systems: Challenges and Solutions
Marie Farrell, Matt Luckcuck, Angelo Ferrando et al. (+28 more)
Engineering reliable autonomous systems is an important and growing topic in computer science. As autonomous systems become more prevalent, easy-to-use techniques for building them reliably are increasingly important. This workshop report captures and expands on the discussions at the Lorentz Center Workshop "Engineering Reliable Autonomous Systems" (ERAS), held from 10 to 14 June 2024. The workshop was co-organised by the organisers of the Workshop on Formal Methods for Autonomous Systems (FMAS) and the Workshop on
Agents and
Robots for reliable Engineered Autonomy (AREA). It brought together members of the FMAS and AREA communities, industry practitioners, and representatives from sectors where autonomous systems pose distinctive engineering challenges. The workshop focused on three main research topics: techniques for verification and validation of autonomous systems; engineering real-world autonomous systems; and software architectures for safe autonomous systems. Its main outcome is a catalogue of challenges in these areas and, most importantly, a pathway to solutions. Some challenges can already be tackled by techniques that are well known in academia but have not yet become regularly used in practice. Other challenges remain unresolved and require further research. This roadmap is intended to support future research and industrial collaboration.
Read the full abstract ▾
All authors Marie Farrell, Matt Luckcuck, Angelo Ferrando, Rafael C. Cardoso, Natasha Alechina, Marco Autili, Diana Benjumea Hernandez, Luciana Brasil Rebelo dos Santos, Daniela Briola, Ana Cavalcanti, Christian Colombo, Louise A. Dennis, Clare Dixon, Michael Fisher, Mario Gleirscher, Taylor Johnson, Charles Lesire, Livia Lestingi, Sven Linker, Brian Logan, Colin Paterson, Fabio Papacchini, Patrizio Pelliccione, Pedro Ribeiro, Maike Schwammberger, Silvia Lizeth Tapia Tarifa, Hazel Taylor, Jim Woodcock, Mengwei Xu, Yi Yang, Huan Zhang
Robotics
Verifiable Foundation Models for Robot Safety
Davide Corsi, Kyungmin Kim and Roy Fox
Deploying foundation models for
robot control raises a central challenge: the expressive power that enables rich,
multimodal perception also makes these models opaque and difficult to analyze formally, rendering them intractable for existing verification tools. In this paper, we present FEARL (Foundation-Enabled Assured
Robot Learning), a framework that addresses this tension through a modular architectural decomposition. FEARL separates the policy into a large Controller (C) responsible for high-dimensional perception and task
reasoning, and a small Safety module (S) that receives low-dimensional observations from dedicated safety sensors together with a bounded context
embedding from C and produces the final action. Since many
robot safety requirements, such as collision avoidance and workspace boundary constraints, can be expressed over these safety sensor observations, formal verification can be applied to S rather than to the full foundation-model backbone. This makes formal analysis tractable with existing tools while preserving the Controller's expressive power for task
reasoning. To show that the decomposed policy remains capable of solving diverse tasks, we evaluate FEARL on three simulated robotic domains using multiple Controller backbones and training procedures, including pretrained off-the-shelf
vision-
language-action models. We further transfer the learned policy from one of our simulated tasks to a physical
robot, suggesting that the low-dimensional safety interface supports practical sim-to-real transfer.
Read the full abstract ▾
Robotics
AutoDex: An Automated Real-World System for Dexterous Grasping Data Collection
Mingi Choi, Gunhee Kim, Jisoo Kim et al. (+4 more)
Learning robust dexterous grasping requires real-world data that records the physical outcomes of grasp attempts. Such data is hard to obtain at scale: teleoperation yields valid physical outcomes but is slow and operator-biased, while simulation-based generation is cheap and scalable but cannot certify contact validity. A natural solution is to generate candidate grasps and verify them on real hardware, but this scales only if the entire collection loop (perception, execution, labeling, and reset) runs without human intervention. We present AutoDex, an automated real-world data-collection system that closes this loop: for each candidate from a replaceable generator, it localizes the object under severe hand-object occlusion with dense 20-camera perception, executes collision-monitored
robot motions, labels lift-and-hold success or failure, and actively resets the object between trials to expose additional candidates across stable poses. The result is a reusable database of physically labeled grasp trials that downstream systems can query by
retrieval and feasibility filtering. Using AutoDex, we collect 3,593 grasp trials across Allegro and Inspire hands on 100 diverse objects, with synchronized multi-view observations and
robot-state logs. For a matched 500-trajectory collection, AutoDex requires 10.3 h versus 49.4 h for teleoperation, yielding a 4.8x throughput improvement, and grasps retrieved from the AutoDex-validated database succeed 76% versus 34% for simulation-only validation. Code and data will be publicly released.
Read the full abstract ▾
All authors Mingi Choi, Gunhee Kim, Jisoo Kim, Taeksoo Kim, Taeyun Ha, Jongbin Lim, Hanbyul Joo
Robotics
LIBERO-Safety: A Comprehensive Benchmark for Physical and Semantic Safety in Vision-Language-Action Models
Rongxu Cui, Zongzheng Zhang, Jingrui Pang et al. (+11 more)
Despite the impressive manipulation capabilities of
Vision-
Language-Action (VLA) models, their operational safety under strict constraints remains largely unverified. To address this, we introduce a parametric safety
benchmark to procedurally generate safety-critical scenarios with comprehensive stochasticity. To overcome the scalability bottlenecks of human teleoperation, we develop a novel keypose-driven data generation pipeline. Leveraging this infrastructure, we curate a large-scale
dataset of 19,664 strictly collision-free demonstrations with extensive domain randomization. We then conduct a systematic cross-paradigm evaluation of eight VLA and two
embodied foundation models. Our analysis reveals a critical generalization-safety tension: although high-diversity training fosters safer trajectories, task success remains fundamentally bottlenecked by sub-optimal trajectory synthesis and semantic misalignment. By providing a scalable pipeline, a robust
dataset, and profound failure-mode insights, LIBERO-Safety establishes a crucial foundation for developing safe and reliable VLA models.
Read the full abstract ▾
All authors Rongxu Cui, Zongzheng Zhang, Jingrui Pang, Haohan Chi, Jinbang Guo, Saining Zhang, Shaoxuan Xie, Xin Jin, Yao Mu, Jiaolong Yang, Guocai Yao, Xianyuan Zhan, Ya-Qin Zhang, Hao Zhao
Robotics
LaST-HD: Learning Latent Physical Reasoning from Scalable Human Data for Robot Manipulation
Jiaming Liu, Yinxi Wang, Chenyang Gu et al. (+15 more)
Human-hand demonstrations provide a direct and scalable source of physical interaction data for
robot learning. While manual retargeting is indispensable for establishing kinematic action correspondence across different morphologies, robust transfer requires going beyond geometry to address the underlying
alignment of physical dynamics between human and
robot manipulation. To address this, we introduce LaST-HD, a novel human-to-
robot action learning paradigm that extends
reasoning-before-acting VLA by aligning human-hand and
robot demonstrations in a shared latent
reasoning space. Rather than mimicking human kinematics, LaST-HD trains an auxiliary action-conditioned world model on unpaired human-hand and
robot trajectories to synthesize unified latent targets. After aligning cross-embodiment representations in this shared forward-dynamics space, these targets supervise LaST-HD's latent
reasoning process, enabling it to internalize shared physical dynamics and drive efficient human-hand action learning. Moreover, we develop Out-of-Lab (OOL) Glove, a low-cost motion-capture glove tailored to LaST-HD for human-hand data collection. The captured human data provide precise keypoints and serve as universal action supervision across grippers and dexterous hands. Armed with the aligned latent space and high-fidelity human-hand data, we develop a progressive mixed-to-human training recipe comprising mixed human-
robot co-training and human-hand online correction post-training. Through mixed co-training, LaST-HD improves generalization to novel objects, scenes, and positions using only human-hand demonstrations. With online correction, LaST-HD further adapts to novel environments and achieves over 90% accuracy using only 20 minutes of OOL glove data.
Read the full abstract ▾
All authors Jiaming Liu, Yinxi Wang, Chenyang Gu, Siyuan Qian, Xiangju Mi, Hao Chen, Jiawei Chen, Qingpo Wuwu, Xiaoqi Li, Nuowei Han, Yiming Zhang, Xuheng Zhang, Yang Yue, Yeqing Yang, Lei Wang, Peng Jia, Hao Tang, Shanghang Zhang
Robotics
CoorDex: Coordinating Body and Hand Priors for Continuous Dexterous Humanoid Loco-Manipulation
Sikai Li, Shuning Li, Zhenyu Wei et al. (+3 more)
Humanoid loco-manipulation is often simplified into a stop-and-go process: walking to an object, stopping to manipulate it, and then resuming locomotion. It also commonly relies on low degree-of-freedom (DoF) end effectors that behave like an open-close grasp primitive. We introduce CoorDex, a learning pipeline that converts high-dimensional body and dexterous hand control into coordinated latent residual control, enabling high-DoF dexterous loco-manipulation on the move. Starting from simulated whole-body and hand demonstrations, CoorDex trains privileged motion tracking teachers for the
humanoid body and dexterous hand, distills them into proprioception-conditioned latent priors, and uses the frozen priors as the action space for downstream residual reinforcement learning. A coordinated latent residual policy composes these priors through shared task context and separate body-hand residual heads, preserving natural whole-body motion while improving finger-level contact reliability. CoorDex enables a Unitree G1
humanoid with a 20-DoF WUJI hand to execute dexterous manipulation while in motion, including non-stop bottle grasping and carrying, fridge door opening on the move, and cube pick-and-turn. Ablations on the walk-grasp-carry task show that joint-space PPO, joint-space hand control, and monolithic latent prediction all fail under the same reward budget, while the latent-prior interface and coordinated residual structure make high-dimensional contact-rich loco-manipulation trainable. Project Page: https://skevinci.github.io/coordex/
Read the full abstract ▾
All authors Sikai Li, Shuning Li, Zhenyu Wei, Yunchao Yao, Chenran Li, Mingyu Ding
Robotics
A Reduced Order Model for Emergent Mechanics in Woven Systems
Anvay A. Pradhan, Evgueni T. Filipov and Talia Y. Moore
Woven structures exhibit rich mechanical behaviors including anisotropic stiffness, shear-induced locking, and crimp interchange that emerge purely from the
geometric arrangement of individual weavers rather than from constituent material properties. Existing models either homogenize these interactions or resolve them at prohibitive computational cost. We introduce a reduced-order model that bridges this gap by representing individual weaver interactions through a system of nodes and four physically interpretable stiffness elements capturing axial deformation, in-plane uncrimping, inter-weaver shear, and frictional slip.
Eigenvalue analysis of the unit
cell confirms that the lowest-energy deformation modes correspond directly to known weave-specific phenomena, and that each element is necessary for a complete kinematic and mechanistic description. Element stiffness parameters are calibrated against empirical three-point bending and shear data, achieving agreement within 5% across varied weaver widths and spacings. The validated model is then applied to demonstrate capabilities beyond the reach of continuum approaches including: the emergent Poisson's response arising from crimp interchange, stepwise force reduction during progressive weaver pullout, stress localization under three distinct tearing configurations, and programmable mechanical anisotropy through spatially graded weaver stiffness. The physical transparency and computational efficiency of the framework position it as a practical tool for the analysis and design of woven architected materials with programmable mechanical response.
Read the full abstract ▾
Robotics
Flatness Preserves Instruction Following in Vision-Language-Action Models
Haochen Zhang and Yonatan Bisk
Vision-
language-action (VLA) models have the potential for open-world generalization by leveraging pretrained
vision-
language representations, yet downstream finetuning on limited
robot data often degrades these representations, leading to brittle policies that ignore
language instructions in favor of visual shortcuts, a failure mode we term instruction blindness. We hypothesize that standard finetuning with limited data applies gradients to a sparse set of points, which manifests as a sharp loss landscape with high-
curvature minima. We propose to address this directly through flatness-preserving optimization while finetuning on the exact same data, where learning a flatter landscape results in a model more robust to perturbations in the weight space. Specifically, we demonstrate that simply applying sharpness-aware minimization during VLA finetuning significantly improves instruction following by over 60% across multiple simulation and real-world
benchmarks without additional data, architectural modification, or retraining. We further analyze the effect of selective sharpness, quantify its effects, and show that our approach is complementary to existing guidance techniques. Project page can be found at https://haochenz11.github.io/papers/flatness-vla/.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
Learning Process Rewards via Success Visitation Matching for Efficient RL
Raymond Tsao, Andrew Wagenmaker and Sergey Levine
In many modern applications of reinforcement learning (RL), the natural reward for a task of interest is inherently sparse: a reward of 0 is given everywhere except when the task is completed, when a reward of +1 is given. Training a policy to maximize such a sparse reward requires solving a challenging credit assignment problem, leading to slow or ineffective RL improvement. We propose a simple approach to transform a sparse outcome reward into a dense process reward. Our approach relies on training a discriminator to distinguish between previous successful and unsuccessful episodes, and using this discriminator to incentivize the RL-learned policy to match the state-action visitations of successful episodes, while avoiding those of unsuccessful episodes. By incentivizing the policy to match the visitations over all states, not just those that correspond to task success, this reward provides dense feedback on whether progress is being made towards task completion, and, we show, provably achieves this without changing the optimal policy. Focusing on finetuning of robotic control policies, we demonstrate that our approach leads to significantly faster RL finetuning performance on both simulated and real-world manipulation tasks, as compared to simply maximizing the sparse outcome reward.
Read the full abstract ▾
Robotics
Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitation
Kuancheng Wang, Vaibhav Saxena, Shuo Cheng et al. (+2 more)
Most imitation learning methods assume full observability in table-top settings. In practice, objects are often occluded, requiring
robots to both search and act, and learning this coupled behavior from limited demonstrations remains challenging. We propose See2Act, an imitation learning approach that conditions action prediction on a sequence of actively inferred viewpoints at test time, by coupling action denoising with viewpoint refinement. The policy is trained using camera poses anchored to keyframe actions from offline demonstrations, enabling implicit learning of where to see, while learning how to act. We empirically demonstrate that in Ravens the policy recovers informative viewpoints under severe occlusions, and on RLBench tasks it improves performance by up to 33% points over prior methods. In the real world, we collect 50 demonstrations in a digital twin and achieve zero-shot sim-to- real transfer on pick-and-place tasks using depth observations. The policy handles significant occlusions, showing that learned viewpoint
reasoning enables robust manipulation under partial observability.
Read the full abstract ▾
All authors Kuancheng Wang, Vaibhav Saxena, Shuo Cheng, Yotto Koga, Danfei Xu
Robotics
dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Models
Yuhao Wu, Yitian Liu, Weijie Shen et al. (+13 more)
Vision-
Language-Action (VLA) models have established a powerful paradigm for generalist robotic manipulation by grounding control into the semantic
reasoning of VLMs. Prevailing architectures typically model actions continuously via
diffusion or flow processes, or discretely through either autoregressive generation or parallel decoding. Recently, Discrete
Diffusion VLAs (dVLAs) have emerged as a distinct alternative, unifying
vision,
language, and action into a single discrete
token space via masked generative modeling. While combining iterative refinement with unified representations, its training has thus far been restricted to Supervised Fine-Tuning (SFT), leaving the potential of Reinforcement Learning (RL) for further policy refinement largely unexplored. A fundamental challenge in RL for dVLAs is that the marginal probability of the final action generated by dVLAs remains intractable. To solve this problem, we propose dVLA-RL, shifting the learning objective from the marginal action probability to the joint probability of the sampled generation path. Specifically, by modeling the denoising process as a Markov Decision Process (MDP), we mathematically formulate this path probability as a product of step-wise transitions. This trajectory-level objective provides a unified formulation that natively accommodates variable denoising steps. Leveraging this intrinsic fexibility, we introduce a unified step scheduling approach for complex multi-task learning, tailoring denoising steps to specific task complexities to maximize both success rates and computational effciency. Extensive evaluations demonstrate that our approach achieves a success rate of 99.7% on LIBERO. Furthermore, it establishes strong VLA-based results on RoboTwin 2.0 by delivering a 30.6% improvement over the SFT baseline, remaining competitive with strong World-Action Model baselines.
Read the full abstract ▾
All authors Yuhao Wu, Yitian Liu, Weijie Shen, Mishuo Han, Wenjie Xu, Haotian Liang, Zhongshan Liu, Yinan Mao, Lei Xu, Xinping Guan, Ru Ying, Ran Zheng, Wei Sui, Xiaokang Yang, Wenbo Ding, Yao Mu
Robotics
RECALL: Recovery Experience Collection for Active Lifelong Learning in Vision-Language-Action Models
Ulas Berk Karli and Tesca Fitzgerald
Vision-
Language-Action (VLA) models are commonly fine-tuned through passive imitation learning, where additional demonstrations are collected for tasks where the policy performs poorly. This approach incurs several downsides: it requires the
robot to fail before data collection is triggered, provides little guidance about which states require supervision, and wastes demonstrator effort on redundant parts of the task where the policy already performs well. In this paper, we propose an active, continual learning paradigm for VLAs. We demonstrate that active, uncertainty-guided data collection leads to more efficient fine-tuning than when using passively-collected demonstrations. However, we also find that fine-tuning only on actively-collected recovery data leads to catastrophic forgetting. We evaluate techniques for continual learning, including replay-based data mixing and elastic weight consolidation, and identify tradeoffs between plasticity to uncertainty-guided recovery data and retention of previously learned behaviors. Overall, our work contributes an empirical study of active continual learning for autoregressive VLAs, establishing that uncertainty-guided recovery demonstrations can improve adaptation efficiency while also revealing open challenges when targeted new data is incorporated into large
robot policies.
Read the full abstract ▾
Robotics
Autonomous Subsea Cable Search and Tracking with Graph-Optimised Priors and Visual Tracking
Ibrahim Fadhil Djauhari, Adrian Bodenmann, Samuel Simmons et al. (+6 more)
Global communications rely on subsea cable infrastructure that remains vulnerable to damage from natural hazards and human activity. Autonomous
underwater vehicles (AUVs) offer an efficient means to inspect long sections of exposed cable, but uncertainty in cable route maps, small cable diameters and partial burial makes continuous tracking a challenge. This paper presents a novel cable search and tracking method that leverages uncertain prior cable route maps. Graph-based optimisation continuously update the cable route to remain consistent with visual observations. Route uncertainty is constrained as a function of distance from observations using physics-based catenary models that account for cable parameters (i.e., lay depth, diameter, and density), bounding the search space to physically feasible regions and improving search efficiency. Cable detection is performed using a semi-supervised classifier running in real-time on-board a camera-equipped AUV. These detections both update the graph-based optimisation and enable visual cable tracking. When tracking is lost due to misclassification, burial or imperfect control, the bounded search space enables efficient recovery. The approach was demonstrated in field trials using the University of Southampton's Smarty200 AUV. The system successfully located the cable despite deliberate errors in it initial cable route map, updating this to be consistent with observations and using visual tracking to inspect up to 59% of a 120m test cable, with successful recovered after tracking loss.
Read the full abstract ▾
All authors Ibrahim Fadhil Djauhari, Adrian Bodenmann, Samuel Simmons, Cailei Liang, David White, Susan Gourvenec, Tom Bennetts, Darryl Newborough, Blair Thornton
Robotics
Real-Time Multimodal Activity-Aware Error Detection in Robot-Assisted Surgery
Seyed Hamid Reza Roodabeh, Zongyu Li and Homa Alemzadeh
Robot-assisted minimally invasive surgery improves
surgical precision but introduces complexity, making technical error detection essential for ensuring patient safety. Current executional error detection methods using
video data often overlook fine-grained contextual descriptions of activities and error types within the hierarchical structure of
surgical procedures. They also under-utilize complementary
multimodal information. We propose a unified framework for executional error detection that leverages
multimodal input, including
video, kinematics, and descriptive textual
prompts. Through activity prompting, we integrate descriptive
language in gesture-level activities, instrument-object interactions, and error definitions. We also introduce activity-aware visual
embeddings derived from
vision encoders pretrained on
surgical activity labels to compare the effectiveness of contrastive
language-image
embeddings with traditional image-based
embeddings for error detection. By seamlessly integrating kinematic data with
video and textual modalities, our framework significantly improves error detection performance. Achieving up to 5% and 16.6% F1 score improvements over state-of-the-art baselines on the JIGSAWS and SAR-RARP50
datasets, respectively, we demonstrate the value of combining curated textual
prompts with
multimodal data for accurate error detection.
Read the full abstract ▾
Robotics
KEMO: Event-Driven Keyframe Memory for Long-Horizon Robot Manipulation with VLA Policies
Yihan Zeng, Minghao Ye, Yiyuan Chen et al. (+4 more)
Long-horizon
robot manipulation remains challenging because similar observations may occur at different execution stages, while the appropriate action depends on previously completed operations.
Memory can address this ambiguity by enabling policies to infer task progress from execution history. However, existing
memory-augmented approaches often either retain dense histories that require compression or rely primarily on recent context that may discard earlier task-relevant events. In this work, we propose propose KEMO, a lightweight plug-in
memory framework that automatically selectively preserves keyframes associated with task-relevant state changes for VLA policies. KEMO combines
robot kinematics with visual filtering to detect events, encodes the selected keyframes as compact temporally ordered
memory tokens, and integrates them with current visual features through cross-
attention and gated residual fusion for VLA training. The detected events also define higher-weight training samples near critical transitions. We evaluate KEMO on various real-world dual-arm manipulation tasks spanning 2 to 6 scored subtasks, and trajectory length ranging from 830 steps to 2846 execution steps (durations from 28 to 95 seconds). Compared with the
memory-free baseline (e.g., $\pi_{0.5}$), KEMO improves aggregate Task Success Rate by 23.6% and Stage Completion Rate by 34.1%. Ablations show that event-driven keyframe selection outperforms uniform sampling and recent-frame retention, while the proposed gated fusion and keyframe-aligned loss weighting provide complementary gains.
Read the full abstract ▾
All authors Yihan Zeng, Minghao Ye, Yiyuan Chen, Yide Shentu, Philipp Wu, Zike Yan, Zhongyu Li
Robotics
A Generative Model for Closed-Loop Microsimulation of Signalized Intersections
Yash Ranjan, Rahul Sengupta, Anand Rangarajan et al. (+1 more)
Traffic microsimulators rely on hand-crafted behavior models that reproduce aggregate flow but miss the heterogeneous interactions between vehicles at signalized intersections. Learned trajectory predictors capture richer interactions but are short-horizon and tend to be unstable when run in closed loop. We present Enactor, an actor-centric generative model for closed-loop intersection microsimulation. The model focuses on vehicles; pedestrians are included as context that can influence vehicle decisions but not predicted. Dynamic actors and lane polylines are encoded in polar coordinates referenced to the intersection center. A
transformer with separate spatial and temporal
attention blocks predicts a distribution over each actor's next-step motion ($s$, $\alpha$). Training uses a closed-loop curriculum so the model is exposed to its own predictions. We evaluate Enactor in two regimes. In a 4000-second simulation-in-the-loop test at two intersection geometries, Enactor controls every dynamic vehicle against a continuously refreshing actor set rather than the fixed cohort that learned trajectory predictors are usually evaluated against. It recovers the SUMO data generator's speed and travel-time distributions with KL divergence over an order of magnitude lower than a recent
transformer baseline on travel time, and substantially lower on speed (roughly $5\times$ lower at Site 1), and reduces red-light violations relative to the same baseline by more than an order of magnitude. An ablation isolates the leader rear-bumper feature as the change with the largest effect on intersection-aware safety metrics. We also evaluate on real-world field data and apply the same architecture to naturalistic vehicle trajectories from a fish-eye camera at a signalized intersection and evaluate it on multi-horizon predictive tasks. Enactor outperforms a constant-velocity baseline at every horizon evaluated.
Read the full abstract ▾
All authors Yash Ranjan, Rahul Sengupta, Anand Rangarajan, Sanjay Ranka
Robotics · Cross-listed from cs.MA
Decentralized Autonomous Traffic Management through Corridor Networks
Jasmine Jerry Aloor, Aadarsh Govada and Hamsa Balakrishnan
As autonomous aircraft are introduced at scale and traffic density increases, centralized management becomes insufficient to coordinate the large numbers of crewed and uncrewed aircraft. Dedicated Advanced Air Mobility (AAM) corridors have therefore been proposed for organizing high-density autonomous traffic flows. The desire to scalably provide autonomous aircraft flexibility in trajectory planning motivates the development of decentralized approaches to traffic management in AAM corridors. In this work, we extend a multi-
agent reinforcement learning (MARL) approach to address the challenge of decentralized traffic flow management in air corridor networks. We test policies trained in a single-corridor setting on increasingly complex multi-corridor networks with combinations of merges and splits in a zero-shot manner. Experimental results demonstrate that learned behaviors transfer well to scenarios with varying traffic density, network geometry, and heterogeneous vehicle performance, without needing centralized coordination or model retraining. We evaluate system-level performance in terms of conformance to corridor boundaries, completion rates, average speeds, distance traveled, and maintenance of inter-aircraft separation. We find that although our policies require only locally coordinated entry, traversal, and exit behaviors, they collectively produce desirable traffic flows through the corridor network.
Read the full abstract ▾
Robotics · Cross-listed from cs.CR
A Watermark for Vision-Language-Action and World Action Models
Yule Liu, Shuai Liu, Jiaheng Wei et al. (+1 more)
Vision-
language-action (VLA) models and world-action models (WAM) are the generative models now driving general-purpose
robot control, turning raw camera input directly into motor commands. They are increasingly deployed as black-box services, where a partner runs the policy through an interface while the owner keeps the weights private. Training such a model takes proprietary data and heavy computational power, making the deployed model itself a valuable intellectual property. To address this, we propose the keyed latent-provenance verification method, which fingerprints the policy through the seed of the Gaussian noise vector that the models draw before generation. At the injection stage, the owner swaps this seed for a keyed one with the same distribution as ordinary noise, so the fingerprinted actions are statistically identical to those of an ordinary run and an adversary watching the output finds no signal to detect or remove. At the verification stage, the owner runs the suspect model under authorized access and records the action channels the
robot executes, a partial and possibly post-processed view of the policy's output. From this view, the verifier recovers the seed by gradient-based maximum a posteriori (MAP) optimization, tests it for the secret key to score each rollout, and aggregates these scores into a single decision on whether the suspect model belongs to the owner. We evaluate the method on two representative models across two
robot suites. The experiments cover detection of the fingerprint, identification of which of several keys a suspect carries, robustness to a range of attacks, and an analysis of why the design works. Across both models, the fingerprint can be detected reliably with little change to task performance, and it remains detectable under output-side removal attacks and weight-level edits.
Read the full abstract ▾
All authors Yule Liu, Shuai Liu, Jiaheng Wei, Xinlei He
Robotics
HoloAgent-0: A Unified Embodied Agent Framework with 3D Spatial Memory
Xiaolin Zhou, Liu Liu, Tingyang Xiao et al. (+9 more)
LLM
agents follow a practical execution loop in digital environments: they reason over structured states, invoke tools, inspect feedback, and revise actions. Extending this loop to physical
robots is difficult because physical execution is continuous, embodiment-dependent, uncertain, and constrained by safety. Existing
embodied-AI systems have advanced manipulation, spatial understanding,
navigation, and
humanoid control, but these capabilities often remain specialized modules or loosely coupled decision loops. In this work, we introduce HoloAgent-0, a unified
embodied agent framework for real-world
robot deployment.
Embodied AgentOS converts
language instructions into executable skill graphs, schedules
robot resources, monitors execution, and triggers clarification or re-planning from runtime feedback. HoloAgent-0 organizes heterogeneous
robot models and controllers through three coupled layers:
Embodied AgentOS for closed-loop execution, 3D spatial
memory for physical world grounding, and
embodied skills for
robot action. We deploy HoloAgent-0 on real hardware and evaluate its spatial
memory, long-horizon
navigation, and closed-loop execution across motion generation, object search, cross-
robot coordination, and mobile manipulation.
Read the full abstract ▾
All authors Xiaolin Zhou, Liu Liu, Tingyang Xiao, Wei Feng, Fa Fu, Xinrui Meng, Xinjie Wang, Jialiang Han, Boyang Yu, Yun Du, Wei Sui, Zhizhong Su
Robotics
BiliVLA: Scene-Aware Vision-Language-Action Model with Reinforcement Learning for Autonomous Biliary Endoscopic Navigation
Jinsong Lin, Chi Kit Ng, Zhiyong Xiong et al. (+8 more)
Endoscopic retrograde cholangiopancreatography (ERCP) demands precise endoscopic
navigation and stable biliary cannulation within a narrow
monocular field characterized by specular reflections, partial occlusions, and frequent tissue contact. Although recent robotic systems and
vision-based assistance techniques improve operator ergonomics and provide perceptual cues, their performance degrades under pronounced anatomical variability and safety-critical visual artifacts, which hinders reliable autonomy in cannulation-grade procedures. Here, we present BiliVLA, a scene-aware
Vision-
Language-Action (VLA) framework that formulates biliary endoscopic
navigation as an instruction-conditioned visuomotor learning problem. Given an endoscopic observation and a stage-specific
language instruction, BiliVLA jointly predicts the target category, a grounded bounding box, and a discrete three-degree-of-freedom (3-DoF) motor command for a continuum endoscope. The proposed framework incorporates scene-aware supervision to improve semantic target consistency and safety-aware recovery supervision to induce conservative retreat behaviors under luminal wall contact. A key component of BiliVLA is a two-stage training paradigm that combines grounding-enhanced supervised fine-tuning (SFT) with Group Relative Policy Optimization (GRPO), thereby improving action reliability and decision consistency during closed-loop
navigation. Across three ERCP subtasks, BiliVLA achieves the best overall performance in physical phantom experiments, with a total mIoU of 0.9625, an overall action precision of 91.96%, and an overall success rate (SR) of 84.85%. These results indicate that integrating semantic grounding, scene-aware learning, and reward-guided optimization strengthens perception–action
alignment and enables more robust autonomous biliary endoscopic
navigation.
Read the full abstract ▾
All authors Jinsong Lin, Chi Kit Ng, Zhiyong Xiong, Zikang Pan, Yihan Hu, Tabassum Tamima, Ziyi Hao, Eddie Cheung, Jiewen Lai, Huxin Gao, Hongliang Ren