Robotics
Active Interaction-Aware Model Predictive Path Integral via Ego-Conditioned Generative Predictions
Khaled A. Mustafa, Mohamed-Khalil Bouzidi, Christian Schlauch et al. (+4 more)
Dense traffic is inherently interactive. The ego vehicle and surrounding
agents continuously influence each other's reactions, making "what-if"
reasoning essential for safe and efficient driving. To enable such an active interaction-aware behavior, we propose a planning framework that integrates an ego-conditioned generative autoregressive prediction model within Model Predictive Path Integral (MPPI) control. The generative prediction model outputs
stochastic, multi-modal predictions of surrounding
agents conditioned on each of the ego's considered future actions. A nested sampling scheme enables tractable evaluation of expected cost and collision risk under the induced distribution. This formulation allows the ego to actively probe how different candidate actions shape the interaction outcomes and to identify actions that reduce ambiguity in uncertain interactions. Closed-loop simulations demonstrate improved safety and efficiency compared to conventional predict-then-plan and passive interaction-aware approaches.
Read the full abstract ▾
All authors Khaled A. Mustafa, Mohamed-Khalil Bouzidi, Christian Schlauch, Ahmad Gazar, Nadja Klein, Joerg Reichardt, Javier Alonso-Mora
Robotics
Enhanced Real-Time 6-DOF Extended Reality Catheter Tracking for Evaluating Potential Improvement in Efficiency, Precision, and Depth Perception for Cardiac Interventions
Mohsen Annabestani, Sandhya Sriram, Andrew Kuzemczak et al. (+3 more)
Despite advances in 3D ultrasound, most percutaneous cardiac interventions still rely on 2D visualization, limiting depth perception and spatial understanding. To address this challenge, we developed an Extended Reality (XR)-based platform that enables real-time six-degree-of-freedom (6-DOF) catheter tracking and visualization within a patient-specific 3D heart model. The system combines a custom machine-
vision algorithm for 5-DOF catheter tracking with a 3D-printed electromechanical encoder that measures catheter roll, providing complete 6-DOF motion reconstruction. In a proof-of-concept study, 20 novice medical students navigated an intracardiac echocardiography (ICE) catheter to six anatomical targets using either immersive 3D visualization or a conventional 2D cathlab-style view. Participants in the 3D condition completed the task in 54.6 seconds and traveled 1,939 mm on average, compared with 267.5 seconds and 7,854 mm in the 2D condition. Therefore, the XR-based 3D system was more than 5x faster and required ~5x less catheter travel. The 3D mode also improved targeting precision and reduced performance variability. Participants consistently rated immersive visualization higher for accuracy, speed, usability, and clinical value. Kinematic analysis showed smoother depth-axis navigation in 3D, whereas 2D users relied on repeated corrective movements. These findings demonstrate that XR-based visualization can substantially improve procedural training efficiency, precision, and motor control.
Read the full abstract ▾
All authors Mohsen Annabestani, Sandhya Sriram, Andrew Kuzemczak, S. Chiu Wong, Alexandros Sigaras, Bobak Mosadegh
Robotics
AdaDexGrasp: Adaptive Dexterous Grasping via 3D Visuo-Tactile Representation Fusion
Xirui Liang, Jiaqi Liang, Jingkai Xu et al. (+6 more)
Humans achieve stable and adaptive grasps by seamlessly integrating visual perception and tactile feedback, a capability that remains challenging to replicate in robotic systems. Existing robotic grasping approaches predominantly rely on visual inputs and lack mechanisms for tactile-guided adaptation after contact, limiting robustness and generalization. To address this challenge, we propose a unified visuo-tactile-fusion grasping framework that integrates grasp generation, feasibility prediction, and adaptive refinement. At its core, our method introduces an efficient visuo-tactile representation that tightly fuses object geometry with tactile feedback by associating tactile signals with finger identities. This unified representation supports contact-aware grasp pose generation during planning and tactile-guided refinement after contact, enabling the system to reason about fine-grained finger-object interactions and adjust grasps dynamically. Comprehensive experiments in both simulation and real-world environments demonstrate that our approach significantly enhances grasp success rates and generalization across diverse objects.
Read the full abstract ▾
All authors Xirui Liang, Jiaqi Liang, Jingkai Xu, Yuran Wang, Ruochong Li, Yuanpei Chen, Masayoshi Tomizuka, Wei Zhan, Ruihai Wu
Robotics
LIRA: Local Cross-Layer Information Routing for Vision-Language-Action Decoding
Zhewei Zhang, Puyue Wang, Guanren Qiao et al. (+10 more)
Vision-
Language-Action (VLA) models transform representations from pretrained
vision-
language models (VLMs) into
robot actions, yet the interface that routes intermediate VLM features into action decoders remains underexplored. Existing designs either expose only a narrow part of the representation hierarchy or rigidly match each decoder block to one VLM layer, restricting access to complementary task evidence across depths. We introduce LIRA, a local cross-layer action-conditioning mechanism that formulates VLM-to-action conditioning as depth-aware information routing. LIRA operates on task-
token features and LIRA Query features derived from intermediate VLM states, then assigns each Parallel Fusion Block a depth-aligned local window centered on its corresponding VLM layer. Parallel Fusion Blocks aggregate neighboring LIRA Query features and integrate them with task-
token features and proprioceptive inputs before action prediction. This routing interface leaves the backbone architecture, action decoder, and supervised training recipe unchanged. Across LIBERO, LIBERO-Plus, CALVIN ABC$\rightarrow$D, and real-world manipulation, LIRA improves the principal aggregate metrics over the VLA-Adapter baseline under the same 0.5B-parameter configuration. In zero-shot transfer to LIBERO-Plus, LIRA increases average success from 59.1% to 78.0%, an 18.9-point gain indicating improved robustness under controlled distribution shifts.
Read the full abstract ▾
All authors Zhewei Zhang, Puyue Wang, Guanren Qiao, Yijie Weng, Jiawei Hu, Guo Li, Lujia Wang, Junyan Wang, Tao Gu, Hongliang Lu, Guiliang Liu, Hong Jia, Xinhu Zheng
Robotics
SoRoMoX: Fast, Differentiable, and Parallelizable Soft Robot Models
Maximilian Stölzle, Solange Gribonval, Daniel Feliu-Talegon et al. (+9 more)
Reduced-order models based on Cosserat-rod theory are now well established, and modeling theory is no longer the primary bottleneck in soft-
robot control. Their implementations, however, do not support the differentiable, GPU-parallel, and control-oriented workflows that underpin advanced rigid-robotics applications. Here, we fill this gap with SoRoMoX (Soft
Robot Models in JAX), a fully numerical, JIT-compilable Python/JAX framework. SoRoMoX implements articulated, Piecewise Constant Strain, and Variable Strain models through a unified, control-ready interface that provides inertia
matrices, gravitational and elastic forces, Jacobians, and their derivatives. To our knowledge, it is the first rod/strain-based soft-
robot modeling framework that runs directly on GPUs, with Warp kernels accelerating parallel continuum-model execution, and is end-to-end differentiable with respect to states, inputs, and parameters. Sequential CPU rollouts are up to 27.0 times faster than SoRoSim, while GPU-parallel GVS rollouts increase throughput by up to 679.7 times. This performance enables workflows that were previously impractical or impossible: static-
equilibrium system identification with 66% lower marker RMSE; residual-force learning with a further 64% reduction; computed-torque tracking with RMSE reduced by a factor of approximately 500 relative to model-free PD; control-gain optimization with up to 98% lower loss than untuned gains; safety-constrained control using high-order control barrier functions to keep the peak contact force within a prescribed 5 N bound, compared with 33.5 N without the safety constraint; and reinforcement-learning policy training up to 7 times faster than a CPU PyElastica discrete-rod baseline through massively parallel rollouts.
Read the full abstract ▾
All authors Maximilian Stölzle, Solange Gribonval, Daniel Feliu-Talegon, Vito Daniele Perfetta, Michele Martini, Chuhan Zhang, Kiwan Wong, Mohammed Tarnini, Anup Teejo Mathew, Federico Renda, Daniela Rus, Cosimo Della Santina
Robotics
Plan-and-Avoid: Real-Time Aircraft Trajectory Coordination in a Multi-Agent Environment
Huseyin Emre Tekaslan, Ella M. Atkins and Natasha Neogi
This paper presents a real-time Plan-and-Avoid (PAA framework for coordinating cooperative multi-
agent airspace operations around a declared priority trajectory. The priority trajectory represents an aircraft flight plan that must be preserved because of constrained maneuverability, an emergency, a mission-critical task, or assigned operational priority. The framework predicts uncertainty-aware, well-clear separation violations with surrounding traffic and, when the priority plan alone cannot maintain separation, generates vehicle-constrained unilateral advisories that modify nearby aircraft trajectories to maintain well-clear separation for all traffic. The approach is applicable to any declared priority trajectory. This paper demonstrates the Plan component using a contingency landing planner to generate candidate priority trajectories. PAA then identifies nearby aircraft passing too close to this priority trajectory and issues Avoid resolution advisories to these aircraft. The framework is tested using real-world Automatic Dependent Surveillance-Broadcast (ADS-B) traffic from the Washington, D.C., airspace across more than 900 forced-landing cases, totaling over 140 hours of simulated flight. The PAA framework generates feasible cooperative advisories for all 575 unique conflict encounters, with a worst-case end-to-end response time of 5.7 s on a personal computer, including priority trajectory planning, advisory generation, and 1 s two-way datalink delay. In total, 93.5% of generated advisories satisfy the 35 s RTCA DO-365 Detect-and-Avoid temporal threshold. These results demonstrate low-latency coordination for preserving priority trajectories while maintaining well-clear separation through real-time automated advisory generation. Future work will quantify advisory-induced delays and their operational impacts.
Read the full abstract ▾
Robotics
SyncSBC: Decentralized Swarm Behavior Prediction for Synchronized Autonomous Control
Varun Raveendra, Connor Mattson and Daniel S. Brown
Robot swarms utilize many independent limited-sensing
agents to produce complex emergent behaviors without requiring centralized control. However, little research explores how
agents can infer swarm-level behavior from purely local perception, a capability critical for detecting faults and behavior changes. In this paper, we introduce Synchronized Swarm Behavior Classification (SyncSBC), which combines improvements in machine learning and distributed consensus to classify collective swarm behavior and synchronize swarm decision-making in an entirely decentralized manner. We show that SyncSBC achieves high classification accuracy and low synchronization delay, making it suitable for real-world deployment. Finally, we use SyncSBC to demonstrate two promising swarm applications on real
robots where we show that swarms utilizing SyncSBC can accurately identify anomalies in
robot behavior and autonomously coordinate collective changes in swarm behavior.
Videos, code and supplemental experiments are available at https://sites.google.com/view/sync-sbc/home.
Read the full abstract ▾
Robotics
A Disturbance in the Force: Force Actuation on the RAVEN II Surgical Robot with Parallel Motor-Cable Units
Haonan Peng, Dun-Tin Chiang, Jordan Hendricks et al. (+5 more)
Difficulty in haptic feedback for surgical
robots has been a long-term problem for decades. In recent years, learning-based force estimation from
robot states suggests desirable accuracy without the necessity of extra sensors. However, challenges remain in obtaining representative training data in which the
robot moves in the workspace under various external forces. In this work, a parallel motor-cable system is developed. With six motor-cable units installed around the
robot workspace, cables with controllable tension connected to the
robot end-effector can provide the desired external force without interfering with the movement of the surgical
robot. The development of the system includes motor-unit hardware, control software, sensor drivers, simulations, and more. Preliminary experiments suggest an accuracy of force actuation with errors less than 1 N.
Read the full abstract ▾
All authors Haonan Peng, Dun-Tin Chiang, Jordan Hendricks, Andrew Lewis, Jared Shing, Haokun Feng, Yun-Hsuan Su, Blake Hannaford
Robotics
Safe and Robust Neural Policy Learning with Statistical Verification for Sim-to-Real Deployment in Robotics
Riccardo Curcio, Hongpeng Cao and Marco Caccamo
Synthesizing safe and robust neural controllers in simulation for reliable sim-to-real deployment remains a critical challenge in robotics. Existing learning-based methods typically lack safety and performance guarantees over an explicitly defined operating region, while post-training verification techniques provide no mechanism to refine controllers when safety violations are detected. To bridge this gap, we propose a curriculum-driven framework that tightly integrates scenario-based
Evolution Strategy with Statistical Model Checking-based verification in a closed-loop procedure. Starting from a candidate region, our approach co-optimizes policy performance while progressively enlarging its safe operating boundaries. Upon termination, it yields a neural controller together with a region over which safety and performance are statistically verified. Extensive evaluations on Cartpole and 3D Quadrotor
benchmarks, showing 6.14x and 224.04x expansions, respectively, of the safe operating region over mathematically certified ones, together with physical experiments under both nominal conditions and severe dynamic perturbations, demonstrate that our learned controllers consistently outperform established control-theoretic and learning-based baselines. Furthermore, we show that the size of the verified region serves as a quantitative indicator of policy quality before deployment. These results establish our framework as an automated pipeline for learning, assessing and deploying safe and robust neural controllers from simulation to reality.
Read the full abstract ▾
Robotics
Fast and Accurate: An Adaptive VLA Inference Framework through Environment-aware Model Selection
Yuewei Sun, Lang Qin, Zechuan Tian et al. (+11 more)
Embodied intelligence demands both long-horizon
reasoning and real-time closed-loop responsiveness. Recent dual-system
Vision-
Language-Action (VLA) architectures combine fast reactive control with slow deliberative
reasoning to balance inference speed and task success rate. However, existing dual-process VLAs tightly couple the fast module to intermediate representations of the slow module, necessitating end-to-end joint training and limiting modularity, extensibility and flexible system switching. In this paper, we propose Environment-aware Model Selection (EMS), an adaptive VLA inference framework that switches between two fully decoupled systems of different scales through environment-aware model selection. The large-scale deliberative system provides globally consistent trajectory planning to ensure task success, while a lightweight reactive system enables high-frequency closed-loop control. A reinforcement-learning-based switching policy dynamically selects which system to invoke based on real-time feedback, enabling sparse use of the slow system and thereby balancing pretrained knowledge utilisation with runtime efficiency. Our design offers three key advantages over prior hierarchical VLA frameworks: (1) a fully decoupled and modular dual-system architecture that supports plug-and-play model replacement; (2) an adaptive, environment-aware switching strategy; (3) high-frequency inference for responsive closed-loop control. We extensively evaluate EMS in both simulation and real-world environments. On the LIBERO
benchmark, EMS achieves success rates comparable to the large-scale baseline while increasing the effective action frequency to 93.4 Hz. The framework further demonstrates strong extensibility in real-world dual-arm manipulation tasks, where it accelerates task completion while maintaining robust performance.
Read the full abstract ▾
All authors Yuewei Sun, Lang Qin, Zechuan Tian, Jingwen Li, Guiqin Wang, Shengzeng Huo, Wenxin Ren, Tao Fang, Xiaochen Zhang, Guanqing Deng, Xiang Wang, Xiaowen Dong, Qinghai Guo, Yuxin Ma
Robotics
$\omega$-0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation
Zhe Li, Zhenzhe Zhang, Yangyang Wei et al. (+8 more)
Humanoid household tasks often require concurrent loco-manipulation, where the
robot must move, adjust posture, maintain balance, and manipulate objects as a single coordinated behavior. Yet existing humanoid policies typically decompose locomotion and manipulation, while recent world-action models remain either arm-centric or
video-centered. We present $\omega$-0, a latent predictive whole-body world-action model for real-world humanoid concurrent loco-manipulation. Given a
language instruction, current visual observation, and
robot proprioceptive state, $\omega$-0 directly predicts controller-compatible whole-body action latents for real-
robot execution. Rather than reconstructing future
videos, $\omega$-0 learns compact future observation
embeddings as a lightweight predictive objective, coupling latent visual foresight with
diffusion-based whole-body action generation. The model supports egocentric RGB, exocentric RGB, and exocentric depth inputs, and leverages controller-based simulation replay to ground human/public visual-motion priors into
robot-executable action latents. We further collect $\omega$-HOME, a 40+ hour real-world household humanoid
dataset with synchronized multi-view observations, whole-body SMPL motions,
robot states, and action latents. Real-world experiments on 11 household tasks demonstrate that a single $\omega$-0 model can produce smooth manipulate-while-moving behaviors and consistently outperform representative imitation learning, VLA, humanoid, and WAM baselines.
Read the full abstract ▾
All authors Zhe Li, Zhenzhe Zhang, Yangyang Wei, Wenjie Zhang, Xichen Yuan, Peiyuan Zhi, Gen Li, Xinying Guo, Fengjie Gao, Jianfei Yang, Shanghang Zhang
Robotics
DyPES-VLA: Learning Shared Dynamics Priors and Embodiment-Specific Control for Cross-Embodiment Manipulation
Junfeng Li, Junjie He, Zhide Zhong et al. (+12 more)
Vision-
Language-Action (VLA) models have become a powerful paradigm for
robot manipulation, but training a single generalist policy for heterogeneous
robot embodiments remains an open problem. Existing methods have two main limitations. First, they underuse dynamics priors shared across diverse visual and interaction data, limiting cross-embodiment transfer. Second, they require extensive manual preprocessing to convert embodiment-specific actions into a common format. To overcome these limitations, we propose DyPES-VLA, a cross-embodiment VLA that learns shared Dynamics Priors and Embodiment-Specific control. First, we learn shared dynamics priors by training the
vision-
language model (VLM) with a future-prediction objective on cross-embodiment data, driving the shared query representation to capture object motion, contact, and interaction-induced scene changes. Second, an embodiment-specific Mixture-of-Experts (MoE) action head translates these shared dynamics priors into executable controls directly in each embodiment's native action space, without manually pre-aligning heterogeneous actions into a common format. This head shares
attention layers to capture common temporal action structures, while its embodiment-specific feed-forward experts resolve the unique kinematic constraints and control semantics of distinct embodiments. As a generalist policy, our \ourmethod achieves state-of-the-art performance across simulation and real-world evaluations, reaching 98.0% success on LIBERO, 59.25% on RoboCasa-GR1, and 89.02% on RoboTwin~2.0.
Read the full abstract ▾
All authors Junfeng Li, Junjie He, Zhide Zhong, Yangyang Zheng, Pingyue Sheng, Jiayu Dong, Ruixin Li, Haodong Yan, Jiaguan Zhu, Tianran Zhang, Runze Yu, Wen Chen, Liuqing Yang, Yuxiang Gao, Haoang Li
Robotics
A Master-Slave Robot Manipulator for Needle-Based Teleoperation in MRI Chamber
Omar Curiel, Jing-Yuan Huang, Po-Chih Chen et al. (+6 more)
We present a MR safe, master-slave
robot manipulator for abdominal interventions in the MRI chamber. A human operated 2+1-DoF master controller manipulator transmits motion and force to a 2+1-DoF slave manipulator via fluid transmission. Jointly, a digital master controller provides
multimodal control capability beyond common split axis or mode switchable hybrid human-digital controller configurations found in previous studies. High input impedance, low-leakage, elastomeric fluid actuators are delegated to remote angulation control. Low-friction graphite piston cylinders are delegated to needle insertion axis remote actuation given the sub-newton force transparency and sub-millimeter motion transmission over bedside fluid piping lengths. The device enables real-time MRI guided interventions allowing manual, digital, hybrid, and collaborative control modes. Collaborative tasks such as assisted tissue penetration, fault-driven virtual fixture, and motion compensation through feedback control are presented in this paper. Preliminary MR scanner results demonstrate manipulator functional viability for an in-vivo pig experiment in bedside, manual control mode configuration.
Read the full abstract ▾
All authors Omar Curiel, Jing-Yuan Huang, Po-Chih Chen, Ji Ma, Qing Dai, Wenqi Zhou, David Lu, Holden H. Wu, Tsu-Chin Tsao
Robotics
GeniWorld: A Generalizable Interactive World Model for Robotic Manipulation via Visual Actions
Chenghao Gu, Hanyang Yu, Jingbo Zhang et al. (+7 more)
Generalist
robot policies exhibit strong capabilities, but their robustness in complex and unseen environments remains limited. Scaling
robot learning and evaluation in diverse real-world environments remains costly and challenging. Action-conditioned world models offer a promising alternative, but they often suffer from limited action controllability and poor generalization to out-of-distribution (OOD) scenarios. To this end, we present GeniWorld, an interactive world model for
robots that generalizes robustly across unseen scenarios. Building on pretrained
video generative models, we use URDF-based rendering to transform numerical actions into visual action representations, enabling spatially grounded action control. By explicitly decoupling embodiment kinematics from environmental dynamics, our model mitigates scene overfitting and facilitates modeling of
robot-environment interactions. To achieve closed-loop control, we construct an autoregressive
video prediction model integrated with high-frequency
robot kinematic control, enabling interaction with both
robot policies and human teleoperators. In our experiments, even when trained solely on limited fixed-scene data, our model achieves superior in-domain performance and robust zero-shot generalization to highly randomized, unseen environments. For downstream applications, GeniWorld serves as a scalable policy evaluator that remains reliable under environmental perturbations. Furthermore, even with limited real-world demonstrations, GeniWorld generates diverse manipulation trajectories within the world model, improving downstream policy performance and robustness in complex environments.
Read the full abstract ▾
All authors Chenghao Gu, Hanyang Yu, Jingbo Zhang, Haitao Lin, Wenyao Zhang, Jinghe Wang, Hanglei Jin, Shuzhao Xie, Jingyan Jiang, Zhi Wang
Robotics
Robot Learning from Human Demonstrations: Handwritten Alphabet Trajectories and Human-Likeness Evaluation
Alperen Kenan, Paul Bremner and Manuel Giuliani
Learning from demonstration (LfD) provides a developmental framework through which
robots can develop motor skills by observing and imitating human dynamics, reducing reliance on explicit programming to teach a skill to a
robot. The resulting human-like
robot motion is recognised as a key factor in building trust and enabling natural collaboration in human-
robot interaction. This paper presents a framework for learning human-like
robot motion from demonstration, including data collection, probabilistic trajectory learning, and perceptual user evaluation. A
dataset of 3,142 handwriting demonstrations was collected from 22 participants across all 52 Latin alphabet character-case combinations via a touchscreen teleoperation interface, capturing planar position, contact force, and timing. Building on the widely used Gaussian Mixture Model and Gaussian Mixture Regression approach for learning from demonstration, the framework is extended in this work by incorporating force and normalised time dimensions to enable richer representation of human dynamics, and adapting it to handle non-continuous, multi-segment trajectories, enabling generalisation across demonstrations. A user study with 21 participants evaluated the perceived human-likeness of the generated trajectories using a continuous scale anchored between robotic and human-like motion, normalised to 0-100 where 50 represents the neutral midpoint. The generated trajectories achieved an overall human-likeness score of 71.50 (SD=22.56), indicating that the majority of trajectories were perceived as more human-like. Participants identified
geometric positioning and trajectory sequence as the most influential perceptual factors, and reported positive attitudes toward human-like
robot behaviour. The
datasets are released as open-source, providing a reproducible
benchmark for developing and evaluating human-like
robot motion methods.
Read the full abstract ▾
Robotics
Design and Evaluation of a Touchscreen-Based Teleoperation Interface for Robotic Manipulators
Juan José García Cárdenas, Alperen Kenan, Hamidreza Raei et al. (+4 more)
Intuitive teleoperation interfaces are crucial for the safe and effective operation of robotic manipulators in challenging environments. In the nuclear industry, surface contact tasks such as swab sampling require precise path and force tracking, obstacle avoidance, and sustained operator
attention, which conventional joystick interfaces struggle to support effectively. This study designs and evaluates a novel touchscreen teleoperation interface that maps continuous finger movements directly to robotic manipulator motions, provides finer velocity control, and integrates control with visualization, enabling more natural, precise, and intuitive surface interaction than conventional controllers. A comparative user study with 20 participants evaluated task performance and workload using the proposed touchscreen, a conventional joystick, and a single-click autonomous mode. Tasks simulated realistic surface manipulation using a Franka Emika Panda arm, remotely controlled from another country. Kinematic, physiological, and behavioral data were recorded to comprehensively assess task performance, cognitive load, and operator trust across each control condition. Participants completed teleoperation tasks more efficiently and accurately with the touchscreen interface, achieving a 53.5% reduction in completion time (median: 2.50 vs. 5.38 min), higher in-area coverage on the sinusoidal path (90.7% vs. 84.1%), and lower overshoot on both path geometries compared with the joystick. Cognitive load, quantified via NASA-TLX (0-100), decreased from joystick to touchscreen (mean TLX 52 to 43; -9 points, -17.3%) and was lowest under the autonomous one-click mode (31; -21 points vs. joystick, -40.4%; -12 vs. touchscreen, -27.9%). This research presents an easy-to-implement touchscreen interface that improves performance in teleoperated surface tasks while reducing cognitive load.
Read the full abstract ▾
All authors Juan José García Cárdenas, Alperen Kenan, Hamidreza Raei, Paul Bremner, Manuel Giuliani, Arash Ajoudani, Adriana Tapus
Robotics
VIDP: Variable Impedance Diffusion Policy for Compliant Robot Manipulation from Diverse Demonstrations
Hisham Khalil, Neil Fernandes, Thomas M. Kwok et al. (+2 more)
Contact-rich manipulation requires precise tracking and mechanical compliance, where variable impedance control can improve robustness in task success, whereas static compliance cannot adapt to varying contact constraints. Variable impedance skills can be learned from demonstrations, avoiding complex modeling, but compliance is a hidden variable in force-agnostic kinematic data. While existing methods infer compliance from trajectory variations, these variations may reflect
geometric adaptation and not intentional compliance when subject to changing spatial layouts. Therefore, this letter introduces Variable Impedance
Diffusion Policy (VIDP), an imitation learning-based variable impedance control framework leveraging a Task-Parameterized Directionality-Aware Mixture Model (TP-DAMM) to extract physically consistent trajectory distributions from diverse demonstrations. By mapping distributions to stiffness profiles, VIDP jointly predicts pose actions and task compliance without force sensors. Real-world experiments show that VIDP significantly outperforms fixed-impedance baselines in task success rate while reducing interaction forces with respect to high stiffness controllers and tracking errors with respect to low stiffness baselines.
Read the full abstract ▾
All authors Hisham Khalil, Neil Fernandes, Thomas M. Kwok, Hsiu-Chin Lin, Yue Hu
Robotics
ErgoSurf: Ergodic Control for the Coverage of Unknown Surfaces
Stefan Schneyer, Timo Bachmann, Maged Iskandar et al. (+4 more)
Contact-centric tasks on surfaces, ranging from inspection and cleaning to sanding and polishing, require
robots to systematically cover the surface while maintaining stable contact. Ergodic control generates trajectories that spend time at a location proportional to a desired, task-specific spatial distribution, enabling efficient information gathering and coverage. However, traditional ergodic control methods rely on prior knowledge of surface geometry or require a
vision sensory input to scan the geometry beforehand, limiting their applicability in real-world scenarios with unknown or dynamic environments. This paper introduces a novel online ergodic control framework that achieves systematic surface coverage while simultaneously reconstructing unknown surface geometry. We employ a Gaussian Process Implicit Surface (GPIS) model that learns global surface geometry from intrinsic tactile sensing during execution. For efficient online planning, we approximate the surface locally using point clouds sampled from tangent planes at observed contact points and iteratively fit them to the Gaussian Process. This approximation simultaneously serves as the sampling domain for both the target and the coverage distributions. We employ a heat-
diffusion analogy to compute potential fields that guide ergodic exploration, translating spatial coverage objectives into smooth
robot trajectories. We demonstrate our framework through simulation and real-
robot experiments, validating simultaneous ergodic coverage and online surface geometry learning with reconstruction errors approaching the ground truth.
Read the full abstract ▾
All authors Stefan Schneyer, Timo Bachmann, Maged Iskandar, Korbinian Nottensteiner, Alin Albu-Schäffer, Freek Stulp, João Silvério
Robotics
Prior-SG: Task and Prior Driven Region Segmentation for Scene Graphs in Arbitrarily-Structured Environments
Giorgio Tonetti, Laurent Kneip, Abel Gawel et al. (+1 more)
Hierarchical 3D scene graphs are a promising representation for high-level spatial
reasoning in autonomous mobile platforms. However, existing extraction frameworks typically rely on purely local visual clustering or strict
geometric heuristics, such as wall-separated rooms, which fail in open-plan or arbitrarily-structured environments. We propose Prior-SG, a task- and prior-driven framework that casts scene graph generation fundamentally as a probabilistic
alignment problem. As the
robot explores, it continuously aggregates an incoming RGB-D sensor stream into a physically grounded Instance Graph utilizing a multi-scale, open-vocabulary feature fusion strategy. The system then infers the high-level functional semantics of this map through a Maximum A Posteriori (MAP) estimate, guided by a Prior Graph-a logical expectation of the environment's structure and task-relevant vocabulary synthesized dynamically by a Large
Language Model. By optimizing a Markov Random Field that fuses heterogeneous experts (visual,
geometric, and discrete objects) with these topological priors, the system resolves local perceptual ambiguities. We validate this approach across diverse simulated residential
datasets and large, open-plan real-world environments. Prior-SG achieves state-of-the-art semantic region segmentation accuracy compared to recent baselines, robustly delineates distant functional boundaries in the absence of physical walls, and uniquely provides zero-shot ontological flexibility, enabling the
robot to entirely restructure its spatial partitioning based on a given high-level task.
Read the full abstract ▾
All authors Giorgio Tonetti, Laurent Kneip, Abel Gawel, Marco Hutter
Robotics
Visual Grounding in Zero-Shot Vision-Language Control
J. de Curtò, Dayani Plasencia, Diego Sánchez et al. (+1 more)
Vision-
language models (VLMs) are increasingly used as zero-shot controllers, but successful trajectories do not necessarily show that decisions are grounded in visual input: simulator dynamics and conservative action priors can produce favourable scores without meaningful perception. We investigate this with an input-ablation battery: blind-image controls, repeated identical inputs, lane-axis reflection, non-visual baselines, and pipeline-integrity checks. Across nine direct-action models, six structured local VLMs, and an exploratory VLM-MPC hierarchy, we analyse 32,874 scored calls over two embodiments and three simulators. The direct-control results are largely negative: a constant-SLOW policy outperforms a scripted
geometric controller, several models are image-invariant or nearly constant, and models that recognize longitudinal hazards still fail to transform LEFT and RIGHT under reflection. No local VLM meets the joint longitudinal and lateral grounding criteria. However, an image-only deterministic positive control estimates the lead gap with 0.090 m MAE and exact mirror equivariance, confirming the stimuli carry sufficient visual information; the failures are modular, not universal. A post-hoc, leakage-controlled
symmetry-consensus guardian selects two models from 16 calibration frames and freezes a 2-of-4 hazard vote across original and reflected views. On 272 held-out frames it reaches 0.954 balanced accuracy (episode-cluster bootstrap 95% CI [0.895,0.990]); nested leave-one-episode-out recovers the same pair and threshold in all 12 folds. Abstaining on ties raises committed balanced accuracy to 0.973 at 0.824 coverage. With deterministic perception retaining lateral authority, offline modular replay achieves 0.934 action agreement and exact mirror equivariance. These results support current VLMs as bounded, selective hazard assistants, not monolithic zero-shot controllers.
Read the full abstract ▾
All authors J. de Curtò, Dayani Plasencia, Diego Sánchez, I. de Zarzà
Robotics
IcFuzz: Fuzzing Isaac Sim with Semantic Stage Guidance and Multi-level Mutation
Zhixiang Chen, Zhuangbin Chen, Ruoxi Jia et al. (+4 more)
Robotics simulators serve as a foundational infrastructure for embodied AI, facilitating safe and scalable robotic system development. NVIDIA Isaac Sim has emerged as one of the most popular simulators, distinguished by its GPU-accelerated physics engine and photorealistic rendering, which enable high-fidelity modeling of complex environments. However, its inherent complexity inevitably introduces software bugs that can compromise simulation reliability. Existing fuzzing approaches struggle to test Isaac Sim effectively due to challenges of context-aware object semantics, hierarchical simulation control, and a vast simulation state space. In this paper, we propose IcFuzz, the first fuzzing approach for Isaac Sim. IcFuzz first performs an LLM-based semantic stage segmentation, decomposing simulation programs into structured stages that capture context-aware object semantics. Guided by this information, IcFuzz designs multi-level mutation operators to systematically exercise the simulator across hierarchical granularities. To efficiently navigate the vast simulation state space, IcFuzz employs a multi-armed bandit
algorithm to adaptively schedule mutation operators. Experimental results show that IcFuzz outperforms the baselines in terms of both code coverage and bug detection. Specifically, IcFuzz achieves approximately 190%–205% of the code coverage of the baselines and detects an average of 3.7 unique crashes over three rounds of 12-hour tests, while no crashes are detected by the baselines. Moreover, IcFuzz has uncovered 11 bugs over approximately four months, 9 of which have been confirmed or fixed by the developers.
Read the full abstract ▾
All authors Zhixiang Chen, Zhuangbin Chen, Ruoxi Jia, Zeqin Liao, Wei Li, Jinyang Liu, Zibin Zheng
Robotics
Topometric Autonomous Vehicle Localization by Combining Visual Embeddings and Feed-Forward 3D Models
Eulogio Quemada-Torres, Alberto Jaenal, Francisco-Angel Moreno et al. (+1 more)
Effective Visual Localization (VL) requires a map of the environment that combines compactness for efficient scalability with robustness against visual appearance changes and metric precision. Through low-dimensional image
embeddings, Visual Place Recognition (VPR) is able to successfully meet the first two requirements, but its low metric accuracy makes it less suitable than standard VL approaches based on local features or neural representations. This limitation can be overcome by integrating VPR with the accurate local trajectory estimates produced by feed-forward neural 3D geometry (FF3D) models. In this paper, we address sequential appearance-based localization through a topometric framework that iteratively combines probabilistic VPR with FF3D metric pose estimation in controlled image sets. Our approach proposes an automatic offline mapping tool that models the topometric pose-appearance interaction in the different parts of the scene. This map is later employed by an online particle filter that estimates the pose from odometry and belief over places for FF3D inference, successfully incorporating neural metric estimation into probabilistic appearance-based localization. We extensively evaluate the framework on three known
benchmarks, demonstrating substantial improvements over existing appearance-based methods. The modularity of our approach allows the descriptor extractor and FF3D model to remain interchangeable, and a focused analysis further shows that sequential belief can mitigate severe failures under perceptual aliasing.
Read the full abstract ▾
All authors Eulogio Quemada-Torres, Alberto Jaenal, Francisco-Angel Moreno, Javier Gonzalez-Jimenez
Robotics
Adaptive-WAM: Quality-Guided Early-Exit Planning from Intermediate Video-Diffusion Features
Sining Ang, Yuguang Yang and Yan Wang
Large
video diffusion models provide rich spatiotemporal priors for autonomous driving, but existing world-action models often inherit the cost of iterative future-
video generation even though deployment only requires an ego trajectory. We ask a more basic question: how much of a
video diffusion model must be executed to make a reliable driving decision? Through a controlled study of
video denoising timesteps and
Diffusion Transformer (DiT) depth, we find that planning performance is largely insensitive to the tested
video-noise levels, whereas strong trajectories can already be decoded from intermediate layers. Based on this observation, we introduce Adaptive-WAM, a quality-aware multi-exit planner built on a Wan2.2-5B backbone. Trajectory
diffusion heads are attached to selected DiT blocks, and a lightweight trajectory-quality scorer terminates inference once the best trajectory decoded so far satisfies a quality threshold; otherwise, computation continues from the cached hidden state to a deeper exit. The deployed planner therefore avoids the iterative classifier-free denoising loop and VAE decoding required for future-
video synthesis, while dynamically allocating backbone depth according to trajectory quality. On NAVSIM, the adaptive single-trajectory planner achieves 90.8 PDMS; a separate fixed-exit variant reaches 92.6 PDMS with 64 proposals. It further obtains 89.9 EPDMS on NAVSIM v2, yielding the best reported results among the compared front-view
video world-model planners. Without target-domain fine-tuning, Adaptive-WAM transfers to nuScenes with 0.88 m average L2 error and a 0.08% collision rate. On an A100, adaptive routing improves PDMS from 90.62 to 90.79 while averaging 170 ms end-to-end planning latency, approximately 10% below the 190 ms fixed block-15 planner and 47% below the 320 ms fixed full-depth planner. Code will be released.
Read the full abstract ▾
Robotics
Beyond Flat Policies: Hierarchical Post-Training for Embodied Agents in Robotic Manipulation
He Kong, Zengjue Chen, Qi Wang et al. (+6 more)
Vision-
language-action (VLA) models have demonstrated remarkable capabilities in robotic manipulation by leveraging pretrained
vision-
language models. However, existing post-training methods predominantly optimize VLA models as flat policies, making it difficult to explicitly model task progression and perform robust long-horizon manipulation. Although hierarchical approaches introduce task decomposition, they mainly rely on supervised learning from offline demonstrations and cannot effectively improve execution through online interaction. To address this limitation, we propose Hierarchical Robotic Control (HiRoC), a hierarchical post-training framework that decouples high-level task planning from low-level action execution. The planner decomposes complex tasks into executable subgoals to provide explicit semantic guidance, while the executor continuously improves subgoal-conditioned action generation through reinforcement learning. To enable effective collaboration between the two modules, we further align the executor with planner-generated subgoals before reinforcement learning, mitigating the distribution misalignment between planning and execution. Extensive experiments across diverse robotic manipulation
benchmarks demonstrate that HiRoC consistently outperforms strong baselines. Comprehensive analyses further validate the effectiveness of hierarchical post-training and the contribution of each key component.
Read the full abstract ▾
All authors He Kong, Zengjue Chen, Qi Wang, Qianli Xing, Runliang Niu, Peidong Liu, Jiawei Li, Shiqi Wang, Yi Chang
Robotics · Cross-listed from cs.LG
Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Control
Xinwei Liu, Junyuan Liang, Jianting Zhang et al. (+1 more)
Sample-efficient policy learning from pixels is a long-standing challenge in reinforcement learning (RL). Recent dynamics-based representation learning methods have significantly improved the sample efficiency of model-free visual RL by learning dynamics-aware representations through auxiliary prediction performed either in latent space (self-prediction) or observation space (observation prediction). However, state-of-the-art methods from both categories still struggle on challenging visual control tasks when training data is limited. We posit that relying on either predictive objective alone may be insufficient. In contrast, observation prediction grounds learned representations in observation-level dynamics, but does not directly regularize the temporal predictability of latent representations over extended horizons. In this paper, we propose Observation-Grounded Self-Predictive Representations (OG-SPR), a model-free visual RL
algorithm for continuous control that learns representations that are both temporally predictive in latent space and grounded in observation-level dynamics. OG-SPR incorporates two core auxiliary objectives: multi-step latent self-prediction and next-observation prediction. We empirically show that directly imposing latent self-prediction on the shared representation may over-constrain it and does not necessarily improve performance. To address this issue, OG-SPR introduces two lightweight adapters for latent self-prediction, allowing the shared representation to benefit from temporally predictive signals without being forced to directly satisfy the self-prediction objective. Experiments on 28 visual control tasks from the DeepMind Control Suite show that OG-SPR improves aggregate performance over state-of-the-art self-predictive and observation-predictive RL methods, with particularly pronounced gains in challenging domains such as dog and humanoid.
Read the full abstract ▾
All authors Xinwei Liu, Junyuan Liang, Jianting Zhang, Wuhui Chen
Robotics
TRACE: Learned Proprioceptive Odometry for Legged Robots under Unreliable Contact Conditions
Taehyeon Kong, Woojin Kim and Jemin Hwangbo
In this paper, we present TRACE (Tokenized Robust
Attention for Contact-Aware Estimation), an end-to-end learned proprioceptive odometry
estimator for legged
robots under unreliable contact conditions. The proposed
estimator directly predicts relative displacement, relative rotation, and body-frame velocity from a recent history of onboard inertial and joint measurements. To improve robustness under unreliable contact conditions, we introduce a foot-aware cross-
attention module that adaptively weights IMU and leg-wise kinematic
tokens without relying on manually defined contact or slip thresholds. The
estimator is trained with direct supervision and two physics-inspired auxiliary losses that promote kinematic consistency and reliable use of leg information. To reduce policy-specific overfitting and consequently improve sim-to-real transfer, simulation training incorporates policy randomization, followed by partial real-world fine-tuning of the temporal encoder and prediction head. Experiments across diverse indoor and outdoor terrains demonstrate consistent reductions in position drift compared with classical filtering-based, hybrid, and purely learning-based baselines. Ablation studies further validate the contributions of the proposed training objectives, policy randomization, and real-world fine-tuning, particularly under unreliable contacts and sim-to-real mismatch.
Read the full abstract ▾
Robotics
SkillMemo: Expert-guided Skill Memory Framework for Compositional Embodied Manipulation
Changyuan Wang, Chubin Zhang, Zhenyu Wu et al. (+8 more)
Embodied visuomotor models, including
Diffusion Policy (DP) and
Vision-
Language-Action (VLA) models, have demonstrated promising performance on robotic manipulation
benchmarks. However, their potential remains fundamentally constrained by the scarcity of large-scale embodied trajectory
datasets, leading to insufficient compositional generalization in out-of-distribution (OOD) scenarios with limited capability to capture reusable skill structures. To address this limitation, we propose Skill-Based
Memory (SkillMemo) framework that implicitly decomposes long-horizon demonstrations into latent atomic skills and integrates skill-level features into a dynamic episodic
memory bank for solving compositional tasks. Specifically, we first introduce an expert-guided trajectory segmentation module built upon a Mixture-of-Experts (MoE) architecture, which implicitly partitions trajectories into distinct skill primitives represented by learned gating coefficients. We further design a skill-level episodic
memory architecture that stores compact skill representations as retrievable key-value pairs. During inference, the
memory bank retrieves the most relevant skill primitives which are subsequently fused with the model's current gating distribution, providing a robust contextual prior to refine action predictions. Extensive experiments on the simulation
benchmark and real-world manipulation tasks demonstrate that SkillMemo consistently enhances both DP and VLA backbones, achieving state-of-the-art performance and outperforming $\pi_{0.5}$, while exhibiting strong compositional generalization to unseen task configurations.
Read the full abstract ▾
All authors Changyuan Wang, Chubin Zhang, Zhenyu Wu, Runhao Li, Angyuan Ma, Ke Chao, Yinan Liang, Xiuwei Xu, Ziwei Wang, Yansong Tang, Jiwen Lu
Robotics · Cross-listed from cs.AI
GAUGE: A Measurement-Grounded Benchmark for Physical Fidelity in Simulation Engines and Video World Models
Shuai Wang, Yaxin Feng, Xuekun Jiang et al. (+12 more)
Physics engines facilitate large-scale training and evaluation for embodied intelligence, while generative
video world models are emerging as implicit simulators of future states and interactions. However, existing evaluations of physical fidelity are often conducted in isolation and rely heavily on perceptual similarity or human judgments, providing limited insight into which physical principles or parameters are violated. We introduce GAUGE, a real-world-grounded diagnostic
benchmark for jointly evaluating how numerical simulators and generative
video world models reproduce or deviate from real-world physics. It comprises 22 controlled task families covering rigid bodies, flexible cables, textiles, and volumetric deformable objects. Grounded in real-world trajectories and paired with calibrated physical metadata, uncertainty annotations, and task-specific observables, these tasks cover fundamental physical processes including collision, friction, momentum transfer, oscillation, self-contact, and deformation across diverse materials and conditions. We
benchmark Isaac Sim, Genesis, and Newton on 14 task families using generalized trajectory errors, and evaluate 6 image-to-
video models on 5 rigid-body tasks by testing physical-law consistency and the temporal stability of inferred parameters. Our results reveal no uniformly faithful physics engine, with the largest discrepancies arising in impulsive contact, rapid textile motion, and volumetric deformation. We further find that
video world models can produce trajectories with the expected equation form while recovering incorrect accelerations, momentum transfer, and oscillation timing. GAUGE lays the groundwork for developing more physically faithful simulators and world models for embodied intelligence.
Read the full abstract ▾
All authors Shuai Wang, Yaxin Feng, Xuekun Jiang, Shihan Tian, Ningyu Yan, Xing Shen, Chaoyang Lyu, Hui Wang, Yunsong Zhou, Hanqing Wang, Jiangmiao Pang, Yang Xiang, Xing Gao, Chunhua Shen, Weinan Zhang
Robotics · Cross-listed from cs.CV
Robust-WAM: Bridging Generative Pretraining and Semantic Foresight in World-Action Models
Haodong Yan, Junfeng Li, Junjie He et al. (+12 more)
Mainstream World-Action Models (WAMs) adapt pretrained
video generation models (VGMs) for
robot control, transferring their learned dynamics prior for action prediction. These VGMs are typically trained in a variational autoencoder (VAE) latent space. However, the VAE latent space is optimized for pixel reconstruction, which rewards fine appearance detail and leaves the action prediction fragile under visual shifts. Recent works build WAMs in semantic latent space, which are more robust to appearance shifts. However, these models cannot leverage the large-scale VGM pretraining that exists only in VAE space. To overcome this dilemma, we propose Robust-WAM, a general post-training method for
video-generation-based WAMs that preserves the VAE-based generative path and adds a lightweight semantic foresight
alignment objective on the action stream. This retains the large-scale VGM pretraining while grounding actions in appearance-invariant dynamics that stay reliable under illumination shifts and other visual out-of-distribution conditions. Specifically, we employ learnable query
tokens to bring future-scene semantics into the action stream by aligning their output hidden states with the semantic foresight of future ground-truth frames. To establish the temporal correspondence between each query and the future step it describes, we give it the positional encoding of the matching action
tokens. Experiments on out-of-distribution generalization simulation
benchmarks and a real-
robot setup show that our Robust-WAM consistently improves the success rates of multiple WAM baselines without sacrificing in-distribution performance.
Read the full abstract ▾
All authors Haodong Yan, Junfeng Li, Junjie He, Zhide Zhong, MingMing Yu, Wenxuan Song, Jiaguan Zhu, Yangyang Zheng, Yuqiao Du, Jiadi You, Yingjie Cai, Xu Yan, Guanyi Zhao, Bingbing Liu, Haoang Li