Robotics
GHOST: Hierarchical Sub-Goal Policies for Generalizing Robot Manipulation
Sriram Krishna, Ben Eisner, Haotian Zhan et al. (+5 more)
We present GHOST, a framework for learning visuomotor manipulation policies that generalize beyond the training distribution. GHOST factorizes control into (i) a high-level policy that predicts the next sub-goal as a distribution over 3D end-effector poses from multi-view RGB-D observations, and (ii) a low-level goal-conditioned controller that executes embodiment-specific actions. To condition image-based policies on 3D goals, we introduce a simple spatial interface that projects predicted goals into the image plane and represents them as end-effector heatmaps. Across a suite of manipulation tasks, this hierarchical factorization consistently improves performance and robustness compared to a flat
Diffusion Policy. Further, we show that this hierarchical interface also makes it easy to incorporate human demonstrations without relying on (noisy) action retargeting. As sub-goals are largely embodiment-agnostic, we train the high-level policy on human
video to specify how learned skills should be applied and composed, while keeping the low-level policy trained purely on
robot data. This hierarchy enables adaptation to novel objects and task variations using a small number of human demonstrations.
Read the full abstract ▾
All authors Sriram Krishna, Ben Eisner, Haotian Zhan, Ying Yuan, Haoyu Zhen, Chuang Gan, Shubham Tulsiani, David Held
Robotics · Cross-listed from cs.CV
Generalized-CVO: Fast and Correspondence-Free Local Point Cloud Registration with Second Order Riemannian Optimization
Ray Zhang, Marcus Greiff, Thomas Lew et al. (+1 more)
We propose a fast and correspondence-free local point cloud registration method that leverages
geometric surface structure and reproducing kernel
Hilbert space (RKHS)
embeddings. The method represents point clouds as continuous functions with point-wise anisotropic kernels that encode local geometry. This formulation improves
alignment along surface normals while relaxing
alignment along tangential directions. To solve the resulting registration problem, we propose a second-order on-
manifold optimization scheme with approximate Riemannian Hessians, achieving a speedup of up to 10x over the first-order solvers used in prior correspondence-free RKHS-based methods. We demonstrate improved frame-to-frame
LiDAR and RGB-D tracking accuracy across diverse indoor and outdoor
datasets. On a
LiDAR tracking registration task in the driving domain, we achieve a reduction of $>55\%$ in both translational and rotational drift in challenging feature-sparse environments. On object registration
benchmarks, we show improved robustness over ICP-based methods and further gains when refining global initialization, particularly under moderate misalignment.
Read the full abstract ▾
All authors Ray Zhang, Marcus Greiff, Thomas Lew, John Subosits
Robotics
Uncertainty-Aware Motion Planning for Autonomous Driving in Mixed Traffic Environment
Ming Cheng, Hao Chen, Ziyi Yang et al. (+2 more)
In mixed-traffic environments where autonomous and human-driven vehicles may co-exist, motion planning for autonomous vehicles requires anticipating the future behaviors of surrounding human drivers. Existing reinforcement learning-based methods generally directly incorporate the predicted human intents into the observation to enable a proactive planning. However, human intent is inherently uncertain due to the behavioral diversity, perception noise, and partial observability. Treating predicted intends as deterministic states can result in unsafe decisions for autonomous vehicles. To address this problem, we propose Uncertainty-Aware Motion Planning (UAMP), which incorporates uncertainty in human intent prediction for AV decision-making. Specifically, UAMP first introduces a proximity-aware uncertainty
estimator to quantify the interaction-conditioned intent uncertainty and constructs an uncertainty-guided joint intent distribution over surrounding human-driven vehicles. Within this uncertainty set, UAMP further introduces Uncertainty-Calibrated Value Learning (UCVL) to correct value function learning biases arising from directly incorporating uncertain human intent predictions into the observation. Extensive experiments in various mixed-traffic scenarios show that UAMP significantly improves safety and driving comfort, while maintaining traffic efficiency compared with existing approaches. The code is released at https://anonymous.4open.science/r/UAMP-5638.
Read the full abstract ▾
All authors Ming Cheng, Hao Chen, Ziyi Yang, Ziluowen Luo, Senzhang Wang
Robotics
MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models
Hao Shi, Weiye Li, Bin Xie et al. (+6 more)
Temporal modeling is essential for robotic manipulation, as effective control requires both
memory of past interactions and imagination of future states. However, most VLA models rely primarily on the current observation and therefore struggle with long-horizon, temporally dependent tasks. Cognitive science suggests that humans rely on working
memory to buffer short-lived context, the hippocampal system to preserve episodic
memory of past experience, and internal models to imagine possible future state
evolution. Inspired by these mechanisms, we propose MemoryVLA++, a full temporal modeling framework that equips VLA models with
memory and imagination for robotic manipulation. A pretrained VLM encodes the current observation into perceptual and cognitive
tokens, forming working
memory. These
tokens query a Perceptual-Cognitive
Memory Bank to retrieve relevant historical context. This bank stores low-level details and high-level semantics from past interactions, and is updated through redundancy-aware consolidation. A world model imagines future states in a denoising latent space, and the imagined latents are integrated under
memory guidance to form full temporal-aware
tokens. The resulting
tokens condition a
diffusion action expert to predict temporally consistent action sequences. We conduct extensive experiments on 5 simulation
benchmarks and 3 categories of real-
robot tasks across 3
robots, covering general manipulation, long-horizon temporal tasks, robustness, and generalization. Our method achieves strong performance across Libero, SimplerEnv, Mikasa-Robo, Calvin, Libero-Plus, and diverse real-
robot tasks, validating the effectiveness of full temporal modeling with
memory and imagination. For example, on real
robots, it achieves +9%, +26%, +28% gains on general,
memory-dependent, and imagination-dependent tasks. Project Page: https://shihao1895.github.io/MemoryVLA-PP-Web
Read the full abstract ▾
All authors Hao Shi, Weiye Li, Bin Xie, Yulin Wang, Renping Zhou, Tiancai Wang, Xiangyu Zhang, Ping Luo, Gao Huang
Robotics
iMaC: Translating Actions into Motion and Contact Images for Embodied World Models
Zhenyu Wu, Xiuwei Xu, Yukun Zhou et al. (+8 more)
Embodied world models have emerged as a pivotal paradigm for visual robotic decision-making and interactive environment simulation. However, conventional
embodied frameworks rely on low-dimensional structured action vectors (e.g., joint angles and end-effector poses), which suffer from limited expressive capacity, poor generalization across diverse embodiments, and unnatural dynamic modeling for complex physical interactions. To address these limitations, this paper proposesiMac (Image as Action Control), a novel unified control paradigm that treats raw visual images as native action representations for
embodied world models. Departing from traditional explicit kinematic action encoding, iMac formulates continuous visual manipulation as image-based action
tokens, which inherently encapsulate spatial motion intentions, interactive
geometric constraints and subtle physical dynamics. We construct a dual-branch
embodied architecture consisting of an image-action encoder and a dynamic world predictor: the encoder compresses target-driven visual images into compact action
embeddings, while the predictor learns environment transition rules conditioned on image actions to achieve high-fidelity future state prediction and closed-loop
embodied control. Extensive experiments are conducted on public
embodied manipulation
benchmarks and real-world robotic scenarios. The results demonstrate that iMac outperforms vector-based action control baselines in prediction accuracy, task success rate and cross-scene generalization ability. Moreover, our image-action design eliminates the reliance on manually defined action spaces, realizing flexible and universal control for heterogeneous
embodied agents. This work provides an innovative visual-action perspective for
embodied world models, offering a simple yet effective paradigm for scalable robotic perception and manipulation.
Read the full abstract ▾
All authors Zhenyu Wu, Xiuwei Xu, Yukun Zhou, Yifan Li, Qiuping Deng, Xiaofeng Wang, Zheng Zhu, Bingyao Yu, Ziwei Wang, Jiwen Lu, Haibin Yan
Robotics
AHA-WAM:Asynchronous Horizon-Adaptive World-Action Modeling with Observation-Guided Context Routing
Jisong Cai, Long Ling, Shiwei Chu et al. (+10 more)
World-action models have emerged as a promising paradigm for
robot manipulation, jointly modeling visual scene dynamics and actions to inject physical priors into policy learning. However, existing world-action models couple world prediction and action execution at the same temporal resolution, forcing the world branch to model near-term frame variations that are redundant and weakly informative. We posit that strictly binding world prediction and action execution to the same temporal rhythm may underutilize the potential of the
video branch for
embodied control. Therefore, we propose AHA-WAM, an Asynchronous Horizon-Adaptive World-Action Model built on a dual
Diffusion Transformer (DiT) architecture that reorganizes world-action modeling around this temporal asymmetry. AHA-WAM instantiates the
video DiT as a low-frequency world planner that maintains rolling key-value
memory over past observations and exposes reusable layerwise latent context encoding long-horizon scene
evolution, while a high-frequency action DiT executes short action chunks in closed loop by querying this context through layerwise joint
attention. To support asynchronous execution, we introduce horizon-adaptive offset training and Observation-Guided
Video-Context Routing (OVCR), which together let the action expert exploit long-horizon world context while remaining responsive to real-time execution state without rerunning the
video DiT. Experiments on RoboTwin and real-world manipulation tasks show that AHA-WAM achieves state-of-the-art performance without any
robot-data pretraining, attaining 92.80% average success on RoboTwin and 78.3% success across 4 real-world tasks, while reaching 24.17 Hz closed-loop control with a 4.59x speedup over Fast-WAM.
Read the full abstract ▾
All authors Jisong Cai, Long Ling, Shiwei Chu, Zhongshan Liu, Jiayue Kang, Zhixuan Liang, Wenjie Xu, Yinan Mao, Weinan Zhang, Xiaokang Yang, Ru Ying, Ran Zheng, Yao Mu
Robotics
SynManDex: Synthesizing Human-like Dexterous Grasps from Synthetic Human Pre-Grasps
Yanming Shao, Zanxin Chen, Wenwei Lin et al. (+5 more)
Human hand-object interactions encode functional intent, but direct transfer to robotic hands often fails under morphology, contact, and reachability constraints. We present SynManDex, a synthetic pipeline that uses generated human pre-grasps as affordance-aware proposals and resolves the final contacts with
robot-native optimization. SynManDex samples object-conditioned digital human pre-grasps, retargets them to dexterous robotic hand poses, optimizes force-closure contacts on the target embodiment, and admits trajectories that pass checks from each step. The resulting keyframes support both grasp-and-lift demonstrations and various prehensile manipulation tasks such as tea pouring, photo taking, and flute playing, designed via VLM
agents. As a result, SynManDex combines high grasp quality (86.4% grasp stability) with 4.67/5 human-likeness (93.4%). It achieves 80.7% successes in simulation and 25/30 (83.3%) real-
robot successes when applied to a 36-DOF bimanual dexterous robotic platform.
Read the full abstract ▾
All authors Yanming Shao, Zanxin Chen, Wenwei Lin, Mingjie Zhou, Tianxing Chen, Xiaokang Yang, Yichen Chi, Yao Mu
Robotics
AetheRock: An Arm-Worn Robot Teaching System for Force-Guided Vision-Tactile Learning
Hong Li, Yue Xu, Yihan Tang et al. (+8 more)
Force and tactile sensing are indispensable in contact-rich manipulation. However, force-aware
robot learning faces critical challenges due to the incompatible assembly of tactile and force sensors in handheld or wearable devices. To address these limitations, we first introduce AetheRock for gripper-force,
vision, and tactile data collection, which is an arm-worn device featuring a modular and easily manufactured visuo-tactile sensor, GelSlim-MiniFab, at the fingertip, a resistive pressure sensor at the human finger contact region, a customized PCB module, and a wearable kit for comfortable and robust collection. Building on this, we propose ForceVT, a representation learning framework that uses force and
vision to guide fidelity-agnostic tactile learning, enabling robust inference in any tactile situation. Real-world experiments show that AetheRock achieves qualified data efficiency and that ForceVT effectively alleviates inefficiencies when visuo-tactile sensors exhibit manufacturing and utilization inconsistencies. Overall, our work mitigates the limitations of gripper-force
vision-tactile
robot learning through innovative hardware design and
algorithms.
Read the full abstract ▾
All authors Hong Li, Yue Xu, Yihan Tang, Yankang Dong, Chenyuan Liu, Chenyang Yu, Xuyang Li, Siyuan Huang, Yujun Shen, Nan Xue, Yong-Lu Li
Robotics
Difference-Aware Retrieval Policies for Imitation Learning
Quinn Pfeifer, Ethan Pronovost, Paarth Shah et al. (+3 more)
Parametric imitation learning via behavior cloning can suffer from poor generalization to out-of-distribution states due to compounding errors during deployment. We show that reusing the training data during inference via a semi-parametric
retrieval-based imitation learning approach can alleviate this challenge. We present Difference-Aware
Retrieval Policies for Imitation Learning (DARP), a semi-parametric
retrieval-based imitation learning approach that addresses this limitation by reparameterizing the imitation learning problem in terms of local neighborhood structure rather than direct state-to-action mappings. Instead of learning a global policy, DARP trains a model to predict actions based on $k$-nearest neighbors from expert demonstrations, their corresponding actions, and the relative distance vectors between neighbor states and query states. DARP requires no additional assumptions beyond those made for standard behavior cloning – it does not require additional data collection, online expert feedback, or task-specific knowledge. We demonstrate consistent performance improvements of 15-46% over standard behavior cloning across diverse domains, including continuous control and robotic manipulation, and across different representations, including high-dimensional visual features. Code and demos are available at https://weirdlabuw.github.io/darp-site/.
Read the full abstract ▾
All authors Quinn Pfeifer, Ethan Pronovost, Paarth Shah, Khimya Khetarpal, Siddhartha Srinivasa, Abhishek Gupta
Robotics
Your Model Already Knows: Attention-Guided Safety Filter for Vision-Language-Action Models
Seongbin Park, Fan Zhang, Baharan Mirzasoleiman et al. (+2 more)
Vision-
Language-Action (VLA) models have demonstrated impressive end-to-end performance across a variety of robotic manipulation tasks. However, these policies offer no guarantees against collisions with task-irrelevant objects in the scene. Existing safety filters sidestep this problem by querying a
vision-
language model (VLM) to identify obstacles and their locations. This, however, is too slow to run in the control loop and can only be invoked at episode initialization, leaving the filter unable to track moving obstacles. We discover that a small number of
attention heads within a VLA model reliably localize the object the policy intends to approach. These heads can be exploited within a training-free safety framework that obtains the active target from the
attention heads at every step, treats the remainder of the scene as obstacles, and feeds these into a Control Barrier Function (CBF) filter. Together with a lightweight real-time object tracker, this allows for collision avoidance for non-static obstacles. We evaluate our framework on SafeLIBERO, which we extend with moving obstacles. On the original static
benchmark, our method performs comparably to an oracle that uses privileged simulator state to identify the target, emulating a VLM-based identification step run once at episode initialization. On the dynamic variant, where the oracle's init-time target assignment becomes stale, our method substantially outperforms it by 43%, on average. Our findings suggest that the perceptual signals needed for real-time safety filtering are already present within VLA policies and can be exploited without additional training or heavy auxiliary models.
Read the full abstract ▾
All authors Seongbin Park, Fan Zhang, Baharan Mirzasoleiman, Shahriar Talebi, Nader Sehatbakhsh
Robotics
ProbeAct: Probe-Guided Training-Free Failure Recovery in Vision-Language-Action Models
Fan Zhang, Seongbin Park, Baharan Mirzasoleiman et al. (+2 more)
Vision-
Language-Action (VLA) models demonstrate strong perfor-1 mance on
language-conditioned robotic manipulation within their training dis-2 tribution, yet their generalization capabilities remain fundamentally limited. They3 lack the robustness required to handle perturbations, frequently failing when con-4 fronted with lighting changes, altered camera viewpoints, or small initial-state5 variations. We propose PROBEACT, a training-free runtime intervention frame-6 work that detects and recovers from grasping and placement failures in pre-7 trained VLA policies without modifying their weights or requiring additional8 demonstrations. PROBEACT combines three components: (i) a lightweight multi-9 target hidden-state probe that predicts the 3D positions of task-relevant objects10 from intermediate VLA features, with Hungarian-matched identity tracking for11 multi-object scenes; (ii) an object-agnostic kinematic state machine that detects12 grasp, transport, and placement failures using only gripper-internal signals and13 end-effector kinematics; and (iii) a hierarchical Control Barrier Function (CBF)14 filter that encodes repeated-failure locations as soft safe-set constraints, mini-15 mally correcting VLA actions while preserving baseline behavior. As a plug-and-16 play, training-free intervention loop, PROBEACT is orthogonal to existing train-17 ing pipelines. Evaluated on the LIBERO-plus
benchmark, our framework acts as18 a universal safety net, improving the success rate of the OpenVLA-OFT model19 from 69.6% to 74.1%, while demonstrating broad applicability to both base and20 fine-tuned VLA policies.
Read the full abstract ▾
All authors Fan Zhang, Seongbin Park, Baharan Mirzasoleiman, Shariar Talebi, Nader Sehatbakhsh
Robotics
Safe Polytope-in-Polytope Motion Planning and Control with Control Barrier Functions
Alejandro Gonzalez-Garcia, Dries Dirckx, Jan Swevers et al. (+1 more)
Autonomous mobile
robots operating in tight environments require motion planning frameworks that account for the physical footprint of the
robot. Simplifying the geometry to a point or a circle is conservative and discards information needed to successfully and safely traverse narrow passages. This work proposes a safe local motion planning and control method that guarantees that a polytopic
robot footprint stays inside a continuously updated convex free-space region. The containment condition is formulated as a set of discrete-time control barrier function constraints within a model predictive controller. The number of safety constraints depends on the complexity of the local free-space geometry and the
robot shape, instead of the number of obstacles. The proposed free-space formulation does not need any obstacle detection or
segmentation. A comparative analysis against a polytope-based obstacle avoidance formulation confirms favorable scaling up to a reduction of 91$\times$ in computation time as the number of obstacles increases. The approach is validated in simulation with an autonomous surface vehicle and on hardware with a non-holonomic mobile
robot, using both occupancy grids and
LiDAR sensing. The experiments demonstrate safe real-time motion planning and control at 10~Hz on an onboard embedded computer, including reactive avoidance of dynamic obstacles.
Read the full abstract ▾
All authors Alejandro Gonzalez-Garcia, Dries Dirckx, Jan Swevers, Wilm Decré
Robotics
Modeling Components and Connections in Cyber-Physical Systems
Kate Sanborn, Tanuj Kenchannavar, Vakul Nath et al. (+1 more)
Text based configuration files for cyber-physical systems show the hierarchy of component modules well but often hide the details of connections and interfaces between modules. A model-based visual approach to these configuration files can better capture this information. The XML structure of
Robot Operating System (ROS) launch files can be improved using a modeling approach. This paper presents ROSLaunchVisual, a model-integrated environment built on WebGME for designing, visualizing, and managing ROS launch files. The tool raises the level of abstraction by allowing developers to create and modify launch files using a graphical interface that represents nodes, publishers, subscribers, and arguments as interconnected components. The tool provides a dynamic system analysis that can then be used in the static development and analysis of new and existing launch files. ROSLaunchVisual incorporates features such as metamodel-driven validation, automatic import/export of launch files, and visual communication mapping. Plugins further enhance functionality by updating libraries, checking for semantic errors, and managing remaps. By making launch file creation more intuitive and less error-prone, ROSLaunchVisual improves development efficiency and system understanding, especially in collaborative or large-scale robotics projects.
Read the full abstract ▾
All authors Kate Sanborn, Tanuj Kenchannavar, Vakul Nath, Jonathan Sprinkle
Robotics
Physics-Aware Sparse Learning and Selective Online Adaptation for Euler-Lagrange Robot Dynamics
Rishabh Dev Yadav, Samaksh Ujjawal, Sihao Sun et al. (+2 more)
Accurate dynamics models are essential for model-based robotic control, yet nominal Euler–Lagrange models often become inaccurate in the presence of payload variation, unmodeled coupling, friction, aerodynamic effects, and changing operating conditions. Most learning-based correction methods improve prediction accuracy by introducing a single additive residual, but do not preserve the internal mechanical structure of Euler–Lagrange systems. This leads to models that do not preserve
symmetry, positive-definiteness, or the coupling between inertia and velocity-dependent terms, which can result in physically inconsistent predictions and reduced reliability when embedded in model-based controllers. We propose a structure-preserving residual learning framework that decomposes model mismatch into an inertia correction, the corresponding induced Coriolis term, and a generalized-force residual. The mechanical component is learned under physical constraints, while the disturbance-sensitive component is represented through a sparse history-dependent latent interaction model and adapted online using
Bayesian linear regression. This separation preserves key mechanical structure while restricting adaptation to the part of the dynamics most affected by changing conditions. Experiments across multiple robotic platforms, including mobile, aerial, and manipulator systems, show that the proposed method improves dynamics prediction and trajectory tracking under coupled and time-varying dynamics. These results highlight the value of combining structured residual modeling, compact latent interaction selection, and selective online adaptation for real-world model-based control.
Read the full abstract ▾
All authors Rishabh Dev Yadav, Samaksh Ujjawal, Sihao Sun, Spandan Roy, Wei Pan
Robotics
ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies
Haodi Hu, Chung-Ta Huang, Jing Liu et al. (+4 more)
Vision-
language-action (VLA) policies provide strong priors for
language-conditioned manipulation, but remain brittle in off-nominal states requiring targeted recovery. We propose ReCoVLA – a failure-conditioned residual recovery framework that keeps a pretrained VLA policy frozen, uses an external
vision-
language model (VLM) to infer the failure mode and recovery stage, and compiles a structured reward from task-relevant components. Rather than using the VLM to generate actions or rewards directly, ReCoVLA uses it as a semantic reward selector: it predicts a recovery descriptor and reward mask for in-simulation residual-policy training, followed by zero-shot sim-to-real deployment of the trained recovery policies. This decouples high-level failure understanding from low-level corrective control to support different VLAs. Experiments across short-horizon, long-horizon, and contact-rich manipulation tasks show that ReCoVLA outperforms the tested baselines on average. In simulation, our reward compiler improves average success from 36.7% for the fine-tuned $\pi_{0.5}$ baseline to 66.7%. In physical zero-shot sim-to-real experiments, ReCoVLA achieves the best average performance, with 61.7% success.
Read the full abstract ▾
All authors Haodi Hu, Chung-Ta Huang, Jing Liu, Ye Wang, Kei Suzuki, Matthew Brand, Toshiaki Koike-Akino
Robotics
Motion planning for hundreds of floating robots
Jan Kamm, Antonio Terpin, Raffaello D'Andrea et al. (+1 more)
Planning collision-free motion for large
robot fleets is difficult because collision avoidance induces strong inter-
agent coupling that grows rapidly with team size. We consider omnidirectional floating
robots on water, where choreographies are specified by sparse keyframes and an interactive tool must generate trajectories within seconds, even when transitions span minutes and thousands of time steps. We propose a scalable pipeline that builds a collision graph from an initialization, decomposes the coupled problem into interaction clusters, and solves clusters independently (and in parallel) with robustness mechanisms for common decomposition pathologies. We validate the approach in simulations up to 500
robots. The synthesized trajectories have also been deployed in two real-world demonstrations, on Lake Zürich with a fleet of 24 Way of Water crafts and at the Time Space Existence 2025 Venice Biennale.
Read the full abstract ▾
All authors Jan Kamm, Antonio Terpin, Raffaello D'Andrea, Aswin Ramachandran
Robotics
DexPIE: Stable Dexterous Policy Improvement from Real-World Experience
Ruizhe Liao, Wenrui Chen, Liangji Zeng et al. (+4 more)
Dexterous manipulation presents substantial challenges for imitation learning due to its high-dimensional action space and complex contact-rich dynamics. Policies trained purely from demonstrations often suffer from compounding errors during deployment and require large amounts of expert data to achieve reliable performance. To move beyond the limitations of demonstration data, in this work, we propose DexPIE, a post-training framework for dexterous policy improvement from experience collected through real-world deployment. First, DexPIE enables effective exploration coverage through a dexterous-hand-adapted intervention system and multi-stage DAgger-style data collection across initial and intermediate task stages. Meanwhile, we enhance consistency between training and inference to reduce the distribution shift between rollouts and demonstration data, better aligning rollout behavior with demonstrations, allowing the critic to learn a value function induced by a more consistent underlying policy. Together, these components provide reliable supervision for policy evaluation. Finally, DexPIE improves the policy through conditioning on a continuous optimality indicator, allowing the policy to leverage the quality of data in a more fine-grained manner. Across three challenging real-world dexterous manipulation tasks, DexPIE achieves a 37.3% improvement in success rate over the demonstration-based reference policy, outperforming all baseline methods and demonstrating stronger robustness. The source code and
dataset will be made publicly available.
Read the full abstract ▾
All authors Ruizhe Liao, Wenrui Chen, Liangji Zeng, Haoran Lin, Fan Yang, Kailun Yang, Yaonan Wang
Robotics
Shape Formation for the Cooperative Transportation of Arbitrary Objects Using Multi-Agent Reinforcement Learning
Mohamed Sayed, Wolfram Burgard and Tanja Katharina Kaiser
Cooperative object transportation is essential in numerous domains, including industrial to domestic services. A popular transportation strategy is to carry objects on top of multi-
robot systems. The corresponding task is typically solved by decomposing it into three interconnected subproblems: formation control, cooperative
navigation, and collision avoidance. A particular challenge posed by real-world objects is their potentially arbitrary shape and non-uniform mass distribution, necessitating
robot formations that securely support the object. In this work, we address the challenge of pattern formation control for transporting such real-world objects by proposing a novel multi-
agent reinforcement learning approach. Our approach enables a multi-
robot system to autonomously position itself underneath an object to support its weight while avoiding obstacles during the formation process. Our evaluations with diverse environments and varying numbers of
robots show that our approach leads to policies that reliably produce balanced formations and generalize to cluttered scenes and objects with complex geometry and non-uniform mass distribution.
Read the full abstract ▾
Robotics
CT-VAM: A Cerebello-Thalamic-Inspired Vision-Action Model for Efficient Visuomotor Control
Jiacheng Li, Yize Guo, Jiabin Guo et al. (+2 more)
Vision-
language-action models have shown strong promise for
robot manipulation, yet raw
language is primarily needed to specify task intent rather than to be repeatedly processed during high-frequency low-level execution. Motivated by this separation, we propose a cerebello-thalamic-inspired
vision-action model (CT-VAM) for efficient task-conditioned visuomotor control. CT-VAM acts as a compact local execution policy that predicts action chunks from dualview visual observations, proprioception, and a lightweight task condition, potentially enabling a practical cloud-edge paradigm in which high-level semantic
reasoning can be handled by large models while fast closed-loop control runs on local hardware. To fuse heterogeneous inputs effectively, CT-VAM introduces TARS (Thalamic Action Routing Stream), a stream-separated conditional
attention decoder that independently routes action, visual and task streams, preventing dense sensory
tokens from overwhelming compact task-relevant conditions. With only 68M parameters, CT-VAM achieves LIBERO success rates competitive with substantially larger VLA models, while reducing inference latency. Together with flow-consistent inpainting for asynchronous chunk execution, CT-VAM supports high-frequency control and demonstrates robust realworld deployment on resource-constrained robotic platforms.
Read the full abstract ▾
All authors Jiacheng Li, Yize Guo, Jiabin Guo, Qingchen Liu, Jiahu Qin
Robotics
Efficient Minimal Solvers for Relative Pose Estimation in Autonomous Driving Applications
Tao Li, Liang Liu, Jianli Han et al. (+1 more)
With the advancement of visual sensing systems, computer
vision is playing an increasingly important role in autonomous driving and
robot navigation. Relative pose estimation in multi-camera systems is essential for accurate vehicle localization and environment perception, demanding high real-time performance and robustness. Existing methods, however, often involve high computational costs and rely heavily on abundant feature matches, limiting their applicability in time-sensitive driving scenarios. To address these limitations, this paper introduces a unified framework for efficient relative pose estimation, built upon a novel translation parameterization and first-order rotation approximation. Within this framework, we propose three efficient minimal solvers specifically designed for autonomous vehicles. The first solver integrates the vertical direction prior from Inertial Measurement Units (IMUs), the second utilizes the rotation axis direction prior during steering maneuvers, and the third is designed for planar motion - a realistic assumption for ground vehicles operating on structured roads. By reducing both the minimal number of point correspondences and the algebraic complexity, our methods enable faster hypothesis generation within RANSAC-based pipelines, improving suitability for real-time systems. Extensive experiments on synthetic
datasets and the KITTI autonomous driving
benchmark demonstrate that the proposed solvers achieve a favorable balance between speed and accuracy compared to existing state-of-the-art
algorithms.
Read the full abstract ▾
All authors Tao Li, Liang Liu, Jianli Han, Weimin Lv
Robotics · Cross-listed from cs.LG
Safe-RULE: Safe Reinforcement UnLEarning
Shixiong Jiang, Taozheng Zhu and Fanxin Kong
Offline safe reinforcement learning (Safe RL) enables policy learning without online interactions, making it suitable for safety-critical systems such as robotics systems. However, its reliance on static
datasets exposes offline Safe RL to data poisoning attacks, where adversaries inject malicious samples that compromise safety and induce unsafe policy behavior. In this work, we propose a new learning paradigm, named safe reinforcement unlearning (Safe-RULE), used as a defense framework to remove the influence of poisoned data without retraining from scratch or requiring access to the original training environment. We further extend reinforcement unlearning to offline Safe RL by explicitly accounting for both task performance and safety constraints during the unlearning process. Experiments across
benchmark Safe RL tasks demonstrate that our approach effectively enhances safety performance against data poisoning attacks.
Read the full abstract ▾
Robotics
Targeting World Models to Compromise Robot Learning Pipelines
Ethan Rathbun, Ahmed Agha, Saaduddin Mahmud et al. (+3 more)
World models have recently seen a rapid growth in both their popularity and capability as more data efficient tools for generating
robot training data or simulating real world environments, with many works proposing their integration into the
robot learning pipeline. While highly practical, in this work we demonstrate that world models introduce a uniquely stealthy and effective data poisoning entry point into the
robot learning supply chain that can result in the deployment of unsafe or otherwise compromised robotic policies despite training on seemingly safe ground truth training data. In contrast to traditional data poisoning techniques which directly implant dangerous trajectories into sold or uploaded
datasets, our novel attack methods inject malicious
prompts or compromising transition dynamics into visibly safe teleoperated
datasets which are only activated once fed through a world model as input. This can result in the generation of synthetic, dangerous
robot training trajectories and subsequently unsafe or compromised
robot policies. We demonstrate the effectiveness of our attacks against both state of the art action conditioned and text conditioned world models, showing a full end-to-end
backdoor on a downstream DRL policy and a proof-of-concept for the VLA setting. Overall these findings necessitate research into more secure world models and reevaluating their position within the
robot learning supply chain.
Read the full abstract ▾
All authors Ethan Rathbun, Ahmed Agha, Saaduddin Mahmud, Christopher Amato, Alina Oprea, Eugene Bagdasarian
Robotics
Goal Sets, Not Goal States: Queryable Robot Goals through Goal-Set Hindsight Relabeling
Carlos Vélez García, Miguel Cazorla and Jorge Pomares
Hindsight relabeling usually turns achieved future states into exact goals, which can overconstrain offline
robot learning when task success depends only on a subset of the state. We propose Goal-Set Hindsight Relabeling (GS-HER), a predicate-level generalization of HER in which achieved states certify query-defined goal sets rather than singleton goal states. A binary query specifies which variables define success, making the goal predicate an inference-time input while leaving the underlying offline GCRL
algorithm unchanged. Across OGBench tasks and five offline goal-conditioned learners, GS-HER improves performance when full-state goals are bottlenecked by nuisance dimensions and turns hindsight relabeling into a reusable goal interface: one checkpoint can answer multiple
robot goal predicates without retraining.
Read the full abstract ▾
Robotics
$\omega$-EVA: Envision, Verify, and Act with Latent Interactive World Models
Zhenguo Sun, Yu Sun, Hande Huang et al. (+1 more)
Embodied policies typically map current observations directly to actions, leaving candidate-action consequences implicit. World models provide predictive supervision, representations, or external simulation, but rarely let a policy inspect the imagined consequence of its own proposal before acting. We introduce $\omega$-EVA, a latent interactive world model that realizes an Envision–Verify–Act loop for
embodied action generation. Its three-stage framework learns action-conditioned latent dynamics, trains a
language-conditioned flow policy on dynamics-aware visual representations, and feeds the policy's proposal back through the world model. A tri-branch refiner jointly reasons over the current state, proposal-conditioned future, and proposed action to produce the final action chunk. Because consequence
reasoning remains in latent feature space, $\omega$-EVA avoids generating future
videos at inference. Evaluations across diverse single-arm, bimanual, long-horizon, and perturbed simulation settings show that the complete interaction pipeline consistently improves the proposal policy, while latent diagnostics indicate meaningful action-conditioned future structure. With approximately 1.2B parameters and no additional
robot-data pretraining, $\omega$-EVA demonstrates a compact and competitive performance–scale–data trade-off, making the world model an active action-feedback module rather than a passive predictor.
Read the full abstract ▾
All authors Zhenguo Sun, Yu Sun, Hande Huang, Alois Knoll
Robotics
Dense Force Estimation with an Event-based Optical Tactile Sensor
Agis Politis, René Zurbrügg and Valentina Cavinato
Humans rely on spatially dense, geometry and force-aware tactile feedback at high temporal resolution for dexterous manipulation. While
vision-based tactile sensors enable dense force estimation, they are limited by camera frame rates, motion blur, and data bandwidth. Event-based optical tactile sensors offer an attractive alternative with microsecond temporal resolution and low motion blur, but existing methods are restricted to predicting only net forces. We introduce the first framework for dense 3D force field reconstruction using event-based optical tactile sensors. Our approach estimates 3D surface displacements from event data and maps them to forces via the inverse Finite Elements Method (iFEM). Shear displacements are recovered through the proposed event-based marker tracking
algorithm, while normal displacements are predicted by a convolutional neural network trained on a collected
dataset of synchronized force-displacement-event data. Experiments demonstrate accurate reconstruction of physically grounded forces, achieving a mean absolute error of (0.14 N, 0.10 N, 0.93 N) over force ranges up to (4 N, 4 N, 20 N), while operating at an average of 100 Hz. This work constitutes a first step toward enabling dense force feedback for high-frequency control in robotic grasping and dexterous manipulation.
Read the full abstract ▾
Robotics
Harness Engineering for Physical AI: Robot Middleware Is the Harness Layer
Sanghoon Lee, Jiyeong Chae and Kyung-Joon Park
Robot middleware faces a new role in the era of Physical AI. Learned policies, planners, and
vision-
language-action (VLA) models now enter deployed
robots as causal participants on the control path, but the layer that integrates them with timing, scheduling, and network has not been named. Recent
language-
agent work names this layer the harness, the external system that mediates tools, manages state, bounds resources, and records execution. The robotics community has not yet adopted this framing, and we propose that
robot middleware is that harness. A Physical AI harness differs from a software harness in where it intervenes. A software harness mediates at tool-call boundaries. A Physical AI harness must mediate at control, computing, and communication simultaneously, because a learned policy's output crosses all three: its commands shift the trajectory, its inference time shifts the schedule, and its payload shifts the bandwidth.
Robot middleware is the lowest
robot-stack layer with mediating abstractions over all three, so it is best positioned to compose their enforcement. It already provides most of what a harness needs but lacks the enforcement for an AI model. We name this missing enforcement as three functions: Projection gates each output at emission, Isolation bounds the model's execution and transmission slot, and Transfer falls back to a verified baseline when checks fail. Each appears today as hand-built application code in deployed
robot systems, built on surfaces
robot middleware already provides.
Robot middleware should host them not as the best single-axis enforcer but as the layer that composes all three. We sketch this as a ROS 2 Harness Profile, a deployment artifact that carries an AI model's declared output region, inference budget, and operating regime while the middleware enforces them across ROS 2, DDS, and Zenoh.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Real-time body pose non-verbal communication with a consistency-based reliability measure
Alina Marcu, Dragos Costea, Cristina Lazar et al. (+1 more)
Body movement communicates intent at distances and in conditions where neither the face, nor
speech can be captured. We study the recognition of communicative intent from 2D body pose alone. We argue that body motion is a reliable signal especially in scenarios that require real time low-cost on-device person-to-
robot communication in long distance environments, such as rescue missions. However, existing resources do not isolate this signal. Affective corpora combine body, face, voice and text, while skeleton action-recognition
benchmarks label the action performed rather than the message conveyed. We release a
dataset of real frames of full-body pose covering ten communicative intents and we compare it against other real (IPC) and synthetic (MotionLCM, VEO3.1, Kimodo) ones that span a range of difficulty. We target systems that can run on a
robot's limited onboard hardware. We
benchmark multiple models, from skeleton graph classifiers to joint motion-
forecasting networks, and report performance metrics together with frame rate on an embedded GPU (NVIDIA Orin~Nano), since speed matters as much as accuracy in our scenario. Finally, we show that a model's own autoregressive self-consistency works as an unsupervised reliability signal. We give a short proof that bounds the probability that a self-consistent prediction is correct, show that this probability grows with the number of consistent steps, and identify the conditions under which a confident prediction can still be false, benchmarked against industry-standard metrics.
Read the full abstract ▾
All authors Alina Marcu, Dragos Costea, Cristina Lazar, Marius Leordeanu
Robotics
ReGIL: Retrieval-Guided Imitation Learning from a Single Demonstration
Yuying Zhang, Francesco Verdoja, Wenyan Yang et al. (+1 more)
Learning
robot manipulation policies with deep neural networks from a single demonstration remains highly challenging, as even small deviations from the demonstrated trajectory can quickly compound into failure, while collecting substantial online interaction data is costly. We propose ReGIL, a
retrieval-guided imitation learning framework that treats a single demonstration as an external
memory. ReGIL repeatedly queries this static
memory throughout training to simultaneously guide exploration, generate the regularization buffer, and construct rewards. Specifically, it computes rewards through local temporal
alignment between the current trajectory and the retrieved segment, providing step-wise and informative feedback for policy improvement. We evaluate ReGIL on robotic manipulation tasks from the LIBERO and Meta-World
benchmarks under the single demonstration setting. ReGIL outperforms prior baselines in both success rate and training efficiency. In real-
robot experiments, using only one demonstration and less than one hour of online training, ReGIL achieves over 75% success rate across three manipulation tasks with randomness in both initial
robot pose and target position. These results demonstrate that leveraging the single demonstration as reusable
memory can provide more than static supervision for efficient
robot learning. More details can be found on our website: https://regil2026.github.io/
Read the full abstract ▾
All authors Yuying Zhang, Francesco Verdoja, Wenyan Yang, Ville Kyrki
Robotics
MosaicIMU: Composing Carrier Experts for Generalizable Neural Inertial Odometry
Junye Zou, Huiyi Yan, Xinning Xu et al. (+4 more)
Robust inertial odometry is essential for various carriers when external sensing is unreliable. Learning-based methods reduce integration drift by capturing local motion priors, but these methods often remain tied to a particular carrier, limiting generalization across heterogeneous platforms. We present MosaicIMU, a carrier-conditioned Mixture-of-Experts (MoE) pretraining-and-adaptation framework for generalizable neural inertial odometry. MosaicIMU uses a prototype-based router to compose carrier-specific expert features, decodes local velocity and uncertainty constraints, and integrates them with a history-aware EKF. For unseen domain adaptation, it freezes the pretrained base model and learns a new lightweight expert residual branch. For edge-deployment, it further reuses the router to select informative online samples for efficient incremental updates. Experiments show that MosaicIMU consistently outperforms learning-based baselines, reducing average ATE and RTE-10s by 40% and 34%, respectively. These results highlight that MosaicIMU provides a scalable pretraining-to-deployment paradigm for generalizable and adaptive neural inertial odometry.
Read the full abstract ▾
All authors Junye Zou, Huiyi Yan, Xinning Xu, Xiaolei Li, Pengkun Zhou, Jinhui Zhang, Ziyang Meng
Robotics
Taming Perception Jitter: Uncertainty-Aware LiDAR Object Detection for Reliable Motion Classification
Cornelius Schröder, Žygimantas Marcinkus and Markus Lienkamp
Reliable motion classification is critical for autonomous driving, as false dynamic predictions of static objects can cascade into unnecessary planner interventions. Unstable bounding box predictions can lead to spurious velocity estimates in tracking and falsely predicted trajectories. We present a deployment-friendly mitigation strategy that augments a 3D object detector with aleatoric uncertainty estimates and applies a two-sample z-test over short observation windows to separate true motion from jitter. Integrated into Autoware with minimal changes, the approach reuses existing data association for minimal compute overhead. Empirical results show parity with velocity thresholding on nuScenes, but substantially fewer false dynamic predictions and unnecessary stops in real-world test drives, explained by the presence of an intermediate jitter band in the recorded data that speed-only rules misclassify. This demonstrates that uncertainty-aware detection and lightweight statistical testing can deliver practical performance gains for autonomous driving in noisier real-world settings.
Read the full abstract ▾