Robotics
Beyond Monotonic Progress: Retry-Supervised Value Learning for Robot Imitation
Xinyao Qin, Junjie Lu, Kaixin Wang et al. (+7 more)
Human demonstrations for
robot imitation learning often contain mistakes and corrective behaviors, such as imprecise grasps, object misalignment, unstable contact, and repeated attempts. While these segments are commonly treated as noisy or suboptimal data, they provide valuable evidence about when execution deviates from a desirable path and how task feasibility can be restored. However, existing reward and value models often rely on monotonic progress assumptions, which capture coarse task advancement but may overlook local execution errors and corrective behaviors in imperfect demonstrations. In this work, we propose ReTVL (ReTry-Supervised Value Learning), a framework for learning mistake-sensitive value functions from mixed-quality
robot demonstrations by leveraging retry events as sparse supervision. ReTVL captures the local degradation-and-recovery structure around mistakes by combining global progress calibration with local pairwise preference learning induced by sparsely annotated retry keypoints. The learned value model is then used to reweight demonstration chunks for downstream behavior cloning, reducing the influence of harmful execution errors while preserving useful corrective behaviors. Experiments on real-
robot manipulation tasks show that ReTVL produces more fine-grained value estimates than progress-based baselines and improves imitation learning from imperfect demonstrations.
Read the full abstract ▾
All authors Xinyao Qin, Junjie Lu, Kaixin Wang, Chuheng Zhang, Sinjae Kang, Kimin Lee, Min Xu, Bin Liang, Jun Yang, Li Zhao
Robotics · Cross-listed from math.OC
Parallel Dynamic Programming for Conic Linear Quadratic Control
Luyao Zhang, Gabriel Bravo-Palacios, Brian Plancher et al. (+1 more)
Linear Quadratic (LQ) control problems are at the heart of linear control theory and Model Predictive Control (MPC). While performant, standard approaches to solving such problems are inherently serial, limiting real-time scalability despite the parallel computing power available on modern multi-core CPUs. Contributing to addressing this challenge and motivated by “divide and conquer” strategies, we present a parallel-in-time approach that solves computationally demanding conic optimal control problems through the use of the alternating direction method of multipliers (ADMM). In particular, we formulate the inner primal update of ADMM as an LQ problem and split the reformulated problem along the time horizon. This enables us to derive a variant of the Riccati recursion using dynamic programming to solve each subproblem in parallel. Numerical
benchmarks on two real-world applications demonstrate as much as a 5x speedup compared to existing related approaches on multi-core CPU hardware.
Read the full abstract ▾
All authors Luyao Zhang, Gabriel Bravo-Palacios, Brian Plancher, Sergio Grammatico
Robotics
Optimization-based Safe Trajectory Planning for Autonomous Ground Vehicle in Multi-Floor Scenarios
Zishang Xiang, Runda Zhang, Runqi Chai et al. (+3 more)
The development of trajectory planning strategies for autonomous ground vehicles (AGVs) represents a prevailing research interest within the domain of intelligent transportation systems. This paper introduces a trajectory planning framework tailored for multi-floor scenarios. The framework consists of two main modules: the task planning module and the trajectory planning module. The task planning module involves a strategic selection phase, where a task planning strategy based on generalized voronoi diagrams (GVD) and multi-objective
algorithms is proposed to select the floor exits for each floor. The trajectory planning module utilizes optimization-based methods to generate high-quality trajectories, and a warm-started hierarchical planning framework is designed to ensure rapid convergence. Additionally, for handling complex obstacle constraints, a correlation constraint calculation method is designed for reducing obstacle constraints in trajectory planning. Finally, the feasibility and effectiveness of the proposed framework are verified through simulations.
Read the full abstract ▾
All authors Zishang Xiang, Runda Zhang, Runqi Chai, Kaiyuan Chen, Senchun Chai, Yuanqing Xia
Robotics
ArtiTwinSplat: Interactable Digital Twin Reconstruction via Gaussian Splatting from RGB-D videos
Pranjal Mishra, René Zurbrügg, Max Wilder-Smith et al. (+4 more)
Deploying
robots in unstructured real-world environments needs accurate, interactive models of the objects. Constructing these models at scale remains a critical bottleneck for robotic system integration. We present ArtiTwinSplat, a framework that automatically constructs articulated, photo-realistic digital twins of objects directly from RGB-D
videos, requiring no CAD models, simulation assets, or manual annotations. Our method is built on 3D Gaussian Splatting that preserve
geometric fidelity and photometric realism, coupled with an unsupervised articulation discovery pipeline that recovers part structure and joint kinematics from observed motion alone. With tracking and optimization stages our method provides stable, queryable digital twins that support real-time rendering, viewpoint control, and interactive manipulation. Unlike prior methods confined to simulation, ArtiTwinSplat operates directly on real-world observations and produces twins that are immediately usable by downstream
robot planning and learning systems. This method offers a practical, scalable pathway toward digital twin construction, lowering the integration barrier for articulated object manipulation in
embodied AI and human-
robot collaboration contexts.
Read the full abstract ▾
All authors Pranjal Mishra, René Zurbrügg, Max Wilder-Smith, Marco Hutter, Marc Pollefeys, Zuria Bauer, Hermann Blum
Robotics
Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement
Xin Liu, Yulin Li, Ziming Li et al. (+7 more)
Simulator-in-the-loop optimization offers a promising inference-time mechanism for
robot manipulation. It uses a physical simulator as a backend rollout engine to evaluate candidate trajectories in parallel and refine nominal actions online, a paradigm shown to be effective in rigid-body manipulation where state and contact are relatively tractable. We bring this paradigm to real-world cloth manipulation from a single RGB input through three pillars. (i) We design a scalable synthetic-data generation and inference-time rollout pipeline built on FLASH, a deformable-object simulator that provides a practical balance among physical fidelity, numerical stability, and rollout efficiency. (ii) We develop a real-to-sim module, trained purely on synthetic data, that maps a single RGB observation to simulation-compatible cloth state by fusing pretrained visual features with learnable canonical
tokens. (iii) We perform online planning by coupling a sparse-mesh rollout backend with prior-guided MPPI, anchored at an offline-distilled policy trajectory, preserving manipulation-relevant deformation and contact while enabling sufficient parallel rollout batches. Real-
robot experiments show higher success rates than baseline methods and closed-loop correction under mid-fold perturbations. Project page: https://silr-cloth.github.io/
Read the full abstract ▾
All authors Xin Liu, Yulin Li, Ziming Li, Pengyu Jing, Zhenhao Huang, Bingyang Zhou, Ziqiu Zeng, Siyuan Luo, Chenkun Qi, Fan Shi
Robotics
Explaining Failures of Cyber-Physical Systems with Actual Causality
Khen Elimelech, Tom Yaacov, David A. Kelly et al. (+2 more)
Modern autonomous Cyber-Physical Systems (CPSs), such as self-driving cars, face increasingly complex demands, and yet are expected to act reliably. The black-box nature often characterizing such systems, especially those relying on neural components, makes it impossible to fully verify the system behavior prior to deployment. Unfortunately, unexpected failures-when the system does not comply with its specification-are inevitable and may have catastrophic implications. To improve trust in the system and facilitate future mitigation after a failure occurs, it is important to try to derive an explanation for the unexpected system behavior. This paper introduces the novel concept of leveraging the framework of actual causality for CPS failure explanation. Up until now, this framework was only used to derive explanations in the context of simple systems, such as image classifiers. This paper addresses the theoretical gaps and provides the guidance needed to allow for correct explanation derivation in the CPS domain. Beyond the theoretical contribution, the paper presents two novel, practical, system-agnostic explanation derivation
algorithms, allowing to prioritize either explanation optimality or derivation efficiency. The approach is demonstrated and evaluated in the context of a neural-network-controlled autonomous car, designed to avoid collisions.
Read the full abstract ▾
All authors Khen Elimelech, Tom Yaacov, David A. Kelly, Hana Chockler, Moshe Y. Vardi
Robotics
Decentralized Pose Graph Riemannian Optimization for Object-based Multi-Robot SLAM
Yixian Zhao, Yan Huang, Yang Xu et al. (+2 more)
Pose graph optimization (PGO) is a key back-end component for state estimation in networked multi-
robot simultaneous localization and mapping (SLAM). In object-based multi-
robot SLAM, the problem becomes more tightly coupled because
robots must jointly estimate both their trajectories and the poses of persistent objects observed by multiple
agents. Existing decentralized solutions often assume that the communication graph closely matches the physical interaction
topology, which is restrictive in realistic deployments where communication is sparse, intermittent, or time-varying. This paper presents a fully decentralized Riemannian optimization framework for object-based multi-
robot PGO that decouples the coupled estimation problem via a consensus mechanism, enabling flexible communication
topologies. To improve convergence under limited communication budgets, we further develop a distributed approximate-Newton scheme that exploits local second-order information while operating directly on the SE(d)
manifold to preserve
geometric consistency, and we establish the convergence to Riemannian first-order stationary points and provide a local condition-number analysis explaining the benefit of approximate second-order information over first-order Riemannian descent. The resulting method reduces iteration count and communication overhead without sacrificing estimation accuracy. Extensive evaluations on public
benchmarks, large-scale simulations, and real-world multi-
robot experiments demonstrate improved accuracy, runtime efficiency, scalability across network
topologies, and robustness to communication failures.
Read the full abstract ▾
All authors Yixian Zhao, Yan Huang, Yang Xu, Liang Li, Jinming Xu
Robotics
Yue Peng, Yongzhe Zhao, Artur Habuda et al. (+5 more)
Vision-
language-action (VLA) models have rapidly advanced generalist
robot manipulation by harnessing the semantic knowledge of pretrained
vision-
language backbones. However, their visual representations are typically grounded in 2D image coordinates rather than calibrated camera geometry. This limitation is especially pronounced in multi-camera systems, where views are coupled by known intrinsics and extrinsics but processed as independent images. We propose G$^3$VLA, a camera-aware
geometric module that injects calibrated structure into the visual-
token stream of a pretrained VLA without altering its action space or original action-learning objective. G$^3$VLA combines intrinsic-conditioned ray
embeddings, projective positional encoding (PRoPE), and bidirectional cross-view fusion to establish geometrically consistent representations across camera views.
Geometric supervision is provided by ground-truth point maps or confidence-gated $\pi^3$X teacher predictions, enabling training without depth sensors or manual
geometric annotations. On $\pi_0$, G$^3$VLA consistently improves performance on LIBERO and RoboCasa24 under both supervision regimes, while ground-truth supervision improves five-task RoboTwin2.0 averages with three cameras. Experiments on camera-shifted LIBERO-Goal and real
robots demonstrate improved robustness to unseen camera viewpoints. Controlled calibration studies across $\pi_0$ and $\pi_{0.5}$ show that these gains depend on correct camera geometry while remaining stable under mild calibration perturbations. These results demonstrate that grounding VLA visual
tokens in calibrated camera geometry provides a simple and general mechanism for improving
geometric consistency, multi-view
reasoning, and viewpoint robustness in pretrained
robot policies.
Read the full abstract ▾
All authors Yue Peng, Yongzhe Zhao, Artur Habuda, Khuyen Pham, Yanheng Zhu, Tran Nguyen Le, Fares Abu-Dakka, Li Guo
Robotics
FT-WBC: Learning Fault-Tolerant Whole-Body Control for Legged Loco-Manipulation
Yudong Zhong, Pengfei Mai, Sikai Guo et al. (+6 more)
Legged manipulators combine the mobility of legged platforms with the manipulation capability of robotic arms. However, arm-induced Center-of-Mass shifts and dynamic disturbances make the system more prone to instability under actuator failures, potentially leading to falls, task failures, or safety risks. Existing fault-tolerant control methods mainly focus on locomotion alone, leaving the coupled problem of whole-body stability and arm reachability in fault-tolerant loco-manipulation largely unaddressed. To bridge this gap, we propose FT-WBC, a fault-tolerant loco-manipulation framework for robust whole-body control of legged manipulators under actuator failures. FT-WBC adopts a decoupled upper- and lower-body policy architecture and introduces two key modules: a Fault
Estimator (FE) and a Posture Adaptation Module (PAM). The FE predicts faulty joints from lower-body proprioceptive histories, while the PAM uses this fault information to adapt the base posture plan generated by the arm policy, converting potentially unstable posture requests into safe and executable base posture commands. Through this fault-aware posture adaptation mechanism, FT-WBC synthesizes compensatory gaits under actuator failures and preserves as much arm workspace as possible while maintaining whole-body stability. Simulation and real-world experiments show that FT-WBC significantly improves survival rate and workspace under weakening or locked failures, and transfers zero-shot to a real legged manipulator in the real world.
Read the full abstract ▾
All authors Yudong Zhong, Pengfei Mai, Sikai Guo, Jiahang Cao, Zhihai Bi, Qiuyue Liu, Ziyan Feng, Jinni Zhou, Jun Ma
Robotics
Varying Bundle Size Reactive Multi-Task Assignment using Selective Cost Estimation for Multi-Agent Systems
Niklas Dahlquist, Shridhar Velhal and George Nikolakopoulos
This paper presents a scalable framework for multi-
robot task allocation in complex environments where estimating task execution costs is computationally expensive. While combinatorial auction-based approaches offer reliable solutions, the exponential complexity of bundle generation typically renders them intractable for real-time reactive applications, particularly when accurate path planning is required for cost validation. We address this through a distributed, two-stage multi-fidelity bundle generation approach.
Agents utilize a local search tree guided by a low-fidelity heuristic (such as euclidean distance) to rapidly explore the bundle space, applying high-fidelity path planning only to the most promising candidates in a best-first manner. These refined bids are then submitted to a central coordinator that solves a set packing problem to ensure global feasibility and maximize the overall utility. Simulation results in multiple environments demonstrate that the framework is able to improve the performance of reactive auction-based task allocation. Overall, the presented framework is shown to enable reactive task allocation with dynamic bundle sizes in multiple settings without exposing the
agents' state and internal cost estimation models.
Read the full abstract ▾
Robotics
NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulation
Soham Patil, Avirup Das, Sourabh Bhosale et al. (+1 more)
Perceiving physical contact is fundamental to dexterous manipulation. While
robots often rely on dedicated hardware tactile sensors, humans exhibit a remarkable ability to infer contact by integrating visual information with an innate sense of their body's pose and movement. Inspired by this
embodied perceptual skill, we investigate whether a
robot can learn to infer contact from
vision, an approach that also offers a scalable alternative to tactile hardware specifically for binary contact estimation, which faces practical challenges in cost, fragility, and integration. We present NoContactNoWorries, a
transformer-based
multimodal framework that fuses RGB-D
vision with the
robot's proprioception to infer binary contact states as a pseudo-tactile signal for hand-object interactions. We validate by training a single contact prediction model on multiple objects and show that the inferred contact signal supports downstream reinforcement learning
agents for in-hand object reorientation, generalizing to novel objects. Experiments in both simulation and on a real-world
robot validate our approach, highlighting the feasibility of inferring contact from
vision and proprioception. Project Page: https://soham2560.github.io/no-contact-no-worries/
Read the full abstract ▾
All authors Soham Patil, Avirup Das, Sourabh Bhosale, Spandan Roy
Robotics
Supervise What Survives: Geometry-Guided VLA Adaptation from Synthetic Robot Videos
Danze Chen, Yanzhe Chen, Qiming Huang et al. (+3 more)
Vision-
Language-Action (VLA) models require large-scale
video-action pairs, yet real teleoperation remains scarce. While generated
robot videos offer a scalable alternative, existing methods treat them as real
robot data by recovering pseudo-actions from synthesized pixels. We argue that deriving low-level control from generated visuals is a mismatched abstraction. A
video captures only geometry: the spatial trajectory representing the where of a task. A real demonstration captures control: the exact motor commands representing the how. Human-to-
robot video generation preserves these unequally: the visible geometry survives the generation process, while the underlying control signals are lost. This Asymmetric Preservation Principle dictates a clean rule: this surviving geometry should solely supervise visual perception, leaving control to real demonstrations. Following this principle, we propose GRA (Geometry-guided Representation Alignment), which extracts the
geometric content as future 2D end-effector waypoints, computed from the source human
video through pose estimation, retargeting, simulation, and calibrated projection, and routes them to the VLA
vision backbone via an auxiliary 2D head. The action head is trained on real demonstrations only. During fine-tuning, the waypoint loss persists as a spatial representation anchor that prevents the backbone from losing its
geometric grounding. On real-
robot tasks, GRA outperforms pseudo-action baselines under matched data budgets and narrows the gap to policies trained with substantially more real demonstrations, suggesting that correctly routed geometry bridges generated
videos to
robot policies more reliably than recovered actions.
Read the full abstract ▾
All authors Danze Chen, Yanzhe Chen, Qiming Huang, Zhijun Cao, Chen Gao, Mike Zheng Shou
Robotics
Legible and Intuitive Multi-modal Robot State and Intent Communication Validated in Online and Real-world Studies
Tim Schreiter, Jens V. Rüppel, Andrey Rudenko et al. (+2 more)
Effective
robot-to-human communication can increase transparency and trust, reduce uncertainty, and contribute to safer collaboration in shared workspaces. Designing and validating an effective
robot communication strategy is challenging due to the varying and often limited communication modalities across
robots, differences in how diverse recipients interpret messages, and the underexplored virtual-to-real gap in studies of communication legibility. We present a systematic, large-scale comparative validation of existing communication strategies for a mobile non-
humanoid robot across message types and settings (online and in-person). Based on the prescribed message types in the existing standards for industrial
robots, we realize and compare a low-expressive, unimodal LED-based strategy with a highly expressive,
multimodal one that leverages robotic gaze, gestures, and voice. For each strategy, we analyze the communication of a turning intention, an
attention request, error status, whether the
robot is stuck, and whether it is functioning normally. We evaluate these strategies in replicated online and in-person experiments. We find strong evidence that highly expressive
multimodal communication is perceived as more legible and intuitive than unimodal LED-based communication. Comparing the online and real-world study findings, we observe a notable decrease in overall legibility, particularly for signaling with LEDs. Similarly, confidence in message interpretation decreases during the real-world evaluation.
Read the full abstract ▾
All authors Tim Schreiter, Jens V. Rüppel, Andrey Rudenko, Martin Magnusson, Achim J. Lilienthal
Robotics
RE4: Transformation-aware Imitation of Object Interactions Using Manipulation Modes
Arsh Chawla and Rahul Shome
Object interaction tasks have been a focus of advances in imitation learning. End-to-end methods, dominated by
diffusion and flow-based variants have shown leaps in performance while sacrificing interpretability. Object-centric and pose-informed variants have had a role in learning from demonstration in manipulation tasks. In this paper, we revisit a few modern imitation learning
benchmarks for object interactions, with the aim of composing a framework that repurposes principled theories of manipulation, preserving both performance and interpretability. For image observations, lightweight training is proposed for model-free pose estimation of the target object, using self-supervision over the demonstration data available for imitation learning. This information is then used to inform a manipulation mode-aware
retrieval of a demonstration, a mode-aware transformation, a replan step that connects to the
retrieval point while preserving mode constraints, and finally rolling out the transformed demonstration. These compose four key steps of the proposed RE4 framework, evaluated over state-based and image-based
benchmarks in Push-T and Robomimic. An
adversarial benchmark that evaluates sparse data regions of image-based Push-T showcases the robustness, further bolstered by indications from low-data regime experiments. The current work shows promise in using simple interpretable building blocks to learn manipulation skills.
Read the full abstract ▾
Robotics · Cross-listed from cs.HC
Average Rankings Mask Per-Subject Optimality: A Friedman-Nemenyi Benchmark of EEG Motor-Imagery BCI Decoders
Xavier Vasques, Paul Barbaste and Olivier Oullier
Electroencephalography (
EEG) is the dominant non-invasive modality for
brain-computer interfaces (BCIs), yet reliable decoding of motor imagery is hampered by inter- and intra-individual variability. A recurring claim is that one decoding pipeline, most often a spatial or Riemannian method, is broadly preferable. We test the weakest version of that claim under the most favourable conditions. Using the Mother of All BCI
Benchmarks (MOABB) framework, we evaluated 1,056 decoding configurations (feature extractor x scaler x classifier), >340,000 subject-level model fits, across three public left-versus-right motor-imagery
datasets (PhysionetMI, 109 participants; Cho2017, 52; Zhou2016, 4) and two frequency bands (8-15 Hz, 8-30 Hz). Every model is fit and tested within a single session of a single participant, the easiest regime, giving every pipeline its best chance. We apply the statistics standard for multi-classifier comparison: Friedman omnibus tests, Nemenyi critical-difference analysis and Wilcoxon signed-rank tests with effect sizes. Covariance tangent-space projection (cov-tgsp) and Common Spatial Patterns (CSP) are the strongest families, but their ordering is
dataset-dependent and, on the largest and most heterogeneous cohort (PhysionetMI), statistically indistinguishable (Nemenyi p = 0.27; Kendall's W = 0.11). At the individual level the single best pipeline is optimal for only 35% of PhysionetMI participants, and nonlinear descriptors are best for roughly one third; matching pipeline to participant adds about seven accuracy points over the best fixed choice. The ranking is not an artefact of dimensionality, and classifier and scaler choices are secondary to the feature representation. Even in the easiest regime, no single pipeline dominates: a lower bound on the personalization problem and a quantitative case for participant-aware model selection rather than a universal decoder.
Read the full abstract ▾
Robotics
PDS Joint: A Parametric Double-Spiral Joint Tailored for Dexterous Hands
Haoyang Li, Yibo Wen, Yixiang Fan et al. (+2 more)
Compliant joints can embed safety and adaptability into dexterous hands, but achieving large-stroke anthropomorphic motion while maintaining joint-specific, directiondependent stiffness and reliable proprioception remains challenging. This paper presents the PDS joint, a parametric doublespiral (PDS) compliant joint that enables systematic shaping of directional stiffness across multiple deformation modes, including flexion/extension, abduction/adduction, and pronation/supination. We instantiate the joint using Archimedean and logarithmic spiral templates for different hand joints and introduce an asymmetry ratio to tailor stiffness distributions for both grasp stability and hyperextension resistance. To make the joint practically usable under large deformation, we co-design embedded inductive proprioception and propose a learningbased calibration pipeline that maps raw inductive signals to joint states using ArUco-marker tracking. Experiments characterize the stiffness landscapes across
geometric parameters and demonstrate a non-monotonic dependence of lateral support on asymmetry, indicating the importance of principled parameter tuning. For joint-state estimation in the most challenging abduction/adduction motion, a learned multilayer-perceptron (MLP) mapping reduces the error compared with conventional curve fitting by 41.6%. Finally, we integrate the proposed joints into an open-source dexterous hand as a demonstration platform, on which the hand grasps a set of nine everyday objects and performs safe, contact-rich human-involved interactions.
Read the full abstract ▾
All authors Haoyang Li, Yibo Wen, Yixiang Fan, Yiheng Xu, Yufeng Yue
Robotics
SlipSense: Multimodal Sensing for Online Slip Detection in Legged Robots
Iris Szu-Yao Liu, Chien Chern Cheah and Meng Yee Michael Chuah
Legged
robots rely on accurate ground interaction awareness to traverse variable terrains, such as slippery surfaces. Existing slip detection methods often rely on kinematics and proprioception, which lack the sensitivity to detect early-stage slips that occur prior to catastrophic instability. Thus, this paper presents SlipSense, a novel framework for online force-based slip detection using a custom lightweight sensorized foot for quadrupeds to detect slip. The framework integrates a
multimodal sensor design with a LSTM-based model to infer ground reaction forces and detect slip-indicative anomalies during locomotion. The proposed framework is deployed on a Unitree Go1 quadruped to demonstrate blind online slip detection over a slippery terrain. Our method detects early-stage slips down to an average displacement of 24.1 +/-6.4mm with an overall accuracy of 85.9%. This represents a 3.3-fold finer detection resolution and a 24% relative accuracy improvement over a standard kinematic baseline that uses foot velocity inferred through state estimation. The work in this paper serves as a foundation for force-aware gait adaptation in legged robotic locomotion, allowing future controllers to estimate terrain friction and adjust constraints, thus improving the overall stability of the system.
Read the full abstract ▾
Robotics
RoBoSR: Structured Scene Representations for Embodied Robotic Reasoning
Kewei Hu, Wanchan Yu, Fangwen Chen et al. (+6 more)
Despite rapid progress,
embodied reasoning under real-world variability remains challenging. Existing approaches rely on demonstration-driven sequential biases, limiting flexibility in open-ended and long-horizon tasks that require structured
reasoning over evolving states. We introduce RoBoSR, an intermediate structural representation that formulates manipulation as step-wise state transitions over semantically grounded, object-centric scene graphs. By modeling object states and their spatial relations at the perception-action interface, RoBoSR disentangles high-level task
reasoning from raw inputs and enables structured
reasoning over preconditions, effects, and goal states. This representation endows the
agent with causal
reasoning capability, enforcing subtask dependencies and supporting coherent long-horizon task planning. To learn such structure-aware
reasoning, we construct Manip-Cognition-1.6M, an open-world
dataset that jointly supervises scene understanding, instruction interpretation, and subtask planning across diverse tasks. Across several
benchmarks and real-world demonstrations, our method consistently outperforms prompting-based methods and classical TAMP baselines in zero-shot generalization and long-horizon tasks. The results underscore structured intermediate representations as a critical inductive bias for scalable
embodied reasoning.
Read the full abstract ▾
All authors Kewei Hu, Wanchan Yu, Fangwen Chen, Jing Jiajian, Zimeng Li, Ying Wei, Tianhao Liu, Michael Zhang, Hanwen Kang
Robotics · Cross-listed from cs.CV
From Open Waters to Enclosed Cabins: ProteusVPR for Cross-Scene Visual Place Recognition in Maritime Perception and Cabin Inspection
Zexi Chen, Zitai Huang, Qiwen Gu et al. (+6 more)
Autonomous robotic inspection in maritime environments presents unique challenges for Visual Place Recognition (VPR) due to cross-scene perceptual shifts.
Robots navigating ship-borne environments must transition between visually distinct domains: open decks with sparse textures and severe illumination changes, and enclosed cabins with repetitive structures and high visual ambiguity. Existing VPR methods, designed primarily for urban or indoor scenes, fail to generalize reliably across these starkly different scenarios. To address this, we propose ProteusVPR, a two-stage
retrieval-refinement framework. The first stage employs any standard VPR model for initial image
retrieval. The second stage introduces a
geometric-visual estimation network that fuses the retrieved image with two temporally preceding frames, incorporating
geometric descriptors, a local affine coordinate system, and camera azimuth encoding to achieve precise localization. To support this task, we introduce the XHZ
dataset, an 8K-panoramic ship-borne
dataset collected from an operational vessel, featuring multi-floor cabin structures, deck transition zones, and strict query-database separation for rigorous evaluation. Extensive experiments on the XHZ
dataset demonstrate that ProteusVPR consistently improves the localization accuracy across multiple VPR backbones, reducing mean localization error by over 60% on average and that ProteusVPR offers an effective and robust solution for precise visual localization in challenging, cross-scene maritime environments.
Read the full abstract ▾
All authors Zexi Chen, Zitai Huang, Qiwen Gu, Zhiqi Li, Shengli Dong, Chenlei Wang, Junqiao Zhao, Hongdong Wang, Bing Han
Robotics
Grounding Generative Policies in Physics: Optimization-Guided Diffusion for Robot Control
Sabrina Bodmer, René Zurbrügg, Tifanny Portela et al. (+5 more)
Diffusion models sample effectively from high-dimensional,
multimodal distributions, but their outputs may violate deployment constraints. For task-space
robot policies, generated grasps, waypoints, or trajectories can be distributionally valid yet infeasible, violating reachability, collision-avoidance, or closed-loop executability requirements. This embodiment gap limits zero-shot deployment across
robots, even when the task-space behavior itself is transferable. We propose an inference-time optimization framework that couples the behavior generation to physical feasibility by formulating
diffusion guidance as a constrained optimization problem. Our key insight is to replace the sampling perturbation in the backward process with an optimized correction, allowing hard constraints or soft penalties to be imposed during sampling without the need to retrain the
diffusion model, while keeping samples close to the learned prior. We evaluate the method on dexterous grasp synthesis with reachability and collision-avoidance constraints, and dynamic manipulation with controller-level trackability constraints. Across settings and
robot embodiments, optimization-guided denoising matches the feasibility of projection- and gradient-guidance baselines while better preserving grasp quality, and improving controller-level executability and task success, with task success improving by up to 20pp. on dexterous grasping and 23pp. on visuomotor manipulation over the best baseline.
Read the full abstract ▾
All authors Sabrina Bodmer, René Zurbrügg, Tifanny Portela, Hao Ma, Alexandre Didier, Marco Hutter, Colin Jones, Melanie Zeilinger
Robotics · Cross-listed from cs.NI
Importance of Intent-Sharing for V2X-based Maneuver Coordination
Rafael Molina-Masegosa, Sergei S. Avedisov, Miguel Sepulcre et al. (+3 more)
This paper examines the critical role of intent-sharing in enabling effective maneuver coordination for connected and automated vehicles (CAVs). Successful maneuver coordinations require vehicles to accurately know other vehicles' driving intentions. Intent-sharing can be achieved by the remote vehicles directly communicating their plans with the ego vehicle, as opposed to the ego vehicle predicting the trajectory on the remote vehicles' behalf. In this paper, we investigate the potential of intent-sharing on maneuver coordination effectiveness by quantifying the percentage of successful coordinations. We analyze the potential of intent-sharing by comparing its effectiveness for coordinated lane changes in a highway scenario with the effectiveness of a trajectory prediction method based on current kinematic data. Our analysis demonstrates in two scenarios substantial improvements in maneuver coordination when CAVs have direct access to the nearby vehicles' driving intentions through intent sharing. These findings highlight the importance of including intent-sharing in the maneuver coordination protocol.
Read the full abstract ▾
All authors Rafael Molina-Masegosa, Sergei S. Avedisov, Miguel Sepulcre, Javier Gozalvez, Yashar Z. Farid, Onur Altintas
Robotics
The Evaluation Cost of Task Specialization in Evolutionary Multi-Robot Systems
Paolo Leopardi, Heiko Hamann, Jonas Kuckling et al. (+1 more)
Task specialization can improve the efficiency of multi-
robot systems (MRSs). Previous works have investigated the emergence of task-specialist
robot controllers through evolutionary optimization and have argued that task specialization is more likely to evolve when subtask behaviors are readily available as building blocks. However, the available evaluation budget must be distributed across all subtasks, whereas a single generalist behavior can exploit the entire budget for its own optimization. We present a cost-benefit analysis of evolving task-specialist versus generalist behaviors in a foraging scenario here. In a physics-based robotics simulator, we study the total evaluation budget required to evolve task-specialist behaviors that outperform generalist behaviors across MRS sizes. We show that with increasing MRS size, a lower total evaluation budget is sufficient to evolve specialists that outperform generalists.
Read the full abstract ▾
All authors Paolo Leopardi, Heiko Hamann, Jonas Kuckling, Tanja Katharina Kaiser
Robotics · Cross-listed from physics.flu-dyn
Efficient Time-Domain Simulation of USV Motions in Short-Crested Irregular Waves Using an IRF-Based Framework
Fei Duan, Zihao Wang, Yaohua Zhou et al. (+1 more)
Traditional time-domain prediction of vessel motions in irregular waves usually relies on superposing responses from many regular-wave components, which is computationally expensive for long-duration simulation and real-time applications. This issue is particularly relevant to unmanned surface vehicles (USVs), for which efficient and realistic motion prediction is needed for seakeeping assessment, simulation-based testing, and control-system development. This study applies an impulse response function (IRF)-based time-domain framework to predict vessel motions in short-crested irregular waves. Froude-Krylov, diffraction, and radiation loads are obtained from frequency-domain analysis and transformed into the time domain. Instantaneous responses are then evaluated directly through convolution-based force reconstruction, reducing the need for repeated regular-wave simulations. Weak nonlinear restoring effects are included by instantaneous wetted-surface pressure integration, and directional wave spectra are used to represent realistic sea states. The framework is validated against model-test measurements of an offshore supply vessel in long-crested beam irregular waves and full-scale measurements of a USV operating in real sea conditions. Predicted significant amplitudes, mean zero-crossing periods, standard deviations, and motion time histories agree well with measurements. The effect of directional-spectrum discretization is also examined. Results show that motion amplitudes are moderately sensitive to directional resolution, whereas motion periods are relatively insensitive. A 30 deg directional interval provides a practical balance between prediction accuracy and computational cost. The proposed framework offers an efficient tool for high-fidelity time-domain prediction of USV motions in realistic directional irregular seas.
Read the full abstract ▾
All authors Fei Duan, Zihao Wang, Yaohua Zhou, Qing Xiao
Robotics
NavWM: A Unified Navigation World Model for Foresight-Driven Planning
Yanghong Mei, Longteng Guo, Ming-Ming Yu et al. (+3 more)
Conventional visual
navigation policies often struggle with myopic decision-making and mode collapse in complex environments. While world models offer a promising alternative, existing paradigms typically isolate perception, generation, and control, failing to capture their shared spatio-temporal dynamics. In this paper, we propose NavWM, a unified
navigation world model that seamlessly integrates latent world
reasoning,
multimodal action prediction, and controllable visual generation. At its core, NavWM leverages latent world
tokens to distill
geometric and semantic priors, endowing the
agent with robust structural understanding. To overcome the limitations of deterministic policies, we introduce an anchor-based
multimodal trajectory
forecasting framework that generates a diverse action space. This inherent diversity explicitly empowers the generative world model to act as a robust closed-loop planner, utilizing visual foresight to evaluate and select the optimal path. Extensive experiments across diverse robotics
datasets demonstrate that NavWM significantly advances the state-of-the-art, delivering remarkable improvements in both high-fidelity future state generation and zero-shot
navigation success.
Read the full abstract ▾
All authors Yanghong Mei, Longteng Guo, Ming-Ming Yu, Guiyu Zhao, Xingjian He, Jing Liu
Robotics
DynaWM: Dynamics-Aware Distillation with World Model and Momentum Targets for Smooth Locomotion over Continuous Stairs
Haidong Hou, Zhangguo Yu, Hengbo Qi et al. (+1 more)
Recent advances in control have enabled bipedal-wheeled
robots to traverse slopes and single-step obstacles, yet long staircase traversal remains challenging as current teacher-student frameworks suffer from weakened dynamics-aware representations and incomplete terrain geometry encoding. To bridge this gap, we propose DynaWM, a dynamics-aware representation learning framework. To enhance terrain encoding capability and enable transparent assessment, we introduce a world model as a regularizer to enforce forward-dynamics awareness, preserving comprehensive terrain geometry while facilitating hierarchical encoding visualization. To stabilize knowledge transfer, we employ a momentum target encoder to provide consistent
distillation targets, preventing dimensional collapse from non-stationary teacher updates. Evaluation of the learned representations through Principal Component Analysis (PCA) visualization and quantitative metrics reveals that our encoder hierarchically captures terrain geometry with higher terrain encoding capability, leading to enhanced terrain adaptability and motion smoothness. Experimental results in simulation and real hardware demonstrate that our method achieves superior terrain adaptability and motion smoothness, enabling bipedal-wheeled
robots to overcome diverse continuous stairs, as shown in Fig. 1.
Read the full abstract ▾
All authors Haidong Hou, Zhangguo Yu, Hengbo Qi, Jianlin Zhang
Robotics
MinInter: Minimizing Trajectory Interpolation During Data Augmentation for Imitation Learning
Qingyang Wang, Xingang Liu, Changwei Yao et al. (+4 more)
Imitation learning enables
robots to acquire complex manipulation skills from demonstrations, but its effectiveness is limited by the cost of collecting high-quality data. Trajectory-level data augmentation methods alleviate this challenge by recombining expert demonstrations under varied initial states. However, such methods typically insert interpolations or other non-expert transition segments between disjoint parts, and such non-expert segments could reduce the quality of the generated data. This paper introduces Minimizing Interpolation (MinInter), an effective trajectory selection method that, for each sampled initial configuration, chooses the source demonstration requiring the least interpolation to form a complete trajectory. By explicitly minimizing interpolations during data generation, MinInter produces higher-quality synthetic demonstrations while remaining compatible with existing data generation frameworks. Experiments on 12 manipulation tasks with 26 variants from the MimicGen
benchmark show that MinInter consistently improves both data generation success rates and policy success rates, with the largest gains on contact-rich, long-horizon and high-variance settings. Compared to the recent SkillGen framework, MinInter achieves higher policy success rates despite its conceptual simplicity, underscoring the value of interpolation minimization for data augmentation.
Read the full abstract ▾
All authors Qingyang Wang, Xingang Liu, Changwei Yao, Zikai Ouyang, Junwei Liu, Haibo Lu, Wei Zhang
Robotics · Cross-listed from cs.CV
ObsGraph: Hierarchical Observation Representation for Embodied Reasoning and Exploration
Taekbeom Lee, Youngseok Jang, Jeonghwa Heo et al. (+2 more)
Embodied reasoning and exploration are increasingly considered crucial abilities for
robots operating in complex and unfamiliar environments. To accomplish tasks in such settings, an
agent must identify and acquire the information necessary for the task through exploration. We propose ObsGraph, an observation-centric hierarchical scene graph that unifies scene representation,
retrieval, and exploration. It retains visual evidence and organizes it into room-view-object layers: rooms provide coarse semantic anchors, views preserve contextual object covisibility, and objects store fine-grained details. On top of this representation, we perform coarse-to-fine hierarchical
retrieval under a bounded budget, and crucially use
retrieval outcomes to structure the exploration candidate space–activating room-level exploration, view refinement, or frontier exploration–thereby tightly coupling representation,
retrieval, and adaptive multi-scale exploration. Experiments across
embodied reasoning and exploration
benchmarks demonstrate improved success and efficiency, highlighting the benefits of structured scene representation and more targeted information gathering driven by identified evidence gaps.
Read the full abstract ▾
All authors Taekbeom Lee, Youngseok Jang, Jeonghwa Heo, Jeongjun Choi, H. Jin Kim
Robotics
SPACE: Enabling Learning from Cross-Robot Data Toward Generalist Policies
Haeone Lee, Byeongguk Jeon, Suchae Jeong et al. (+2 more)
In
robot learning, scaling training
datasets across diverse embodiments and environments has become a dominant paradigm for learning generalizable
robot policies. These policies are commonly trained via behavior cloning to imitate actions from pre-collected demonstrations. However, since
robot actions are tied to the dynamics of the data collection
robot, different
robots may require different actions to achieve the same motion. This discrepancy hinders both policy training and deployment across diverse
robots. To address this, we propose using Cartesian state delta as a universal action representation across
robots, and introduce State Prediction and Adaptive Command Execution (SPACE) framework. SPACE handles
robot dynamics variation at three levels: across different embodiments, across hardware units of the same embodiment, and within a single
robot during operation. It consists of two components: (i) a Cartesian state delta policy that predicts
geometric end-effector displacement, and (ii) Action Adapter, which converts the predicted Cartesian state delta into
robot-specific control commands. Experiments show that SPACE substantially outperforms policies that directly predict control commands when learning from data collected across different embodiments and across hardware units of the same embodiment. SPACE also remains robust under dynamics shifts at deployment, including changes in control frequency, object weight, and controller gains. The project page is available at http://haeone.site/space-website/.
Read the full abstract ▾
All authors Haeone Lee, Byeongguk Jeon, Suchae Jeong, Jian Kim, Kimin Lee
Robotics
TurboMPC: Fast, Scalable, and Differentiable Model Predictive Control on the GPU
Gabriel Bravo-Palacios, Jianghan Zhang, Zachary Pestrikov et al. (+2 more)
Robotics increasingly relies on GPUs for parallel simulation, large-scale learning, and neural-network inference. For model predictive control (MPC) to scale with this paradigm, solvers must run efficiently on this hardware while remaining fast, differentiable, and compatible with expressive MPC formulations used in robotics. We present TurboMPC, a differentiable MPC solver that runs entirely on the GPU and supports state and control inequality constraints, implicit integrators, cross-time-coupled costs, and slack variables. TurboMPC combines sequential quadratic programming (SQP), an alternating direction method of multipliers (ADMM) inner solver, implicit differentiation, and a co-designed JAX-CUDA implementation for efficiency and ease of use. In simulation, we validate TurboMPC on constrained planning,
humanoid imitation learning, and reinforcement learning with neural-network cost function tasks, achieving up to $15\times$ and $58\times$ speedups over state-of-the-art CPU and GPU differentiable solvers, respectively. We deploy TurboMPC on a full-scale car for minimum-time racing and find that batched, GPU-accelerated tuning of MPC parameters via
Bayesian optimization yields significantly faster driving than a hand-tuned baseline. TurboMPC also scales to planning horizons of over $8000$
knot points while maintaining control of the vehicle. We open-source TurboMPC at: https://github.com/ToyotaResearchInstitute/turbompc
Read the full abstract ▾
All authors Gabriel Bravo-Palacios, Jianghan Zhang, Zachary Pestrikov, Brian Plancher, Thomas Lew
Robotics
Sim-to-Real Betting on the E-Process: Bringing "simulators" to anytime-valid confidence sequences
Yujia Chen and Bowen Weng
This note describes an integration of the sim-to-real performance estimate with betting (from Chen et al.) and the safe anytime-valid inference (from Ramdas et al.). Using the scaled simulators. The method produces efficient, reliable certificates for the mean estimate, an approach that is especially valuable in
robot performance testing. This note gives a primary, self-contained account of the construction; preliminaries of the respective methods are kept at a minimum, and one shall refer to the original works for full detail. Some synthetic examples demonstrating the proposed
algorithm can be found at https://github.com/ISUSAIL/Bet4Sim2Real-EProcess.
Read the full abstract ▾