Robotics · Cross-listed from cs.LG
Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates
Anish Diwan, Davide Tateo, Christopher E. Mower et al. (+3 more)
Inverse reinforcement learning (IRL) is typically formulated as maximizing
entropy subject to matching the distribution of expert trajectories. Classical (dual-ascent) IRL guarantees monotonic performance improvement but requires fully solving an RL problem each iteration to compute dual gradients. More recent
adversarial methods avoid this cost at the expense of stability and monotonic dual improvement, by directly optimizing the primal problem and using a discriminator to provide rewards. In this work, we bridge the gap between these approaches by enabling monotonic improvement of the reward function and policy without having to fully solve an RL problem at every iteration. Our key theoretical insight is that a trust-region-optimal policy for a reward function update can be globally optimal for a smaller update in the same direction. This smaller update allows us to explicitly optimize the dual objective while only relying on a local search around the current policy. In doing so, our approach avoids the training instabilities of
adversarial methods, offers monotonic performance improvement, and learns a reward function in the traditional sense of IRL–one that can be globally optimized to match expert demonstrations. Our proposed
algorithm, Trust Region Inverse Reinforcement Learning (TRIRL), outperforms state-of-the-art imitation learning methods across multiple challenging tasks by a factor of 2.4x in terms of aggregate inter-quartile mean, while recovering reward functions that generalize to system dynamics shifts.
Read the full abstract ▾
All authors Anish Diwan, Davide Tateo, Christopher E. Mower, Haitham Bou-Ammar, Jan Peters, Oleg Arenz
Robotics · Cross-listed from cs.LG
ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network
Qian Chen, Junqiao Zhao, Hongtu Zhou et al. (+4 more)
Long-horizon, sparse-reward tasks pose a fundamental challenge for reinforcement learning, since single-step TD learning suffers from bootstrapping error accumulation across successive Bellman updates. Actor-critic methods with action chunking address this by operating over temporally extended actions, which reduce the effective horizon, enable fast value backups, and support temporally consistent exploration. However, existing methods rely on a fixed chunk size and therefore cannot adaptively balance reactivity against temporal consistency. A large fixed chunk size reduces responsiveness to new observations, while a small one produces incoherent motions, forcing task-specific tuning of the chunk size. To address this limitation, we propose Adaptive Chunk Size Actor-Critic (ACSAC). ACSAC leverages a causal
Transformer critic to evaluate expected returns for action chunks of different sizes. At each chunk boundary, it adaptively selects the chunk size that maximizes the expected return, supporting flexible, state-dependent chunk sizes. We prove that the ACSAC Bellman operator is a $\gamma$-contraction whose unique fixed point is the action-value function of the adaptive policy. Experiments on OGBench demonstrate that ACSAC achieves strong performance on long-horizon, sparse-reward manipulation and
navigation tasks across both offline RL and offline-to-online RL settings.
Read the full abstract ▾
All authors Qian Chen, Junqiao Zhao, Hongtu Zhou, Hang Yu, Yanping Zhao, Chen Ye, Guang Chen
Robotics
Above and Below: Heterogeneous Multi-robot SLAM Across Surface and Underwater Domains
John McConnell, Armon Shariati, Paul Szenher et al. (+1 more)
Multi-
robot simultaneous localization and mapping (SLAM) is a fundamental task in multi-
robot operations.
Robots must have a common understanding of their location and that of their team members to complete coordinated actions. However, multi-
robot SLAM between Uncrewed Surface Vessels (USVs) and Autonomous
Underwater Vehicles (AUVs) has primarily been achieved through acoustic pinging between
robots to retrieve range measurements; a measurement technique requires that
robots to be in similar locations simultaneously, have an uninterrupted path for signal propagation, and may necessitate synchronized clocks. This is especially challenging in complex, cluttered maritime environments, where structures may impede signals. However, these same structures may be observable above and below the water's surface, presenting an opportunity for inter-
robot SLAM loop closure between USV and AUV data streams. This work builds upon recent research on inter-
robot SLAM loop closure between USV and AUV data, extending it to propose a centralized multi-
robot SLAM system. Each
robot performs its state estimation, and we detect loop closures between each AUV and the USV data. These inter-
robot loop closures are used to merge each
robot's state estimate into a centralized graph, yielding estimates for the whole time history of the USV and all AUVs in the system. Validation is performed using real-world perceptual data in three different environments. Results show improved errors for AUVs in the multi-
robot SLAM system compared to single-
robot SLAM over the same trajectories. To our knowledge, this is the first instance of a multi-
robot SLAM system with AUVs and USVs built on loop closures rather than acoustic distance measurements.
Read the full abstract ▾
All authors John McConnell, Armon Shariati, Paul Szenher, Yaxuan Li
Robotics
Efficient Multi-Robot Motion Planning with Precomputed Translation-Invariant Edge Bundles
Himanshu Gupta, Paul Motter, Aritra Chakrabarty et al. (+5 more)
Solving multi-
robot motion planning (MRMP) requires generating collision-free kinodynamically feasible trajectories for multiple interacting
robots. We introduce Kinodynamic Translation-Invariant Edge Bundles or KiTE-Extend, a planner-agnostic action selection mechanism for sampling-based kinodynamic motion planning. KiTE-Extend uses a library of trajectory segments computed offline to guide action selection during online planning, improving the ability of existing planners to identify feasible motion segments without altering state propagation, collision checking, or cost evaluation, and without changing their theoretical guarantees. While KiTE-Extend can modestly improve single-
agent planners, its benefits are most clear in the multi-
agent setting, where it is able to explore more effectively and significantly improve planning through the dense spatiotemporal constraints introduced by
robot-
robot interaction. Through experiments on multiple kinodynamic systems and environments, we show that KiTE-Extend reduces planning time and improves scalability across the three most common MRMP paradigms: centralized, prioritized, and conflict-based.
Read the full abstract ▾
All authors Himanshu Gupta, Paul Motter, Aritra Chakrabarty, Rishabh Sodani, Srikrishna Bangalore Raghu, Alessandro Roncone, Bradley Hayes, Zachary Sunberg
Robotics
Zero-Shot Sim-to-Real Robot Learning: A Dexterous Manipulation Study on Reactive Catching
Kejia Ren, Gaotian Wang, Andrew S. Morgan et al. (+1 more)
Dexterous manipulation is physics-intensive and highly sensitive to modeling errors and perception noise, making sim-to-real transfer prohibitively challenging. Domain randomization (DR) is commonly used to improve the robustness of learned policies for such tasks, but conventional DR randomizes one instance per episode, offering very limited exposure to the variability of real-world dynamics. To this end, we propose Domain-Randomized Instance Set (DRIS), which represents and propagates a set of randomized instances simultaneously, providing richer approximation of uncertain dynamics and enabling policies to learn actions that account for multiple possible outcomes. Supported by theoretical analysis, we show that DRIS yields more robust policies and alleviates the need for real-world fine-tuning, even with a modest number of instances (e.g., 10). We demonstrate this on a challenging reactive catching task. Unlike traditional catching setups that use end-effectors designed to mechanically stabilize the object (e.g., curved or enclosing surfaces), our system uses a flat plate that offers no passive stabilization, making the task highly sensitive to noise and requiring rapid reactive motions. The learned policies exhibit strong robustness to uncertainties and achieve reliable zero-shot sim-to-real transfer.
Read the full abstract ▾
All authors Kejia Ren, Gaotian Wang, Andrew S. Morgan, Kaiyu Hang
Robotics · Cross-listed from eess.SY
Safe Exploration for Nonlinear Processes Using Online Gaussian Process Learning
Stefano Tonini, Soroush Rastegarpour, Hamid Reza Feyzmahdavian et al. (+2 more)
This paper proposes a safe data-driven control framework for nonlinear systems with partially known dynamics. The method ensures stability and constraint satisfaction during online learning, assuming only a stabilizable linear approximation of the process is available. Unmodeled nonlinear dynamics are captured by a Gaussian process residual learned in real time. Safety is enforced through a probabilistic control-invariant set derived from Lyapunov theory, guaranteeing high-probability stability. A convex quadratic program computes control inputs that maximize information gain while respecting probabilistic safety constraints. The framework provides finite-sample safety guarantees and allows adaptive expansion of the invariant set as uncertainty decreases. Numerical results validate the approach, demonstrating safe and informative exploration under model uncertainty: the safe set expands by about 30% while the Gaussian process root-mean-square error drops from 1.11 to 0.03.
Read the full abstract ▾
All authors Stefano Tonini, Soroush Rastegarpour, Hamid Reza Feyzmahdavian, Nicola Bastianello, Karl Henrik Johansson
Robotics
MVB-Grasp: Minimum-Volume-Box Filtering of Diffusion-based Grasps for Frontal Manipulation
Bibek Poudel, Abdul Basit and Muhammad Shafique
State-of-the-art 6-DoF grasp generators excel on tabletop
benchmarks with overhead cameras but struggle in frontal grasping scenarios on low-cost manipulators with constrained workspaces, where kinematic limits and approach-direction constraints cause high failure rates. We address this challenge for the Unitree Z1 arm by proposing MVB-Grasp, a novel grasping stack that injects a Minimum Volume Bounding Box (MVBB)
geometric prior into
diffusion-based grasp generation to dramatically improve success rates in frontal, workspace-constrained settings. Our key scientific contributions are threefold: (i) an MVBB-based
geometric filter that exploits oriented bounding-box face normals to reject grasps approaching through the table or misaligned with accessible object faces in O(N) time; (ii) a combined re-scoring function that blends learned discriminator scores with face-
alignment geometry {\alpha}=0.85, specifically calibrated for the Z1's frontal workspace and kinematic constraints; and (iii) a systematic MuJoCo evaluation protocol measuring grasp success across object types, distances, lateral positions, and pitch orientations to validate embodiment-specific performance. We implement MVB-Grasp on a Unitree Z1 arm with an Intel RealSense D405 camera, integrating YOLOv8 object detection, GraspGen for candidate generation, Principal Component Analysis (PCA)-based MVBB fitting, and inverse-kinematics trajectory planning. Experiments across 81 MuJoCo episodes (cylinder, asymmetric box, waterbottle) demonstrate that MVB-Grasp achieves 59.3% success versus 24.7% for vanilla GraspGen, a 2.4x improvement, by filtering geometrically infeasible candidates and prioritizing face-aligned grasps suited to the Z1's frontal approach constraints. Real-world trials confirm that the MVBB prior substantially improves grasp reliability on constrained, low-cost manipulators without requiring model retraining.
Read the full abstract ▾
Robotics
Towards Generative Predictive Display for Vision-Based Teleoperation: A Zero-Shot Benchmark of Off-the-Shelf Video Models
Aws Khalil and Jaerock Kwon
Teleoperation systems are fundamentally limited by communication latency, which degrades situational awareness and control performance. Predictive display aims to mitigate this limitation by presenting an estimate of the current visual state rather than delayed observations. While recent advances in generative
video models enable high-quality
video synthesis, their suitability for latency-sensitive predictive display remains unclear. This paper presents a zero-shot
benchmark of off-the-shelf generative
video models for short-horizon predictive display, without task-specific fine-tuning. We formulate the problem as rollout-based future frame prediction and develop a unified benchmarking pipeline using simulated driving data from the CARLA simulator. Five publicly released
video models spanning
transformer-based and
diffusion-based families are evaluated across two resolutions and two conditioning regimes (multi-frame and single-frame). Performance is assessed using prediction accuracy (mean absolute difference), per-rollout latency, peak GPU
memory usage, and temporal error
evolution across the prediction horizon. On this zero-shot
benchmark, no tested model simultaneously achieves low rollout error, non-divergent per-step error behavior, and real-time inference at the source frame rate. Increasing model scale or resolution yields limited and, in some cases, inverted improvements. These findings highlight a gap between general-purpose generative
video synthesis and the requirements of predictive display in teleoperation, suggesting that practical deployment will require either explicit short-horizon temporal supervision, in-domain adaptation, or aggressive inference optimization rather than direct application of off-the-shelf models. Code, configurations, and qualitative results are released on the project page: https://bimilab.github.io/paper-GenPD
Read the full abstract ▾
Robotics
SACK : Safe Active Continual Koopman Learning for Uncertain Systems with Contractive Guarantees
Chandan Kumar Sah, Rajpal Singh and Jishnu Keshavan
Koopman operator theory provides a powerful framework for representing nonlinear dynamics through a linear operator acting on lifted observables, enabling the use of linear control techniques for nonlinear systems. However, Koopman models are typically learned from data and often degrade in performance under model uncertainty and distributional shifts between training and deployment. Although several works have explored online adaptation to address this issue, many rely on neural network-based updates that introduce significant computational overhead and lack formal safety guarantees, limiting their suitability for real-time and safety-critical robotic applications. In this work, we propose SACK, a unified framework for continual adaptive Koopman learning that enables safe and efficient online refinement of learned models during task execution. A Koopman model is first learned offline and subsequently refined online through a contractive adaptation law, which provides theoretical convergence guarantees under distributional shifts and model uncertainty. To improve data efficiency and accelerate model refinement, the adaptation mechanism is integrated with an active learning strategy that drives the system to collect informative data while accomplishing task objectives. The resulting control problem is formulated as a nonconvex optimization problem incorporating both active learning objectives and safety constraints. We further derive theoretical bounds on model approximation error and show how these bounds can be incorporated within a robust Model Predictive Control (MPC) framework to provide formal safety guarantees. To reduce conservatism in practice, we also introduce a conformal prediction-based tightening mechanism that calibrates safety margins online from observed residuals. Extensive simulation and experimental studies demonstrate efficacy of the proposed scheme.
Read the full abstract ▾
Robotics
ORICF – Open Robotics Inference and Control Framework
Andrés Meseguer Valenzuela and Luís Miguel Bartolín Arnau
Recent advances in artificial intelligence (AI) have enabled effective perception and
language models for
robots, but their deployment remains computationally expensive, increasing latency and energy use. This work presents the Open Robotics Inference and Control Framework (ORICF), a modular, declarative, and model-agnostic platform for composing
multimodal robotic inference pipelines. ORICF integrates input/output (I/O) adapters, pluggable inference back ends, and post-processing logic, while lightweight YAML specifications allow models, hardware targets, and data channels to be changed without code modification. The framework also supports edge offloading, i.e., executing inference on nearby external computers instead of onboard the
robot. ORICF is evaluated on a mobile
robot that answers spoken queries about people detected in its camera stream by combining automatic
speech recognition (ASR), a large
language model (LLM), and a convolutional neural network (CNN) detector through
Robot Operating System 2 (ROS2). Compared with onboard execution, ORICF-based edge deployment reduces
robot-side compute utilization by up to 83.16% and estimated energy consumption by 65.8%, while preserving modularity and reproducibility.
Read the full abstract ▾
Robotics
Minimizing Worst-Case Weighted Latency for Multi-Robot Persistent Monitoring: Theory and RL-Based Solutions
Weizhen Wang, Ziheng Wang, Jianping He et al. (+2 more)
We study multi-
robot persistent monitoring on weighted graphs, where node weights encode monitoring priorities and edge weights encode travel distances. The goal is to design joint
robot trajectories that minimize the worst-case weighted latency across all nodes over an infinite time horizon. The widely adopted worst-case latency objective evaluates team performance over the entire time horizon and therefore may fail to distinguish strategies with poor transient behavior but strong asymptotic performance. To address this limitation, we propose a family of tail-performance objectives that generalize the standard objective and study the resulting functional optimization problems. We establish several key theoretical properties, including the existence of optimal strategies, relationships among the proposed objectives and their corresponding optimization problems, approximation by periodic solutions to arbitrary accuracy, and reductions to event-driven decision models with discretized waiting times. Building on these results, we construct an equivalent event-driven Markov decision process (MDP), called the Tail Worst-case Latency-Optimizing Markov Decision Process (TWLO-MDP), which reformulates the tail-performance objective as a standard average-reward criterion. We then develop reinforcement-learning-based solution methods for the TWLO-MDP and introduce the multi-
robot monitoring
benchmark (M2Bench), a unified platform that supports the evaluation and comparison of heuristic and learning-based monitoring
algorithms. Experiments on synthetic and realistic monitoring scenarios show that our methods effectively reduce the worst-case weighted latency and outperform representative baselines.
Read the full abstract ▾
All authors Weizhen Wang, Ziheng Wang, Jianping He, Xinping Guan, Xiaoming Duan
Robotics
SABER: A Scalable Action-Based Embodied Dataset for Real-World VLA Adaptation
Narsimha Menga, Parikshit Sakurikar, Amirreza Rouhi et al. (+6 more)
Robotic deployment in real-world environments depends on rich, domain-specific action data as much as on strong model architecture. General-purpose
robot foundation models show modest performance in complex unseen tasks such as manipulation in a retail domain when applied out of the box. The root cause is a data gap: retail environments are structurally absent from general
robot pretraining distributions, and the path to filling that gap through teleoperation is prohibitively expensive, logistically constrained, and difficult to scale. We introduce SABER, a high-fidelity retail robotics action
dataset built from over 100 hours of natural in-store capture across multiple real grocery environments.
Egocentric footage from head-mounted cameras records fine-grained hand activity at the point of interaction, while exocentric 360-degree scene footage from DreamVu's ALIA camera simultaneously observes all actors and activities across the entire space. This combination yields a uniquely complete picture of human retail behavior: dexterous hand activity, whole-body motion, and scene dynamics, all captured without staging, scripting, or teleoperation overhead. The SABER corpus contains 44.8K training samples across three action representation streams: 25K latent action sequences via LAPA-style encoding, 18.6K dexterous hand-pose trajectories retargeted to
robot joint space, and 1.2K whole-body synchronized motion sequences retargeted to a
humanoid embodiment. When applied to GR00T N1.6 via a shared-backbone multi-task post-training recipe, SABER yields a mean success rate of 29.3% across ten retail manipulation tasks – more than 2.19x over fine-tuning baselines (13.4%). SABER demonstrates that the path to capable retail
robots runs through better data, which can be collected today, at scale, without a
robot in the loop. The
dataset and code are available at https://dreamvu.ai/saber
Read the full abstract ▾
All authors Narsimha Menga, Parikshit Sakurikar, Amirreza Rouhi, Satya Sai Reddy, Anirudh Govil, Sri Harsha Chittajallu, Rajat Aggarwal, Anoop Namboodiri, Sashi Reddi
Robotics · Cross-listed from cs.NE
Neuromorphic Reinforcement Learning for Quadruped Locomotion Control on Uneven Terrain
Zhuangyu Han and Abhronil Sengupta
Reinforcement learning (RL) has enabled robust quadruped locomotion over complex terrain, but most learned controllers are trained offline with backpropagation in massively parallel simulation and deployed as fixed policies, limiting adaptation to terrain variation, payload changes, actuator wear, and other real-world conditions under onboard power constraints. Local learning provides a potential path toward energy-aware on-
robot adaptation by replacing global backpropagation graphs with updates driven by local neural states, making the learning rule more compatible with neuromorphic and in-
memory computing substrates. This work proposes an
equilibrium-propagation (EP)-based proximal policy optimization (PPO) framework for uneven-terrain quadruped locomotion. The controller combines a bio-inspired central pattern generator (CPG) policy with a residual postural adjustment policy, while replacing conventional backpropagation-trained policy and value networks with EP-enabled local learning. To train
stochastic continuous-control policies with EP, we derive an EP-compatible PPO output-nudging signal and introduce a two-sided ratio clipping mechanism that stabilizes policy updates during relaxation. Experiments on a 12-DoF A1 quadruped show that the proposed controller achieves stable policy convergence in a two-stage uneven terrain locomotion task. Its locomotion performance is comparable to a backpropagation-trained PPO baseline in success rate, velocity tracking, actuator power, and body stability, while improving GPU
memory efficiency by 4.3\(\times\) compared with backpropagation through time (BPTT). These results suggest that local
equilibrium-based learning can support high-dimensional
embodied locomotion and provide an algorithmic foundation for low-power on-
robot adaptation and fine-tuning.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
DeformMaster: An Interactive Physics-Neural World Model for Deformable Objects from Videos
Can Li, Zhoujian Li, Ren Li et al. (+4 more)
World models for deformable objects should recover not only geometry and appearance, but also underlying physical dynamics, interaction grounding, and material behavior. Learning such a model from real
videos is challenging because deformable linear, planar, and volumetric objects evolve under high-dimensional deformation, noisy interactions, and complex material response. The model must therefore infer a physical state from visual observations, roll it forward under new interactions, and render the resulting dynamics with high visual fidelity. We present DeformMaster, a
video-derived interactive physics-neural world model that turns real interaction
videos into an online interactive model of deformable objects within a unified dynamics-and-appearance framework. DeformMaster preserves structured physical rollout while using a neural residual to compensate for unmodeled effects, grounds sparse hand motion as distributed compliant actuator for hand-continuum interaction, represents material response with spatially varying constitutive experts, and drives high-fidelity 4D appearance from the predicted physical
evolution. Experiments on real-world deformable-object sequences demonstrate DeformMaster's ability to roll out future dynamics and render dynamic appearance, outperforming state-of-the-art baselines while supporting novel action rollout, material-parameter variation, and dynamic novel-view synthesis. Project page: https://can-lee.github.io/deformmaster-web/
Read the full abstract ▾
All authors Can Li, Zhoujian Li, Ren Li, Jie Gu, Lei Lei, Jingmin Chen, Lei Sun
Robotics · Cross-listed from cs.CV
PhysHanDI: Physics-Based Reconstruction of Hand-Deformable Object Interactions
Jihyun Lee, Changmin Lee, Donghwan Kim et al. (+1 more)
While existing methods for reconstructing hand-object interactions have made impressive progress, they either focus on rigid or part-wise rigid objects-limiting their ability to model real-world objects (e.g., cloth, stuffed animals) that exhibit highly non-rigid deformations-or model deformable objects without full 3D hand reconstruction. To bridge this gap, we present PhysHanDI (Physics-based Reconstruction of Hand and Deformable Object Interactions), a framework that enables full 3D reconstruction of both interacting hands and non-rigid objects. Our key idea is to physically simulate object deformations driven by forces induced from densely reconstructed 3D hand motions, ensuring that the reconstructed object dynamics are both physically plausible and coherent with the interacting hand movements. Furthermore, we demonstrate that such simulation of object deformations can, in turn, refine and improve hand reconstruction via inverse physics. In experiments, PhysHanDI outperforms the state-of-the-art baseline across reconstruction and future prediction.
Read the full abstract ▾
All authors Jihyun Lee, Changmin Lee, Donghwan Kim, Tae-Kyun Kim
Robotics
Drift is a Sampling Error: SNR-Aware Power Distributions for Long-Horizon Robotic Planning
Kewei Chen, Yayu Long and Mingsheng Shang
Despite rapid progress in
Vision-
Language-Action (VLA) models for robotic control, instruction drift remains a persistent failure mode in long-horizon tasks. This paper reconceptualizes this phenomenon, positing that instruction drift is fundamentally a systematic sampling error: local greedy sampling is prone to collapsing into "Negative Pivotal Windows"–irreversible local optima with high local probability that sever global success pathways. To address this, we propose Context-Aware Power Sampling (CAPS), a training-free inference-time computation framework. CAPS leverages power distributions to sharpen global trajectory probabilities, enabling lookahead search over the model's conditional generative trajectory distribution. Furthermore, we introduce a metacognitive control mechanism based on Signal-to-Noise Ratio (SNR). This mechanism triggers adaptive MCMC search solely when drift risk is detected, enabling a dynamic transition from "intuitive fast thinking" to "rational slow search." Experiments on RoboTwin, Simpler-WindowX, and Libero-long
benchmarks show that CAPS achieves substantial improvements over strong baselines, including OpenVLA and TACO, without parameter updates. These results support the effectiveness of adaptive inference-time computation for improving long-horizon robustness in
embodied control.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
QueST: Persistent Queries as Semantic Monitors for Drift Suppression in Long-Horizon Tracking
Mayank Anand, Mohammad Saqlain, Kyan Mahajan et al. (+3 more)
Tracking points in
videos is typically formulated as frame-to-frame correspondence, where each point is matched locally to the next frame. While this works over short horizons, errors accumulate under articulation, occlusion, and viewpoint change, leading to silent semantic drift that existing trackers cannot detect or correct. In this work, we revisit long-horizon tracking from a monitoring perspective and introduce QueST, a monitoring-by-design framework that treats interaction-relevant entities as persistent semantic queries rather than transient point tracks. Instead of local propagation, each query attends globally over spatio-temporal
video features at every time-step, providing a stable semantic anchor across time. We further constrain query trajectories with lightweight 3D physical grounding, using
geometric plausibility to suppress unbounded drift under occlusion. We evaluate QueST on long-horizon articulated sequences from PartNet-Mobility in SAPIEN and compare against RAFT-3D, CoTracker, and TAP-Net. QueST substantially reduces terminal drift achieving a 67.7% Absolute Point Error (APE) improvement over TAP-Net while better preserving identity over extended horizons. Our results show that
embedding semantic monitoring directly into perception enables more reliable long-horizon tracking under distribution shift.
Read the full abstract ▾
All authors Mayank Anand, Mohammad Saqlain, Kyan Mahajan, Priya Shukla, Gora Chand Nandi, Andrew Melnik
Robotics
LASSA Architecture-Based Autonomous Fault-Tolerant Control of Unmanned Underwater Vehicles
Hong Chen, Zixiang Tang, Yuanbao Chen et al. (+1 more)
Unmanned
underwater vehicles (UUVs) operate persistently in communication-constrained environments, thus requiring high-level autonomous fault-tolerant control under faulty operating conditions. Existing approaches rely heavily on predefined hard-coded rules and struggle to achieve effective fault-tolerant control against unforeseen faults. Although large
language models (LLMs) possess powerful cognitive and
reasoning capabilities, their inherent
hallucinations remain a major obstacle to their application in UUV control systems. This paper proposes an intelligent control method based on the LASSA (LLM-based
Agent with Solver, Sensor and Actuator) architecture. Within this architecture, an LLM identifies unknown faults and accomplishes task replanning via autonomous
reasoning without hard-coded rules; the intelligent
agent undertakes perception, scheduling and decision evaluation; the solver verifies physical boundary feasibility constraints prior to command transmission to the actuators. This architecture suppresses physically infeasible LLM
hallucinations and ensures interpretable, verifiable decision-making. Moreover, it enables fast-slow dual closed-loop collaborative control, where the slow loop undertakes high-level dynamic decision-making and the fast loop guarantees high-frequency real-time control, simultaneously balancing decision intelligence and control timeliness. Lake experiments under normal and lower-rudder-fault conditions show that the framework detects trajectory tracking abnormalities, replans the route by adjusting the turning radius from 4m to 12m and reducing speed from 2kn to 1kn, passes all three solver constraints on the first invocation, and guides the UUV to complete the full mission; under normal conditions no false fault alarms are raised throughout the run.
Read the full abstract ▾
All authors Hong Chen, Zixiang Tang, Yuanbao Chen, Yu Liu
Robotics
High Precision Hydraulic Excavator Control for Heavy-Duty Grading
Lennart Werner, Pol Eyschen, Sean Costello et al. (+2 more)
High-precision heavy-duty grading is a common step in earthworks, traditionally carried out manually by skilled operators. Removing a significant amount of material while achieving a high-precision surface requires substantial machine-specific experience. Different hydraulic architectures react differently to operator inputs and soil interaction forces, which makes generalizable controllers challenging. In this paper, we present an autonomous controller that achieves high-precision grading at expert-operator speed on Load Sensing and Negative Flow Control machines alike. We split our controller into two parts: (1) a hydraulic-aware low-level loop that is hydraulic architecture-specific and (2) a path-tracking layer that coordinates joint motions and responses. Through a calibration process, our technique is applicable to load-sensing and negative-flow-control machinery. To showcase its versatility, we
benchmark our approach on two excavators with different hydraulics and compare it against a commercial state-of-the-art solution. Our technique (RMSE 1.8~cm) outperforms the commercial solution (RMSE 4.7~cm) in precision by a factor of 2.6 and improves machine usage by leveraging the maximum function pressure, as opposed to commercial solutions that stall prematurely.
Read the full abstract ▾
All authors Lennart Werner, Pol Eyschen, Sean Costello, Andrei Cramariuc, Marco Hutter
Robotics
Beyond Isolation: A Unified Benchmark for General-Purpose Navigation
Samson Sun, Tianyi Yang, Tengyue Wang et al. (+6 more)
The pursuit of general-purpose
embodied agents is hindered by fragmented evaluation protocols that isolate
navigation skills and fixate on specific
robot morphologies, failing to reflect real-world scenarios where
agents must orchestrate diverse behaviors across varying embodiments. To bridge this gap, we introduce OmniNavBench, a
benchmark for cross-skill coordination and cross-embodiment generalization. OmniNavBench introduces three paradigm shifts: (1) Compositional Complexity. We propose composite instructions that interleave sub-tasks from 6 categories (PointNav, VLN, ObjectNav, SocialNav, Human Following and EQA), compelling
agents to transition between exploration, interaction, and social compliance within a single episode. (2) Morphological Universality and Sensor Flexibility. We present a simulation platform that breaks the reliance on single-morphology evaluation, enabling generalization tests across
humanoid, quadrupedal, and wheeled
robots, with a modular sensor interface and 170 environments blending synthetic assets with real-world scans. (3) Demonstrations Quality. Moving beyond shortest-path
algorithms, we curate 1779 expert trajectories via human teleoperation, capturing behavioral nuances such as exploratory glance and anticipatory avoidance. Extensive evaluations demonstrate that current methods, despite their claimed unified design, struggle with the complex, interleaved nature of general-purpose
navigation. This exposes a critical disparity between existing capabilities and real-world deployment demands, underscoring OmniNavBench as a testbed for the next generation of generalist navigators.
Dataset, code, and leaderboard are available at http://omninavbench.cloud-ip.cc.
Read the full abstract ▾
All authors Samson Sun, Tianyi Yang, Tengyue Wang, Yikai Xue, Zhengjie Xu, Lingming Zhang, Qichen Zhang, Chao Liang, Zhipeng Zhang
Robotics · Cross-listed from cs.CV
MAG-VLAQ: Multi-modal Aerial-Ground Query Aggregation for Cross-View Place Recognition
Zhengyi Xu, Yuhang Ming, Zhihao Zhan et al. (+3 more)
Multi-modal cross-view place recognition remains a fundamental challenge in computer
vision and robotics due to the severe viewpoint, modality, and spatial-structure discrepancies between ground observations and aerial references. To address this challenge, we present MAG-VLAQ, a foundation-model-enhanced query aggregation framework for multi-modal aerial-ground cross-view place recognition. Specifically, our approach leverages pre-trained foundation models to extract dense visual
tokens from both ground and aerial images, as well as expressive
geometric tokens from ground
LiDAR observations. These heterogeneous
tokens are then projected into a shared
embedding space for cross-modal
alignment and fusion. As our main contribution, we propose ODE-conditioned VLAQ, which tightly couples neural ordinary differential equations (ODE)-based RGB-
LiDAR fusion with vectors of locally aggregated queries (VLAQ). In this design, the VLAQ query centers are dynamically adapted according to the fused multi-modal state. This mechanism allows the final global descriptor to preserve globally learned
retrieval prototypes while remaining responsive to scene-specific visual and
geometric evidence, significantly improving aerial-ground matching. Extensive experiments on KITTI360-AG and nuScenes-AG validate the effectiveness of our proposed MAG-VLAQ. Notably, on KITTI360-AG, our MAG-VLAQ nearly doubles the state-of-the-art performance, achieving 61.1 Recall@1 in the satellite setting, compared with 34.5 from the closest competing approach.
Read the full abstract ▾
All authors Zhengyi Xu, Yuhang Ming, Zhihao Zhan, Hanyu Zhu, Javier Civera, Wanzeng Kong
Robotics
RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models
Weijia Liufu, Xiaoyu Guo, Ruiyi Chen et al. (+16 more)
Vision-
Language-Action (VLA) models remain brittle in long-horizon, contact-rich manipulation because success-only imitation provides little supervision for execution drift, while failed rollouts are often discarded. We introduce RePO-VLA, a recovery-driven policy optimization framework that assigns distinct roles to success, recovery, and failure trajectories. RePO-VLA first applies Recovery-Aware Initialization (RAI), slicing each corrective suffix into a new episode so corrective actions depend on the current adverse state rather than the preceding failure. It then learns a Progress-Aware Semantic Value Function (PAS-VF), aligning spatiotemporal trajectory features with instructions and successful references. The resulting labels salvage useful failure prefixes via reliability decay, while low-value labels mark drift and terminal breakdowns, teaching differences among nominal, failed, and corrective actions. The data engine turns adverse states into planner-generated or human-collected corrective rollouts, teaching recovery to the success
manifold. Value-Conditioned Refinement (VCR) trains the policy to prefer high-progress actions. At deployment, a fixed high-value
token biases actions toward the learned success
manifold without online failure detectors or heuristic retries. We introduce the Failure-Recovery
Benchmark (FRBench), with standardized error injection and recovery-focused evaluation. Across simulated and real-world bimanual tasks, RePO-VLA improves robustness, raising
adversarial success from 20% to 75% on average and up to 80% in scaled real-world trials. Code and
video results are available at https://1018weijia.github.io/RePO-VLA-Web/.
Read the full abstract ▾
All authors Weijia Liufu, Xiaoyu Guo, Ruiyi Chen, Jingzhi Liu, Kaidong Zhang, Xiwen Liang, Jianqi Lin, Dawei Sun, Yuze Wang, Rongtao Xu, Bingqian Lin, Bowen Yang, Tongtong Cao, Bowen Peng, Dongyu Zhang, Guangrun Wang, Min Wang, Liang Lin, Xiaodan Liang
Robotics · Cross-listed from cs.AI
NEXUS: Learning Safety Constraints and Action Models for Safe and Robust Embodied Planning
Tiehan Cui, Peipei Liu, Yanxu Mao et al. (+3 more)
Large
language models (LLMs) enable
embodied agents to plan from naturallanguage instructions, yet their actions may be physically infeasible or hazardous. Existing neuro-symbolic learners primarily refine execution models, while formal safety methods acquire or enforce safety constraints. This separation limits the reuse of both execution corrections and safety knowledge across tasks. We present NEXUS, a modular neuro-symbolic framework that jointly maintains a PDDL domain model and an LTL safety specification. Execution failures are used to repair action schemas, whereas detected hazards are converted into reusable LTL constraints through hierarchical semantic and translation sampling with equivalence consensus. Accumulated constraints are enforced through LTL progression during symbolic search; comparing constrained and unconstrained search outcomes distinguishes physical unreachability from safety-induced refusal. Experiments on SafeAgentBench evaluation set show that NEXUS achieves the best overall trade-off between safetask completion and unsafe-task refusal, while maintaining low violation rates under jailbreak attacks. Ablations confirm the complementary benefits of PDDLdomain refinement and LTL-constraint accumulation.
Read the full abstract ▾
All authors Tiehan Cui, Peipei Liu, Yanxu Mao, Gaosheng Wang, Mingzhe Xing, Datao You
Robotics
Safety-Critical LiDAR-Inertial Odometry with On-Manifold Deterministic Protection Level
Yueqi Zhu, Yan Pan, Chufan Rui et al. (+3 more)
In safety-critical scenarios, the protection level of the autonomous
navigation system is crucial for enabling mobile
robots to perform safe tasks. However, existing studies on probabilistic
navigation systems for
robots usually perform offline accuracy evaluations using limited
datasets and assume that the results can be applied to unknown real-world environments. As a result, current autonomous mobile
robots often lack protection levels for online safety assessment. To fill this gap, we propose a safety-critical
LiDAR-inertial odometry (LIO) that provides deterministic protection levels based on on-
manifold deterministic state estimation. By adopting the unknown but bounded assumption, we derive a neat closed-form relationship between point cloud noise and the uncertainty of the estimation from the iterated closest point
algorithm. Using this relationship, we design an on-
manifold ellipsoidal set-membership filter and implement it within the LIO system. Leveraging the properties of the set-membership filter, our system offers the feasible sets of the estimated locations as the deterministic protection levels, serving as safety references for the
robots' downstream autonomous operations. The experimental results show that our system can provide effective deterministic online safety references for diverse
robots in various environments.
Read the full abstract ▾
All authors Yueqi Zhu, Yan Pan, Chufan Rui, Jiasheng Luo, Shihua Li, Bo Zhou
Robotics
Mismatch-Aware Adaptive Constraint Tightening for Bicycle-Model Trajectory Optimization
Lingxue Lyu and Zihui Liu
Trajectory optimization for autonomous vehicles usually relies on the kinematic bicycle model because of its computational simplicity. However, when the planned trajectory is executed under the true vehicle dynamics, which include lateral slip, tire stiffness and yaw-lateral coupling, safety constraints can be violated owing to the model mismatch. In this paper, we make three theoretical contributions. First, we derive a characteristic speed $v_c=\sqrt{C_\alpha L/M}$ which separates two different mismatch regimes: below $v_c$ the dynamic bicycle initially oversteers inward (safe); above $v_c$ it understeers outward (safety-critical). Second, we prove that the peak outward deviation $\varepsilon^*$ follows a $T^2$ horizon scaling whose coefficient transitions between a transient bound $\frac{1}{2}(v^2-v_c^2)\kappa$ and a steady-state bound. Third, we obtain a simulation-free analytical coefficient $a_2^{\mathrm{anal}}=\frac{1}{2}(1-v_c^2/v_{\max}^2)T^2$ that is computable from vehicle parameters and the planning horizon alone. Putting these together, we propose Mismatch-Aware Adaptive Constraint Tightening (MACT), $\epsilon(v,\kappa)=a_2 v^2|\kappa|$, which replaces a fixed worst-case margin by a state-dependent one that is large at high speed/
curvature but nearly zero on gentle paths. Eight numerical experiments confirm the scaling laws. MACT reaches 100% safety with 84% less wasted margin than a fixed-margin baseline on the 2-DOF vehicle, extends to a nonlinear leaning bicycle, and in a closed-loop direct-shooting MPC comparison it cuts the applied margin by 34% compared with tube MPC while keeping the same safety.
Read the full abstract ▾
Robotics
PECMAN: Perception-enabled Collaborative Multi-Agent Navigation in Unknown Environments
Tianchonghui Fang, Shaunak Roy and Shalabh Gupta
Most path planners assume fully known, static environments, assumptions that fail when
robots navigate in dynamic and partially observable environments. SMART-3D addresses these issues by real-time replanning, where it morphs the underlying RRT* tree whenever new obstacles or structures are discovered in the environment. Instead of rebuilding the tree entirely from scratch, SMART-3D prunes invalid nodes and edges and subsequently repairs the disjoint subtrees at hot-nodes to find a new path, thus providing high computational efficiency for real-time adaptability. We extend SMART-3D to perception-enabled collaborative multi-
agent navigation (PECMAN) in unknown environments. PECMAN is built upon distributed tree morphing and shared perception strategies, where each
agent reacts to environmental changes and morphs its respective tree to replan its path, while simultaneously broadcasting newly discovered structures to other
agents, thus enabling them to proactively replan even in areas that have not yet been explored by them. This approach reduces redundant reactions and unnecessary replannings of the
agents due to improved situational awareness. The performance of PECMAN was evaluated by 28,000 multi-
agent simulations on seven 2D scenarios with different case studies. The results show that PECMAN achieves up to 52% reduction in the team-completion time, while maintaining near 100% success rates. Finally, PECMAN was tested by real experiments on two autonomous
robots in a building environment.
Read the full abstract ▾