Robotics
Hierarchical Floorplan-Guided Vision-Language Exploration for Embodied Question Answering
Albert Gassol Puigjaner and Kostas Alexis
Embodied Question Answering (EQA) requires an
agent to explore a previously unseen environment, gather relevant information, and answer questions about the scene. Recent approaches leverage
Vision-
Language Models (VLMs) together with semantic maps or scene graphs to guide exploration. However, exploration is typically driven only by local observations, while structural priors about the environment remain largely unused. We propose HFLEX-EQA, a hierarchical EQA framework that combines online scene graph construction, VLM- based planning, semantic frontier exploration, and floorplan priors. The system incrementally builds a hierarchical scene graph and an open-vocabulary occupancy map from RGB-D observations, enabling a VLM to jointly reason over the scene graph, task-relevant visual observations, exploration history, and an estimated topological floorplan. Furthermore, we introduce a room-discovery strategy that leverages the floorplan and open-vocabulary frontier semantics to guide exploration toward semantically relevant yet currently unobserved room types. We evaluate HFLEX-EQA on the OpenEQA and ExploreEQA
benchmarks and demonstrate deployment on a quadruped
robot in real indoor environments. Our results demonstrate the benefit of combining VLM-based hierarchical planning with structural floorplan priors for the EQA task.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Leveraging Vision-Based Point Cloud Map Priors for Camera-Based 3D Object Detection and Online Vectorized HD Mapping
Markus Käppeler, Rohit Mohan and Abhinav Valada
Camera-based 3D object detection and online vectorized HD mapping provide compact scene representations for autonomous driving, but both depend on accurate metric geometry and remain limited by depth ambiguity. Over long-term deployment, observations from repeated traversals can be accumulated into persistent point cloud priors that provide
geometric context beyond the current observations. Existing explicit point cloud prior approaches, however, rely on
LiDAR-based map construction and therefore require expensive 3D ranging sensors. We propose a framework that constructs a static point cloud prior map from previous camera traversals using Pi3X and augments each point with DINOv3 features. At runtime, a local prior patch is retrieved using global localization, encoded with a sparse voxel backbone, and fused in bird's-eye view (BEV) with lifted multi-view camera features. Task-specific sparse
transformer heads then predict 3D objects and vectorized map elements from the fused representation. On Argoverse 2, the
vision-based prior improves a strong baseline from 0.287 to 0.299 CDS and from 0.669 to 0.750 vectorized mapping mAP. Ablations show that semantic DINOv3 features are particularly important for vectorized mapping. These results demonstrate that
vision-built
geometric-semantic priors provide an effective form of long-term scene
memory for camera-based perception, improving both tasks without
LiDAR for prior-map construction or online inference.
Read the full abstract ▾
Robotics
ArborSplat: Online Semantic Gaussian Splatting SLAM for Orchards
Alessandro Masini, Matteo Frosi, Mirko Usuelli et al. (+1 more)
Orchard
robots need maps that preserve small but semantically important structures such as trunks, trellises, and fruit. 3D Gaussian Splatting (3DGS) SLAM achieves high photometric fidelity. However, its optimization remains appearance-driven, and transferring image semantics to 3D points is unreliable for thin structures, whose pixels may receive depth from background surfaces. We present ArborSplat, an online semantic 3DGS SLAM system that tracks with
LiDAR odometry and optimizes semantics directly on the Gaussian map, constrained by class-specific height bands above a ground plane fitted to each keyframe's stereo point cloud, and fuses multi-view evidence into a semantic point cloud online while rejecting labels inconsistent with the local ground surface or with monocular depth. Class-constrained refinement reserves Gaussian capacity for underrepresented structures and, under reduced budgets, increases training-view accuracy on tree classes. We evaluate the approach on apple and pear orchards during dormancy, flowering, and harvesting. On full routes, it keeps ATE below 0.5 m on all 12 traversals. On shared 301-frame segments, it exceeds SGS-SLAM and GS3LAM by 0.23 to 0.50 training-view and 0.15 to 0.36 held-out mIoU while running 1.7 to 7.5 times faster, whereas SemGauss-SLAM runs out of GPU
memory on all six.
Read the full abstract ▾
All authors Alessandro Masini, Matteo Frosi, Mirko Usuelli, Matteo Matteucci
Robotics
TriWorldBench: A Tri-View Consistency Perspective on Embodied World Models
Xuanyi Liu, Haofeng Wang, Ruiqi Li et al. (+9 more)
Embodied world models predict the outcomes of
robot actions to support learning and planning. For
robots equipped with head and wrist cameras, this requires complementary views: the head view captures the overall task, while wrist views reveal local gripper-object interactions. However, evaluating these views independently cannot determine whether they describe the same action and object state. We introduce TRIWORLDBENCH, a
benchmark for evaluating embodied world models through synchronized head, left-wrist, and right-wrist
videos. It contains 500 episodes across 50 bimanual manipulation tasks and uses 19 metrics to assess tri-view consistency, task
alignment, physical and 3D coherence, motion quality, temporal consistency, and visual quality. By combining cross-view checks with measurements tailored to each camera, the
benchmark evaluates whether plausible individual
videos also form a consistent prediction of the intended task. We summarize overall performance with TWB-Score and retain per-view results to identify where predictions fail. This extends world-model evaluation beyond single-view visual quality. Code, data, and metric definitions are available at https://github.com/TriWorldBench/TriWorldBench.
Read the full abstract ▾
All authors Xuanyi Liu, Haofeng Wang, Ruiqi Li, Danni Yu, Rui Wan, Ruixu Zhang, Siyu Tao, Xue Yang, Shaofeng Zhang, Zicheng Zhang, Jiaqi Zhang, Siwei Ma
Robotics
SafeLoop: Risk-Aware Rollback for Vision-Language-Action Manipulation
Zeyu Lou, Tianran Zhang, Xinquan Yue et al. (+2 more)
Recent
vision-
language-action (VLA) models are promising for general-purpose manipulation, but long-horizon execution remains fragile. Small state-estimation or control errors can lead to irreversible failures (e.g., collisions and object drops). Avoiding these risks requires a proactive safety mechanism capable of anticipating hazards. In this paper, we introduce SafeLoop, a non-invasive external wrapper that adds hazard prediction and rollback-based recovery to a VLA model without changing its parameters. SafeLoop trains a risk predictor from
vision and proprioception to output four values: the probability and time-to-hazard for body collisions and for object failures. A lightweight controller then chooses one of three actions based on the predicted risk: continue execution (noop), save a safety checkpoint (record), or retreat in joint space (rollback). Rollback moves the
robot back to a recent safe waypoint and queries the base policy again, which may yield an alternative continuation. Across 24 LIBERO tasks (16 random seeds each) and three real-
robot tasks (25 rollouts each), SafeLoop achieves a stronger overall safety-success trade-off than alternative methods, reducing hazard cases by roughly 70% while preserving task success and the base-policy control rate. Project code is available at https://github.com/Loule0-0/SafeLoop/tree/release/safeloop.
Read the full abstract ▾
All authors Zeyu Lou, Tianran Zhang, Xinquan Yue, Ya Jing, Chenyang Si
Robotics
Shaft-Configuration-Adaptive Catheter Tip Position Estimation via Motor-History Conditioned Residual Learning
Peihan Zhang, Michael C. Yip, Ankur Kapoor et al. (+1 more)
Tendon-driven continuum manipulators are widely used in medical applications, where accurate tip-position estimation is essential for precise navigation and instrument positioning. However, patient anatomy and procedural setup impose task-dependent unknown shaft configurations, while friction, slack, and compliance introduce hysteresis, making tip estimation challenging. This paper presents a motor-history-conditioned gated recurrent unit (GRU) residual
estimator for three-dimensional catheter tip estimation without direct shaft-configuration sensing. First, an initial multidirectional sweep strategy is applied to calibrate a
geometric catheter model backbone, and encode the motor-angle and drive-torque response into a shaft-configuration context vector. During subsequent motion, the context conditions a GRU that predicts a task-space residual correcting this backbone, relying on motor measurements alone. The context remains fixed for the current shaft configuration, while the recurrent state captures the evolving actuation history. Across four disposable intra-cardiac echocardiography catheters and 16 bent shaft configurations, the method achieves 3.3mm open-loop tip RMSE, a 59% reduction relative to the constant-
curvature baseline.
Read the full abstract ▾
All authors Peihan Zhang, Michael C. Yip, Ankur Kapoor, Young-Ho Kim
Robotics
RoboTwin-Phys: Do WAMs and VLAs Understand the Physical World?
Jiaqi Zhang, Feng Ye, Mingjia Yang et al. (+6 more)
Physical-condition diversity is largely missing from current
benchmarks for
robot manipulation. While large-scale simulation
benchmarks increasingly incorporate variations in object appearance, scene layout, and visual observations, they typically keep the underlying physical parameters fixed. As a result, important sources of real-world variability, such as changes in mass, friction, and joint dynamics, remain largely untested. We introduce RoboTwin-Phys, a physics-diverse
benchmark that treats physical-condition diversity as an explicit dimension of
robot manipulation evaluation. The
benchmark continuously varies 13 physical attributes within physically plausible ranges, providing a unified setting for evaluating policies across diverse physical operating conditions. We further release more than 5,000 expert demonstrations with ground-truth physical parameters, enabling physical-attribute estimation, condition-aware modeling, and physics-conditioned policy training. Evaluations of representative WAMs and VLAs reveal a substantial robustness gap: models that remain effective under existing visual and layout randomization can degrade markedly under changes in physical conditions. RoboTwin-Phys provides the
benchmark, data, and evaluation protocol needed to systematically measure and improve robustness to physical-condition diversity in
robot manipulation.
Read the full abstract ▾
All authors Jiaqi Zhang, Feng Ye, Mingjia Yang, Zhihong Chen, Mingkang Xiang, Xinglin Yao, Yanbin Li, Siwei Ma, Chuanmin Jia
Robotics
Manipulation of Deformable Linear Objects Using Model Predictive Path Integral Control with Bidirectional Long Short-Term Memory Learning
Lukas Zeh, Johannes Meiwaldt, Zexu Zhou et al. (+2 more)
The manipulation of Deformable Linear Objects (DLOs) such as cables poses a significant challenge for automation due to their infinite degrees of freedom and non-linear dynamics. In this paper we present a machine learning based optimal control approach for the manipulation of DLOs. This approach is divided into two main components: modeling and control. For modeling the dynamics of the DLO, we propose a learning based approach using a bidirectional Long Short-Term
Memory (biLSTM) network. The biLSTM network is trained on synthetic data generated by the MuJoCo physics engine. For manipulating the DLO, a model predictive control strategy that employs Model Predictive Path Integral (MPPI) control is selected. The proposed approach is evaluated through simulation and experiments. The results demonstrate the effectiveness of the proposed method in achieving accurate and efficient manipulation of DLOs.
Read the full abstract ▾
All authors Lukas Zeh, Johannes Meiwaldt, Zexu Zhou, Armin Lechler, Alexander Verl
Robotics
Towards Intent-Aware Human-Robot Teaming: A Platform for Search-and-Rescue Operations
Rohith Prem Maben, Ayesha Jena, Björn Olofsson et al. (+4 more)
We investigate the challenges of enabling effective collaboration between human operators and heterogeneous autonomous
agents in complex, dynamic environments by developing an interaction platform that allows study of operator behavior and supports intent inference and decision-making using state-of-the-art frameworks. We demonstrate the extent to which the operator's perception, decisions, and actions could be supported by autonomous systems during search-and-rescue operations with our platform.
Read the full abstract ▾
All authors Rohith Prem Maben, Ayesha Jena, Björn Olofsson, Stefan Reitmann, Jacek Malec, Rogier Woltjer, Elin Anna Topp
Robotics · Cross-listed from cs.MA
MATES: Learning Multi-Agent Interactions by Transforming Observations for Frozen Single-Agent Policies
Elie Abboud and Oren Gal
Multi-
agent reinforcement learning (MARL) commonly trains decentralized policies from scratch, requiring
agents to acquire individual task competence and coordination simultaneously. Yet many multi-
agent problems admit a compatible single-
agent counterpart in which the underlying task can be learned in isolation. We introduce Multi-
Agent Observation Transformation for Existing Single-
Agent Policies (MATES), an input-side adaptation framework for tasks whose multi-
agent observations preserve the solo-task information while exposing separately identifiable neighbor information. From multi-
agent experience, MATES learns a small adapter that maps this observation into the format expected by a frozen single-
agent policy, inducing actions suited to the shared environment without updating the single-
agent policy itself. MATES leaves the pretrained policy's internal architecture unchanged and retains the objectives and update procedures of the underlying MARL
algorithm. We evaluate MATES using both on- and off-policy
algorithms on lifelong pathfinding, navigation, and cooperative discovery, spanning discrete and continuous observation and action spaces. Across all evaluated settings, MATES optimizes only 3.5-7.3% as many parameters as full-policy training while consistently outperforming MARL training from scratch. It approaches the performance of full fine-tuning, remains competitive overall with demonstration-based baselines, and retains strong task performance at team sizes not encountered during training. These results provide evidence that, under this observation structure, effective multi-
agent behavior can be learned without modifying the policy that encodes individual competence.
Read the full abstract ▾
Robotics
Skytopia: Monocular Drone Navigation with Action-Conditioned Latent World Models
Yuhang Zhang, Rangya Zhang, Yujing Shang et al. (+6 more)
Monocular drone navigation requires reaching a goal in an unseen environment from a single forward-facing camera, which offers few cues for depth and scale. World models address this by modelling how observations evolve under actions, but they are built to be executed: the prediction is produced at deployment and fed back into action generation at every control step. We argue that what a policy needs from a world model is not the prediction but the representation required to produce it: in flight the executed action explains almost all of the change between observations, so prediction reduces to reprojecting a static scene under a known displacement. We therefore introduce skytopia, a policy built on an action-conditioned latent world model, and the 3D Gaussian Splatting platform on which it is trained. A forward objective predicts the representation of the next observation from the intended motion, and an inverse objective recovers that motion from the predicted transition. Because the prediction never reaches action generation, the predictor is discarded and one policy serves point-goal, image-goal, and goal-free navigation. Simulation experiments show that skytopia outperforms every baseline under all three specifications, attaining 57.8%, 66.0%, and 49.0% success rate, while discarding the predictor removes 59.4% of the inference cost. The same policy is subsequently deployed on a physical drone without fine-tuning and reaches goals in indoor, open outdoor, and woodland environments.
Read the full abstract ▾
All authors Yuhang Zhang, Rangya Zhang, Yujing Shang, Zhuoyuan Yu, Weiying Wang, Steven Yang, Qingsong Yan, Chao Yan, Mir Feroskhan
Robotics
Manipulation with Stability Guarantees: Linear Deformable Objects with Non-negligible Physical Response Grasped at Multiple Location
Daniel Feliu-Talegon and Cosimo Della Santina
Most research on the manipulation of deformable objects focuses on lightweight systems with negligible mechanical response, effectively restricting
attention to quasi-static regimes. This assumption excludes a broad class of practically relevant objects, such as hoses, pipes, and wiring harnesses, whose dynamics cannot be ignored during manipulation. In this work, we address this limitation by introducing a closed-loop control architecture that explicitly accounts for object dynamics and recasts manipulation as a shape-regulation problem. Control is achieved by modulating forces and torques applied at multiple fixed points along the object. This approach builds on three methodological contributions: a fully dynamic model of linear deformable objects based on discrete strain parameterizations; an extension of the notion of actuation coordinates to SE(3), yielding a structured and inherently underactuated control architecture; and nonlinear feedback strategies providing explicit conditions for steady-state convergence to desired configurations. Extensive simulations on representative manipulation tasks demonstrate the performance gains enabled by the proposed modelbased formulation. We finally validate the approach experimentally through a real-time closed-loop implementation with online shape estimation, confirming its practical feasibility and effectiveness
Read the full abstract ▾
Robotics
Safety-Constrained Model Predictive Control for an Omnidirectional Walking Assistive Robot Using Control Barrier Function
Andrea Fortuna, Marta Lorenzini, Elisa Motta et al. (+3 more)
Providing safe and effective mobility assistance plays a crucial role in restoring independence and enhancing the quality of life for individuals with motor impairments. In this context, robotic walking assistive devices have recently emerged as promising solutions to provide physically compliant interaction while ensuring user safety and support. This paper presents a novel control framework for an omnidirectional Walking Assistive
Robot (I-WANDER) that integrates a Control Barrier Function (CBF) formulation into a Model Predictive Control (MPC) scheme to explicitly enforce collision-avoidance safety constraints while optimizing for energy efficiency and smooth human-
robot collaboration. The method was experimentally evaluated with 12 healthy participants performing two different walking tasks using both the proposed CBF-based MPC controller (CB-MPC) and a variable admittance controller (AC). The first task involved structured navigation through a U-shaped corridor, whereas the second consisted of a single-obstacle avoidance task performed blindfolded to ensure the obstacle was unexpected. Comparative results show that the CB-MPC architecture significantly reduces energy consumption and mechanical work (p < 0.01) without compromising motion smoothness, while also decreasing the number of obstacle collisions. Overall, the findings highlight the potential of the proposed control architecture to enhance both safety and efficiency in robotic walking assistance.
Read the full abstract ▾
All authors Andrea Fortuna, Marta Lorenzini, Elisa Motta, Alberto Ranavolo, Elena De Momi, Arash Ajoudani
Robotics
Predict Before You Step: Auditable Occupancy Forecasting for Dynamic Obstacle Avoidance under Sparse Guidance
Yuhui Mao, Fen Liu, Shenghai Yuan et al. (+3 more)
Legged
robots under sparse waypoint guidance must avoid moving obstacles using partial, rapidly changing
LiDAR observations. We present LOOP (Latent-recurrent Occupancy rollOut Policy), a local avoidance policy that connects sparse waypoint guidance to a frozen locomotion controller at 50 Hz. From occupancy and ego-velocity histories, a recurrent predictor forecasts future occupancy over a 1 s horizon by warping the current map with learned flow and visibility gates. These maps guide velocity selection through map-derived features and
geometric risk estimates, providing an explicit interface for inspecting and replacing predictions. In encounter-synchronised Isaac Lab evaluations, LOOP achieves 57.1% head-on success at obstacle speeds of 2.5-3.2 m/s, exceeding a retrained reactive baseline by 8.2 percentage points. Comparisons with a rollout-free BEV policy show smaller, scenario-dependent gains from the prediction branch, including improved crossing success and reduced variability across training seeds at the highest head-on speeds. The adapter runs onboard a Unitree Go2 in 14.5 ms per step and completes all 16 real-world crossing trials without collision, demonstrating deployment feasibility.
Read the full abstract ▾
All authors Yuhui Mao, Fen Liu, Shenghai Yuan, Tianxin Hu, Ruimeng Liu, Rong Su
Robotics
An Action Is Worth One Patch: Unified World-Action Modeling with PatchWAM
Tianheng Wang, Zhou Xie, Heng Jia et al. (+3 more)
Generative visual models offer a foundation for learning representations of physical dynamics, yet their extension to continuous control raises a fundamental question: do visual prediction and action generation require separate computational pathways? Existing approaches usually introduce trainable action heads or separate action experts to bridge low-dimensional states and high-dimensional visual representations. In this work, we explore whether the visual backbone's existing capacity can also support control when actions are expressed in a compatible representation. Thus, we introduce PatchWAM (Patch World-Action Model), which treats continuous actions as another type of patch through a fixed mapping called Action-as-Patch. This allows a single model to predict both how the
robot should move and what the scene may look like afterward. Visual prediction and action generation become parts of the same generative process, without a dedicated action head or separate action expert. Experiments with subsampled training windows show gains over a matched dual-expert control, while
benchmark evaluations reach 91.8% success rate on LIBERO-Plus and 96.12% on RoboTwin 2.0 in a full-data setting with additional augmented demonstrations. More broadly, the result suggests that capability need not be added where it can be inherited: the constraint on extending a generative backbone is the interface a new signal is written in, not the capacity to model it.
Read the full abstract ▾
All authors Tianheng Wang, Zhou Xie, Heng Jia, Jianhua Xu, Tong Zhang, Kaicheng Yu
Robotics
Destination Support Restoration for Finite-Set Multimodal Trajectory Prediction
Fengrui Liu, Jiajun Peng, Duo Peng et al. (+1 more)
Robots operating around pedestrians often reason over a finite set of predicted human futures. Repeated online updates can concentrate this limited prediction budget on dominant destinations and leave plausible alternatives underrepresented or absent, removing those alternatives from the finite representation available to downstream decision making. We introduce Destination Support Restoration (DSR), a causal post-selection operator that repairs destination support without retraining the host predictor or increasing the maintained set size. At a repair step, DSR evaluates a temporary destination-stratified candidate bank from the observed prefix, converts candidate evidence into integer target counts, protects representatives of active modes, and reallocates redundant surplus hypotheses to deficient modes. The maintained and returned sets retain exactly $N$ hypotheses, and DSR replaces at most $\lceil\rho N\rceil$ entries. Protected representatives preserve current categorical support; lineage-aware particle filters also preserve surviving resampling ancestors. Each replacement reduces the allocation mismatch to the evidence-driven target by one. On the complete 3,719-trajectory Edinburgh protocol over three seeds, DSR reduces MIF weighted ADE and FDE by 13.36% and 13.30% at $N=64$. Paired integrations with CLiFF, PPT, causal GDTS, Social Informer, and PECNet improve both metrics in every evaluated pair. These results show that finite-set support allocation is a useful prediction-side control point when a fixed hypothesis set serves as the interface to downstream systems.
Read the full abstract ▾
All authors Fengrui Liu, Jiajun Peng, Duo Peng, Feng Liu
Robotics
Unsigned Distance Maps on 2D Point Cloud Registration
Ricardo B. Sousa, Giorgio Grisetti, Héber Miguel Sobreira et al. (+2 more)
2D point cloud registration arises in
laser odometry and Simultaneous Localization and Mapping (SLAM) for mobile
robots. Iterative Closest Point (ICP) is one of the most widely used approaches. Still, its iterative procedure recomputes correspondences via nearest-neighbor search at every iteration, whereas correspondence-free alternatives focus on scan-to-map
alignment. This paper proposes a 2D point cloud registration approach based on unsigned distance maps, precomputing the Euclidean distance to the nearest reference point, along with its spatial derivatives, over a discrete grid, replacing the per-iteration search with O(1) lookups. Moreover, point-to-point and point-to-plane error formulations are derived on the SE(2)
manifold and solved via Gauss-Newton optimization. On a synthetic
benchmark and the real-world IILABS 3D
dataset, the precomputed point-to-point variant outperforms its analytical counterparts, achieving competitive
laser-odometry drift compared to point-to-plane formulations, as the precomputed gradient regularizes correspondences in the presence of sensor noise.
Read the full abstract ▾
All authors Ricardo B. Sousa, Giorgio Grisetti, Héber Miguel Sobreira, Carlos André Silva, António Paulo Moreira
Robotics
Vision-based Underwater Formation Control With Input Saturations via Barrier Lyapunov Functions
Nicola De Carli, João Zenário, Victor Nan Fernandez-Ayala et al. (+1 more)
In this work, we propose a communication-free framework for
vision-based formation control of fully actuated underwater
robots subject to sensing constraints, collision-avoidance requirements, and input saturations. Recentered barrier Lyapunov functions encode sensing and collision-avoidance constraints, while command-filtered backstepping extends the design to the second-order vehicle dynamics. The resulting control objective is enforced through a quadratic program that explicitly accounts for actuator limits. Conservative sensing domains provide margins from the physical limits and are adaptively relaxed when necessary, allowing temporary violation of the conservative bounds. The proposed approach is validated through realistic Software-in-the-Loop (SITL) simulations in Gazebo.
Read the full abstract ▾
All authors Nicola De Carli, João Zenário, Victor Nan Fernandez-Ayala, Dimos V. Dimarogonas
Robotics
Control Barrier Functions for Safe Free-Flying Robotic Spacecraft Operations in Tumbling Target Capture
Alexander Meinert, Peter Stadler, Niklas Baldauf et al. (+1 more)
This paper presents a modular control barrier function (CBF) framework for safe free-flying robotic spacecraft operations during tumbling target capture. Motivated by latest ESA guidelines for safe close proximity operations, safety zones and requirements are translated into dedicated CBFs. The 13-DoF system is decomposed into translational, attitude, and robotic subsystems, each equipped with a safety filter that minimally modifies nominal control inputs in a lightweight quadratic program. The filters enforce a conical approach corridor, collision avoidance zone, attitude line-of-sight pointing, angular velocity limits, robotic joint limits, link-base collision avoidance, and actuator constraints. Dynamic coupling between subsystems is handled by treating upstream safe control commands as known interconnection inputs in the downstream safety filters, preserving modularity while supporting system-level safety. The framework is validated in an on-orbit servicing scenario, including final approach, angular rate synchronization, and tumbling target grasping, using the high-fidelity astrodynamics simulator Basilisk. Monte Carlo simulation results demonstrate runtime efficiency and operational safety for various tumbling rates.
Read the full abstract ▾
All authors Alexander Meinert, Peter Stadler, Niklas Baldauf, Alen Turnwald
Robotics
You Should Be Properly Scoring Your Odometry
Ola Rønning, Usama Saqib and Andrzej Wąsowski
When we evaluate the performance of our odometry, it is common practice to score the estimated track against a ground truth. Unfortunately, scoring uses point metrics, such as the root mean square error, that ignore the covariance
matrix which
estimators like filters and smoothers already report. Using the covariance matters for two reasons. First, the covariance encodes the
estimator's uncertainty, so it tells us whether the
estimator trusts its own output. An overconfident
estimator will not report itself lost. Second, the covariance weights the error in each direction of the estimate. Without the covariance, an
estimator is unduly penalized for a high error in an uncertain direction. Instead of point metrics, we should use strictly proper scoring rules. These rules score the estimate together with its reported uncertainty. Strictly proper scoring rules recover the point metrics when no covariance is reported, and they diagnose covariance inconsistency when covariance is reported. Using a one-sided pairwise test, we show that two
estimators can expose overconfidence in at least one of them without a ground truth. Strictly proper scoring rules and our pairwise test are available in our open-source framework smfeval. As a case study, we use smfeval to assess the uncertainty quality of the translational component of ground-based
LiDAR-inertial odometry. Across four filters we find overconfidence - the worst case reports centimeter certainty with kilometer error. Knowing the filters are overconfident, we investigate the mechanism. The investigation traces overconfidence to filters crediting
LiDAR measurements with more new information than they carry.
Read the full abstract ▾
Robotics
Robust Active-Perception Control for Global-State-Free Aerial-Ground Cooperation
Mingxuan Zhang, Jiajun Yu, Baozhe Zhang et al. (+5 more)
Aerial-ground cooperation requires real-time
UAV–UGV relative-state information. Instead of maintaining global estimates for both
robots, direct control in a UGV-attached non-inertial frame avoids reliance on global localization.
Vision-based relative pose estimation with a passive marker offers a low-cost and effective solution. However, a fixed camera may lose sight of the moving UGV when the required
UAV attitude conflicts with the field-of-view (FOV) constraint. To address this, we propose COPA, a robust active-perception framework for global-state-free aerial-ground cooperation. We use a single-axis gimbal to decouple the camera optical axis from the
UAV pitch attitude. We derive an active-perception model that relates
UAV motion, gimbal angle, and UGV motion to the target image-plane state.A Temporal Convolutional Network (TCN) predicts short-horizon UGV acceleration and angular velocity from recent motion history without global-state measurements. The model predictive control (MPC) uses these predictions to jointly optimize
UAV and gimbal control. Simulations show that COPA maintains continuous target visibility, while ablation studies confirm that the TCN reduces peak errors during UGV motion transitions. Real-world experiments with UGV accelerations up to 3m/s^2 and yaw rates up to 1.0rad/s demonstrate robust tracking.
Read the full abstract ▾
All authors Mingxuan Zhang, Jiajun Yu, Baozhe Zhang, Pengxiang Zhou, Wentao Liu, Fei Gao, Chao Xu, Yanjun Cao
Robotics
What is the Better Curriculum: Controller-Shaped Grasping Behavior for Contact Force-Sensitive Manipulation
Ziyan Feng, Zizhao Yuan, Yulong Fu et al. (+6 more)
How should a
robot learn to manipulate objects so fragile that sub-Newton contact forces can cause irreversible damage? Existing visuo-tactile policy learning typically treats tactile sensing as an additional policy input. In direct-contact force-sensitive manipulation, however, the bottleneck can arise earlier, during data collection: manual gripper control is too delayed and coarse-grained to reliably maintain the narrow force range required for stable grasping. We therefore use a deterministic 25 Hz tactile reflex controller as a collection-time teacher, producing demonstrations with controller-shaped grasping behavior for tactile-free policy learning. On Action Chunking with
Transformers (ACT), policies trained from reflex-shaped demonstrations recover the teacher's grasping profile and achieve 95% stable grasps on the nominal plastic-cup task, substantially outperforming visually screened manual demonstrations. The same intervention improves in-distribution stability on $\pi_{0.5}$ and shows a favorable exploratory trend on an unseen paper-cup variant. Under randomized external disturbance, however, the reflex-data $\pi_{0.5}$ policy still fails in 45% of policy-only trials, whereas a deployment-time reflex arbiter retains all grasps. These results reveal a new role for tactile feedback in force-sensitive manipulation: rather than integrating tactile into the policy, we use it as a collection-time teacher that shapes grasping behavior in demonstrations for policy learning, while disturbance rejection remains controller-dependent, revealing the boundary of tactile-free policy.
Read the full abstract ▾
All authors Ziyan Feng, Zizhao Yuan, Yulong Fu, Yuxin He, Zhiyuan Zhang, Zhengjie Zhang, Jinni Zhou, Renjing Xu, Qiang Nie
Robotics · Cross-listed from cs.CV
MatchFusion: Explicit-Implicit Instance Matching for Spatio-Temporal Multimodal Autonomous Driving
Xiaoyu Li, Jiajia Fu, Long Shi et al. (+7 more)
Sparse instance representations provide a compact interface for spatial
LiDAR-camera and temporal past-current interaction in
multimodal perception and E2EAD. Effective interaction requires reliable instance correspondences despite
geometric discrepancies and heterogeneous semantic representations.
Attention-based methods exploit contextual semantics but often require specialized representation
alignment, increasing computational overhead. In contrast, association based on structured object states is efficient and interpretable but lacks contextual evidence to resolve ambiguous matches. To combine these complementary strengths, we propose MatchFusion, a learnable instance matching and fusion module for spatio-temporal
multimodal autonomous driving. MatchFusion initializes pairwise affinities using
geometric similarity and category consistency, then selectively refines structurally plausible associations using instance
embeddings. The resulting soft matchmap guides a common residual aggregation operator for adaptive information exchange. This unified matching-fusion formulation supports spatial
LiDAR-camera and temporal past-current interaction, using multi-view image-plane geometry and motion-compensated BEV geometry as the respective structural priors. Experiments on nuScenes demonstrate consistent perception gains across diverse front-end configurations. Compared with a prior instance-centric fusion method, the MatchFusion-equipped system achieves higher perception accuracy while reducing FLOPs by 55.3% and GPU
memory usage by 39.3%, with the matching-fusion module accounting for only 3.7% of total perception latency. Integrating temporal MatchFusion into SparseDrive further improves perception within an E2E framework without additional supervision. These results establish explicit-implicit matching as an effective and efficient mechanism for spatio-temporal instance interaction.
Read the full abstract ▾
All authors Xiaoyu Li, Jiajia Fu, Long Shi, Tianyu Du, Ruihang Li, Xian Wu, Lijun Zhao, Yingtao Zhang, Lining Sun, Ruifeng Li
Robotics
Sometimes You Gotta Run Before You Can Walk: Run-then-Walk Scheduling Strategy for VLM Autonomous Driving
Yuqi Ye, Shangkun Sun, Junhong Lin et al. (+6 more)
Recent VLM-based autonomous driving planners adopt GRPO-style reinforcement learning to optimize driving performance. However, existing GRPO recipes either optimize driving efficiency, risking progress-seeking but unsafe behavior, or enforce early safety constraints, leading to overly conservative behavior; both require lengthy training. To solve these problems, we first reveal two distinct RL regimes: a progress regime (Run-GRPO) that aggressively explores high progress, and a safety regime (Walk-GRPO) that restores safety under stable progress. Based on this finding, we propose $\textit{Run-then-Walk}$, a simple yet effective two-stage reward scheduling strategy for GRPO, achieving both better performance and faster convergence. Unlike one-stage RL, which may focus on progress, safety, or a mixture of both within a single training phase, this schedule explicitly separates progress discovery from safety repair. In the $\textit{Run}$ phase, we focus on progress, allowing the policy to escape the conservative bias and discover high-progress modes. In the subsequent $\textit{Walk}$ phase, we introduce endpoint and safety strategy to repair unsafe behaviors from the Run phase. This reversed schedule overcomes the conservatism of Walk-first methods and the unsafe progress-seeking of joint optimization. We validate it with various VLM-based planners on multiple
benchmarks: NAVSIMv1, NAVSIMv2, Navhard, and nuScenes. Extensive experiments demonstrate improved driving performance while requiring 40–50% fewer RL training epochs than the baselines. Code is available at https://github.com/haha-yuki-haha/AutoDrive-P3_with_Run-then-walk.
Read the full abstract ▾
All authors Yuqi Ye, Shangkun Sun, Junhong Lin, Jiayi Zhao, Changhao Peng, Wei Zheng, Guoqing Liu, Tiesong Zhao, Wei Gao
Robotics
Beyond Reconstruction Error: Analytical and Data-Driven Action Tokenization for Autoregressive Vision-Language-Action Models
Yuxin Yang, Gaohan He, Changxue Guan et al. (+1 more)
Discrete action tokenization is central to autoregressive
vision-
language-action (VLA) models, yet action representations are often evaluated primarily through reconstruction fidelity. We ask which representation properties actually matter for closed-loop control by comparing fixed analytical, data-driven linear, and nonlinear neural representations under a unified tokenization interface. Across rate-distortion analysis, sequence-modeling diagnostics, and 3,500 LIBERO rollouts, representation rankings change with the evaluation criterion. PCA achieves lower nominal reconstruction error than Temporal-DCT, but produces less predictable
token sequences and 3.0 percentage points lower mean seen-task success across three policy-training seeds, with the policy ordering reversing in one seed. In a matched seed-42 ablation, an autoencoder further reduces reconstruction error yet does not yield the strongest policy and exhibits greater sensitivity to discrete
token perturbations. These findings show that reconstruction fidelity alone cannot reliably select action representations for autoregressive control, motivating joint evaluation of
geometric fidelity, sequence predictability, decoder stability, and closed-loop performance.
Read the full abstract ▾
All authors Yuxin Yang, Gaohan He, Changxue Guan, Hangming Liu
Robotics
MOLA LiDAR-Inertial Odometry (MOLA-LIO) on the COMFORT Localization Benchmark
Jose Luis Blanco-Claraco
This short report documents our entry to the COMFORT Localization
Benchmark (IROS 2026), evaluated on the GrandTour
dataset recorded with the Boxi payload. It extends MOLA-LO into a
LiDAR-inertial system that also ingests IMU and, optionally, legged kinematic odometry. We describe the architecture, the streams consumed, the local protocol that selected the submitted configuration, and the measurements backing our real-time claim.
Read the full abstract ▾
Robotics
VisForce: Visual Grounding of Current and Desired Forces for Goal-Conditioned Dexterous Manipulation
Jung-Woo Lee and Soo-Chul Lim
Vision-
Language-Action (VLA) models have emerged as general-purpose robotic manipulation policies. However, in dexterous hand manipulation, contact forces are typically provided as separate states or force-specific representations, making it difficult to explicitly represent the spatial correspondence between force and their corresponding visual locations. In this work, we propose VisForce, which visually grounds the current and desired forces at their corresponding fingertip locations. VisForce renders current and desired visual force cues on the current wrist image and a task-specific goal image, and combines the two representations through goal-conditioned cross-
attention to generate force-aware actions. We evaluate VisForce using a real UR10
robot equipped with an RH56F1 dexterous hand through force-conditioned grasping and three multi-stage manipulation tasks. In force-conditioned grasping experiments, VisForce exhibited a consistent grip-force response as the desired force increased, and achieved grasp-and-lift success rates of 70% and 80% for an egg and a toothpaste tube, respectively. It further achieved final success rates of 70%, 55%, and 40% on cup insertion/bottle pouring, tong-assisted bread transfer, and slip-modulated peg-in-hole, respectively. These results show that fingertip-aligned visual force representations can be effectively used for force-aware conditioning in VLA-based dexterous hand manipulation.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
Minimal Recurrent Behavioral Memory for Imitation under Partial Observability
Xianyao Li, Fang Xu, Rui Min et al. (+2 more)
What is the least recurrent
memory needed to reproduce a specified expert under partial observability? The instantaneous requirement is the conditional
entropy of the expert's behavioral quotient, but recurrence must also preserve distinctions that future observations will not restore before use. We characterize this minimal recurrent behavioral
memory by a compatibility relation: under transitivity its classes attain the exact minimum, while the general case is an
entropy minimization over closed compatible state assignments, with exact certificates on finite instances. A sole-carrier measurement protocol separates behavioral sufficiency, excess code rate, and information carried by observations or other
memory paths; experimental bit requirements refer to the induced symbolic behavioral model under the stated occupancy. Across manipulation tasks, learned code rates remain near zero- and two-bit requirements as hidden modes grow to $512$, and anticipatory
memory follows a $2\to1\to0$ requirement despite zero instantaneous demand during waiting. Learning this representation remains difficult: event-agnostic future-behavior supervision yields $36/40$ sufficient seeds with one frozen configuration and improves the longest-horizon pixel setting from $0/8$ to $6/8$ sufficient held-out seeds (closed-loop success from $0.08$ to $0.57$). On unmodified community
benchmarks, the protocol certifies delay-independent requirements, which sufficient codes match at mid-delay. The supervision aids commitment but can induce predictive surplus; annealing it lets imitation and rate training reduce that surplus, separating the information-theoretic target from the ability to learn it.
Read the full abstract ▾
All authors Xianyao Li, Fang Xu, Rui Min, Ruitong Tian, Jing Du
Robotics
MedVLA: A Hierarchical Vision-Language-Action Framework for Closed-Loop Precision Medical Robot Manipulation
Junjie Xie, Angen Ye, Chuxuan He et al. (+2 more)
Precision medical robotics demands adaptive decision-making under strict safety, interpretability, and execution constraints. Although recent
Vision-
Language-Action (VLA) models show strong
multimodal reasoning ability, their continuous action generation paradigm is not well suited for precision medical tasks, where reliable closed-loop operation may also depend on non-action system function calls. To address this gap, we propose MedVLA, a hierarchical framework that couples high-level
multimodal reasoning with low-level function-constrained execution. We further introduce a scalable multi-
agent pipeline to generate skill-oriented chain-of-thought(CoT) data for structured training. Built on different
multimodal large-model backbones, MedVLA consistently improves performance after fine-tuning, demonstrating the effectiveness of the proposed framework across model variants. Under identical initial conditions, we perform 100 closed-loop flexible electrode implantation trials. The results show that MedVLA achieves a 95.0% task success rate, substantially outperforming representative VLA baselines, including OpenVLA (8%) and $\pi_0$ (15%), in accuracy, stability, and safety. These results indicate that structured
reasoning with constrained function-level execution is a practical route toward deployable precision medical robotics.
Read the full abstract ▾
All authors Junjie Xie, Angen Ye, Chuxuan He, Yujia Song, Dapeng Zhang
Robotics
PLAT: Sparse Timed Keyframe Motion Tracking for Humanoid Control via Privileged Latent Transition Learning
Zepeng Wang, Jiangxing Wang, Chao Ma et al. (+2 more)
Humanoid motion tracking policies rely on dense frame-by-frame references, limiting their use as high-level motion controllers for planning and interactive motion generation. We study Sparse Timed Keyframe Motion Tracking, where a policy receives only sparse future keyframes and their desired arrival times, and must execute stable whole-body motions that reach successive goals. We propose PLAT, a three-stage sparse timed keyframe motion tracking policy learning framework with Privileged LAtent Transition learning. PLAT bridges dense motion tracking and sparse goal-conditioned control by exploiting dense goal sequences as privileged supervision during training while requiring only sparse timed keyframe commands at deployment. A pretrained dense tracking expert first provides robust motion priors. A privileged latent prior is then learned through DAgger-style imitation, followed by latent residual reinforcement learning that refines latent transitions instead of directly optimizing actions. Extensive simulation experiments demonstrate that PLAT maintains accurate and stable sparse timed keyframe tracking across varying planning horizons, with particularly strong performance under long-horizon commands. Successful deployment on a Unitree G1 humanoid
robot further demonstrates the effectiveness and practicality of PLAT for sparse humanoid motion control.
Read the full abstract ▾
All authors Zepeng Wang, Jiangxing Wang, Chao Ma, Xiaochuan Shi, Zongqing Lu