Robotics
Perception-Aware Time-Optimal Planning for Quadrotor Waypoint Flight
Chao Qin, Jiaxu Xing, Rudolf Reiter et al. (+4 more)
Agile quadrotor flight pushes the limits of control, actuation, and onboard perception. While time-optimal trajectory planning has been extensively studied, existing approaches typically neglect the tight coupling between vehicle dynamics, environmental geometry, and the visual requirements of onboard state estimation. As a result, trajectories that are dynamically feasible may fail in closed-loop execution due to degraded visual quality. This paper introduces a unified time-optimal trajectory optimization framework for
vision-based quadrotors that explicitly incorporates perception constraints alongside full nonlinear dynamics, rotor actuation limits, aerodynamic effects, camera field-of-view constraints, and convex
geometric gate representations. The proposed formulation solves minimum-time lap trajectories for arbitrary racetracks with diverse gate shapes and orientations, while remaining numerically robust and computationally efficient. We derive an information-theoretic position uncertainty metric to quantify visual state-estimation quality and integrate it into the planner through three perception objectives: position uncertainty minimization, sequential field-of-view constraints, and look-ahead
alignment. This enables systematic exploration of the trade-offs between speed and perceptual reliability. To accurately track the resulting perception-aware trajectories, we develop a model predictive contouring tracking controller that separates lateral and progress errors. Experiments demonstrate real-world flight speeds up to 9.8 m/s with 0.07 m average tracking error, and closed-loop success rates improved from 55% to 100% on a challenging Split-S course. The proposed system provides a scalable
benchmark for studying the fundamental limits of perception-aware, time-optimal autonomous flight.
Read the full abstract ▾
All authors Chao Qin, Jiaxu Xing, Rudolf Reiter, Angel Romero, Yifan Lin, Hugh H.-T. Liu, Davide Scaramuzza
Robotics
Compliant In-hand Rolling Manipulation Using Tactile Sensing
Huan Weng, Yifei Chen and Kevin M. Lynch
We investigate in-hand rolling manipulation using a multifingered
robot hand, where each finger is compliant and equipped with a tactile fingertip providing contact location and wrench information. We derive the equations of motion for compliant quasistatic in-hand rolling manipulation and formulate a fingertip rolling manipulation controller for multiple fingers to achieve a desired object twist within a grasp. The contact mechanics are demonstrated in simulation and the controller is tested on an experimental
robot system.
Read the full abstract ▾
Robotics
OmniPlanner: Universal Exploration and Inspection Path Planning Across Robot Morphologies
Angelos Zacharia, Mihir Dharmadhikari, Mohit Singh et al. (+1 more)
Autonomous robotic systems are increasingly deployed for mapping, monitoring, and inspection in complex and unstructured environments. However, most existing path planning approaches remain domain-specific (i.e., in the air, on land, or at sea), limiting their scalability and cross-platform applicability. This article presents OmniPlanner, a unified planning framework for autonomous exploration and inspection across aerial, ground, and
underwater robots. The method integrates volumetric exploration and viewpoint-based inspection, alongside target reach behaviors within a single modular architecture, complemented by a platform abstraction layer that captures morphology-specific sensing, traversability and motion constraints. This enables the same planning strategy to generalize across distinct mobility domains with minimal retuning. The framework is validated through extensive simulation studies and field deployments in underground mines, industrial facilities, forests, submarine bunkers, and structured outdoor environments. Across these diverse scenarios, OmniPlanner demonstrates robust performance, consistent cross-domain generalization, and improved exploration and inspection efficiency compared to representative state-of-the-art baselines.
Videos presenting the OmniPlanner framework and demonstrating its field deployments across aerial, ground, and
underwater robots are available at https://ntnu-arl.github.io/omniplanner, and the source code is publicly available at https://github.com/ntnu-arl/gbplanner_ros/tree/gbplanner3.
Read the full abstract ▾
All authors Angelos Zacharia, Mihir Dharmadhikari, Mohit Singh, Kostas Alexis
Robotics
VehAnchor: Metadata-Free Metric Scale Recovery from Vehicle Cues in Aerial Imagery
Yifei Chen, Chenqian Le, Jiayi Cheng et al. (+1 more)
Autonomous aerial
robots operating in GPS-denied or communication-degraded environments frequently lose access to camera metadata and telemetry, leaving onboard perception systems unable to recover the absolute metric scale of the scene. As LLM/VLM-based planners are increasingly adopted as high-level
agents for
embodied systems, their ability to reason about physical dimensions becomes safety-critical – yet our experiments show that five state-of-the-art VLMs suffer from spatial scale
hallucinations, with median area estimation errors exceeding 50%. We propose VehAnchor, a lightweight, deterministic
Geometric Perception Skill designed as a callable tool that any LLM-based
agent can invoke to recover Ground Sample Distance (GSD) from ubiquitous environmental anchors: small vehicles detected via oriented bounding boxes, whose modal pixel length is robustly estimated through kernel density estimation and converted to GSD using a pre-calibrated reference length. The tool returns both a GSD estimate and a composite confidence score, enabling the calling
agent to autonomously decide whether to trust the measurement or fall back to alternative strategies. On the DOTA~v1.5
benchmark, VehAnchor achieves 6.87% median GSD error on 306~images. Integrated with SAM-based
segmentation for downstream area measurement, the pipeline yields 19.7% median error on a 100-entry
benchmark – with 2.6$\times$ lower category dependence and 4$\times$ fewer catastrophic failures than the best VLM baseline – demonstrating that equipping
agents with deterministic
geometric tools is essential for safe autonomous spatial
reasoning.
Read the full abstract ▾
All authors Yifei Chen, Chenqian Le, Jiayi Cheng, Xupeng Chen
Robotics
RoboLight: A Dataset with Linearly Composable Illumination for Robotic Manipulation
Shutong Jin, Jin Yang, Muhammad Zahid et al. (+1 more)
In this paper, we introduce RoboLight, the first real-world robotic manipulation
dataset capturing synchronized episodes under systematically varied lighting conditions. RoboLight consists of two components. (a) RoboLight-Real contains 2,800 real-world episodes collected in our custom Light Cube setup, a calibrated system equipped with eight programmable RGB LED lights. It includes structured illumination variation along three independently controlled dimensions: color, direction, and intensity. Each dimension is paired with a dedicated task featuring objects of diverse geometries and materials to induce perceptual challenges. All image data are recorded in high-dynamic-range (HDR) format to preserve radiometric accuracy. Leveraging the linearity of light transport, we introduce (b) RoboLight-Synthetic, comprising 196,000 episodes synthesized through interpolation in the HDR image space of RoboLight-Real. In principle, RoboLight-Synthetic can be arbitrarily expanded by refining the interpolation granularity. We further verify the
dataset quality through qualitative analysis and real-world policy roll-outs, analyzing task difficulty, distributional diversity, and the effectiveness of synthesized data. We additionally demonstrate three representative use cases of the proposed
dataset. The full
dataset, along with the system software and hardware design, will be released as open-source to support continued research.
Read the full abstract ▾
All authors Shutong Jin, Jin Yang, Muhammad Zahid, Florian T. Pokorny
Robotics
AMP2026: A Multi-Platform Marine Robotics Dataset for Tracking and Mapping
Edwin Meriaux, Shuo Wen, David Widhalm et al. (+11 more)
Marine environments present significant challenges for perception and autonomy due to dynamic surfaces, limited visibility, and complex interactions between aerial, surface, and submerged sensing modalities. This paper introduces the Aerial Marine Perception
Dataset (AMP2026), a multi-platform marine robotics
dataset collected across multiple field deployments designed to support research in two primary areas: multi-view tracking and marine environment mapping. The
dataset includes synchronized data from aerial drones, boat-mounted cameras, and submerged robotic platforms, along with associated localization and telemetry information. The goal of this work is to provide a publicly available
dataset enabling research in marine perception and multi-
robot observation scenarios. This paper describes the data collection methodology, sensor configurations,
dataset organization, and intended research tasks supported by the
dataset.
Read the full abstract ▾
All authors Edwin Meriaux, Shuo Wen, David Widhalm, Zhizun Wang, Junming Shi, Mariana Sosa Guzmán, Kalvik Jakkala, Bennett Carley, Elias Sokolova, Yogesh Girdhar, Monika Roznere, Jason O'Kane, Junaed Sattar, Gregory Dudek
Robotics
PRAM-R: A Perception-Reasoning-Action-Memory Framework with LLM-Guided Modality Routing for Adaptive Autonomous Driving
Yi Zhang, Xian Zhang, Saisi Zhao et al. (+4 more)
Multimodal perception enables robust autonomous driving but incurs unnecessary computational cost when all sensors remain active. This paper presents PRAM-R, a unified Perception-
Reasoning-Action-
Memory framework with LLM-Guided Modality Routing for adaptive autonomous driving. PRAM-R adopts an asynchronous dual-loop design: a fast reactive loop for perception and control, and a slow deliberative loop for
reasoning-driven modality selection and
memory updates. An LLM router selects and weights modalities using environmental context and sensor diagnostics, while a hierarchical
memory module preserves temporal consistency and supports long-term adaptation. We conduct a two-stage evaluation: (1) synthetic stress tests for stability analysis and (2) real-world validation on the nuScenes
dataset. Synthetic stress tests confirm 87.2% reduction in routing oscillations via hysteresis-based stabilization. Real-world validation on nuScenes shows 6.22% modality reduction with 20%
memory recall while maintaining comparable trajectory accuracy to full-modality baselines in complex urban scenarios. Our work demonstrates that LLM-augmented architectures with hierarchical
memory achieve efficient, adaptive
multimodal perception in autonomous driving.
Read the full abstract ▾
All authors Yi Zhang, Xian Zhang, Saisi Zhao, Yinglei Song, Chengdong Wu, Nenad Petrovic, Alois Knoll
Robotics
GSeg3D: A High-Precision Grid-Based Algorithm for Safety-Critical Ground Segmentation in LiDAR Point Clouds
Muhammad Haider Khan Lodhi and Christoph Hertzberg
Ground
segmentation in point cloud data is the process of separating ground points from non-ground points. This task is fundamental for perception in autonomous driving and robotics, where safety and reliable operation depend on the precise detection of obstacles and navigable surfaces. Existing methods often fall short of the high precision required in safety-critical environments, leading to false detections that can compromise decision-making. In this work, we present a ground
segmentation approach designed to deliver consistently high precision, supporting the stringent requirements of autonomous vehicles and robotic systems operating in real-world, safety-critical scenarios.
Read the full abstract ▾
Robotics
Learning Hip Exoskeleton Control Policy via Predictive Neuromusculoskeletal Simulation
Ilseung Park, Changseob Song and Inseung Kang
Developing exoskeleton controllers that generalize across diverse locomotor conditions typically requires extensive motion-capture data and biomechanical labeling, limiting scalability beyond instrumented laboratory settings. Here, we present a physics-based neuromusculoskeletal learning framework that trains a hip-exoskeleton control policy entirely in simulation, without motion-capture demonstrations, and deploys it on hardware via policy
distillation. A reinforcement learning teacher policy is trained using a muscle-synergy action prior over a wide range of walking speeds and slopes through a two-stage curriculum, enabling direct comparison between assisted and no-exoskeleton conditions. In simulation, exoskeleton assistance reduces mean muscle activation by up to 3.4% and mean positive joint power by up to 7.0% on level ground and ramp ascent, with benefits increasing systematically with walking speed. On hardware, the assistance profiles learned in simulation are preserved across matched speed-slope conditions (r: 0.82, RMSE: 0.03 Nm/kg), providing quantitative evidence of sim-to-real transfer without additional hardware tuning. These results demonstrate that physics-based neuromusculoskeletal simulation can serve as a practical and scalable foundation for exoskeleton controller development, substantially reducing experimental burden during the design phase.
Read the full abstract ▾
Robotics
Mingleyang Li, Yuran Wang, Yue Chen et al. (+6 more)
Garment manipulation has attracted increasing
attention due to its critical role in home-assistant robotics. However, the majority of existing garment manipulation works assume an initial state consisting of only one garment, while piled garments are far more common in real-world settings. To bridge this gap, we propose a novel garment
retrieval pipeline that can not only follow
language instruction to execute safe and clean
retrieval but also guarantee exactly one garment is retrieved per attempt, establishing a robust foundation for the execution of downstream tasks (e.g., folding, hanging, wearing). Our pipeline seamlessly integrates
vision-
language reasoning with visual affordance perception, fully leveraging the high-level
reasoning and planning capabilities of VLMs alongside the generalization power of visual affordance for low-level actions. To enhance the VLM's comprehensive awareness of each garment's state within a garment pile, we employ visual
segmentation model (SAM2) to execute object
segmentation on the garment pile for aiding VLM-based
reasoning with sufficient visual cues. A mask fine-tuning mechanism is further integrated to address scenarios where the initial
segmentation results are suboptimal. In addition, a dual-arm cooperation framework is deployed to address cases involving large or long garments, as well as excessive garment sagging caused by incorrect grasping point determination, both of which are strenuous for a single arm to handle. The effectiveness of our pipeline are consistently demonstrated across diverse tasks and varying scenarios in both real-world and simulation environments. Project page: https://garmentpile2.github.io/.
Read the full abstract ▾
All authors Mingleyang Li, Yuran Wang, Yue Chen, Tianxing Chen, Jiaqi Liang, Zishun Shen, Haoran Lu, Ruihai Wu, Hao Dong
Robotics
HBRB-BoW: A Retrained Bag-of-Words Vocabulary for ORB-SLAM via Hierarchical BRB-KMeans
Minjae Lee, Sang-Min Choi, Gun-Woo Kim et al. (+1 more)
In visual simultaneous localization and mapping (SLAM), the quality of the visual vocabulary is fundamental to the system's ability to represent environments and recognize locations. While ORB-SLAM is a widely used framework, its binary vocabulary, trained through the k-majority-based bag-of-words (BoW) approach, suffers from inherent precision loss. The inability of conventional binary clustering to represent subtle feature distributions leads to the degradation of visual words, a problem that is compounded as errors accumulate and propagate through the hierarchical tree structure. To address these structural deficiencies, this paper proposes hierarchical binary-to-real-and-back (HBRB)-BoW, a refined hierarchical binary vocabulary training
algorithm. By integrating a global real-valued flow within the hierarchical clustering process, our method preserves high-fidelity descriptor information until the final binarization at the leaf nodes. Experimental results demonstrate that the proposed approach yields a more discriminative and well-structured vocabulary than traditional methods, significantly enhancing the representational integrity of the visual dictionary in complex environments. Furthermore, replacing the default ORB-SLAM vocabulary file with our HBRB-BoW file is expected to improve performance in loop closing and relocalization tasks.
Read the full abstract ▾
All authors Minjae Lee, Sang-Min Choi, Gun-Woo Kim, Suwon Lee
Robotics
Modeling and Control of a Pneumatic Soft Robotic Catheter Using Neural Koopman Operators
Yiyao Yue, Noah Barnes, Lingyun Di et al. (+4 more)
Catheter-based interventions are widely used for the diagnosis and treatment of cardiac diseases. Recently, robotic catheters have attracted
attention for their ability to improve precision and stability over conventional manual approaches. However, accurate modeling and control of soft robotic catheters remain challenging due to their complex, nonlinear behavior. The Koopman operator enables lifting the original system data into a linear "lifted space", offering a data-driven framework for predictive control; however, manually chosen basis functions in the lifted space often oversimplify system behaviors and degrade control performance. To address this, we propose a neural network-enhanced Koopman operator framework that jointly learns the lifted space representation and Koopman operator in an end-to-end manner. Moreover, motivated by the need to minimize radiation exposure during X-ray fluoroscopy in cardiac ablation, we investigate open-loop control strategies using neural Koopman operators to reliably reach target poses without continuous imaging feedback. The proposed method is validated in two experimental scenarios: interactive position control and a simulated cardiac ablation task using an atrium-like cavity. Our approach achieves average errors of 2.1 +- 0.4 mm in position and 4.9 +- 0.6 degrees in orientation, outperforming not only model-based baselines but also other Koopman variants in targeting accuracy and efficiency. These results highlight the potential of the proposed framework for advancing soft robotic catheter systems and improving catheter-based interventions.
Read the full abstract ▾
All authors Yiyao Yue, Noah Barnes, Lingyun Di, Olivia Young, Ryan D. Sochol, Jeremy D. Brown, Axel Krieger
Robotics
Swimming Under Constraints: A Safe Reinforcement Learning Framework for Quadrupedal Bio-Inspired Propulsion
Xinyu Cui, Fei Han, Hang Xu et al. (+9 more)
Bio-inspired aquatic propulsion offers high thrust and maneuverability but is prone to destabilizing forces such as lift fluctuations, which are further amplified by six-degree-of-freedom (6-DoF) fluid coupling. We formulate quadrupedal swimming as a constrained optimization problem that maximizes forward thrust while minimizing destabilizing fluctuations. Our proposed framework, Accelerated Constrained Proximal Policy Optimization with a PID-regulated Lagrange multiplier (ACPPO-PID), enforces constraints with a PID-regulated Lagrange multiplier, accelerates learning via conditional asymmetric clipping, and stabilizes updates through cycle-wise
geometric aggregation. Initialized with imitation learning and refined through on-hardware towing-tank experiments, ACPPO-PID produces control policies that transfer effectively to quadrupedal free-swimming trials. Results demonstrate improved thrust efficiency, reduced destabilizing forces, and faster convergence compared with state-of-the-art baselines, underscoring the importance of constraint-aware safe RL for robust and generalizable bio-inspired locomotion in complex fluid environments.
Read the full abstract ▾
All authors Xinyu Cui, Fei Han, Hang Xu, Yongcheng Zeng, Luoyang Sun, Ruizhi Zhang, Jian Zhao, Haifeng Zhang, Weikun Li, Hao Chen, Jun Wang, Dixia Fan
Robotics
SaFeR: Safety-Critical Scenario Generation for Autonomous Driving Test via Feasibility-Constrained Token Resampling
Jinlong Cui, Fenghua Liang, Guo Yang et al. (+2 more)
Safety-critical scenario generation is crucial for evaluating autonomous driving systems. However, existing approaches often struggle to balance three conflicting objectives:
adversarial criticality, physical feasibility, and behavioral realism. To bridge this gap, we propose SaFeR: safety-critical scenario generation for autonomous driving test via feasibility-constrained
token resampling. We first formulate traffic generation as a discrete next
token prediction problem, employing a
Transformer-based model as a realism prior to capture naturalistic driving distributions. To capture complex interactions while effectively mitigating
attention noise, we propose a novel differential
attention mechanism within the realism prior. Building on this prior, SaFeR implements a novel resampling strategy that induces
adversarial behaviors within a high-probability trust region to maintain naturalism, while enforcing a feasibility constraint derived from the Largest Feasible Region (LFR). By approximating the LFR via offline reinforcement learning, SaFeR effectively prevents the generation of theoretically inevitable collisions. Closed-loop experiments on the Waymo Open Motion
Dataset and nuPlan demonstrate that SaFeR significantly outperforms state-of-the-art baselines, achieving a higher solution rate and superior kinematic realism while maintaining strong
adversarial effectiveness.
Read the full abstract ▾
All authors Jinlong Cui, Fenghua Liang, Guo Yang, Chengcheng Tang, Jianxun Cui
Robotics
Sim2Sea: Sim-to-Real Policy Transfer for Maritime Vessel Navigation in Congested Waters
Xinyu Cui, Xuanfa Jin, Xue Yan et al. (+7 more)
Autonomous
navigation in congested maritime environments is a critical capability for a wide range of real-world applications. However, it remains an unresolved challenge due to complex vessel interactions and significant environmental uncertainties. Existing methods often fail in practical deployment due to a substantial sim-to-real gap, which stems from imprecise simulation, inadequate situational awareness, and unsafe exploration strategies. To address these, we propose Sim2Sea, a comprehensive framework designed to bridge simulation and real-world execution. Sim2Sea advances in three key aspects. First, we develop a GPU-accelerated parallel simulator for scalable and accurate maritime scenario simulation. Second, we design a dual-stream spatiotemporal policy that handles complex dynamics and multi-modal perception, augmented with a velocity-obstacle-guided action masking mechanism to ensure safe and efficient exploration. Finally, a targeted domain randomization scheme helps bridge the sim-to-real gap. Simulation results demonstrate that our method achieves faster convergence and safer trajectories than established baselines. In addition, our policy trained purely in simulation successfully transfers zero-shot to a 17-ton unmanned vessel operating in real-world congested waters. These results validate the effectiveness of Sim2Sea in achieving reliable sim-to-real transfer for practical autonomous maritime
navigation.
Read the full abstract ▾
All authors Xinyu Cui, Xuanfa Jin, Xue Yan, Yongcheng Zeng, Luoyang Sun, Siying Wei, Ruizhi Zhang, Jian Zhao, Haifeng Zhang, Jun Wang
Robotics · Cross-listed from cs.CV
Long-Term Visual Localization in Dynamic Benthic Environments: A Dataset, Footprint-Based Ground Truth, and Visual Place Recognition Benchmark
Martin Kvisvik Larsen and Oscar Pizarro
Long-term visual localization has the potential to reduce cost and improve mapping quality in optical benthic monitoring with autonomous
underwater vehicles (AUVs). Despite this potential, long-term visual localization in benthic environments remains understudied, primarily due to the lack of curated
datasets for benchmarking. Moreover, limited georeferencing accuracy and image footprints necessitate precise
geometric information for accurate ground-truthing. In this work, we address these gaps by presenting a curated
dataset for long-term visual localization in benthic environments and a novel method to ground-truth visual localization results for near-nadir
underwater imagery. Our
dataset comprises georeferenced AUV imagery from five benthic reference sites, revisited over periods up to six years, and includes raw and color-corrected stereo imagery, camera calibrations, and sub-decimeter registered camera poses. To our knowledge, this is the first curated
underwater dataset for long-term visual localization spanning multiple sites and photic-zone habitats. Our ground-truthing method estimates 3D seafloor image footprints and links camera views with overlapping footprints, ensuring that ground-truth links reflect shared visual content. Building on this
dataset and ground truth, we
benchmark eight state-of-the-art visual place recognition (VPR) methods and find that Recall@K is significantly lower on our
dataset than on established terrestrial and
underwater benchmarks. Finally, we compare our footprint-based ground truth to a traditional location-based ground truth and show that distance-threshold ground-truthing can overestimate VPR Recall@K at sites with rugged terrain and altitude variations. Together, the curated
dataset, ground-truthing method, and VPR
benchmark provide a stepping stone for advancing long-term visual localization in dynamic benthic environments.
Read the full abstract ▾
Robotics
HE-VPR: Height Estimation Enabled Aerial Visual Place Recognition Against Scale Variance
Mengfan He, Xingyu Shao, Chunyu Li et al. (+4 more)
In this work, we propose HE-VPR, a visual place recognition (VPR) framework that incorporates height estimation. Our system decouples height inference from place recognition, allowing both modules to share a frozen DINOv2 backbone. Two lightweight bypass adapter branches are integrated into our system. The first estimates the height partition of the query image via
retrieval from a compact height database, and the second performs VPR within the corresponding height-specific sub-database. The adaptation design reduces training cost and significantly decreases the search space of the database. We also adopt a center-weighted masking strategy to further enhance the robustness against scale differences. Experiments on two self-collected challenging multi-altitude
datasets demonstrate that HE-VPR achieves up to 6.1% Recall@1 improvement over state-of-the-art ViT-based baselines and reduces
memory usage by up to 90%. These results indicate that HE-VPR offers a scalable and efficient solution for height-aware aerial VPR, enabling practical deployment in GNSS-denied environments. All the code and
datasets for this work have been released on https://github.com/hmf21/HE-VPR.
Read the full abstract ▾
All authors Mengfan He, Xingyu Shao, Chunyu Li, Chao Chen, Liangzheng Sun, Ziyang Meng, Yuanqing Wu
Robotics
Force-Aware Residual DAgger via Trajectory Editing for Precision Insertion with Impedance Control
Yiou Huang, Ning Ma, Weichu Zhao et al. (+4 more)
Imitation learning (IL) has shown strong potential for contact-rich precision insertion tasks. However, its practical deployment is often hindered by
covariate shift and the need for continuous expert monitoring to recover from failures during execution. In this paper, we propose Trajectory Editing Residual
Dataset Aggregation (TER-DAgger), a scalable and force-aware human-in-the-loop imitation learning framework that mitigates
covariate shift by learning residual policies through optimization-based trajectory editing. This approach smoothly fuses policy rollouts with human corrective trajectories, providing consistent and stable supervision. Second, we introduce a force-aware failure anticipation mechanism that triggers human intervention only when discrepancies arise between predicted and measured end-effector forces, significantly reducing the requirement for continuous expert monitoring. Third, all learned policies are executed within a Cartesian impedance control framework, ensuring compliant and safe behavior during contact-rich interactions. Extensive experiments in both simulation and real-world precision insertion tasks show that TER-DAgger improves the average success rate by over 37% compared to behavior cloning, human-guided correction, retraining, and fine-tuning baselines, demonstrating its effectiveness in mitigating
covariate shift and enabling scalable deployment in contact-rich manipulation.
Read the full abstract ▾
All authors Yiou Huang, Ning Ma, Weichu Zhao, Zinuo Liu, Jun Sun, Qiufeng Wang, Yaran Chen
Robotics
Self-adapting Robotic Agents through Online Continual Reinforcement Learning with World Model Feedback
Fabian Domberg and Georg Schildbach
As learning-based robotic controllers are typically trained offline and deployed with fixed parameters, their ability to cope with unforeseen changes during operation is limited. Biologically inspired, this work presents a framework for online Continual Reinforcement Learning that enables automated adaptation during deployment. Building on DreamerV3, a model-based Reinforcement Learning
algorithm, the proposed method leverages world model prediction residuals to detect out-of-distribution events and automatically trigger finetuning. Adaptation progress is monitored using both task-level performance signals and internal training metrics, allowing convergence to be assessed without external supervision and domain knowledge. The approach is validated on a variety of contemporary continuous control problems, including a quadruped
robot in high-fidelity simulation, and a real-world model vehicle. Relevant metrics and their interpretation are presented and discussed, as well as resulting trade-offs described. The results sketch out how autonomous robotic
agents could once move beyond static training regimes toward adaptive systems capable of self-reflection and -improvement during operation, just like their biological counterparts.
Read the full abstract ▾
Robotics · Cross-listed from cs.DC
Lambdas at the Far Edge: a Tale of Flying Lambdas and Lambdas on Wheels
Giorgio Audrito, Daniele Bortoluzzi, Ferruccio Damiani et al. (+13 more)
Aggregate Programming (AP) is a paradigm for programming the collective behaviour of sets of distributed devices, possibly situated at the network far edge, by relying on asynchronous proximity-based interactions. The eXchange Calculus (XC), a recently proposed foundational model for AP, is essentially a typed lambda calculus extended with an operator (the exchange operator) providing an implicit communication mechanism between neighbour devices. This paper provides a gentle introduction to XC and to its implementation as a C++ library, called FCPP. The FCPP library and toolchain has been mainly developed at the Department of Computer Science of the University of Turin, where Stefano Berardi spent most of his academic career conducting outstanding research about logical foundation of computer science and transmitting his passion for research to students and young researchers, often exploiting typed lambda calculi. An FCCP program is essentially a typed lambda term, and FCPP has been used to write code that has been deployed on devices at the far edge of the network, including rovers and (soon) Uncrewed Aerial Vehicles (
UAVs); hence the title of the paper.
Read the full abstract ▾
All authors Giorgio Audrito, Daniele Bortoluzzi, Ferruccio Damiani, Giordano Scarso, Gianluca Torta, Andrea Basso, Monica Cochi, Lorenzo Gusman, Lorenzo Comba, Paolo Gay, Paola Dal Zovo, Giada Galati, Francesco Gallo, Aljaž Grdadolnik, Massimo Pescarollo, Paola Pisano
Robotics
Map-Agnostic And Interactive Safety-Critical Scenario Generation via Multi-Objective Tree Search
Wenyun Li, Zejian Deng and Chen Sun
Generating safety-critical scenarios is essential for validating the robustness of autonomous driving systems, yet existing methods often struggle to produce collisions that are both realistic and diverse while ensuring explicit interaction logic among traffic participants. This paper presents a novel framework for traffic-flow level safety-critical scenario generation via multi-objective Monte Carlo Tree Search (MCTS). We reframe trajectory feasibility and naturalistic behavior as optimization objectives within a unified evaluation function, enabling the discovery of diverse collision events without compromising realism. A hybrid Upper Confidence Bound (UCB) and Lower Confidence Bound (LCB) search strategy is introduced to balance exploratory efficiency with risk-averse decision-making. Furthermore, our method is map-agnostic and supports interactive scenario generation with each vehicle individually powered by SUMO's microscopic traffic models, enabling realistic
agent behaviors in arbitrary geographic locations imported from OpenStreetMap. We validate our approach across four high-risk accident zones in Hong Kong's complex urban environments. Experimental results demonstrate that our framework achieves an 85% collision failure rate while generating trajectories with superior feasibility and comfort metrics. The resulting scenarios exhibit greater complexity, as evidenced by increased vehicle mileage and CO\(_2\) emissions. Our work provides a principled solution for stress testing autonomous vehicles through the generation of realistic yet infrequent corner cases at traffic-flow level.
Read the full abstract ▾
Robotics
Right in Time: Reactive Reasoning in Regulated Traffic Spaces
Simon Kohaut, Benedict Flade, Julian Eggert et al. (+2 more)
Exact inference in probabilistic First-Order Logic offers a promising yet computationally costly approach for regulating the behavior of autonomous
agents in shared traffic spaces. While prior methods have combined logical and probabilistic data into decision-making frameworks, their application is often limited to pre-flight checks due to the complexity of
reasoning across vast numbers of possible universes. In this work, we propose a reactive mission design framework that jointly considers uncertain environmental data and declarative, logical traffic regulations. By synthesizing Probabilistic Mission Design (ProMis) with reactive
reasoning facilitated by Reactive
Circuits (RC), we enable online, exact probabilistic inference over hybrid domains. Our approach leverages the Frequency of Change inherent in heterogeneous data streams to subdivide inference formulas into memoized, isolated tasks, ensuring that only the specific components affected by new sensor data are re-evaluated. In experiments involving both real-world vessel data and simulated drone traffic in dense urban scenarios, we demonstrate that our approach provides orders of magnitude in speedup over ProMis without reactive paradigms. This allows intelligent transportation systems, such as Unmanned Aircraft Systems (UAS), to actively assert safety and legal compliance during operations rather than relying solely on preparation procedures.
Read the full abstract ▾
All authors Simon Kohaut, Benedict Flade, Julian Eggert, Kristian Kersting, Devendra Singh Dhami
Robotics
Structural Action Transformer for 3D Dexterous Manipulation
Xiaohan Lei, Min Wang, Bohong Weng et al. (+2 more)
Achieving human-level dexterity in
robots via imitation learning from heterogeneous
datasets is hindered by the challenge of cross-embodiment skill transfer, particularly for high-DoF robotic hands. Existing methods, often relying on 2D observations and temporal-centric action representation, struggle to capture 3D spatial relations and fail to handle embodiment heterogeneity. This paper proposes the Structural Action
Transformer (SAT), a new 3D dexterous manipulation policy that challenges this paradigm by introducing a structural-centric perspective. We reframe each action chunk not as a temporal sequence, but as a variable-length, unordered sequence of joint-wise trajectories. This structural formulation allows a
Transformer to natively handle heterogeneous embodiments, treating the joint count as a variable sequence length. To encode structural priors and resolve ambiguity, we introduce an
Embodied Joint Codebook that embeds each joint's functional role and kinematic properties. Our model learns to generate these trajectories from 3D point clouds via a continuous-time flow matching objective. We validate our approach by pre-training on large-scale heterogeneous
datasets and fine-tuning on simulation and real-world dexterous manipulation tasks. Our method consistently outperforms all baselines, demonstrating superior sample efficiency and effective cross-embodiment skill transfer. This structural-centric representation offers a new path toward scaling policies for high-DoF, heterogeneous manipulators.
Read the full abstract ▾
All authors Xiaohan Lei, Min Wang, Bohong Weng, Wengang Zhou, Houqiang Li
Robotics
ArthroCut: Autonomous Policy Learning for Robotic Bone Resection in Knee Arthroplasty
Xu Lu, Yiling Zhang, Wenquan Cheng et al. (+3 more)
Despite rapid commercialization of
surgical robots, their autonomy and real-time decision-making remain limited in practice. To address this gap, we propose ArthroCut, an autonomous policy learning framework that upgrades knee arthroplasty
robots from assistive execution to context-aware action generation. ArthroCut fine-tunes a Qwen–VL backbone on a self-built, time-synchronized
multimodal dataset from 21 complete cases (23,205 RGB–D pairs), integrating preoperative CT/MR, intraoperative NDI tracking of bones and end effector, RGB–D
surgical video,
robot state, and textual intent. The method operates on two complementary
token families – Preoperative Imaging
Tokens (PIT) to encode patient-specific anatomy and planned resection planes, and Time-Aligned
Surgical Tokens (TAST) to fuse real-time visual,
geometric, and kinematic evidence – and emits an interpretable action grammar under grammar/safety-constrained decoding. In bench-top experiments on a knee prosthesis across seven trials, ArthroCut achieves an average success rate of 86% over the six standard resections, significantly outperforming strong baselines trained under the same protocol. Ablations show that TAST is the principal driver of reliability while PIT provides essential anatomical grounding, and their combination yields the most stable multi-plane execution. These results indicate that aligning preoperative geometry with time-aligned intraoperative perception and translating that
alignment into tokenized, constrained actions is an effective path toward robust, interpretable autonomy in orthopedic robotic surgery.
Read the full abstract ▾
All authors Xu Lu, Yiling Zhang, Wenquan Cheng, Longfei Ma, Fang Chen, Hongen Liao
Robotics
Jaewon Lee, Jaeseok Heo, Gunmin Lee et al. (+3 more)
Safe visual
navigation is critical for indoor mobile
robots operating in cluttered environments. Existing
benchmarks, however, often neglect collisions or are designed for outdoor scenarios, making them unsuitable for indoor visual
navigation. To address this limitation, we introduce the reactive visual
navigation benchmark (RVN-Bench), a collision-aware
benchmark for indoor mobile
robots. In RVN-Bench, an
agent must reach sequential goal positions in previously unseen environments using only visual observations and no prior map, while avoiding collisions. Built on the Habitat 2.0 simulator and leveraging high-fidelity HM3D scenes, RVN-Bench provides large-scale, diverse indoor environments, defines a collision-aware
navigation task and evaluation metrics, and offers tools for standardized training and benchmarking. RVN-Bench supports both online and offline learning by offering an environment for online reinforcement learning, a trajectory image
dataset generator, and tools for producing negative trajectory image
datasets that capture collision events. Evaluations demonstrate that policies trained on RVN-Bench generalize effectively across unseen simulated environments. Furthermore, initial physical experiments using a Jackal UGV indicate promising sim-to-real transfer. Code and additional materials are available at: https://sequor-robotics-research.github.io/projects/RVN-Bench/.
Read the full abstract ▾
All authors Jaewon Lee, Jaeseok Heo, Gunmin Lee, Howoong Jun, Jeongwoo Oh, Songhwai Oh
Robotics
Lightweight Visual Reasoning for Socially-Aware Robots
Alessio Galatolo, Ronald Cumbal, Alexandros Rouchitsas et al. (+3 more)
Robots operating in shared human environments must not only navigate, interact, and detect their surroundings, they must also interpret and respond to dynamic, and often unpredictable, human behaviours. Although recent advances have shown promise in enhancing robotic perception and instruction-following using
Vision-
Language Models (VLMs), they remain limited in addressing the complexities of
multimodal human-
robot interactions (HRI). Motivated by this challenge, we introduce a lightweight
language-to-
vision feedback module that closes the loop between an LLM and the
vision encoder in VLMs. The module projects image-
token hidden states through a gated Multi-Layer Perceptron (MLP) back into the encoder input, prompting a second pass that reinterprets the scene under text context. We evaluate this approach on three robotics-centred tasks:
navigation in a simulated environment (Habitat), sequential scene description (Mementos-Robotics), and human-intention recognition (our HRI
dataset). Results show that our method improves Qwen 2.5 (7B) by $3.3\%$ (less distance), $+0.057$ description score, and $+2.93\%$ accuracy, with less than $3\%$ extra parameters; Gemma 3 (4B) and LLaVA OV 1.5 (4B) show mixed
navigation results but gains $+0.111,+0.055$ and $+10.81\%,+4.79\%$ on the latter two tasks. Code is available at https://github.com/alessioGalatolo/VLM-
Reasoning-for-Robotics
Read the full abstract ▾
All authors Alessio Galatolo, Ronald Cumbal, Alexandros Rouchitsas, Katie Winkle, Didem Gürdür Broo, Ginevra Castellano
Robotics · Cross-listed from cs.CV
DISC: Dense Integrated Semantic Context for Large-Scale Open-Set Semantic Mapping
Felix Igelbrink, Lennart Niecksch, Martin Atzmueller et al. (+1 more)
Open-set semantic mapping enables
language-driven robotic perception, but current instance-centric approaches are bottlenecked by context-depriving and computationally expensive crop-based feature extraction. To overcome this fundamental limitation, we introduce DISC (Dense Integrated Semantic Context), featuring a novel single-pass, distance-weighted extraction mechanism. By deriving high-fidelity CLIP
embeddings directly from the
vision transformer's intermediate layers, our approach eliminates the latency and domain-shift artifacts of traditional image cropping, yielding pure, mask-aligned semantic representations. To fully leverage these features in large-scale continuous mapping, DISC is built upon a fully GPU-accelerated architecture that replaces periodic offline processing with precise, on-the-fly voxel-level instance refinement. We evaluate our approach on standard
benchmarks (Replica, ScanNet) and a newly generated large-scale-mapping
dataset based on Habitat-Matterport 3D (HM3DSEM) to assess scalability across complex scenes in multi-story buildings. Extensive evaluations demonstrate that DISC significantly surpasses current state-of-the-art zero-shot methods in both semantic accuracy and query
retrieval, providing a robust, real-time capable framework for robotic deployment. The full source code, data generation and evaluation pipelines will be made available at https://github.com/DFKI-NI/DISC.
Read the full abstract ▾
All authors Felix Igelbrink, Lennart Niecksch, Martin Atzmueller, Joachim Hertzberg
Robotics
IROSA: Interactive Robot Skill Adaptation using Natural Language
Markus Knauer, Samuel Bustamante, Thomas Eiband et al. (+3 more)
Foundation models have demonstrated impressive capabilities across diverse domains, while imitation learning provides principled methods for
robot skill adaptation from limited data. Combining these approaches holds significant promise for direct application to robotics, yet this combination has received limited
attention, particularly for industrial deployment. We present a novel framework that enables open-vocabulary skill adaptation through a tool-based architecture, maintaining a protective abstraction layer between the
language model and
robot hardware. Our approach leverages pre-trained LLMs to select and parameterize specific tools for adapting
robot skills without requiring fine-tuning or direct model-to-
robot interaction. We demonstrate the framework on a 7-DoF torque-controlled
robot performing an industrial bearing ring insertion task, showing successful skill adaptation through natural
language commands for speed adjustment, trajectory correction, and obstacle avoidance while maintaining safety, transparency, and interpretability.
Read the full abstract ▾
All authors Markus Knauer, Samuel Bustamante, Thomas Eiband, Alin Albu-Schäffer, Freek Stulp, João Silvério
Robotics · Cross-listed from eess.SY
Dual-Interaction-Aware Cooperative Control Strategy for Alleviating Mixed Traffic Congestion
Zhengxuan Liu, Yuxin Cai, Yijing Wang et al. (+3 more)
As Intelligent Transportation System (ITS) develops, Connected and Automated Vehicles (CAVs) are expected to significantly reduce traffic congestion through cooperative strategies, such as in bottleneck areas. However, the uncertainty and diversity in the behaviors of Human-Driven Vehicles (HDVs) in mixed traffic environments present major challenges for CAV cooperation. This paper proposes a Dual-Interaction-Aware Cooperative Control (DIACC) strategy that enhances both local and global interaction perception within the Multi-
Agent Reinforcement Learning (MARL) framework for Connected and Automated Vehicles (CAVs) in mixed traffic bottleneck scenarios. The DIACC strategy consists of three key innovations: 1) A Decentralized Interaction-Adaptive Decision-Making (D-IADM) module that enhances actor's local interaction perception by distinguishing CAV-CAV cooperative interactions from CAV-HDV observational interactions. 2) A Centralized Interaction-Enhanced Critic (C-IEC) that improves critic's global traffic understanding through interaction-aware value estimation, providing more accurate guidance for policy updates. 3) A reward design that employs softmin aggregation with temperature annealing to prioritize interaction-intensive scenarios in mixed traffic. Additionally, a lightweight Proactive Safety-based Action Refinement (PSAR) module applies rule-based corrections to accelerate training convergence. Experimental results demonstrate that DIACC significantly improves traffic efficiency and adaptability compared to rule-based and
benchmark MARL models.
Read the full abstract ▾
All authors Zhengxuan Liu, Yuxin Cai, Yijing Wang, Xiangkun He, Chen Lv, Zhiqiang Zuo
Robotics
SkillVLA: Tackling Combinatorial Diversity in Dual-Arm Manipulation via Skill Reuse
Xuanran Zhai, Zekai Huang, Longyan Wu et al. (+5 more)
Recent progress in
vision-
language-action (VLA) models has demonstrated strong potential for dual-arm manipulation, enabling complex behaviors and generalization to unseen environments. However, mainstream bimanual VLA formulations largely overlook the critical challenge of combinatorial diversity. Different pairings of single-arm behaviors can induce qualitatively distinct task behaviors, yet existing models do not explicitly account for this structure. We argue that effective bimanual VLAs should support skill reuse - the ability to recombine previously learned single-arm skills across novel left-right pairings - thereby avoiding the need to separately learn every possible combination. Current VLA designs entangle skills across arms, preventing such recomposition and limiting scalability. To address this limitation, we propose SkillVLA, a framework explicitly designed to enable skill reuse in dual-arm manipulation. Extensive experiments demonstrate that SkillVLA substantially improves skill composition, increasing overall success rate from 0% to 51%, and achieves strong performance on cooperative and long-horizon tasks.
Read the full abstract ▾
All authors Xuanran Zhai, Zekai Huang, Longyan Wu, Qianyou Zhao, Qiaojun Yu, Jieji Ren, Ce Hao, Harold Soh