Robotics
Model-Reference Adaptive Flight Control of a 95-mg Insect-Scale Flapping-Wing Aerial Robot
Francisco M. F. R. Gonçalves, Conor K. Trygstad and Néstor O. Pérez-Arancibia
Due to the system's scale and complex fabrication, the model describing the dynamics of a flapping-wing insect-scale aerial
robot is subject to parameter uncertainty; for example, in the inertia
matrix and the actuator mapping of the flier. Furthermore, due to its low inertia, this type of
robot is greatly affected by
stochastic and systematic disturbances during flight, including power-wire tension, gusts, and undesired aerodynamic forces produced by wing misalignment. Therefore, the high-performance execution of complex maneuvers at the subdecigram scale requires the
robot to adapt its behavior to counteract disturbances and model uncertainty. Toward this objective, we introduce a model-reference adaptive control (MRAC) architecture for high-performance position control of flapping-wing robotic insects that can be modeled as rigid bodies in the three-dimensional (3D) space. In addition, we demonstrate how the implementation of a hybrid multiplicative extended Kálmán filter for estimating current and desired angular velocities during flight significantly dampens attitude vibrations, especially along the roll and pitch degrees of freedom (DOFs), and also improves flight performance. To show the suitability, functionality, and high performance of the proposed approach, we conducted real-time hovering and trajectory-tracking 6-DOF flight control experiments with a 95-mg insect-scale aerial
robot.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
Quantile-Coupled Flow Matching for Distributional Reinforcement Learning
Michael Groom, Victor-Alexandru Darvariu, Lars Kunze et al. (+2 more)
Unlike standard expected-return Reinforcement Learning (RL), Distributional RL (DRL) models the full return distribution, making it better-suited for uncertainty-aware and risk-sensitive decision-making. Conditional Flow Matching (CFM) critics have recently attracted
attention for modelling continuous, multi-modal return distributions. Despite this interest, there remains a substantial metric mismatch: DRL theory relies on the distributional Bellman operator being contractive in the $p$-Wasserstein distance, yet existing CFM critics are trained with arbitrary source-target couplings, so their flow-matching losses are not Wasserstein-aligned surrogates for matching Bellman target return distributions. In this work, we address this mismatch by proposing FlowIQN, a CFM critic that sorts source and Bellman target samples within each mini-batch to approximate the monotone optimal transport coupling, replacing arbitrary pairings with
quantile-aligned flow paths. We prove that the loss of our
quantile-coupled CFM critic yields a Wasserstein-aligned approximate projection compatible with the foundations of DRL. To our knowledge, FlowIQN is the first flow-matching distributional critic with an explicit Wasserstein-aligned projection guarantee. We further extend FlowIQN with shortcut models for efficient inference. Empirical results show that FlowIQN improves Wasserstein return-distribution accuracy over other CFM critics. It also yields competitive performance on offline RL
benchmarks across multiple policy extraction methods, providing a theoretically grounded CFM critic that is readily compatible with DRL pipelines. Code: https://github.com/ori-goals/flowIQN.
Read the full abstract ▾
All authors Michael Groom, Victor-Alexandru Darvariu, Lars Kunze, James Wilson, Nick Hawes
Robotics
Trajectory-Consistent Flow Matching for Robust Visuomotor Policy Learning
Riad Ahmed, Sujosh Nag, Moniruzzaman Akash et al. (+2 more)
Flow matching policies learn continuous velocity fields that transport noise to actions, enabling fast deterministic inference for
robot manipulation. However, standard training optimizes a pointwise velocity objective while inference requires numerical integration of that field – a mismatch that causes compounding trajectory errors. We propose four complementary remedies: (1) auxiliary rectified flow velocity regression that provides uniform temporal supervision across the full time interval; (2) multi-step trajectory consistency training that supervises the integrated displacement of the velocity field over trajectory segments, directly closing the train-inference gap; (3) velocity field regularization that enforces temporal smoothness, preventing oscillations that destabilize integration; and (4) fourth-order Runge-Kutta (RK4) inference that reduces global discretization error by orders of magnitude over Euler methods. Critically, these components are not independently sufficient – RK4 without a smooth velocity field fails, and smoothness without trajectory-level supervision still drifts, as our ablation study confirms. We further pair these with a dual-view 3D point cloud encoder using two independent PointNet encoders for complementary spatial perception. On four real-
robot tasks across a Franka arm and a Boston Dynamics Spot, our method achieves 70% and 60% overall success on two long-horizon multi-phase tasks where both baselines score 0%, and reaches 100% on precision tool placement. Three MetaWorld simulation tasks confirm consistent improvements, validating that trajectory-level supervision is essential for reliable policy execution.
Read the full abstract ▾
All authors Riad Ahmed, Sujosh Nag, Moniruzzaman Akash, Mostafa Hussein, Momotaz Begum
Robotics
DynSSM: A Physics-Aware State-Space Memory Framework for Learning Vehicle Dynamics
Musabbir Ahmed Arrafi, Malik Ali, Nicholas M. Stiffler et al. (+1 more)
Accurate modeling of nonlinear vehicle dynamics is essential for high-speed autonomous racing, where controllers operate at the handling limits. Model-based methods are interpretable but rely on simplifying assumptions, while purely learned models capture nonlinearities yet often lack physical consistency, generalization, and adaptability to changing operating conditions. This paper presents DynSSM, a physics-aware state-space
memory framework that combines learned temporal representations with a structured vehicle dynamics model. The proposed approach integrates state-space sequence modeling and recurrent encoders to capture long- and short-term dynamic behavior. It simultaneously adapts tire and vehicle-dynamics parameters within bounded ranges to preserve physical plausibility. A residual correction mechanism compensates for remaining unmodeled dynamics while preserving the underlying physics-based structure. DynSSM is evaluated on both simulated small-scale racing data and real-world full-scale autonomous Indy racecar data. When evaluated on an unseen real-world track, DynSSM reduces one-step prediction RMSE by up to $27.2\%$ in longitudinal velocity, $73.9\%$ in lateral velocity, and $88.8\%$ in yaw rate compared with the state-of-the-art (\sota{}) baselines. Component-wise ablation studies demonstrate the importance of temporal
memory, parameter adaptation, and residual correction for predictive performance and robustness. Further, closed-loop simulations using nonlinear model predictive control demonstrate that DynSSM remains feasible across the evaluated tracks and achieves up to a 13.2% reduction in one-lap completion time compared with \sota{} baselines. These results indicate that combining temporal
memory, bounded physics-guided parameter adaptation, and residual correction provides an accurate, interpretable, and control-ready dynamics model for autonomous racing.
Read the full abstract ▾
All authors Musabbir Ahmed Arrafi, Malik Ali, Nicholas M. Stiffler, Krishna Bhavithavya Kidambi
Robotics
Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models
Meng Zheng, Samhita Marri, Anwesa Choudhuri et al. (+6 more)
Vision-
language-action (VLA) models provide a promising paradigm for scalable robotic manipulation, yet their reliance on success-only behavioral cloning leaves them brittle; lacking corrective training signals, minor execution errors rapidly compound into unrecoverable, out-of-distribution failures. To address this limitation, we propose Adaptive Failure-Informed Learning (AFIL), an end-to-end framework that leverages failure trajectories as adaptive negative guidance for
diffusion- and flow-based VLA policies. AFIL uses a pretrained VLA to generate failure rollouts online, avoiding the need for handcrafted failure-mode design or human-in-the-loop recovery. It then jointly trains Dual Action Generators (DAGs) for successful and failed behaviors while sharing a common
vision-
language backbone, enabling efficient failure-aware policy learning with limited parameter overhead. During sampling, the failure generator adaptively steers action generation away from failure-prone regions and toward more reliable success modes, with guidance strength determined by the per-
diffusion-step distance between success and failure distributions. Experiments across in-domain and out-of-domain robotic manipulation tasks, covering both short- and long-horizon settings, show that AFIL consistently improves task success rates and robustness over existing VLA baselines, demonstrating its effectiveness, efficiency, and generality.
Read the full abstract ▾
All authors Meng Zheng, Samhita Marri, Anwesa Choudhuri, Benjamin Planche, Zhongpai Gao, Van Nguyen Nguyen, Terrence Chen, Girish Chowdhary, Ziyan Wu
Robotics
Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning
Karolina Źróbek, Tessa Pulli, Paweł Gajewski et al. (+2 more)
We present a hierarchical
language-driven framework for robotic task and motion planning to improve natural, intuitive human-
robot interaction in service and assistance scenarios. The proposed system employs two large
language model (LLM) modules: a high-level planning
agent and a low-level spatial
reasoning sub-module. The primary
agent processes natural
language commands and generates action sequences using a ReAct-style
prompt, interacting with tools for object perception and manipulation (e.g., pick, place, release). For precise spatial placement, such as interpreting "place the mug next to the plate", a separate sub-prompting module handles 3D
reasoning based on object geometry and scene layout. The system integrates YOLOX-GDRNet for object detection and pose estimation, along with a motion execution stub. We evaluated the system in 24 test scenarios, ranging from simple spatial commands to high-level instructions and infeasible requests. The system achieved an overall task success rate of 86%.
Read the full abstract ▾
All authors Karolina Źróbek, Tessa Pulli, Paweł Gajewski, Antonio Galiza Cerdeira Gonzalez, Bipin Indurkhya
Robotics
Anatomical Landmark-Guided Deep Reinforcement Learning for Autonomous Gastric Navigation
Haoxuan Wu, Sishen Yuan, Haitao Gao et al. (+3 more)
Wireless capsule endoscopy (WCE) enables painless visualization of the gastrointestinal tract, but its diagnostic potential is limited by incomplete mucosal coverage and poor transferability of existing
navigation methods across patient anatomies. We propose a transferable, anatomical landmarkguided deep reinforcement learning (AL-DRL) framework for autonomous gastric
navigation. Leveraging a lightweight edgecontour-depth fusion module, our policy operates on stable, lowdimensional landmark coordinates rather than high-dimensional
video streams, effectively bridging the sim-to-real gap. In simulations across eight patient-derived models, the method achieves over 97% coverage within 50 seconds, significantly outperforming vanilla PPO, SAC, and DQN
agents. A two-stage sim-to-real pipeline with an adaptive dynamic programming controller actively mitigates physical disturbances. Ex-vivo experiments demonstrate a mean coverage of 87% and a 53% reduction in procedure time compared with expert manual control.
Read the full abstract ▾
All authors Haoxuan Wu, Sishen Yuan, Haitao Gao, Zhen Li, Xiuli Zuo, Hongliang Ren
Robotics
123D: Unifying Multi-Modal Autonomous Driving Data at Scale
Daniel Dauner, Valentin Charraut, Bastian Berle et al. (+10 more)
The pursuit of autonomous driving has produced one of the richest sensor data collections in all of robotics. However, its scale and diversity remain largely untapped. Each
dataset adopts different 2D and 3D modalities, such as cameras,
lidar, ego states, annotations, traffic lights, and HD maps, with different rates and synchronization schemes. They come in fragmented formats requiring complex dependencies that cannot natively coexist in the same development environment. Further, major inconsistencies in annotation conventions prevent training or measuring generalization across multiple
datasets. We present 123D, an open-source framework that unifies such multi-modal driving data through a single API. To handle synchronization, we store each modality as an independent timestamped event stream with no prescribed rate, enabling synchronous or asynchronous access across arbitrary
datasets. Using 123D, we consolidate eight real-world driving
datasets spanning 3,300 hours and 90,000 kilometers, together with a synthetic
dataset with configurable collection scripts, and provide tools for data analysis and visualization. We conduct a systematic study comparing annotation statistics and assessing each
dataset's pose and calibration accuracy. Further, we showcase two applications 123D enables: cross-
dataset 3D object detection transfer and reinforcement learning for planning, and offer recommendations for future directions. Code and documentation are available at https://github.com/kesai-labs/py123d.
Read the full abstract ▾
All authors Daniel Dauner, Valentin Charraut, Bastian Berle, Tianyu Li, Long Nguyen, Jiabao Wang, Changhui Jing, Maximilian Igl, Holger Caesar, Boris Ivanovic, Yiyi Liao, Andreas Geiger, Kashyap Chitta
Robotics · Cross-listed from cs.CV
6D Pose Estimation via Keypoint Heatmap Regression with RGB-D Residual Neural Networks
Ismail Aljosevic, Amir Masoud Almasi, Ana Parovic et al. (+1 more)
In this paper, we propose a modular framework for 6D pose estimation based on keypoint heatmap regression. Our approach combines YOLOv10m for object detection with a ResNet18-based network that predicts 2D heatmaps from RGB images. Keypoints extracted from these heatmaps are used to estimate the 6D object pose via the PnP RANSAC
algorithm. We compare different keypoint selection strategies to assess their impact on pose accuracy. Additionally, we extend the baseline by incorporating depth data using a cross-fusion architecture, which enables interaction between RGB and depth features at multiple stages. We further explore general training improvements, such as experimenting with activation functions and learning rate scheduling strategies to improve model performance. Our best RGB-only model achieved a mean ADD-based accuracy of 84.50%, while the RGB-D fusion model reached 92.41% on the LINEMOD
dataset. The code is available at https://github.com/ameermasood/HeatNet.
Read the full abstract ▾
All authors Ismail Aljosevic, Amir Masoud Almasi, Ana Parovic, Ashkan Shafiei
Robotics
Active Embodiment Identification with Reinforcement Learning for Legged Robots
Nico Bohlinger and Jan Peters
We present an active embodiment identification method for legged
robots that jointly learns information-seeking behavior and explicit embodiment prediction. Using a history-augmented URMA architecture, the method infers joint-level and global embodiment parameters through interaction with the environment in simulation across different morphologies.
Read the full abstract ▾
Robotics
Evaluation of an Actuated Spine in Agile Quadruped Locomotion
Nico Bohlinger, Piotr Kicki, Davide Tateo et al. (+2 more)
The spine plays a crucial role in the dynamic locomotion of quadrupedal animals, improving the stability, speed, and efficiency of their gait, especially for fast-paced and highly agile movements. Therefore, the spine is also a promising and natural way to extend the capabilities of quadruped
robots. This paper empirically investigates the benefits of an actuated spine for learning agile quadruped locomotion. We evaluate whether the use of the spine brings benefits in terms of high-speed running, climbing stairs, climbing high-angle slopes, hurdling, and crawling scenarios. We conducted an empirical study in MuJoCo simulation using the Silver Badger
robot from MAB Robotics with an actuated 1-DOF spine in the sagittal plane. The obtained results show that the use of the spine provides the
robot with increased agility and allows it to overcome higher stairs, steeper slopes, higher obstacles, and smaller passages.
Read the full abstract ▾
All authors Nico Bohlinger, Piotr Kicki, Davide Tateo, Krzysztof Walas, Jan Peters
Robotics
TAVIS: A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning
Giacomo Spigler
Active
vision – where a policy controls its own gaze during manipulation – has emerged as a key capability for imitation learning, with multiple independent systems demonstrating its benefits in the past year. Yet there is no shared
benchmark to compare approaches or quantify what active
vision contributes, on which task types, and under what conditions. We introduce TAVIS, evaluation infrastructure for active-
vision imitation learning, with two complementary task suites – TAVIS-Head (5 tasks, global search via pan/tilt necks) and TAVIS-Hands (3 tasks, local occlusion via wrist cameras) – on two
humanoid torso embodiments (GR1T2, Reachy2), built on IsaacLab. TAVIS provides three evaluation primitives: a paired headcam-vs-fixedcam protocol on identical demonstrations; GALT (Gaze-Action Lead Time), a novel metric grounded in cognitive science and HRI that quantifies anticipatory gaze in learned policies; and procedural ID/OOD splits. Baseline experiments with
Diffusion Policy and $\pi_0$ reveal that (i) active
vision generally helps, consistently across training runs, but benefits are task-conditional rather than uniform; (ii) multi-task policies degrade sharply under controlled distribution shifts on both suites; and (iii) imitation alone yields anticipatory gaze that lands on the task-relevant object, with lead times comparable to those of the human demonstrations, although head motion is less smooth than in the demonstrations, an effect of action chunking that success rate does not reveal. Code and evaluation scripts are released at https://github.com/spiglerg/tavis; demonstrations (LeRobot v3.0; ~2200 episodes) and trained baselines at https://huggingface.co/tavis-
benchmark.
Read the full abstract ▾
Robotics
AERO-VIS: Asynchronous Event-based Real-time Onboard Visual-Inertial SLAM
Yannick Burkhardt, Sebastián Barbas Laina, Simon Boche et al. (+2 more)
The robustness of event cameras to high dynamic range and motion blur holds the potential to improve visual odometry systems in challenging environments. Although their high temporal resolution does not require synchronous processing, most event-based odometry methods still run at fixed rates, which simplifies system design but restricts latency and throughput. In this work, we present AERO-VIS, a stereo event-inertial SLAM system with an integrated, data-driven, robust, and performance-optimized keypoint detector. By decoupling event preprocessing from state estimation, our architecture operates asynchronously at the maximum rate supported by available hardware, dynamically adapting to downstream runtime demands and ensuring low-latency, real-time performance. When deploying AERO-VIS on a
UAV, we achieve unprecedented accuracy in onboard event-based SLAM. These unique characteristics enable us to present the first purely event-based inertial SLAM system that demonstrates closed-loop
UAV control and large-scale state estimation while relying solely on onboard compute.
Videos of the experiments, source code, and additional results are available at https://ethz-mrl.github.io/AERO-VIS/.
Read the full abstract ▾
All authors Yannick Burkhardt, Sebastián Barbas Laina, Simon Boche, Leonard Freißmuth, Stefan Leutenegger
Robotics
Melding LLM and temporal logic for reliable human-swarm collaboration in complex scenarios
Junfeng Chen, Yuxiao Zhu, An Zhuo et al. (+6 more)
Robot swarms promise scalable assistance in complex and hazardous environments. Task planning lies at the core of human-swarm collaboration, translating the operator's intent into coordinated swarm actions and helping determine when validation or intervention is required during execution. In long-horizon missions under dynamic scenarios, however, reliable task planning becomes difficult to maintain: emerging events and changing conditions demand continual adaptation, and sustained operator oversight imposes substantial cognitive burden. Existing LLM-based planning tools can support plan generation, yet they remain susceptible to invalid task orderings and infeasible
robot actions, resulting in frequent manual adjustment. Here we introduce a neuro-symbolic framework for long-horizon human-swarm collaboration that tightly melds verifiable task planning with context-grounded LLM
reasoning. We formalize mission goals and operational rules as temporal logic formulas and admissible task orderings as task automata. Conditioned on these formal constraints and live perceptual context, LLMs generate executable subtask sequences that satisfy mission rules and remain grounded in the current scene. An uncertainty-aware scheduler then assigns subtasks across the heterogeneous swarm to maximize parallelisms while remaining resilient to disruptions. An event-triggered interaction protocol further limits operator involvement to sparse, high-level confirmation and guidance. Deployment on a heterogeneous robotic fleet yields similar results while remaining robust to hardware-specific actuation and communication uncertainties. Together, these results support a formal and scalable paradigm for reliable and low-overhead human-swarm collaboration in dynamic environments
Read the full abstract ▾
All authors Junfeng Chen, Yuxiao Zhu, An Zhuo, Xintong Zhang, Shuo Zhang, Guanghui Wen, Xiwang Dong, Meng Guo, Zhongkui Li
Robotics
Many-to-Many Multi-Agent Pickup and Delivery
Ethan Schneider, Jingkai Chen, Tianyi Gu et al. (+3 more)
Multi-
robot systems in automated warehouses must manage continuous streams of pickup-and-delivery tasks while ensuring efficiency and safety. Prior work on Multi-
Agent Pickup-and-Delivery (MAPD) has largely focused on the one-to-one variant, where each task has a fixed pickup and delivery location. In contrast, real warehouses often present many-to-many MAPD scenarios, where items, tracked by stock keeping unit (SKU) identifiers, can be retrieved from or stored at multiple locations, resulting in an NP-hard four-dimensional assignment problem. To solve the many-to-many MAPD problem, we contribute our
algorithm: Many-to-Many Multi-
Agent Pickup and Delivery (M2M). We experiment with two variants of our
algorithm: one that minimizes estimated task durations (M2M), and one which incorporates SKU distribution into the objective function (M2M-wSKU). Simulation results over 8-hour warehouse operations show that our method consistently matches or outperforms prior state of the art, with M2M completing up to 22,000 more tasks on average across different environments and warehouse inventory densities.
Read the full abstract ▾
All authors Ethan Schneider, Jingkai Chen, Tianyi Gu, Kunlei Lian, Seth Hutchinson, Sonia Chernova
Robotics · Cross-listed from cs.CV
Text-to-CAD Evaluation with CADTests
Dimitrios Mallis, Marco Wang, Ahmet Serdar Karadeniz et al. (+3 more)
Text-to-CAD has recently emerged as an important task with the potential to substantially accelerate design workflows. Despite its significance, there has been surprisingly little work on Text-to-CAD evaluation, and assessing CAD model generation performance remains a considerable challenge. In this work, we introduce a new evaluation perspective for Text-to-CAD based on automated testing. We propose CADTestBench, the first test-based
benchmark for Text-to-CAD, based on CADTests, executable software tests that verify whether a generated CAD model satisfies the
geometric and topological requirements of the input
prompt. Using CADTestBench, we conduct comprehensive benchmarking of recent Text-to-CAD methods and further demonstrate that CADTests can also guide CAD model generation, yielding simple baselines that surpass performance of current methods. CADTestBench code and data are available at GitHub and Hugging Face
dataset.
Read the full abstract ▾
All authors Dimitrios Mallis, Marco Wang, Ahmet Serdar Karadeniz, Elisa Ricci, Anis Kacem, Djamila Aouada
Robotics
NoiseGate: Learning Per-Latent Timestep Schedules as Information Gating in World Action Models
Wen Huang, Haoran Sun, Yongjian Guo et al. (+8 more)
World Action Models (WAMs) are an emerging family of policies that tie
robot action generation to future-observation modeling. In this work, we focus on the joint
video–action modeling paradigm, where actions and imagined future observations are co-generated along a shared denoising or flow trajectory, so that perception, prediction, and control are coupled within one generative process. Existing WAMs typically realize this paradigm with a Mixture-of-
Transformers (MoT), where
video and action
tokens interact through shared self-
attention. This architecture can in principle assign a separate timestep $t_f$ to each predicted latent frame, yet current systems collapse this degree of freedom onto a single shared scalar $t$. Under the noise-as-masking view of
Diffusion Forcing, this shared schedule imposes the unjustified prior that every predicted latent is equally reliable for action generation. We instead view the per-latent schedule as a learnable information-gating policy: by changing a latent frame's noise level, the policy modulates the reliability of its Key/Value contribution to the action
tokens. We propose NoiseGate, which combines independent per-latent timestep sampling during backbone training, a lightweight Gating Policy Network that emits per-latent time increments during denoising, and task-reward optimization that trains the schedule policy without hand-crafted shape priors. Built on a joint
video–action MoT backbone, NoiseGate delivers consistent gains on diverse RoboTwin random-scene manipulation tasks.
Read the full abstract ▾
All authors Wen Huang, Haoran Sun, Yongjian Guo, Yunxuan Ma, Haoran Li, Jing Long, Zhouying Mo, Zhong Guan, Yucheng Guo, Shuai Di, Junwu Xiong
Robotics
Sensitivity-Based Robust NMPC for Close-Proximity Offshore Wind Turbine Inspection with a Tilted Multirotor
Giuseppe Silano and Martin Saska
Close-proximity offshore wind turbine inspection requires strict clearance control around large cylindrical structures under wind and model mismatch. Nominal Nonlinear Model Predictive Control (NMPC) may violate safety constraints when mass, inertia, thrust effectiveness, drag, or wind conditions differ from nominal assumptions. We propose a sensitivity-based robust NMPC for a tilted multirotor that robustifies the tower-clearance constraint via online constraint tightening. First-order parametric state sensitivities provide a structured-uncertainty margin, while bounded gusts are handled by a stage-dependent additive margin. The formulation augments the nominal NMPC with sensitivity propagation and margin evaluation only, leaving the receding-horizon optimization structure unchanged. Monte-Carlo evaluation over 500 uncertainty realizations on a boundary-critical helical inspection trajectory shows that the proposed controller eliminates the clearance violations observed under nominal NMPC at the cost of a moderate increase in solve time.
Read the full abstract ▾
Robotics
CommandSwarm: Safety-Aware Natural Language-to-Behavior-Tree Generation for Robotic Swarms
Mohammed Majid and Amjad Yousef Majid
Natural-
language interfaces can make swarm robotics more accessible to non-expert operators, but they must translate ambiguous user intent into executable swarm behaviors without unsupported actions, malformed programs, or unsafe plans. This paper presents CommandSwarm, a safety-aware
language-to-behavior-tree pipeline for generating XML behavior trees (BTs) from
speech or text commands. The system combines
multilingual translation, command-level safety filtering, constrained prompting, a LoRA-adapted large
language model (LLM), and deterministic parser validation against a whitelist of executable swarm primitives. We evaluate eleven open 6.7B–14B parameter LLMs, all using 4-bit quantization, on representative swarm-control scenarios under zero-shot, one-shot, and two-shot prompting. Falcon3-Instruct-10B and Mistral-7B-v3 are the strongest
prompt-engineered candidates, reaching BLEU scores above 0.60 and high syntactic validity in few-shot settings. LoRA adaptation of Falcon3-Instruct-10B on a 2,063-example synthetic instruction–BT corpus improves zero-shot BLEU from 0.267 to 0.663, ROUGE-L from 0.366 to 0.692, and parser-accepted syntactic validity from 0% to 72%. Translation experiments further show that SeamlessM4T v2-large and EuroLLM-9B provide the best quality-latency trade-offs for the
multilingual front end. The results indicate that compact, quantized, domain-adapted LLMs can generate useful swarm BTs when embedded in a validated systems pipeline. They also show that parser acceptance and safety filtering remain necessary execution gates; generation quality alone is not sufficient for autonomous deployment.
Read the full abstract ▾
Robotics
Offline-Online Hierarchical 3D Global Relocalization With Synthetic LiDAR Sensing and Descriptor-Space Retrieval
Jiahua Ren, Kai Shen, Muhua Zhang et al. (+1 more)
3D global relocalization is one of the key capabilities for mobile
robots in practical applications. However, in large scale spaces, existing methods often suffer from prolonged online relocalization time due to factors such as the massive pose search space and high computational overhead. To address these issues, this paper proposes an offline-online hierarchical framework that decouples the search space. In the offline phase, candidate positions and their corresponding
geometric descriptor indices are generated in the map by simulating
LiDAR scans within the grid map. In the online phase, a coarse pose estimate is first obtained via global
retrieval, followed by point cloud registration to output precise 6-DoF pose estimates. Real-world experiments demonstrate that the proposed method achieves an average relocalization time of 3 s and an average localization accuracy of 8 cm in 3D environments. Compared with existing global relocalization methods, the proposed method achieves an order-of-magnitude improvement in computational efficiency while delivering comparable relocalization accuracy.
Read the full abstract ▾
All authors Jiahua Ren, Kai Shen, Muhua Zhang, Lei Ma
Robotics · Cross-listed from cs.LG
Drifting Field Policy: A One-Step Generative Policy via Wasserstein Gradient Flow
Juil Koo, Mingue Park, Jiwon Choi et al. (+2 more)
We propose Drifting Field Policy (DFP), a non-ODE one-step generative policy built on the drifting model paradigm. We frame the policy update as a reverse-KL Wasserstein-2 gradient flow toward a soft target policy, so that each DFP update corresponds to a gradient step in probability space. By construction, this gradient is decomposed into an ascent toward higher action-value regions and a score matching with the anchor policy as a trust region. We further derive a simple, tractable surrogate of the otherwise intractable update loss, akin to behavior cloning on top-K critic-selected actions. We find empirically that this mechanism uniquely benefits the drifting backbone owing to its non-ODE parameterization. With one-step inference, DFP achieves state-of-the-art performance on several manipulation tasks across Robomimic and OGBench, outperforming ODE-based policies.
Read the full abstract ▾
All authors Juil Koo, Mingue Park, Jiwon Choi, Yunhong Min, Minhyuk Sung
Robotics · Cross-listed from cs.AI
Finite-Time Analysis of MCTS in Continuous POMDP Planning
Da Kong and Vadim Indelman
This paper presents a finite-time analysis for Monte Carlo Tree Search (MCTS) in Partially Observable Markov Decision Processes (POMDPs), with probabilistic concentration bounds in both discrete and continuous observation spaces. While MCTS-style solvers such as POMCP achieve empirical success in many applications, rigorous finite-time guarantees remain an open problem due to the nonstationarity and the interdependencies induced by heuristic action selection (e.g., UCB). In the discrete setting, we address these challenges by extending the
polynomial exploration bonus to UCB in POMDP setting, yielding
polynomial concentration bounds for the empirical value estimation at the root node. For continuous observation spaces, we introduce an abstract partitioning framework and propose a finite-time bound on partitioning loss. Under mild conditions, we prove highprobability bound on value estimates in POMDPs with continuous observation space. Specifically, we propose Voro-POMCPOW, a variant of POMCPOW with f inite-time guarantees that adaptively partitions the continuous observation space using Voronoi
cells. This approach maintains a finite branching factor while preserving the original observation generator. Empirical validation demonstrates that the proposed Voro-POMCPOW shows competitive performance while providing theoretical guarantees. Although our analysis focuses on continuous POMDPs, the techniques developed herein are also applicable to continuous MDPs, closing another gap on the MDP side.
Read the full abstract ▾
Robotics
PhySPRING: Structure-Preserving Reduction of Physics-Informed Twins via GNN
Yixiong Jing, Xingyuan Chen, Guangming Wang et al. (+3 more)
Physics-based digital twins aim to predict the dynamics of real-world objects under interaction, enabling real-to-sim-to-real applications in robotics. Current approaches reconstruct such twins as explicit physical models (such as spring-mass systems) to predict the dynamics, but the resulting models often inherit the resolution of the visual reconstruction rather than being reduced to the physical complexity required to reproduce task-relevant dynamics. This mismatch introduces redundant
topology, making repeated forward-dynamics rollouts unnecessarily expensive. To address this challenge, we present PhySPRING, an fully differentiable GNN-based method to reduce complexity in spring–mass digital twins. PhySPRING jointly learns a hierarchy of coarsened graph
topologies and their mechanical parameters from observations. At each reduction level, PhySPRING merges nodes with similar learned dynamic responses to optimize the
topology, while maintaining every reduced layer as an explicit spring–mass system. On the PhysTwin
benchmark, PhySPRING improves dense reconstruction and prediction accuracy over PhysTwin, while reduced models retain stable physical and visual fidelity with up to a 2.30 times speed-up. We further demonstrate the effectiveness of PhySPRING in a Real2Sim
robot policy-evaluation pipeline, where the reduced models are substituted zero-shot into ACT and $\pi_0$ evaluations, maintaining comparable manipulation success rates across downsampling levels while improving action-sampling effectiveness. Together, PhySPRING enables efficient and structure-preserving spring–mass reduction without sacrificing fidelity or robotic utility.
Read the full abstract ▾
All authors Yixiong Jing, Xingyuan Chen, Guangming Wang, Olaf Wysocki, Haibing Wu, Brian Sheil
Robotics · Cross-listed from cs.CV
Operating Within the Operational Design Domain: Zero-Shot Perception with Vision-Language Models
Berkehan Ünal, Hauke Dierend, Dren Fazlija et al. (+1 more)
Over the last few years, research on autonomous systems has matured to such a degree that the field is increasingly well-positioned to translate research into practical, stakeholder-driven use cases across well-defined domains. However, for a wide-scale practical adoption of autonomous systems, adherence to safety regulations is crucial. Many regulations are influenced by the Operational Design Domain (ODD), which defines the specific conditions in which an autonomous
agent can function. This is especially relevant for Automated Driving Systems (ADS), as a dependable perception of ODD elements is essential for safe implementation and
auditing.
Vision-
language models (VLMs) integrate visual recognition and
language reasoning, functioning without task-specific training data, which makes them suitable for adaptable ODD perception. To assess whether VLMs can function as zero-shot "ODD sensors" that adapt to evolving definitions, we contribute (i) an empirical study of zero-shot ODD classification and detection using four VLMs on a custom
dataset and Mapillary Vistas, along with failure analyses; (ii) an ablation of zero-shot optimization strategies with a cost-performance overview; and (iii) a suite of reusable prompting templates with guidance for adaptation. Our findings indicate that definition-anchored chain-of-thought prompting with persona decomposition performs best, while other methods may result in reduced recall. Overall, our results pave the way for transparent and effective ODD-based perception in safety-critical applications.
Read the full abstract ▾
All authors Berkehan Ünal, Hauke Dierend, Dren Fazlija, Christopher Plachetka
Robotics
BrickCraft: Visuomotor Skill Composition with Situated Manual Guidance for Long-Horizon Interlocking Brick Assembly
Jichuan Yu, Bowei Li, Zhenran Tang et al. (+4 more)
Autonomous robotic assembly of interlocking bricks demands seamless integration of long-horizon task
reasoning, spatial grounding, and fine-grained manipulation. This paper presents BrickCraft, a compositional framework designed for long-horizon and generalizable interlocking brick assembly. BrickCraft models the assembly process using a relative formulation, where each step is anchored to a reference brick within the partial structure, thereby decomposing complex tasks into a finite set of reusable primitive skills. BrickCraft bridges the gap between high-level assembly plans and physical execution through situated manuals, which provide explicit spatial guidance for learned visuomotor skills by projecting the assembly intent onto real-time
robot observations. Finally, BrickCraft employs a modular execution pipeline that chains these spatially grounded skills to accomplish long-horizon assembly tasks. Extensive experimental validations demonstrate that BrickCraft acquires proficient assembly skills from a limited set of demonstrations and effectively composes them for diverse long-horizon assembly tasks. The project website is available at https://intelligent-control-lab.github.io/BrickCraft.
Read the full abstract ▾
All authors Jichuan Yu, Bowei Li, Zhenran Tang, Guanxing Lu, Chuxiong Hu, Ruixuan Liu, Changliu Liu
Robotics
MemCompiler: Compile, Don't Inject – State-Conditioned Memory for Embodied Agents
Xin Ding, Xinrui Wang, Yifan Yang et al. (+9 more)
Existing
memory systems for
embodied agents typically inject retrieved
memory as static context at episode start, a paradigm we term Ahead-of-time Monolithic
Memory Injection (AMMI). However, this static design quickly becomes misaligned with the
agent's evolving state and may degrade lightweight executors below the no-
memory baseline. To address this, we propose MemCompiler, which reframes
memory utilization as State-Conditioned
Memory Compilation. A learned
Memory Compiler reads a structured Brief State capturing the
agent's current execution state and dynamically selects and compiles only relevant
memory into executable guidance. This guidance is delivered through a text channel and a latent Soft-Mem channel that preserves perceptual information not expressible in text. Across Alf World, EmbodiedBench, and ScienceWorld, MemCompiler consistently improves over no-
memory across open-source backbones (up to +129%), matches or approaches frontier closed-source systems, and reduces per-step latency by 60%, demonstrating that state-aware
memory compilation improves both effectiveness and efficiency.
Read the full abstract ▾
All authors Xin Ding, Xinrui Wang, Yifan Yang, Hao Wu, Shiqi Jiang, Qianxi Zhang, Liang Mi, Hanxin Zhu, Kun Li, Yunxin Liu, Zhibo Chen, Ting Cao
Robotics
How to Utilize Failure Demo Data?: Effective Data Selection for Imitation Learning Using Distribution Differences in Attention Mechanism
Kana Miyamoto, Kanata Suzuki and Tetsuya Ogata
Imitation learning for robotic tasks has relied primarily on policies trained only on successful demonstrations, although failures are unavoidable during human data collection. Many existing approaches for exploiting failure data require additional data processing or iterative policy updates through autonomous rollouts, making it difficult to directly and stably utilize failure data accumulated during data collection. In this work, we propose a method that learns latent representations of success-failure discrepancies and incorporates them into the
attention mechanism. During inference, an appropriate latent mode is selected from the initial observation to improve action stability. Furthermore, we introduce a post-training metric that quantifies the
attention discrepancy between each failure sample and successful demonstrations to select failure data. Simulation results show that the proposed method improves task success rates when trained with failure data and that the proposed metric identifies failure samples that are beneficial for learning when combined with successful demonstrations. These results suggest that the proposed method can support more efficient use of collected demonstrations in robotic data collection pipelines.
Read the full abstract ▾
Robotics
Search-based Robustness Testing of Laptop Refurbishing Robotic Software
Erblin Isaku, Hassan Sartaj, Shaukat Ali et al. (+2 more)
The Danish Technological Institute (DTI) focuses on transferring advanced technologies (including
robots) to the industry and the public sector. One key application is laptop refurbishment using specialized
robots, aimed at promoting reuse, reducing electronic waste, and supporting the European Circular Economy Action Plan. The software of such
robots often includes features that use object detection models to detect objects for various purposes, such as identifying screws for laptop disassembly or detecting stickers to remove them. Ensuring the robustness of such models to small input variations remains a critical challenge, and addressing it is important to avoid potential damage to laptops during refurbishment. In this paper, we propose PROBE, a search-based robustness testing approach that leverages multi-objective optimization to identify minimal, localized perturbations that expose failures in object detection models used in the software of laptop refurbishing
robots. PROBE employs NSGA-II to systematically explore the perturbation space, optimizing for failure induction considering both localization and confidence, and perturbation magnitude, while enabling the discovery of diverse failure cases. Results show that PROBE is 3$\times$ to 7$\times$ more effective than random search in generating failure-inducing perturbations, while requiring smaller perturbation magnitudes, and that the generated perturbations transfer across models. We further show that metamorphic relations provide additional insights into model robustness, enabling the assessment of stability even in non-failing cases.
Read the full abstract ▾
All authors Erblin Isaku, Hassan Sartaj, Shaukat Ali, Malaika Din Hashmi, Francois Picard
Robotics
Is the Future Compatible? Diagnosing Dynamic Consistency in World Action Models
Bo-Kai Ruan, Teng-Fang Hsiao, Ling Lo et al. (+1 more)
World Action Models (WAMs) enable decision-making through imagined rollouts by predicting future observations and actions. However, the reliability of these imagined futures remains under-examined: is a generated future merely visually plausible, or is it dynamically compatible with the action sequence it claims to model? In this work, we identify action-state consistency, the
alignment between predicted actions and induced state transitions, as a missing reliability axis for WAMs. Through a systematic study across representative joint-prediction and inverse-dynamics models, we find that action-state consistency systematically separates successful and failed rollouts across many tasks and follows similar success-failure trends as learned value estimates. These results suggest that consistency captures decision-relevant structure beyond visual realism. We further identify background collapse as an important boundary condition, where low-dynamics failed trajectories can become deceptively consistent because static futures are easier to predict. Building on these findings, we introduce a value-free consensus strategy for test-time selection, which ranks candidate rollouts by agreement among predicted futures. This strategy improves success rates on RoboCasa and RoboTwin 2.0 without additional training or reward modeling. Taken together, our findings establish action-state consistency as both a diagnostic tool for evaluating WAM reliability and a practical signal for value-free planning.
Read the full abstract ▾
All authors Bo-Kai Ruan, Teng-Fang Hsiao, Ling Lo, Hong-Han Shuai
Robotics
PathPainter: Transferring the Generalization Ability of Image Generation Models to Embodied Navigation
Yijin Wang, Yuru Tian, Xijie Huang et al. (+5 more)
Bird's-eye-view (BEV) images have been demonstrated to provide valuable prior information for
navigation. Given the global information provided by such views, two key challenges remain: how to fully exploit this information and how to reliably use it during execution. In this paper, we propose a
navigation system that uses BEV images as global priors and is designed for ground and near-ground robotic platforms. The system employs an image generation model to interpret human intent from natural
language, identify the target destination, and generate traversability masks. During execution, we introduce cross-view localization to align the
robot's odometry with the BEV map and mitigate long-term drift in conventional odometry. We conduct extensive
benchmark experiments to evaluate the proposed method and further validate its effectiveness on both
UAV and quadruped
robot platforms. Using only a conventional local motion planner, the quadruped
robot successfully completes a 285-meter outdoor long-range
navigation task. This work demonstrates how the world-understanding capabilities of foundation models can be transferred to
embodied navigation, enabling
robots to benefit from the strong generalization ability of existing image generation models.
Read the full abstract ▾
All authors Yijin Wang, Yuru Tian, Xijie Huang, Weiqi Gai, Mo Zhu, Xin Zhou, Yuze Wu, Fei Gao