Robotics
TIC-VLA: A Think-in-Control Vision-Language-Action Model for Robot Navigation in Dynamic Environments
Zhiyu Huang, Yun Zhang, Johnson Liu et al. (+3 more)
Robots in dynamic, human-centric environments must follow
language instructions while maintaining real-time reactive control.
Vision-
language-action (VLA) models offer a promising framework, but they assume temporally aligned
reasoning and control, despite semantic inference being inherently delayed relative to real-time action. We introduce Think-in-Control (TIC)-VLA, a latency-aware framework that explicitly models delayed semantic
reasoning during action generation. TIC-VLA defines a delayed semantic-control interface that conditions action generation on delayed
vision-
language semantic states and explicit latency metadata, in addition to current observations, enabling policies to compensate for asynchronous
reasoning. We further propose a latency-consistent training pipeline that injects
reasoning inference delays during imitation learning and online reinforcement learning, aligning training with asynchronous deployment. To support realistic evaluation, we present DynaNav, a physics-accurate, photo-realistic simulation suite for
language-guided
navigation in dynamic environments. Extensive experiments in simulation and on a real
robot show that TIC-VLA consistently outperforms prior VLA models while maintaining robust real-time control under multi-second
reasoning latency. Project website: https://ucla-mobility.github.io/TIC-VLA/
Read the full abstract ▾
All authors Zhiyu Huang, Yun Zhang, Johnson Liu, Rui Song, Chen Tang, Jiaqi Ma
Robotics
Relationship-Aware Hierarchical 3D Scene Graph for Task Reasoning
Albert Gassol Puigjaner, Angelos Zacharia and Kostas Alexis
Representing and understanding 3D environments in a structured manner is crucial for autonomous
agents to navigate and reason about their surroundings. While traditional Simultaneous Localization and Mapping (SLAM) methods generate metric reconstructions and can be extended to metric-semantic mapping, they lack a higher level of abstraction and relational
reasoning. To address this gap, 3D scene graphs have emerged as a powerful representation for capturing hierarchical structures and object relationships. In this work, we propose an enhanced hierarchical 3D scene graph that integrates open-vocabulary features across multiple abstraction levels and supports object-relational
reasoning. Our approach leverages a
Vision Language Model (VLM) to infer semantic relationships. Notably, we introduce a task
reasoning module that combines Large
Language Models (LLM) and a VLM to interpret the scene graph's semantic and relational information, enabling
agents to reason about tasks and interact with their environment more intelligently. We validate our method by deploying it on a quadruped
robot in multiple environments and tasks, highlighting its ability to reason about them.
Read the full abstract ▾
Robotics
World-Gymnast: Training Robots with Reinforcement Learning in a World Model
Ansh Kumar Sharma, Yixiang Sun, Ninghao Lu et al. (+3 more)
Robot learning from interacting with the physical world is fundamentally bottlenecked by the cost of physical interaction. The two alternatives, supervised finetuning (SFT) from expert demonstrations and reinforcement learning (RL) in a software-based simulator, are limited by the amount of expert data available and the sim-to-real gap for manipulation. With the recent emergence of world models learned from real-world
video-action data, we ask the question of whether training a policy in a world model can be more effective than supervised learning or software simulation in achieving better real-
robot performance. We propose World-Gymnast, which performs RL finetuning of a
vision-
language-action (VLA) policy by rolling out the policy in an action-conditioned
video world model and rewarding the rollouts with a
vision-
language model (VLM). On the Bridge
robot setup, World-Gymnast outperforms SFT by as much as 18x and outperforms software simulator by as much as 2x. More importantly, World-Gymnast demonstrates intriguing capabilities of RL with a world model, including training on diverse
language instructions and novel scenes from the world model, test-time training in a novel scene, and online iterative world model and policy improvement. Our results suggest learning a world model and training
robot policies in the cloud could be the key to bridging the gap between
robots that work in demonstrations and
robots that can work in anyone's household.
Read the full abstract ▾
All authors Ansh Kumar Sharma, Yixiang Sun, Ninghao Lu, Yunzhe Zhang, Jiarao Liu, Sherry Yang
Robotics
3D Foundation Model-Based Loop Closing for Decentralized Collaborative SLAM
Pierre-Yves Lajoie, Benjamin Ramtoula, Daniele De Martini et al. (+1 more)
Decentralized Collaborative Simultaneous Localization And Mapping (C-SLAM) techniques often struggle to identify map overlaps due to significant viewpoint variations among
robots. Motivated by recent advancements in 3D foundation models, which can register images despite large viewpoint differences, we propose a robust loop closing approach that leverages these models to establish inter-
robot measurements. In contrast to resource-intensive methods requiring full 3D reconstruction within a centralized map, our approach integrates foundation models into existing SLAM pipelines, yielding scalable and robust multi-
robot mapping. Our contributions include: (1) integrating 3D foundation models to reliably estimate relative poses from
monocular image pairs within decentralized C-SLAM; (2) introducing robust outlier mitigation techniques critical to the use of these relative poses; and (3) developing specialized pose graph optimization formulations that efficiently resolve scale ambiguities. We evaluate our method against state-of-the-art approaches, demonstrating improvements in localization and mapping accuracy, alongside significant gains in computational and
memory efficiency. These results highlight the potential of our approach for deployment in large-scale multi-
robot scenarios.
Read the full abstract ▾
All authors Pierre-Yves Lajoie, Benjamin Ramtoula, Daniele De Martini, Giovanni Beltrame
Robotics
Multi-Agent Monte Carlo Tree Search for Makespan-Efficient Object Rearrangement in Cluttered Spaces
Hanwen Ren, Junyong Kim, Aathman Tharmasanthiran et al. (+1 more)
Object rearrangement planning in complex, cluttered environments is a common challenge in warehouses, households, and rescue sites. Prior studies largely address monotone instances, whereas real-world tasks are often non-monotone-objects block one another and must be temporarily relocated to intermediate positions before reaching their final goals. In such settings, effective multi-
agent collaboration can substantially reduce the time required to complete tasks. This paper introduces Centralized, Asynchronous, Multi-
agent Monte Carlo Tree Search (CAM-MCTS), a novel framework for general-purpose makespan-efficient object rearrangement planning in challenging environments. CAM-MCTS combines centralized task assignment-where
agents remain aware of each other's intended actions to facilitate globally optimized planning-with an asynchronous task execution strategy that enables
agents to take on new tasks at appropriate time steps, rather than waiting for others, guided by a one-step look-ahead cost estimate. This design minimizes idle time, prevents unnecessary synchronization delays, and enhances overall system efficiency. We evaluate CAM-MCTS across a diverse set of monotone and non-monotone tasks in cluttered environments, demonstrating consistent reductions in makespan compared to strong baselines. Finally, we validate our approach on a real-world multi-
agent system under different configurations, further confirming its effectiveness and robustness.
Read the full abstract ▾
All authors Hanwen Ren, Junyong Kim, Aathman Tharmasanthiran, Ahmed H. Qureshi
Robotics
SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-body Manipulation
Mu Huang, Hui Wang, Kerui Ren et al. (+5 more)
Simulating deformable objects under rich interactions remains a fundamental challenge for real-to-sim
robot manipulation, with dynamics jointly driven by environmental effects and
robot actions. Existing simulators rely on predefined physics or data-driven dynamics without
robot-conditioned control, limiting accuracy, stability, and generalization. This paper presents SoMA, a 3D Gaussian Splat simulator for soft-body manipulation. SoMA couples deformable dynamics, environmental forces, and
robot joint actions in a unified latent neural space for end-to-end real-to-sim simulation. Modeling interactions over learned Gaussian splats enables controllable, stable long-horizon manipulation and generalization beyond observed trajectories without predefined physical models. SoMA improves resimulation accuracy and generalization on real-world
robot manipulation by 20%, enabling stable simulation of complex tasks such as long-horizon cloth folding.
Read the full abstract ▾
All authors Mu Huang, Hui Wang, Kerui Ren, Linning Xu, Yunsong Zhou, Mulin Yu, Bo Dai, Jiangmiao Pang
Robotics
PRISM: Performer RS-IMLE for Single-pass Multisensory Imitation Learning
Amisha Bhaskar, Pratap Tokekar, Stefano Di Cairano et al. (+1 more)
Robotic imitation learning typically requires models that capture
multimodal action distributions while operating at real-time control rates and accommodating multiple sensing modalities. Although recent generative approaches such as
diffusion models, flow matching, and Implicit Maximum Likelihood Estimation (IMLE) have achieved promising results, they often satisfy only a subset of these requirements. To address this, we introduce PRISM, a single-pass policy based on a batch-global rejection-sampling variant of IMLE. PRISM couples a temporal multisensory encoder (integrating RGB, depth, tactile, audio, and proprioception) with a linear-
attention generator using a Performer architecture. We demonstrate the efficacy of PRISM on a diverse real-world hardware suite, including loco-manipulation using a Unitree Go2 with a 7-DoF arm D1 and tabletop manipulation with a UR5 manipulator. Across challenging physical tasks such as pre-manipulation parking, high-precision insertion, and multi-object pick-and-place, PRISM outperforms state-of-the-art
diffusion policies by 10-25% in success rate while maintaining high-frequency (30-50 Hz) closed-loop control. We further validate our approach on large-scale simulation
benchmarks, including CALVIN, MetaWorld, and Robomimic. In CALVIN (10% data split), PRISM improves success rates by approximately 25% over
diffusion and approximately 20% over flow matching, while simultaneously reducing trajectory jerk by 20x-50x. These results position PRISM as a fast, accurate, and multisensory imitation policy that retains
multimodal action coverage without the latency of iterative sampling.
Read the full abstract ▾
All authors Amisha Bhaskar, Pratap Tokekar, Stefano Di Cairano, Alexander Schperberg
Robotics
Mapping-Guided Task Discovery and Allocation for Robotic Inspection of Underwater Structures
Marina Ruediger and Ashis G. Banerjee
Task generation for
underwater multi-
robot inspections without prior knowledge of existing geometry can be achieved and optimized through examination of simultaneous localization and mapping (SLAM) data. By considering hardware parameters and environmental conditions, a set of tasks is generated from SLAM meshes and optimized through expected keypoint scores and distance-based
pruning. In-water tests are used to demonstrate the effectiveness of the
algorithm and determine the appropriate parameters. These results are compared to simulated Voronoi partitions and boustrophedon patterns for inspection coverage on a model of the test environment. The key benefits of the presented task discovery method include adaptability to unexpected geometry and distributions that maintain coverage while focusing on areas more likely to present defects or damage.
Read the full abstract ▾
Robotics
TTT-Parkour: Rapid Test-Time Training for Perceptive Robot Parkour
Shaoting Zhu, Baijun Ye, Jiaxuan Wang et al. (+5 more)
Achieving highly dynamic parkour with
humanoid robots on complex terrains unseen during the pre-training stage remains a challenge in robotics. Although general locomotion policies demonstrate capabilities across broad terrain distributions, they often struggle with arbitrary and highly challenging environments. To overcome this limitation, we propose a real-to-sim-to-real framework for rapid test-time training (TTT): environment-specific policy fine-tuning in simulation before deployment on a newly encountered terrain. We adopt a two-stage end-to-end learning paradigm: a policy is first pre-trained on diverse procedurally generated terrains, followed by rapid fine-tuning on high-fidelity meshes reconstructed from real-world captures. Specifically, we develop an efficient, feed-forward, and high-fidelity geometry reconstruction pipeline using RGB-D inputs, ensuring both speed and quality during test-time training. We demonstrate that TTT-Parkour empowers
humanoid robots to master complex obstacles, including wedges, stakes, boxes, trapezoids, and narrow beams. The entire pipeline, including capture, reconstruction, and test-time training, takes approximately 10 minutes for most tested terrains. Experiments show improved traversal on the tested terrains after fine-tuning, with direct sim-to-real deployment and no further policy updates on the
robot.
Read the full abstract ▾
All authors Shaoting Zhu, Baijun Ye, Jiaxuan Wang, Jiakang Chen, Ziwen Zhuang, Linzhan Mou, Runhan Huang, Hang Zhao
Robotics · Cross-listed from cs.SE
Before Autonomy Takes Control: Software Testing in Robotics
Nils Chur, Thiago Santos de Moura, Argentina Ortega et al. (+4 more)
Robotic systems are complex and safety-critical software systems. As such, they need to be tested thoroughly. Unfortunately,
robot software is intrinsically hard to test compared to traditional software, mainly since the software needs to closely interact with hardware, account for uncertainty in its operational environment, handle disturbances, and act highly autonomously. However, given the large space in which
robots operate, anticipating possible failures when designing tests is challenging. This paper presents a mapping study by considering robotics testing papers and relating them to the software testing theory. We consider 247 robotics testing papers and map them to software testing, discussing the state-of-the-art software testing in robotics with an illustrated example, and discuss current challenges. Forming the basis to introduce both the robotics and software engineering communities to software testing challenges. Finally, we identify open questions and lessons learned.
Read the full abstract ▾
All authors Nils Chur, Thiago Santos de Moura, Argentina Ortega, Sven Peldszus, Thorsten Berger, Nico Hochgeschwender, Yannic Noller
Robotics
Bridging the Sim-to-Real Gap with multipanda_ros2: A Real-Time ROS2 Framework for Multimanual Systems
Jon Škerlj, Seongjin Bien, Abdeldjallil Naceri et al. (+1 more)
We present $multipanda\_ros2$, a novel open-source ROS2 architecture for multi-
robot control of Franka Robotics
robots. Leveraging ros2 control, this framework provides native ROS2 interfaces for controlling any number of
robots from a single process. Our core contributions address key challenges in real-time torque control, including interaction control and
robot-environment modeling. A central focus of this work is sustaining a 1kHz control frequency, a necessity for real-time control and a minimum frequency required by safety standards. Moreover, we introduce a controllet-feature design pattern that enables controller-switching delays of $\le 2$ ms, facilitating reproducible benchmarking and complex multi-
robot interaction scenarios. To bridge the simulation-to-reality (sim2real) gap, we integrate a high-fidelity MuJoCo simulation with quantitative metrics for both kinematic accuracy and dynamic consistency (torques, forces, and control errors). Furthermore, we demonstrate that real-world inertial parameter identification can significantly improve force and torque accuracy, providing a methodology for iterative physics refinement. Our work extends approaches from soft robotics to rigid dual-arm, contact-rich tasks, showcasing a promising method to reduce the sim2real gap and providing a robust, reproducible platform for advanced robotics research.
Read the full abstract ▾
All authors Jon Škerlj, Seongjin Bien, Abdeldjallil Naceri, Sami Haddadin
Robotics
Adaptive Control in Autonomous Driving via Real-Time Recurrent RL
Julian Lemmel, Felix Resch, Mónika Farsang et al. (+3 more)
We study online fine-tuning of pretrained control policies for autonomous driving using Real-Time Recurrent Reinforcement Learning (RTRRL), a
memory-efficient
algorithm that updates policy parameters at every time step without backpropagation through time. We extend RTRRL to support LrcSSM, a recently proposed nonlinear diagonal state-space model, and combine offline behavioral cloning with online RTRRL fine-tuning to adapt policies to distribution shifts at deployment. We validate the approach in the CarRacing simulation and on a 1:10-scale RoboRacer platform equipped with an event camera, where a pretrained policy is fine-tuned online during real-world line-following. To our knowledge, this is the first demonstration of online RL fine-tuning with event-camera observations on standard (non-spiking) hardware in closed-loop control. LrcSSM-based policies improve fastest and most consistently across both settings.
Read the full abstract ▾
All authors Julian Lemmel, Felix Resch, Mónika Farsang, Ramin Hasani, Daniela Rus, Radu Grosu
Robotics · Cross-listed from cs.CV
LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation
Bo Miao, Weijia Liu, Jun Luo et al. (+8 more)
Language-conditioned goal
navigation (LGN) requires
embodied agents to locate user-specified targets without step-by-step guidance. However, existing
benchmarks largely focus on category-level goals or rely on instance descriptions generated by
vision-
language models, which often contain ambiguities and semantic errors, limiting systematic and reliable evaluation. We introduce HieraNav, an open-vocabulary LGN task with goals at four hierarchical semantic levels: scene, room, region, and instance. We present
Language as a Map (LangMap), the first LGN
benchmark to enrich real-world indoor 3D scans with human-verified semantic annotations supporting tasks across all four goal levels. Built on HM3D using a contrastive annotation protocol that compares same-scene regions and instances, LangMap provides region labels and discriminative region and instance descriptions covering 414 object categories and contains over 18K tasks. Each target has concise and detailed descriptions, enabling evaluation across instruction styles. Automated and human evaluations validate our annotation quality: our descriptions improve text-to-view matching accuracy over GOAT-Bench's by 23 points on all shared annotated instances, and an independent human audit yields 92.5% unique-and-correct matches. We also propose PlaNaVid, an RGB-only baseline that combines Bounded Diverse
Memory with high-level planning to
prime a reactive policy for multi-goal
navigation, achieving top-tier success rates without depth, 3D scene representations, or object masks. Further analyses reveal that exploration and hierarchical disambiguation failures become more prominent at finer goal levels, while long-tail categories, small objects, distant targets, timely stopping, and multi-goal completion remain challenging.
Benchmark and code: https://bo-miao.github.io/LangMap
Read the full abstract ▾
All authors Bo Miao, Weijia Liu, Jun Luo, Lachlan Shinnick, Jian Liu, Thomas Hamilton-Smith, Yuhe Yang, Zijie Wu, Vanja Videnovic, Feras Dayoub, Anton van den Hengel
Robotics
Extending the Law of Intersegmental Coordination: Implications for Powered Prosthetic Controls
Elad Siman Tov and Nili E. Krausz
Powered prostheses are capable of providing net positive work to amputees and have advanced in the past two decades. However, reducing amputee metabolic cost of walking remains an open problem. The Law of Intersegmental Coordination (ISC) has been observed across gaits and previously implicated in energy expenditure of walking, yet it has rarely been analyzed or applied within the context of lower-limb amputee gait. This law states that the elevation angles of the thigh, shank and foot over the gait cycle covary. In this work, we developed a method to analyze intersegmental coordination for lower-limb 3D kinematic data, to simplify ISC analysis. Moreover, inspired by motor control, biomechanics and robotics literature, we used our method to extend ISC to a new law of coordination of moments. We find these Elevation Space Moments (ESM), and present results showing a moment-based coordination for able bodied gait. We also analyzed ISC for amputee gait with powered and passive prostheses, and found that while elevation angles remained planar, the ESM lacked planar coordination. We present an ISC-driven powered prosthetic control framework, using healthy coordination as a constraint to predict the shank angles/moments to compensate for alterations due to a passive foot. We developed the ISC3d toolbox that is freely available online, which may be used to compute kinematic and kinetic ISC in 3D. This provides a means to further study the role of coordination in gait and may help address fundamental questions of the neural control of human movement.
Read the full abstract ▾
Robotics · Cross-listed from eess.SP
Real-Time 2D LiDAR Object Detection Using Three-Frame RGB Scan Encoding
Soheil Behnam Roudsari, Alexandre S. Brandão and Felipe N. Martins
Indoor service
robots need perception that is robust, more
privacy-friendly than RGB
video, and feasible on embedded hardware. We present a camera-free 2D
LiDAR object detection pipeline that encodes short-term temporal context by stacking three consecutive scans as RGB channels, yielding a compact YOLOv8n input without occupancy-grid construction while preserving angular structure and motion cues. Evaluated in Webots across 160 randomized indoor scenarios with strict scenario-level holdout, the method achieves 98.4% mAP@0.5 (0.778 mAP@0.5:0.95) with 94.9% precision and 94.7% recall on four object classes. On a Raspberry Pi 5, it runs in real time with a mean post-warm-up end-to-end latency of 47.8ms per frame, including scan encoding and postprocessing. Relative to a closely related occupancy-grid
LiDAR-YOLO pipeline reported on the same platform, the proposed representation is associated with substantially lower reported end-to-end latency. Although results are simulation-based, they suggest that lightweight temporal encoding can enable accurate and real-time
LiDAR-only detection for embedded indoor robotics without capturing RGB appearance.
Read the full abstract ▾
Robotics
FD-VLA: Force-Distilled Vision-Language-Action Model for Contact-Rich Manipulation
Ruiteng Zhao, Wenshuo Wang, Yicheng Ma et al. (+4 more)
Force sensing is a crucial modality for
Vision-
Language-Action (VLA) frameworks, as it enables fine-grained perception and dexterous manipulation in contact-rich tasks. We present Force-Distilled VLA (FD-VLA), a novel framework that integrates force awareness into contact-rich manipulation without relying on physical force sensors. The core of our approach is a Force
Distillation Module (FDM), which distills force by mapping a learnable query
token, conditioned on visual observations and
robot states, into a predicted force
token aligned with the latent representation of actual force signals. During inference, this distilled force
token is injected into the pretrained VLM, enabling force-aware
reasoning while preserving the integrity of its
vision-
language semantics. This design provides two key benefits: first, it allows practical deployment across a wide range of
robots that lack expensive or fragile force-torque sensors, thereby reducing hardware cost and complexity; second, the FDM introduces an additional force-
vision-state fusion prior to the VLM, which improves cross-modal
alignment and enhances perception-action robustness in contact-rich scenarios. Surprisingly, our physical experiments show that the distilled force
token outperforms direct sensor force measurements as well as other baselines, which highlights the effectiveness of this force-distilled VLA approach.
Read the full abstract ▾
All authors Ruiteng Zhao, Wenshuo Wang, Yicheng Ma, Xiaocong Li, Francis E.H. Tay, Marcelo H. Ang Jr., Haiyue Zhu
Robotics
Frictional Contact Solving for Material Point Method
Etienne Ménager and Justin Carpentier
Accurately handling contact with friction remains a core bottleneck for Material Point Method (MPM), from reliable contact point detection to enforcing frictional contact laws (non-penetration, Coulomb friction, and maximum dissipation principle). In this paper, we introduce a frictional-contact pipeline for implicit MPM that is both precise and robust. During the collision detection phase, contact points are localized with particle-centric
geometric primitives; during the contact resolution phase, we cast frictional contact as a Nonlinear Complementarity Problem (NCP) over contact impulses and solve it with an Alternating Direction Method of Multipliers (ADMM) scheme. Crucially, the formulation reuses the same implicit MPM linearization, yielding efficiency and numerical stability. The method integrates seamlessly into the implicit MPM loop and is agnostic to modeling choices, including material laws, interpolation functions, and transfer schemes. We evaluate it across seven representative scenes that span elastic and elasto-plastic responses, simple and complex deformable geometries, and a wide range of contact conditions. Overall, the proposed method enables accurate contact localization, reliable frictional handling, and broad generality, making it a practical solution for MPM-based simulations in robotics and related domains.
Read the full abstract ▾
Robotics
Bandwidth-Efficient Multi-Agent Communication through Information Bottleneck and Vector Quantization
Ahmad Farooq and Kamran Iqbal
Multi-
agent reinforcement learning systems deployed in real-world robotics applications face severe communication constraints that significantly impact coordination effectiveness. We present a framework that combines information bottleneck theory with vector quantization to enable selective, bandwidth-efficient communication in multi-
agent environments. Our approach learns to compress and discretize communication messages while preserving task-critical information through principled information-theoretic optimization. We introduce a gated communication mechanism that dynamically determines when communication is necessary based on environmental context and
agent states. Experimental evaluation on challenging coordination tasks demonstrates that our method achieves 181.8% performance improvement over no-communication baselines while reducing bandwidth usage by 71.4%. Pareto frontier analysis shows dominance across the entire success-bandwidth spectrum, with an area under the curve of 0.198 vs 0.142 for next-best methods. Our approach significantly outperforms existing communication strategies and establishes a theoretically grounded framework for deploying multi-
agent systems in bandwidth-constrained environments such as robotic swarms, autonomous vehicle fleets, and distributed sensor networks.
Read the full abstract ▾
Robotics
Synchronized Online Friction Estimation and Adaptive Grasp Control for Robust Gentle Grasp
Zhenwei Niu, Xiaoyi Chen, Jiayu Hu et al. (+3 more)
We introduce a unified framework for gentle robotic grasping that synergistically couples real-time friction estimation with adaptive grasp control. We propose a new particle filter-based method for real-time estimation of the friction coefficient using
vision-based tactile sensors. This estimate is seamlessly integrated into a reactive controller that dynamically modulates grasp force to maintain a stable grip. The two processes operate synchronously in a closed-loop: the controller uses the current best estimate to adjust the force, while new tactile feedback from this action continuously refines the estimation. This creates a highly responsive and robust sensorimotor cycle. The reliability and efficiency of the complete framework are validated through extensive robotic experiments.
Read the full abstract ▾
All authors Zhenwei Niu, Xiaoyi Chen, Jiayu Hu, Zhaoyang Liu, Tang Jian, Xiaozu Ju
Robotics
Reformulating AI-based Multi-Object Relative State Estimation for Aleatoric Uncertainty-based Outlier Rejection of Partial Measurements
Thomas Jantos, Giulio Delama, Stephan Weiss et al. (+1 more)
Precise localization with respect to a set of objects of interest enables mobile
robots to perform various tasks. With the rise of edge devices capable of deploying deep neural networks (DNNs) for real-time inference, it stands to reason to use artificial intelligence (AI) for the extraction of object-specific, semantic information from raw image data, such as the object class and the relative six degrees of freedom (6-DoF) pose. However, fusing such AI-based measurements in an Extended Kalman Filter (EKF) requires quantifying the DNNs' uncertainty and outlier rejection capabilities. This paper presents the benefits of reformulating the measurement equation in AI-based, object-relative state estimation. By deriving an EKF using the direct object-relative pose measurement, we can decouple the position and rotation measurements, thus limiting the influence of erroneous rotation measurements and allowing partial measurement rejection. Furthermore, we investigate the performance and consistency improvements for state
estimators provided by replacing the fixed measurement covariance
matrix of the 6-DoF object-relative pose measurements with the predicted aleatoric uncertainty of the DNN.
Read the full abstract ▾
All authors Thomas Jantos, Giulio Delama, Stephan Weiss, Jan Steinbrener
Robotics
A Unified Control Architecture for Macro-Micro Manipulation using a Active Remote Center of Compliance for Manufacturing Applications
Patrick Frank and Christian Friedrich
Macro-micro manipulators combine a macro manipulator with a large workspace, such as an industrial
robot, with a lightweight, high-bandwidth micro manipulator. This enables highly dynamic interaction control while preserving the wide workspace of the
robot. Traditionally, position control is assigned to the macro manipulator, while the micro manipulator handles the interaction with the environment, limiting the achievable interaction control bandwidth. To solve this, we propose a novel control architecture that incorporates the macro manipulator into the active interaction control. This leads to a increase in control bandwidth by a factor of 2.1 compared to the state of the art architecture, based on the leader-follower approach and factor 12.5 compared to traditional
robot-based force control. Further we propose surrogate models for a more efficient controller design and easy adaptation to hardware changes. We validate our approach by comparing it against the other control schemes in different experiments, like collision with an object, following a force trajectory and industrial assembly tasks.
Read the full abstract ▾
Robotics
Towards Exploratory and Focused Manipulation with Bimanual Active Perception: A New Problem, Benchmark and Strategy
Yuxin He, Ruihao Zhang, Tianao Shen et al. (+2 more)
Recently, active
vision has reemerged as an important concept for manipulation, since visual occlusion occurs more frequently when main cameras are mounted on the
robot heads. We reflect on the visual occlusion issue and identify its essence as the absence of information useful for task completion. Inspired by this, we come up with the more fundamental problem of Exploratory and Focused Manipulation (EFM). The proposed problem is about actively collecting information to complete challenging manipulation tasks that require exploration or focus. As an initial attempt to address this problem, we establish the EFM-10
benchmark that consists of 4 categories of tasks that align with our definition (10 tasks in total). We further come up with a Bimanual Active Perception (BAP) strategy, which leverages one arm to provide active
vision and another arm to provide force sensing while manipulating. Based on this idea, we collect a
dataset named BAPData for the tasks in EFM-10. With the
dataset, we successfully verify the effectiveness of the BAP strategy in an imitation learning manner. We hope that the EFM-10
benchmark along with the BAP strategy can become a cornerstone that facilitates future research towards this direction. Project website: EFManipulation.github.io.
Read the full abstract ▾
All authors Yuxin He, Ruihao Zhang, Tianao Shen, Cheng Liu, Qiang Nie
Robotics
LIEREx: Language-Image Embeddings for Robotic Exploration
Felix Igelbrink, Lennart Niecksch, Marian Renz et al. (+2 more)
Semantic maps allow a
robot to reason about its surroundings to fulfill tasks such as navigating known environments, finding specific objects, and exploring unmapped areas. Traditional mapping approaches provide accurate
geometric representations but are often constrained by pre-designed symbolic vocabularies. The reliance on fixed object classes makes it impractical to handle out-of-distribution knowledge not defined at design time. Recent advances in
Vision-
Language Foundation Models, such as CLIP, enable open-set mapping, where objects are encoded as high-dimensional
embeddings rather than fixed labels. In LIEREx, we integrate these VLFMs with established 3D Semantic Scene Graphs to enable target-directed exploration by an autonomous
agent in partially unknown environments.
Read the full abstract ▾
All authors Felix Igelbrink, Lennart Niecksch, Marian Renz, Martin Günther, Martin Atzmueller
Robotics
ForSim: Stepwise Forward Simulation for Traffic Policy Fine-Tuning
Keyu Chen, Wenchao Sun, Hao Cheng et al. (+2 more)
As the foundation of closed-loop training and evaluation in autonomous driving, traffic simulation still faces two fundamental challenges:
covariate shift introduced by open-loop imitation learning and limited capacity to reflect the
multimodal behaviors observed in real-world traffic. Although recent frameworks such as RIFT have partially addressed these issues through group-relative optimization, their forward simulation procedures remain largely non-reactive, leading to unrealistic
agent interactions within the virtual domain and ultimately limiting simulation fidelity. To address these issues, we propose ForSim, a stepwise closed-loop forward simulation paradigm. At each virtual timestep, the traffic
agent propagates the virtual candidate trajectory that best spatiotemporally matches the reference trajectory through physically grounded motion dynamics, thereby preserving
multimodal behavioral diversity while ensuring intra-modality consistency. Other
agents are updated with stepwise predictions, yielding coherent and interaction-aware
evolution. When incorporated into the RIFT traffic simulation framework, ForSim operates in conjunction with group-relative optimization to fine-tune traffic policy. Extensive experiments confirm that this integration consistently improves safety while maintaining efficiency, realism, and comfort. These results underscore the importance of modeling closed-loop
multimodal interactions within forward simulation and enhance the fidelity and reliability of traffic simulation for autonomous driving. Project Page: https://currychen77.github.io/ForSim/
Read the full abstract ▾
All authors Keyu Chen, Wenchao Sun, Hao Cheng, Zheng Fu, Sifa Zheng
Robotics
Multi-Task Learning for Robot Perception with Imbalanced Data
Ozgur Erkent
Multi-task problem solving has been shown to improve the accuracy of the individual tasks, which is an important feature for
robots, as they have a limited resource. However, when the number of labels for each task is not equal, namely imbalanced data exist, a problem may arise due to insufficient number of samples, and labeling is not very easy for mobile
robots in every environment. We propose a method that can learn tasks even in the absence of the ground truth labels for some of the tasks. We also provide a detailed analysis of the proposed method. An interesting finding is related to the interaction of the tasks. We show a methodology to find out which tasks can improve the performance of other tasks. We investigate this by training the teacher network with the task outputs such as depth as inputs. We further provide empirical evidence when trained with a small amount of data. We use semantic
segmentation and depth estimation tasks on different
datasets, NYUDv2 and Cityscapes.
Read the full abstract ▾
Robotics
Path Tracking with Dynamic Control Point Blending for Autonomous Vehicles: An Experimental Study
Alexandre Lombard, Florent Perronnet, Nicolas Gaud et al. (+1 more)
This paper presents an experimental study of a path-tracking framework for autonomous vehicles in which the lateral control command is applied to a dynamic control point along the wheelbase. Instead of enforcing a fixed reference at either the front or rear axle, the proposed method continuously interpolates between both, enabling smooth adaptation across driving contexts, including low-speed maneuvers and reverse motion. The lateral steering command is obtained by barycentric blending of two complementary controllers: a front-axle Stanley formulation and a rear-axle
curvature-based
geometric controller, yielding continuous transitions in steering behavior and improved tracking stability. In addition, we introduce a
curvature-aware longitudinal control strategy based on virtual track borders and ray-tracing, which converts upcoming
geometric constraints into a virtual obstacle distance and regulates speed accordingly. The complete approach is implemented in a unified control stack and validated in simulation and on a real autonomous vehicle equipped with GPS-RTK, radar, odometry, and IMU. The results in closed-loop tracking and backward maneuvers show improved trajectory accuracy, smoother steering profiles, and increased adaptability compared to fixed control-point baselines.
Read the full abstract ▾
All authors Alexandre Lombard, Florent Perronnet, Nicolas Gaud, Abdeljalil Abbas-Turki
Robotics
Giulio Antonio Abbo, Senne Lenaerts and Tony Belpaeme
In this work, we explore how
multimodal large
language models can support real-time context- and value-aware decision-making. To do so, we combine the GPT-4o
language model with a TurtleBot 4 platform simulating a smart
vacuum cleaning
robot in a home. The model evaluates the environment through
vision input and determines whether it is appropriate to initiate cleaning. The system highlights the ability of these models to reason about domestic activities, social norms, and user preferences and take nuanced decisions aligned with the values of the people involved, such as cleanliness, comfort, and safety. We demonstrate the system in a realistic home environment, showing its ability to infer context and values from limited visual input. Our results highlight the promise of
multimodal large
language models in enhancing robotic autonomy and situational awareness, while also underscoring challenges related to consistency, bias, and real-time performance.
Read the full abstract ▾
Robotics
BTGenBot-2: Efficient Behavior Tree Generation with Small Language Models
Riccardo Andrea Izzo, Gianluca Bardaro and Matteo Matteucci
Recent advances in
robot learning increasingly rely on LLM-based task planning, leveraging their ability to bridge natural
language with executable actions. While prior works showcased great performances, the widespread adoption of these models in robotics has been challenging as 1) existing methods are often closed-source or computationally intensive, neglecting the actual deployment on real-world physical systems, and 2) there is no universally accepted, plug-and-play representation for robotic task generation. Addressing these challenges, we propose BTGenBot-2, a 1B-parameter open-source small
language model that directly converts natural
language task descriptions and a list of
robot action primitives into executable behavior trees in XML. Unlike prior approaches, BTGenBot-2 enables zero-shot BT generation, error recovery at inference and runtime, while remaining lightweight enough for resource-constrained
robots. We further introduce the first standardized
benchmark for LLM-based BT generation, covering 52
navigation and manipulation tasks in NVIDIA Isaac Sim. Extensive evaluations demonstrate that BTGenBot-2 consistently outperforms GPT-5, Claude Opus 4.1, and larger open-source models across both functional and non-functional metrics, achieving average success rates of 90.38% in zero-shot and 98.07% in one-shot, while delivering up to 16x faster inference compared to the previous BTGenBot.
Read the full abstract ▾
Robotics
Onboard UAV State Estimation for Fast Flights Without External Localization Systems: A2RL Drone Racing Finalist Approach
Filip Novák, Matěj Petrlík, Matej Novosad et al. (+3 more)
Fast flights with aggressive maneuvers in cluttered GNSS-denied environments require fast, reliable, and accurate
UAV state estimation. In this paper, we present an approach for onboard state estimation of a high-speed
UAV using a
monocular RGB camera and an IMU. Our approach fuses data from Visual-Inertial Odometry (VIO), an onboard landmark-based camera measurement system, and an IMU to produce an accurate state estimate. Using onboard measurement data, we estimate and compensate for VIO drift through a novel mathematical drift model. State-of-the-art approaches often rely on more complex hardware (e.g., stereo cameras or rangefinders) and use uncorrected drifting VIO velocities, orientation, and angular rates, leading to errors during fast maneuvers. In contrast, our method corrects all VIO states (position, orientation, linear and angular velocity), resulting in accurate state estimation even during rapid and dynamic motion. Our approach was thoroughly validated through 1600 simulations and numerous real-world experiments. Furthermore, we applied the proposed method in the A2RL Drone Racing Challenge 2025, where our team advanced to the final four out of 210 teams and earned a medal.
Read the full abstract ▾
All authors Filip Novák, Matěj Petrlík, Matej Novosad, Parakh M. Gupta, Robert Pěnička, Martin Saska
Robotics
Concept-Based Dictionary Learning for Inference-Time Safety in Vision Language Action Models
Siqi Wen, Shu Yang, Shaopeng Fu et al. (+3 more)
Vision Language Action (VLA) models close the perception action loop by translating
multimodal instructions into executable behaviors, but this very capability magnifies safety risks: jailbreaks that merely yield toxic text in LLMs can trigger unsafe physical actions in
embodied systems. Existing defenses
alignment, filtering, or
prompt hardening intervene too late or at the wrong modality, leaving fused representations exploitable. We introduce a concept based dictionary learning framework for inference time safety control. By learning sparse, interpretable dictionaries from hidden activations, our method identifies harmful concept directions and attenuates risky components when the estimated risk exceeds a threshold. Experiments on Libero-Harm, BadRobot, RoboPair, and IS-Bench show that our approach achieves state-of-the-art defense performance, cutting attack success rates by over 70% while maintaining task success. Crucially, the framework is plug-in and model-agnostic, requiring no retraining and integrating seamlessly with diverse VLAs. To our knowledge, this is the first inference time concept based safety method for
embodied systems, advancing both interpretability and safe deployment of VLA models.
Read the full abstract ▾
All authors Siqi Wen, Shu Yang, Shaopeng Fu, Jingfeng Zhang, Lijie Hu, Di Wang