Robotics · Cross-listed from cs.AI
General-purpose LLMs as Models of Human Driver Behavior: The Case of Simplified Merging
Samir H.A. Mohammad, Wouter Mooi and Arkady Zgonnikov
Human behavior models are essential as behavior references and for simulating human
agents in virtual safety assessment of automated vehicles (AVs), yet current models face a trade-off between interpretability and flexibility. General-purpose large
language models (LLMs) offer a promising alternative: a single model potentially deployable without parameter fitting across diverse scenarios. However, what LLMs can and cannot capture about human driving behavior remains poorly understood. We address this gap by
embedding two general-purpose LLMs (OpenAI o3 and Google Gemini 2.5 Pro) as standalone, closed-loop driver
agents in a simplified one-dimensional merging scenario and comparing their behavior against human data using quantitative and qualitative analyses. Both models reproduce human-like intermittent operational control and tactical dependencies on spatial cues. However, neither consistently captures the human response to dynamic velocity cues, and safety performance diverges sharply between models. A systematic
prompt ablation study reveals that
prompt components act as model-specific inductive biases that do not transfer across LLMs. These findings suggest that general-purpose LLMs could potentially serve as standalone, ready-to-use human behavior models in AV evaluation pipelines, but future research is needed to better understand their failure modes and ensure their validity as models of human driving behavior.
Read the full abstract ▾
Robotics · Cross-listed from cs.MA
Decoupling Geometric Planning and Execution in Scalable Multi-Agent Path Finding
Fernando Salanova, Eduardo Montijano and Cristian Mahulea
Multi-
Agent Path Finding (MAPF) requires collision-free trajectories for multiple
agents on a shared graph, often with the objective of minimizing the sum-of-costs (SOC). Many optimal and bounded-suboptimal solvers rely on time-expanded models and centralized conflict resolution, which limits scalability in large or dense instances. We propose a hybrid prioritized framework that separates
geometric planning from execution-time conflict resolution. In the first stage,
Geometric Conflict Preemption (GCP) plans
agents sequentially with A* on the original graph while inflating costs for transitions entering vertices used by higher-priority paths, encouraging spatial detours without explicit time
reasoning. In the second stage, a Decentralized Local Controller (DLC) executes the
geometric paths using per-vertex FIFO authorization queues and inserts wait actions to avoid vertex and edge-swap conflicts. Experiments on standard
benchmark maps with up to 1000
agents show that the method scales with an near-linear runtime trend and attains a 100% success rate on instances satisfying the
geometric feasibility assumption. Page of the project: https://sites.google.com/unizar.es/multi-
agent-path-finding/home
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
COT-FM: Cluster-wise Optimal Transport Flow Matching
Chiensheng Chiang, Kuan-Hsun Tu, Jia-Wei Liao et al. (+2 more)
We introduce COT-FM, a general framework that reshapes the probability path in Flow Matching (FM) to achieve faster and more reliable generation. FM models often produce curved trajectories due to random or batchwise couplings, which increase discretization error and reduce sample quality. COT-FM fixes this by clustering target samples and assigning each cluster a dedicated source distribution obtained by reversing pretrained FM models. This divide-and-conquer strategy yields more accurate local transport and significantly straighter vector fields, all without changing the model architecture. As a plug-and-play approach, COT-FM consistently accelerates sampling and improves generation quality across 2D
datasets, image generation
benchmarks, and robotic manipulation tasks.
Read the full abstract ▾
All authors Chiensheng Chiang, Kuan-Hsun Tu, Jia-Wei Liao, Cheng-Fu Chou, Tsung-Wei Ke
Robotics
Vision-Based Hand Shadowing for Robotic Manipulation via Inverse Kinematics
Hendrik Chiche, Antoine Jamme, Trevor Rigoberto Martinez et al. (+1 more)
Teleoperation of low-cost robotic manipulators remains challenging due to the difficulty of retargeting human hand motion to
robot joint commands. We present an offline hand-shadowing inverse-kinematics (IK) retargeting pipeline driven by a single
egocentric RGB-D camera mounted on 3D-printed glasses. The pipeline detects 21 hand landmarks per hand using MediaPipe Hands, deprojects them into 3D via depth sensing, transforms them into the
robot coordinate frame, and solves a damped-least-squares IK problem to produce joint commands for the SO-ARM101
robot (5 arm + 1 gripper joints). A gripper controller maps thumb-index finger geometry to grasp aperture with a multi-level fallback hierarchy. Actions are previewed in a physics simulation before replay on the physical
robot. We evaluate the pipeline on a structured pick-and-place
benchmark (5-tile grid, 10 grasps per tile, 3 independent runs) achieving an 86.7% +/- 4.2% success rate, and compare it against four
vision-
language-action (VLA) policies (ACT, SmolVLA, pi_0.5, GR00T N1.5) trained on leader-follower teleoperation data. We provide a quantitative error analysis of the pipeline, reporting a mean IK position error of 36.4 mm, trajectory smoothness metrics showing 57-68% jerk reduction from EMA smoothing, and an ablation study over the smoothing parameter. We also test the pipeline in unstructured real-world environments (grocery store, pharmacy) and find that success is reduced to 9.3% due to hand occlusion by surrounding objects. To mitigate this, we integrate WiLoR as an alternative hand detector, achieving an 8% improvement in hand detection rate over MediaPipe, highlighting both the promise and current limitations of marker-free analytical retargeting.
Read the full abstract ▾
All authors Hendrik Chiche, Antoine Jamme, Trevor Rigoberto Martinez, Gabriel Gomes
Robotics
D-SLAMSpoof: An Environment-Agnostic LiDAR Spoofing Attack using Dynamic Point Cloud Injection
Rokuto Nagata, Kenji Koide, Kazuma Ikeda et al. (+2 more)
In this work, we introduce Dynamic SLAMSpoof (D-SLAMSpoof), a novel attack that compromises
LiDAR SLAM even in feature-rich environments. The attack leverages
LiDAR spoofing, which injects spurious measurements into
LiDAR scans through external
laser interference. By designing both spatial injection shapes and temporally coordinated dynamic injection patterns guided by scan-matching principles, D-SLAMSpoof significantly improves attack success rates in real-world, feature-rich environments such as urban areas and indoor spaces, where conventional
LiDAR spoofing methods often fail. Furthermore, we propose a practical defense method, ISD-SLAM, that relies solely on inertial dead reckoning signals commonly available in autonomous systems. We demonstrate that ISD-SLAM accurately detects
LiDAR spoofing attacks, including D-SLAMSpoof, and effectively mitigates the resulting position drift. Our findings expose inherent vulnerabilities in
LiDAR-based SLAM and introduce the first practical defense against
LiDAR-based SLAM spoofing using only standard onboard sensors, providing critical insights for improving the security and reliability of autonomous systems.
Read the full abstract ▾
All authors Rokuto Nagata, Kenji Koide, Kazuma Ikeda, Ozora Sako, Kentaro Yoshioka
Robotics
MirrorDrift: Actuated Mirror-Based Attacks on LiDAR SLAM
Rokuto Nagata, Kenji Koide, Kazuma Ikeda et al. (+3 more)
LiDAR SLAM provides high-accuracy localization but is fragile to point-cloud corruption because scan matching assumes
geometric consistency. Prior physical attacks on
LiDAR SLAM largely rely on
LiDAR spoofing via external signal injection, which requires sensor-specific timing knowledge and is increasingly mitigated by modern defense mechanisms such as timing obfuscation and injection rejection. In this work, we show that specular reflection offers an injection-free alternative and demonstrate an attack, MirrorDrift, that uses an actuated planar mirror to cause ghost points in
LiDAR scans and systematically bias scan-matching correspondences. MirrorDrift optimizes mirror placement,
alignment, and actuation. In simulation, it increases the average pose error (APE) by 6.1x over random placement, degrading three SLAM systems to 2.29-3.31 m mean APE. In real-world experiments on a modern
LiDAR with state-of-the-art interference mitigation, it induces localization errors of up to 6.03 m. To the best of our knowledge, this is the first successful SLAM-targeted attack against production-grade secure
LiDARs.
Read the full abstract ▾
All authors Rokuto Nagata, Kenji Koide, Kazuma Ikeda, Ozora Sako, Shion Horie, Kentaro Yoshioka
Robotics
Novelty Adaptation Through Hybrid Large Language Model (LLM)-Symbolic Planning and LLM-guided Reinforcement Learning
Hong Lu, Pierrick Lorang, Timothy R. Duggan et al. (+2 more)
In dynamic open-world environments, autonomous
agents often encounter novelties that hinder their ability to find plans to achieve their goals. Specifically, traditional symbolic planners fail to generate plans when the
robot's planning domain lacks the operators that enable it to interact appropriately with novel objects in the environment. We propose a neuro-symbolic architecture that integrates symbolic planning, reinforcement learning, and a large
language model (LLM) to learn how to handle novel objects. In particular, we leverage the common sense
reasoning capability of the LLM to identify missing operators, generate plans with the symbolic AI planner, and write reward functions to guide the reinforcement learning
agent in learning control policies for newly identified operators. Our method outperforms the state-of-the-art methods in operator discovery as well as operator learning in continuous robotic domains.Our webpage and code can be access here: https://helenlu66.github.io/hybridLLMguided/
Read the full abstract ▾
All authors Hong Lu, Pierrick Lorang, Timothy R. Duggan, Jivko Sinapov, Matthias Scheutz
Robotics · Cross-listed from cs.CV
Learning to Assist: Physics-Grounded Human-Human Control via Multi-Agent Reinforcement Learning
Yuto Shibata, Kashu Yamazaki, Lalit Jayanti et al. (+3 more)
Humanoid robotics has strong potential to transform daily service and caregiving applications. Although recent advances in general motion tracking within physics engines (GMT) have enabled virtual characters and
humanoid robots to reproduce a broad range of human motions, these behaviors are primarily limited to contact-less social interactions or isolated movements. Assistive scenarios, by contrast, require continuous awareness of a human partner and rapid adaptation to their evolving posture and dynamics. In this paper, we formulate the imitation of closely interacting, force-exchanging human-human motion sequences as a multi-
agent reinforcement learning problem. We jointly train partner-aware policies for both the supporter (assistant)
agent and the recipient
agent in a physics simulator to track assistive motion references. To make this problem tractable, we introduce a partner policies initialization scheme that transfers priors from single-human motion-tracking controllers, greatly improving exploration. We further propose dynamic reference retargeting and contact-promoting reward, which adapt the assistant's reference motion to the recipient's real-time pose and encourage physically meaningful support. We show that AssistMimic is the first method capable of successfully tracking assistive interaction motions on established
benchmarks, demonstrating the benefits of a multi-
agent RL formulation for physically grounded and socially aware
humanoid control.
Read the full abstract ▾
All authors Yuto Shibata, Kashu Yamazaki, Lalit Jayanti, Yoshimitsu Aoki, Mariko Isogawa, Katerina Fragkiadaki
Robotics
ADMM-based Continuous Trajectory Optimization in Graphs of Convex Sets
Lukas Pries, Jon Arrizabalaga, Zachary Manchester et al. (+1 more)
This paper presents a numerical solver for computing continuous trajectories in non-convex environments. Our approach relies on a customized implementation of the Alternating Direction Method of Multipliers (ADMM) built upon two key components: First, we parameterize trajectories as
polynomials, allowing the primal update to be computed in closed form as a minimum-control-effort problem. Second, we introduce the concept of a spatio-temporal allocation graph based on a mixed-integer formulation and pose the slack update as a shortest-path search. The combination of these ingredients results in a solver with several distinct advantages over the state of the art. By jointly optimizing over both discrete and continuous domains, our method accesses a larger search space than existing decoupled approaches, enabling the discovery of superior trajectories. Additionally, the solver's structural robustness ensures reliable convergence from naive initializations, removing the bottleneck of complex warm starting in non-convex environments.
Read the full abstract ▾
All authors Lukas Pries, Jon Arrizabalaga, Zachary Manchester, Markus Ryll
Robotics
Distributed Kalman–Consensus Filtering with Adaptive Uncertainty Weighting for Multi-Object Tracking in Mobile Robot Networks
Niusha Khosravi, Rodrigo Ventura and Meysam Basiri
This paper presents an implementation and evaluation of a Distributed Kalman–Consensus Filter (DKCF) for Multi-Object Tracking (MOT) in mobile
robot networks operating under partial observability and heterogeneous localization uncertainty. A key challenge in such systems is the fusion of information from
agents with differing localization quality, where frame misalignment can lead to inconsistent estimates, track duplication, and ghost tracks. To address this issue, we build upon the MOTLEE framework and retain its frame-
alignment methodology, which uses consistently tracked dynamic objects as transient landmarks to improve relative pose estimates between
robots. On top of this framework, we propose an uncertainty-aware adaptive consensus weighting mechanism that dynamically adjusts the influence of neighbor information based on the covariance of the transmitted estimates, thereby reducing the impact of unreliable data during distributed fusion. Local tracking is performed using a Kalman Filter (KF) with a Constant Velocity Model (CVM) and Global Nearest Neighbor (GNN) data association. simulation results demonstrate that adaptive weighting effectively protects local estimates from inconsistent data, yielding a MOTA improvement of 0.09 for
agents suffering from localization drift, although system performance remains constrained by communication latency.
Read the full abstract ▾
Robotics
A Causal Approach to Predicting and Improving Human Perceptions of Social Navigation Robots
Maximilian Diehl, Nathan Tsoi, Gustavo Chavez et al. (+2 more)
As mobile
robots are increasingly deployed in human environments, enabling them to predict how people perceive them is critical for socially adaptable
navigation. Predicting perceptions is challenging for two main reasons: (1) HRI prediction models must learn from limited data, and (2) the obtained models must be interpretable to enable safe and effective interactions. Interpretability is particularly important when a
robot is perceived as incompetent (e.g., when the
robot suddenly stops or rotates away from the goal), as it allows the
robot to explain its
reasoning and identify controllable factors to improve performance, requiring causal rather than associative
reasoning. To address these challenges, we propose a Causal
Bayesian Network designed to predict how people perceive a mobile
robot's competence and how they interpret its intent during
navigation. Additionally, we introduce a novel method to improve perceived
robot competence employing a combinatorial search, guided by the proposed causal model, to identify better
navigation behaviors. Our method enhances interpretability and generates counterfactual
robot motions while achieving comparable or superior predictive performance to state-of-the-art methods, reaching an F1-score of 0.78 and 0.75 for competence and intention on a binary scale. To further assess our method's ability to improve the perceived
robot competence, we conducted an online evaluation in which users rated
robot behaviors on a 5-point Likert scale. Our method statistically significantly increased the perceived competence of low-competent
robot behavior by 83%.
Read the full abstract ▾
All authors Maximilian Diehl, Nathan Tsoi, Gustavo Chavez, Karinne Ramirez-Amaro, Marynel Vázquez
Robotics · Cross-listed from eess.SY
Multi-Robot Multitask Gaussian Process Estimation and Coverage
Lai Wei, Andrew McDonald and Vaibhav Srivastava
Coverage control is essential for the optimal deployment of
agents to monitor or cover areas with sensory demands. While traditional coverage involves single-task
robots, increasing autonomy now enables multitask operations. This paper introduces a novel multitask coverage problem and addresses it for both the cases of known and unknown sensory demands. For known demands, we design a
federated multitask coverage
algorithm and establish its convergence properties. For unknown demands, we employ a multitask Gaussian Process (GP) framework to learn sensory demand functions and integrate it with the multitask coverage
algorithm to develop an adaptive
algorithm. We introduce a novel notion of multitask coverage regret that compares the performance of the adaptive
algorithm against an oracle with prior knowledge of the demand functions. We establish that our
algorithm achieves sublinear cumulative regret, and numerically illustrate its performance.
Read the full abstract ▾
Robotics
Robust Co-design Optimisation for Agile Fixed-Wing UAVs
Adrian Andrei Buda, Xavier Chen, Nicolò Botteghi et al. (+1 more)
Co-design optimisation of autonomous systems has emerged as a powerful alternative to sequential approaches by jointly optimising physical design and control strategies. However, existing frameworks often neglect the robustness required for autonomous systems navigating unstructured, real-world environments. For agile Unmanned Aerial Vehicles (
UAVs) operating at the edge of the flight envelope, this lack of robustness yields designs that are sensitive to perturbations and model mismatch. To address this, we propose a robust co-design framework for agile fixed-wing
UAVs that integrates parametric uncertainty and wind disturbances directly into the concurrent optimisation process. Our bi-level approach optimises physical design in a high-level loop while discovering nominal solutions via a constrained trajectory planner and evaluating performance across a
stochastic Monte Carlo ensemble using feedback LQR control. Validated across three agile flight missions, our strategy consistently outperforms deterministic baselines. The results demonstrate that our robust co-design strategy inherently tailors aerodynamic features, such as wing placement and aspect ratio, to achieve an optimal trade-off between mission performance and disturbance rejection.
Read the full abstract ▾
All authors Adrian Andrei Buda, Xavier Chen, Nicolò Botteghi, Urban Fasel
Robotics
ResWM: Residual-Action World Model for Visual RL
Jseen Zhang, Gabriel Adineera, Jinzhou Tan et al. (+1 more)
Learning predictive world models from raw visual observations is a central challenge in reinforcement learning (RL), especially for robotics and continuous control. Conventional model-based RL frameworks directly condition future predictions on absolute actions, which makes optimization unstable: the optimal action distributions are task-dependent, unknown a priori, and often lead to oscillatory or inefficient control. To address this, we introduce the Residual-Action World Model (ResWM), a new framework that reformulates the control variable from absolute actions to residual actions – incremental adjustments relative to the previous step. This design aligns with the inherent smoothness of real-world control, reduces the effective search space, and stabilizes long-horizon planning. To further strengthen the representation, we propose an Observation Difference Encoder that explicitly models the changes between adjacent frames, yielding compact latent dynamics that are naturally coupled with residual actions. ResWM is integrated into a Dreamer-style latent dynamics model with minimal modifications and no extra hyperparameters. Both imagination rollouts and policy optimization are conducted in the residual-action space, enabling smoother exploration, lower control variance, and more reliable planning. Empirical results on the DeepMind Control Suite demonstrate that ResWM achieves consistent improvements in sample efficiency, asymptotic returns, and control smoothness, significantly surpassing strong baselines such as Dreamer and TD-MPC. Beyond performance, ResWM produces more stable and energy-efficient action trajectories, a property critical for robotic systems deployed in real-world environments. These findings suggest that residual action modeling provides a simple yet powerful principle for bridging algorithmic advances in RL with the practical requirements of robotics.
Read the full abstract ▾
All authors Jseen Zhang, Gabriel Adineera, Jinzhou Tan, Jinoh Kim
Robotics · Cross-listed from cs.CV
RC-NF: Robot-Conditioned Normalizing Flow for Real-Time Anomaly Detection in Robotic Manipulation
Shijie Zhou, Bin Zhu, Jiarui Yang et al. (+3 more)
Recent advances in
Vision-
Language-Action (VLA) models have enabled
robots to execute increasingly complex tasks. However, VLA models trained through imitation learning struggle to operate reliably in dynamic environments and often fail under Out-of-Distribution (OOD) conditions. To address this issue, we propose
Robot-Conditioned Normalizing Flow (RC-NF), a real-time monitoring model for robotic anomaly detection and intervention that ensures the
robot's state and the object's motion trajectory align with the task. RC-NF decouples the processing of task-aware
robot and object states within the normalizing flow. It requires only positive samples for unsupervised training and calculates accurate robotic anomaly scores during inference through the probability density function. We further present LIBERO-Anomaly-10, a
benchmark comprising three categories of robotic anomalies for simulation evaluation. RC-NF achieves state-of-the-art performance across all anomaly types compared to previous methods in monitoring robotic tasks. Real-world experiments demonstrate that RC-NF operates as a plug-and-play module for VLA models (e.g., pi0), providing a real-time OOD signal that enables state-level rollback or task-level replanning when necessary, with a response latency under 100 ms. These results demonstrate that RC-NF noticeably enhances the robustness and adaptability of VLA-based robotic systems in dynamic environments.
Read the full abstract ▾
All authors Shijie Zhou, Bin Zhu, Jiarui Yang, Xiangyu Zhao, Jingjing Chen, Yu-Gang Jiang
Robotics
Thousand-GPU Large-Scale Training and Optimization Recipe for AI-Native Cloud Embodied Intelligence Infrastructure
Yongjian Guo, Yunxuan Ma, Haoran Sun et al. (+22 more)
Embodied intelligence is a key step towards Artificial General Intelligence (AGI), yet its development faces multiple challenges including data, frameworks, infrastructure, and evaluation systems. To address these issues, we have, for the first time in the industry, launched a cloud-based, thousand-GPU distributed training platform for
embodied intelligence, built upon the widely adopted LeRobot framework, and have systematically overcome bottlenecks across the entire pipeline. At the data layer, we have restructured the data pipeline to optimize the flow of
embodied training data. In terms of training, for the GR00T-N1.5 model, utilizing thousand-GPU clusters and data at the scale of hundreds of millions, the single-round training time has been reduced from 15 hours to just 22 minutes, achieving a 40-fold speedup. At the model layer, by combining variable-length FlashAttention and Data Packing, we have moved from sample redundancy to sequence integration, resulting in a 188% speed increase; {\pi}-0.5
attention optimization has accelerated training by 165%; and FP8 quantization has delivered a 140% speedup. On the infrastructure side, relying on high-performance storage, a 3.2T RDMA network, and a Ray-driven elastic AI data lake, we have achieved deep synergy among data, storage, communication, and computation. We have also built an end-to-end evaluation system, creating a closed loop from training to simulation to assessment. This framework has already been fully validated on thousand-GPU clusters, laying a crucial technical foundation for the development and application of next-generation autonomous intelligent
robots, and is expected to accelerate the arrival of the era of human-machine integration.
Read the full abstract ▾
All authors Yongjian Guo, Yunxuan Ma, Haoran Sun, Zhong Guan, Shuai Di, Jing Long, Wanting Xu, Xiaodong Bai, Wen Huang, Yucheng Guo, Chen Zhou, Qiming Yang, Mingxi Luo, Tianyun Zhao, Hedan Yang, Song Wang, Xiaomeng Tian, Xiaolong Xiang, Zhen Sun, Yu Wei, Luqiao Wang, Yuzhen Li, Chenfeng Gu, Junwu Xiong, Yicheng Gong
Robotics · Cross-listed from cs.AI
A Survey of Reasoning in Autonomous Driving Systems: Open Challenges and Emerging Paradigms
Kejin Yu, Yuhan Sun, Taiqiang Wu et al. (+5 more)
The development of high-level autonomous driving (AD) is shifting from perception-centric limitations to a more fundamental bottleneck, namely, a deficit in robust and generalizable
reasoning. Although current AD systems manage structured environments, they consistently falter in long-tail scenarios and complex social interactions that require human-like judgment. Meanwhile, the advent of large
language and
multimodal models (LLMs and MLLMs) presents a transformative opportunity to integrate a powerful cognitive engine into AD systems, moving beyond pattern matching toward genuine comprehension. However, a systematic framework to guide this integration is critically lacking. To bridge this gap, we provide a comprehensive review of this emerging field and argue that
reasoning should be elevated from a modular component to the system's cognitive core. Specifically, we first propose a novel Cognitive Hierarchy to decompose the monolithic driving task according to its cognitive and interactive complexity. Building on this, we further derive and systematize seven core
reasoning challenges, such as the responsiveness-
reasoning trade-off and social-game
reasoning. Furthermore, we conduct a dual-perspective review of the state-of-the-art, analyzing both system-centric approaches to architecting intelligent
agents and evaluation-centric practices for their validation. Our analysis reveals a clear trend toward holistic and interpretable "glass-box"
agents. In conclusion, we identify a fundamental and unresolved tension between the high-latency, deliberative nature of LLM-based
reasoning and the millisecond-scale, safety-critical demands of vehicle control. For future work, a primary objective is to bridge the symbolic-to-physical gap by developing verifiable neuro-symbolic architectures, robust
reasoning under uncertainty, and scalable models for implicit social negotiation.
Read the full abstract ▾
All authors Kejin Yu, Yuhan Sun, Taiqiang Wu, Ruixu Zhang, Zhiqiang Lin, Yuxin Meng, Junjie Wang, Yujiu Yang
Robotics
Edge-Assisted Multi-Robot Visual-Inertial SLAM with Efficient Communication
Xin Liu, Shuhuan Wen, Jing Zhao et al. (+2 more)
The integration of cloud computing and edge computing is an effective way to achieve global consistent and real-time multi-
robot Simultaneous Localization and Mapping (SLAM). Cloud computing effectively solves the problem of limited computing, communication and storage capacity of terminal equipment. However, limited bandwidth and extremely long communication links between terminal devices and the cloud result in serious performance degradation of multi-
robot SLAM systems. To reduce the computational cost of feature tracking and improve the real-time performance of the
robot, a lightweight SLAM method of optical flow tracking based on pyramid IMU prediction is proposed. On this basis, a centralized multi-
robot SLAM system based on a
robot-edge-cloud layered architecture is proposed to realize real-time collaborative SLAM. It avoids the problems of limited on-board computing resources and low execution efficiency of single
robot. In this framework, only the feature points and keyframe descriptors are transmitted and lossless encoding and compression are carried out to realize real-time remote information transmission with limited bandwidth resources. This design reduces the actual bandwidth occupied in the process of data transmission, and does not cause the loss of SLAM accuracy caused by data compression. Through experimental verification on the EuRoC
dataset, compared with the current most advanced local feature compression method, our method can achieve lower data volume feature transmission, and compared with the current advanced centralized multi-
robot SLAM scheme, it can achieve the same or better positioning accuracy under low computational load.
Read the full abstract ▾
All authors Xin Liu, Shuhuan Wen, Jing Zhao, Tony Z. Qiu, Hong Zhang
Robotics · Cross-listed from cs.CV
DynVLA: Learning World Dynamics for Action Reasoning in Autonomous Driving
Shuyao Shang, Bing Zhan, Yunfei Yan et al. (+9 more)
We propose DynVLA, a driving VLA model that introduces a new CoT paradigm termed Dynamics CoT. DynVLA forecasts compact world dynamics before action generation, enabling more informed and physically grounded decision-making. To obtain compact dynamics representations, DynVLA introduces a Dynamics Tokenizer that compresses future
evolution into a small set of dynamics
tokens. Considering the rich environment dynamics in interaction-intensive driving scenarios, DynVLA decouples ego-centric and environment-centric dynamics, yielding more accurate world dynamics modeling. We then train DynVLA to generate dynamics
tokens before actions through SFT and RFT, improving decision quality while maintaining latency-efficient inference. Compared to Textual CoT, which lacks fine-grained spatiotemporal understanding, and Visual CoT, which introduces substantial redundancy due to dense image prediction, Dynamics CoT captures the
evolution of the world in a compact, interpretable, and efficient form. Extensive experiments on NAVSIM, Bench2Drive, and a large-scale in-house
dataset demonstrate that DynVLA consistently outperforms Textual CoT and Visual CoT methods, validating the effectiveness and practical value of Dynamics CoT. Project Page: https://yaoyao-jpg.github.io/dynvla.
Read the full abstract ▾
All authors Shuyao Shang, Bing Zhan, Yunfei Yan, Yuqi Wang, Yingyan Li, Yasong An, Xiaoman Wang, Jierui Liu, Lu Hou, Lue Fan, Zhaoxiang Zhang, Tieniu Tan
Robotics
PPGuide: Steering Diffusion Policies with Performance Predictive Guidance
Zixing Wang, Devesh K. Jha, Ahmed H. Qureshi et al. (+1 more)
Diffusion policies have shown to be very efficient at learning complex, multi-modal behaviors for robotic manipulation. However, errors in generated action sequences can compound over time which can potentially lead to failure. Some approaches mitigate this by augmenting
datasets with expert demonstrations or learning predictive world models which might be computationally expensive. We introduce Performance Predictive Guidance (PPGuide), a lightweight, classifier-based framework that steers a pre-trained
diffusion policy away from failure modes at inference time. PPGuide makes use of a novel self-supervised process: it uses
attention-based multiple instance learning to automatically estimate which observation-action chunks from the policy's rollouts are relevant to success or failure. We then train a performance predictor on this self-labeled data. During inference, this predictor provides a real-time gradient to guide the policy toward more robust actions. We validated our proposed PPGuide across a diverse set of tasks from the Robomimic and MimicGen
benchmarks, demonstrating consistent improvements in performance.
Read the full abstract ▾
All authors Zixing Wang, Devesh K. Jha, Ahmed H. Qureshi, Diego Romeres
Robotics
Learning Adaptive Force Control for Contact-Rich Sample Scraping with Heterogeneous Materials
Cenk Cetin, Shreyas Pouli and Gabriella Pizzuto
The increasing demand for accelerated scientific discovery, driven by global challenges, highlights the need for advanced AI-driven robotics. Deploying robotic chemists in human-centric labs is key for the next horizon of autonomous discovery, as complex tasks still demand the dexterity of human scientists. Robotic manipulation in this context is uniquely challenged by handling diverse chemicals (granular, powdery, or viscous
liquids), under varying lab conditions. For example, humans use spatulas for scraping materials from vial walls. Automating this process is challenging because it goes beyond simple robotic insertion tasks and traditional lab automation, requiring the execution of fine-granular movements within a constrained environment (the sample vial). Our work proposes an adaptive control framework to address this, relying on a low-level Cartesian impedance controller for stable and compliant physical interaction and a high-level reinforcement learning
agent that learns to dynamically adjust interaction forces at the end-effector. The
agent is guided by perception feedback, which provides the material's location. We first created a task-representative simulation environment with a Franka Research 3
robot, a scraping tool, and a sample vial containing heterogeneous materials. To facilitate the learning of an adaptive policy and model diverse characteristics, the sample is modelled as a collection of spheres, where each sphere is assigned a unique dislodgement force threshold, which is procedurally generated using Perlin noise. We train an
agent to autonomously learn and adapt the optimal contact wrench for a sample scraping task in simulation and then successfully transfer this policy to a real robotic setup. Our method was evaluated across five different material setups, outperforming a fixed-wrench baseline by an average of 10.9%.
Read the full abstract ▾
Robotics
ContactExplorer: Contact Coverage-Guided Exploration for General-Purpose Dexterous Manipulation
Zixuan Liu, Ruoyi Qiao, Chenrui Tie et al. (+5 more)
Reinforcement learning explores effectively in domains such as Atari games,
navigation, and locomotion, where novelty over states or dynamics is a sufficient signal. In contrast, dexterous manipulation requires rich physical hand–object interactions, but existing methods often suffer from unstable contact-based novelty signals, inefficient distance novelty signals, or reliance on task-specific priors. We propose ContactExplorer, a general exploration method for dexterous manipulation tasks. ContactExplorer represents contact as the intersection between object surface points and hand keypoints, encouraging dexterous hands to discover diverse and novel contact patterns, namely which fingers contact which object regions. It maintains a contact counter conditioned on discretized object states obtained via learned hash codes. This counter is leveraged in two complementary ways: (1) a count-based contact coverage reward that promotes exploration of novel contact patterns, and (2) an energy-based reaching reward that guides the
agent toward under-explored contact regions. We evaluate ContactExplorer on seven contact-rich manipulation tasks and five dexterous hand embodiments. Experimental results show that ContactExplorer substantially improves sample efficiency and success rates over existing exploration methods, that it reduces the need for task-specific priors, and that it remains effective across hand embodiments and transfers to the real world. Project page is https://contact-explorer.github.io.
Read the full abstract ▾
All authors Zixuan Liu, Ruoyi Qiao, Chenrui Tie, Xuanwei Liu, Yunfan Lou, Chongkai Gao, Zhixuan Xu, Lin Shao
Robotics · Cross-listed from cs.SE
STADA: Specification-based Testing for Autonomous Driving Agents
Joy Saha, Trey Woodlief, Sebastian Elbaum et al. (+1 more)
Simulation-based testing has become a standard approach to validating autonomous driving
agents prior to real-world deployment. A high-quality validation campaign will exercise an
agent in diverse contexts comprised of varying static environments, e.g., lanes, intersections, signage, and dynamic elements, e.g., vehicles and pedestrians. To achieve this, existing test generation techniques rely on template-based, manually constructed, or random scenario generation. When applied to validate formally specified safety requirements, such methods either require significant human effort or run the risk of missing important behavior related to the requirement. To address this gap, we present STADA, a Specification-based Test generation framework for Autonomous Driving
Agents that systematically generates the space of scenarios defined by a formal specification expressed in temporal logic (LTLf). Given a specification, STADA constructs all distinct initial scenes, a diverse space of continuations of those scenes, and simulations that reflect the behaviors of the specification. Evaluation of STADA on a variety of LTLf specifications formalized in SCENEFLOW using three complementary coverage criteria demonstrates that STADA yields more than 2x higher coverage than the best baseline on the finest criteria and a 75% increase for the coarsest criteria. Moreover, it matches the coverage of the best baseline with 6 times fewer simulations. While set in the context of autonomous driving, the approach is applicable to other domains with rich simulation environments.
Read the full abstract ▾
All authors Joy Saha, Trey Woodlief, Sebastian Elbaum, Matthew B. Dwyer
Robotics · Cross-listed from cs.CV
Lifelong Imitation Learning with Multimodal Latent Replay and Incremental Adjustment
Fanqi Yu, Matteo Tiezzi, Tommaso Apicella et al. (+2 more)
We introduce a lifelong imitation learning framework that enables continual policy refinement across sequential tasks under realistic
memory and data constraints. Our approach departs from conventional experience replay by operating entirely in a
multimodal latent space, where compact representations of visual, linguistic, and
robot's state information are stored and reused to support future learning. To further stabilize adaptation, we introduce an incremental feature adjustment mechanism that regularizes the
evolution of task
embeddings through an angular margin constraint, preserving inter-task distinctiveness. Our method establishes a new state of the art in the LIBERO
benchmarks, achieving 10-17 point gains in AUC and up to 65% less forgetting compared to previous leading methods. Ablation studies confirm the effectiveness of each component, showing consistent gains over alternative strategies. The code is available at: https://github.com/yfqi/lifelong_mlr_ifa.
Read the full abstract ▾
All authors Fanqi Yu, Matteo Tiezzi, Tommaso Apicella, Cigdem Beyan, Vittorio Murino
Robotics
A gripper for flap separation and opening of sealed bags
Sergi Foix, Jaume Oriol, Carme Torras et al. (+1 more)
Separating thin, flexible layers that must be individually grasped is a common but challenging manipulation primitive for most off-the-shelf grippers. A prominent example arises in clinical settings: the opening of sterile flat pouches for the preparation of the operating room, where the first step is to separate and grasp the flaps. We present a novel gripper design and opening strategy that enables reliable flap separation and robust seal opening. This capability addresses a high-volume repetitive hospital procedure in which nurses manually open up to 240 bags per shift, a physically demanding task linked to musculoskeletal injuries. Our design combines an active dented-roller fingertip with compliant fingers that exploit environmental constraints to robustly grasp thin flexible flaps. Experiments demonstrate that the proposed gripper reliably grasps and separates sealed bag flaps and other thin-layered materials from the hospital, the most sensitive variable affecting performance being the normal force applied. When two copies of the gripper grasp both flaps, the system withstands the forces needed to open the seals robustly. To our knowledge, this is one of the first demonstrations of robotic assistance to automate this repetitive, low-value, but critical hospital task.
Read the full abstract ▾
All authors Sergi Foix, Jaume Oriol, Carme Torras, Júlia Borràs