Robotics · Cross-listed from cs.CV
A Survey on Efficient Vision-Language-Action Models
Zhaoshu Yu, Bo Wang, Pengpeng Zeng et al. (+7 more)
Vision-
Language-Action models (VLAs) represent a significant frontier in
embodied intelligence, aiming to bridge digital knowledge with physical-world interaction. Despite their remarkable performance, foundational VLAs are hindered by the prohibitive computational and data demands inherent to their large-scale architectures. To this end, recent studies improve VLA efficiency from different views, e.g., real-time inference, training computation, and scalable data collection. However, these efforts are mostly studied separately. A unified view is still missing for understanding how efficiency should be optimized across the full VLA lifecycle. To bridge this gap, this survey presents the first comprehensive review of Efficient
Vision-
Language-Action models (Efficient VLAs) across the entire model-training-data pipeline. Specifically, we introduce a unified taxonomy to systematically organize the disparate efforts in this domain, categorizing current techniques into three core pillars: (1) Efficient Model Design, focusing on efficient architectures and model compression; (2) Efficient Training, which reduces computational burdens during model learning; and (3) Efficient Data Collection, which addresses the bottlenecks in acquiring and utilizing robotic data. Through a critical review of state-of-the-art methods within this framework, this survey provides an organized reference for the community and summarizes representative applications, delineates key challenges, and charts a roadmap for future research. We maintain a continuously updated project page to track our latest developments: https://evla-survey.github.io/.
Read the full abstract ▾
All authors Zhaoshu Yu, Bo Wang, Pengpeng Zeng, Haonan Zhang, Ji Zhang, Zheng Wang, Lianli Gao, Jingkuan Song, Nicu Sebe, Heng Tao Shen
Robotics
Adaptive Keyframe Selection for Scalable 3D Scene Reconstruction in Dynamic Environments
Raman Jha, Yang Zhou and Giuseppe Loianno
In this paper, we propose an adaptive keyframe selection method for improved 3D scene reconstruction in dynamic environments. The proposed method integrates two complementary modules: an error-based selection module utilizing photometric and structural similarity (SSIM) errors, and a momentum-based update module that dynamically adjusts keyframe selection thresholds according to scene motion dynamics. By dynamically curating the most informative frames, our approach addresses a key data bottleneck in real-time perception. This allows for the creation of high-quality 3D world representations from a compressed data stream, a critical step towards scalable
robot learning and deployment in complex, dynamic environments. Experimental results demonstrate significant improvements over traditional static keyframe selection strategies, such as fixed temporal intervals or uniform frame skipping. These findings highlight a meaningful advancement toward adaptive perception systems that can dynamically respond to complex and evolving visual scenes. We evaluate our proposed adaptive keyframe selection module on two recent state-of-the-art 3D reconstruction networks, Spann3r and CUT3R, and observe consistent improvements in reconstruction quality across both frameworks. Furthermore, an extensive ablation study confirms the effectiveness of each individual component in our method, underlining their contribution to the overall performance gains.
Read the full abstract ▾
Robotics
Stand, Walk, Navigate: Recovery-Aware Visual Navigation on a Low-Cost Wheeled Quadruped
Jans Solano and Diego Quiroz
Wheeled-legged
robots combine the efficiency of wheels with the obstacle negotiation of legs, yet many state-of-the-art systems rely on costly actuators and sensors, and fall-recovery is seldom integrated, especially for wheeled-legged morphologies. This work presents a recovery-aware visual-inertial
navigation system on a low-cost wheeled quadruped. The proposed system leverages
vision-based perception from a depth camera and deep reinforcement learning policies for robust locomotion and autonomous recovery from falls across diverse terrains. Simulation experiments show agile mobility with low-torque actuators over irregular terrain and reliably recover from external perturbations and self-induced failures. We further show goal directed
navigation in structured indoor spaces with low-cost perception. Overall, this approach lowers the barrier to deploying autonomous
navigation and robust locomotion policies in budget-constrained robotic platforms.
Read the full abstract ▾
Robotics · Cross-listed from cs.MA
Coordinated Autonomous Drones for Human-Centered Fire Evacuation in Partially Observable Urban Environments
Maria G. Mendoza, Addison Kalanther, Daniel Bostwick et al. (+3 more)
Autonomous drone technology holds significant promise for enhancing search and rescue operations during evacuations by guiding humans toward safety and supporting broader emergency response efforts. However, their application in dynamic, real-time evacuation support remains limited. Existing models often overlook the psychological and emotional complexity of human behavior under extreme stress. In real-world fire scenarios, evacuees frequently deviate from designated safe routes due to panic and uncertainty. To address these challenges, this paper presents a multi-
agent coordination framework in which autonomous Unmanned Aerial Vehicles (
UAVs) assist human evacuees in real-time by locating, intercepting, and guiding them to safety under uncertain conditions. We model the problem as a Partially Observable Markov Decision Process (POMDP), where two heterogeneous
UAV agents, a high-level rescuer (HLR) and a low-level rescuer (LLR), coordinate through shared observations and complementary capabilities. Human behavior is captured using an
agent-based model grounded in empirical psychology, where panic dynamically affects decision-making and movement in response to environmental stimuli. The environment features
stochastic fire spread, unknown evacuee locations, and limited visibility, requiring
UAVs to plan over long horizons to search for humans and adapt in real-time. Our framework employs the Proximal Policy Optimization (PPO)
algorithm with recurrent policies to enable robust decision-making in partially observable settings. Simulation results demonstrate that the
UAV team can rapidly locate and intercept evacuees, significantly reducing the time required for them to reach safety compared to scenarios without
UAV assistance.
Read the full abstract ▾
All authors Maria G. Mendoza, Addison Kalanther, Daniel Bostwick, Emma Stephan, Chinmay Maheshwari, Shankar Sastry
Robotics · Cross-listed from eess.SY
Modeling and Scheduling of Fusion Patterns in Autonomous Driving Systems (Extended Version)
Hoora Sobhani and Hyoseung Kim
In Autonomous Driving Systems (ADS), Directed Acyclic Graphs (DAGs) are widely used to model complex data dependencies and inter-task communication. However, existing DAG scheduling approaches oversimplify data fusion tasks by assuming fixed triggering mechanisms, failing to capture the diverse fusion patterns found in real-world ADS software stacks. In this paper, we propose a systematic framework for analyzing various fusion patterns and their performance implications in ADS. Our framework models three distinct fusion task types: timer-triggered, wait-for-all, and immediate fusion, which comprehensively represent real-world fusion behaviors. Our Integer Linear Programming (ILP)-based approach enables an optimization of multiple real-time performance metrics, including reaction time, time disparity, age of information, and response time, while generating deterministic offline schedules directly applicable to real platforms. Evaluation using real-world ADS case studies, Raspberry Pi implementation, and randomly generated DAGs demonstrates that our framework handles diverse fusion patterns beyond the scope of existing work, and achieves substantial performance improvements in comparable scenarios.
Read the full abstract ▾
Robotics
Motivating Students' Self-study with Goal Reminder and Emotional Support
Hyung Chan Cho, Go-Eum Cha, Yanfu Liu et al. (+1 more)
While the efficacy of social
robots in supporting people in learning tasks has been extensively investigated, their potential impact in assisting students in self-studying contexts has not been investigated much. This study explores how a social
robot can act as a peer study companion for college students during self-study tasks by delivering task-oriented goal reminder and positive emotional support. We conducted an exploratory Wizard-of-Oz study to explore how these robotic support behaviors impacted students' perceived focus, productivity, and engagement in comparison to a
robot that only provided physical presence (control). Our study results suggest that participants in the goal reminder and the emotional support conditions reported greater ease of use, with the goal reminder condition additionally showing a higher willingness to use the
robot in future study sessions. Participants' satisfaction with the
robot was correlated with their perception of the
robot as a social other, and this perception was found to be a predictor for their level of goal achievement in the self-study task. These findings highlight the potential of socially assistive
robots to support self-study through both functional and emotional engagement.
Read the full abstract ▾
All authors Hyung Chan Cho, Go-Eum Cha, Yanfu Liu, Sooyeon Jeong
Robotics
RoboOmni: Proactive Robot Manipulation in Omni-modal Context
Siyin Wang, Jinlan Fu, Feihong Liu et al. (+11 more)
Recent advances in
Multimodal Large
Language Models (MLLMs) have driven rapid progress in
Vision-
Language-Action (VLA) models for robotic manipulation. Although effective in many scenarios, current approaches largely rely on explicit instructions, whereas in real-world interactions, humans rarely issue instructions directly. Effective collaboration requires
robots to infer user intentions proactively. In this work, we introduce cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands. To address this new setting, we present RoboOmni, a Perceiver-Thinker-Talker-Executor framework based on end-to-end omni-modal LLMs that unifies intention recognition, interaction confirmation, and action execution. RoboOmni fuses auditory and visual signals spatiotemporally for robust intention recognition, while supporting direct
speech interaction. To address the absence of training data for proactive intention recognition in robotic manipulation, we build OmniAction, comprising 140k episodes, 5k+ speakers, 2.4k event sounds, 640 backgrounds, and six contextual instruction types. Experiments in simulation and real-world settings show that RoboOmni surpasses text- and ASR-based baselines in success rate, inference speed, intention recognition, and proactive assistance.
Read the full abstract ▾
All authors Siyin Wang, Jinlan Fu, Feihong Liu, Xinzhe He, Huangxuan Wu, Junhao Shi, Kexin Huang, Zhaoye Fei, Jingjing Gong, Zuxuan Wu, Yu-Gang Jiang, See-Kiong Ng, Tat-Seng Chua, Xipeng Qiu
Robotics · Cross-listed from cs.CV
Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infilling
Shuhong Zheng, Ashkan Mirzaei and Igor Gilitschenski
Current 3D/4D generation methods are usually optimized for photorealism, efficiency, and aesthetics. However, they often fail to preserve the semantic identity of the subject across different viewpoints. Adapting generation methods with one or few images of a specific subject (also known as Personalization or Subject-driven generation) allows generating visual content that align with the identity of the subject. However, personalized 3D/4D generation is still largely underexplored. In this work, we introduce TIRE (Track, Inpaint, REsplat), a novel method for subject-driven 3D/4D generation. It takes an initial 3D asset produced by an existing 3D generative model as input and uses
video tracking to identify the regions that need to be modified. Then, we adopt a subject-driven 2D inpainting model for progressively infilling the identified regions. Finally, we resplat the modified 2D multi-view observations back to 3D while still maintaining consistency. Extensive experiments demonstrate that our approach significantly improves identity preservation in 3D/4D generation compared to state-of-the-art methods. Our project website is available at https://zsh2000.github.io/track-inpaint-resplat.github.io/.
Read the full abstract ▾
Robotics
UrbanVLA: A Vision-Language-Action Model for Urban Micromobility
Anqi Li, Zhiyong Wang, Jiazhao Zhang et al. (+5 more)
Urban micromobility applications, such as delivery
robots, demand reliable
navigation across large-scale urban environments while following long-horizon route instructions. This task is particularly challenging due to the dynamic and unstructured nature of real-world city areas, yet most existing
navigation methods remain tailored to short-scale and controllable scenarios. Effective urban micromobility requires two complementary levels of
navigation skills: low-level capabilities such as point-goal reaching and obstacle avoidance, and high-level capabilities, such as route-visual
alignment. To this end, we propose UrbanVLA, a route-conditioned
Vision-
Language-Action (VLA) framework designed for scalable urban
navigation. Our method explicitly aligns noisy route waypoints with visual observations during execution, and subsequently plans trajectories to drive the
robot. To enable UrbanVLA to master both levels of
navigation, we employ a two-stage training pipeline. The process begins with Supervised Fine-Tuning (SFT) using simulated environments and trajectories parsed from web
videos. This is followed by Reinforcement Fine-Tuning (RFT) on a mixture of simulation and real-world data, which enhances the model's safety and adaptability in real-world settings. Experiments demonstrate that UrbanVLA surpasses strong baselines by more than 55% in the SocialNav task on MetaUrban. Furthermore, UrbanVLA achieves reliable real-world
navigation, showcasing both scalability to large-scale urban environments and robustness against real-world uncertainties.
Read the full abstract ▾
All authors Anqi Li, Zhiyong Wang, Jiazhao Zhang, Minghan Li, Yunpeng Qi, Zhibo Chen, Zhizheng Zhang, He Wang
Robotics
RobotArena $\infty$: Scalable Robot Benchmarking via Real-to-Sim Translation
Yash Jangir, Yidi Zhang, Pang-Chi Lo et al. (+7 more)
The pursuit of
robot generalists,
agents capable of performing diverse tasks across diverse environments, demands rigorous and scalable evaluation. Yet real-world testing of
robot policies remains fundamentally constrained: it is labor-intensive, slow, unsafe at scale, and difficult to reproduce. As policies expand in scope and complexity, these barriers only intensify, since defining "success" in robotics often hinges on nuanced human judgments of execution quality. We introduce RobotArena Infinity, a new benchmarking framework that overcomes these challenges by shifting
vision-
language-action (VLA) evaluation into large-scale simulated environments augmented with online human feedback. Leveraging advances in
vision-
language models, 2D-to-3D generative modeling, and differentiable rendering, our approach automatically converts
video demonstrations from widely used
robot datasets into simulated counterparts. Within these digital twins, we assess VLA policies using both automated
vision-
language-model-guided scoring and scalable human preference judgments collected from crowdworkers, transforming human involvement from tedious scene setup, resetting, and safety supervision into lightweight preference comparisons. To measure robustness, we systematically perturb simulated environments along multiple axes, including textures and object placements, stress-testing policy generalization under controlled variation. The result is a continuously evolving, reproducible, and scalable
benchmark for real-world-trained
robot manipulation policies, addressing a critical missing capability in today's robotics landscape.
Read the full abstract ▾
All authors Yash Jangir, Yidi Zhang, Pang-Chi Lo, Kashu Yamazaki, Chenyu Zhang, Kuan-Hsun Tu, Tsung-Wei Ke, Lei Ke, Yonatan Bisk, Katerina Fragkiadaki
Robotics · Cross-listed from cs.CV
DPGLA: Bridging the Gap between Synthetic and Real Data for Unsupervised Domain Adaptation in 3D LiDAR Semantic Segmentation
Wanmeng Li, Simone Mosco, Daniel Fusaro et al. (+1 more)
Annotating real-world
LiDAR point clouds for use in intelligent autonomous systems is costly. To overcome this limitation, self-training-based Unsupervised Domain Adaptation (UDA) has been widely used to improve point cloud semantic
segmentation by leveraging synthetic point cloud data. However, we argue that existing methods do not effectively utilize unlabeled data, as they either rely on predefined or fixed confidence thresholds, resulting in suboptimal performance. In this paper, we propose a Dynamic Pseudo-Label Filtering (DPLF) scheme to enhance real data utilization in point cloud UDA semantic
segmentation. Additionally, we design a simple and efficient Prior-Guided Data Augmentation Pipeline (PG-DAP) to mitigate domain shift between synthetic and real-world point clouds. Finally, we utilize data mixing consistency loss to push the model to learn context-free representations. We implement and thoroughly evaluate our approach through extensive comparisons with state-of-the-art methods. Experiments on two challenging synthetic-to-real point cloud semantic
segmentation tasks demonstrate that our approach achieves superior performance. Ablation studies confirm the effectiveness of the DPLF and PG-DAP modules. We release the code of our method in this paper.
Read the full abstract ▾
All authors Wanmeng Li, Simone Mosco, Daniel Fusaro, Alberto Pretto
Robotics
Explicit Memory through Online 3D Gaussian Splatting Improves Class-Agnostic Video Segmentation
Anthony Opipari, Aravindhan K Krishnan, Shreekant Gayaka et al. (+4 more)
Remembering where object segments were predicted in the past is useful for improving the accuracy and consistency of class-agnostic
video segmentation algorithms. Existing
video segmentation algorithms typically use either no object-level
memory (e.g. FastSAM) or they use implicit
memories in the form of recurrent neural network features (e.g. SAM2). In this paper, we augment both types of
segmentation models using an explicit 3D
memory and show that the resulting models have more accurate and consistent predictions. For this, we develop an online 3D Gaussian Splatting (3DGS) technique to store predicted object-level segments generated throughout the duration of a
video. Based on this 3DGS representation, a set of fusion techniques are developed, named FastSAM-Splat and SAM2-Splat, that use the explicit 3DGS
memory to improve their respective foundation models' predictions. Ablation experiments are used to validate the proposed techniques' design and hyperparameter settings. Results from both real-world and simulated benchmarking experiments show that models which use explicit 3D
memories result in more accurate and consistent predictions than those which use no
memory or only implicit neural network
memories. Project Page: https://topipari.com/projects/FastSAM-Splat/
Read the full abstract ▾
All authors Anthony Opipari, Aravindhan K Krishnan, Shreekant Gayaka, Min Sun, Cheng-Hao Kuo, Arnie Sen, Odest Chadwicke Jenkins
Robotics
Localising under the drape: proprioception in the era of distributed surgical robotic system
Martin Huber, Nicola A. Cavalcanti, Ayoob Davoodi et al. (+15 more)
Despite their mechanical sophistication,
surgical robots remain blind to their surroundings. This lack of spatial awareness causes collisions, system recoveries, and workflow disruptions, issues that will intensify with the introduction of distributed
robots with independent interacting arms. Existing tracking systems rely on bulky infrared cameras and reflective markers, providing only limited views of the
surgical scene and adding hardware burden in crowded operating rooms. We present a marker-free proprioception method that enables precise localisation of
surgical robots under their sterile draping despite associated obstruction of visual cues. Our method solely relies on lightweight stereo-RGB cameras and novel
transformer-based deep learning models. It builds on the largest multi-centre spatial robotic surgery
dataset to date (1.4M self-annotated images from human cadaveric and preclinical in vivo studies). By tracking the entire
robot and
surgical scene, rather than individual markers, our approach provides a holistic view robust to occlusions, supporting
surgical scene understanding and context-aware control. We demonstrate an example of potential clinical benefits during in vivo breathing compensation with access to tissue dynamics, unobservable under state of the art tracking, and accurately locate in multi-
robot systems for future intelligent interaction. In addition, and compared with existing systems, our method eliminates markers and improves tracking visibility by 25%. To our knowledge, this is the first demonstration of marker-free proprioception for fully draped
surgical robots, reducing setup complexity, enhancing safety, and paving the way toward modular and autonomous robotic surgery.
Read the full abstract ▾
All authors Martin Huber, Nicola A. Cavalcanti, Ayoob Davoodi, Ruixuan Li, Christopher E. Mower, Fabio Carrillo, Christoph J. Laux, Francois Teyssere, Thibault Chandanson, Antoine Harlé, Elie Saghbiny, Mazda Farshad, Guillaume Morel, Emmanuel Vander Poorten, Philipp Fürnstahl, Sébastien Ourselin, Christos Bergeles, Tom Vercauteren
Robotics
Dexbotic: Open-Source Vision-Language-Action Toolbox
Bin Xie, Erjin Zhou, Fan Jia et al. (+36 more)
In this paper, we present Dexbotic, an open-source
Vision-
Language-Action (VLA) model toolbox based on PyTorch. It aims to provide a one-stop VLA research service for professionals in the field of
embodied intelligence. It offers a codebase that supports multiple mainstream VLA policies simultaneously, allowing users to reproduce various VLA methods with just a single environment setup. The toolbox is experiment-centric, where the users can quickly develop new VLA experiments by simply modifying the Exp script. Moreover, we provide much stronger pretrained models to achieve great performance improvements for state-of-the-art VLA policies. Dexbotic will continuously update to include more of the latest pre-trained foundation models and cutting-edge VLA models in the industry.
Read the full abstract ▾
All authors Bin Xie, Erjin Zhou, Fan Jia, Hao Shi, Haoqiang Fan, Haowei Zhang, Hebei Li, Jianjian Sun, Jie Bin, Junwen Huang, Kai Liu, Kaixin Liu, Kefan Gu, Lin Sun, Meng Zhang, Peilong Han, Ruitao Hao, Ruitao Zhang, Saike Huang, Songhan Xie, Tiancai Wang, Tianle Liu, Wenbin Tang, Wenqi Zhu, Yang Chen, Yingfei Liu, Yizhuang Zhou, Yu Liu, Yucheng Zhao, Yunchao Ma, Yunfei Wei, Yuxiang Chen, Ze Chen, Zeming Li, Zhao Wu, Ziheng Zhang, Ziming Liu, Ziwei Yan, Ziyu Zhang
Robotics
Logic-Guided Socially-aware Robot Navigation World Model
Weizheng Wang, Obi Ike, Soyun Choi et al. (+3 more)
Social
robot navigation increasingly relies on large
language models for
reasoning, path planning, and enabling movement in dynamic human spaces. However, relying solely on LLMs for planning often leads to unpredictable and unsafe behaviors, especially in dynamic human spaces, due to limited physical grounding and weak logical consistency. In this work, we introduce NaviWM, a socially-aware
robot Navigation World Model that augments LLM
reasoning with a structured world model and a logic-driven chain-of-thought process. NaviWM consists of two main components: (1) a spatial-temporal world model that captures the positions, velocities, and activities of
agents in the environment, and (2) a deductive
reasoning module that guides LLMs through a multi-step, logic-based inference process. This integration enables the
robot to generate
navigation decisions that are both socially compliant and physically safe, under well-defined constraints such as personal space, collision avoidance, and timing. Unlike previous methods based on prompting or fine-tuning, NaviWM encodes social norms as first-order logic, enabling interpretable and verifiable
reasoning. Experiments show that NaviWM improves success rates and reduces social violations, particularly in crowded environments. These results demonstrate the benefit of combining formal
reasoning with LLMs for robust social
navigation. Additional experimental details and demo
videos for this work can be found at: https://sites.google.com/view/NaviWM.
Read the full abstract ▾
All authors Weizheng Wang, Obi Ike, Soyun Choi, Sungeun Hong, Aniket Bera, Byung-Cheol Min
Robotics
COOPERA: Continual Open-Ended Human-Robot Assistance
Chenyang Ma, Kai Lu, Ruta Desai et al. (+3 more)
To understand and collaborate with humans,
robots must account for individual human traits, habits, and activities over time. However, most robotic assistants lack these abilities, as they primarily focus on predefined tasks in structured environments and lack a human model to learn from. This work introduces COOPERA, a novel framework for COntinual, OPen-Ended human-
Robot Assistance, where simulated humans, driven by psychological traits and long-term intentions, interact with
robots in complex environments. By integrating continuous human feedback, our framework, for the first time, enables the study of long-term, open-ended human-
robot collaboration (HRC) in different collaborative tasks across various time-scales. Within COOPERA, we introduce a
benchmark and an approach to personalize the
robot's collaborative actions by learning human traits and context-dependent intents. Experiments validate the extent to which our simulated humans reflect realistic human behaviors and demonstrate the value of inferring and personalizing to human intents for open-ended and long-term HRC. Project Page: https://dannymcy.github.io/coopera/
Read the full abstract ▾
All authors Chenyang Ma, Kai Lu, Ruta Desai, Xavier Puig, Andrew Markham, Niki Trigoni
Robotics
Full-Dynamics Real-Time Nonlinear Model Predictive Control of Heavy-Duty Hydraulic Manipulator for Trajectory Tracking Tasks
Alvaro Paz, Mahdi Hejrati, Pauli Mustalahti et al. (+1 more)
Heavy-duty hydraulic manipulators (HHMs) operate under strict physical and safety-critical constraints due to their large size, high power, and complex nonlinear dynamics. Ensuring that both joint-level and end-effector trajectories remain compliant with actuator capabilities, such as force, velocity, and position limits, is essential for safe and reliable operation, yet remains largely underexplored in real-time control frameworks. This paper presents a nonlinear model predictive control (NMPC) framework designed to guarantee constraint satisfaction throughout the full nonlinear dynamics of HHMs, while running at a real-time control frequency of 1 kHz. The proposed method combines a multiple-shooting strategy with real-time sensor feedback, and is supported by a robust low-level controller based on virtual decomposition control (VDC) for precise joint tracking. Experimental validation on a full-scale hydraulic manipulator shows that the NMPC framework not only enforces actuator constraints at the joint level, but also ensures constraint-compliant motion in Cartesian space for the end-effector. These results demonstrate the method's capability to deliver high-accuracy trajectory tracking while strictly respecting safety-critical limits, setting a new
benchmark for real-time control in large-scale hydraulic systems.
Read the full abstract ▾
All authors Alvaro Paz, Mahdi Hejrati, Pauli Mustalahti, Jouni Mattila
Robotics
T-ESKF: Transformed Error-State Kalman Filter for Consistent Visual-Inertial Navigation
Chungeng Tian, Ning Hao and Fenghua He
This paper presents a novel approach to address the inconsistency problem caused by observability mismatch in visual-inertial
navigation systems (VINS). The key idea involves applying a linear time-varying transformation to the error-state within the Error-State Kalman Filter (ESKF). This transformation ensures that \textrr{the unobservable subspace of the transformed error-state system} becomes independent of the state, thereby preserving the correct observability of the transformed system against variations in linearization points. We introduce the Transformed ESKF (T-ESKF), a consistent VINS
estimator that performs state estimation using the transformed error-state system. Furthermore, we develop an efficient propagation technique to accelerate the covariance propagation based on the transformation relationship between the transition and accumulated
matrices of T-ESKF and ESKF. We validate the proposed method through extensive simulations and experiments, demonstrating better (or competitive at least) performance compared to state-of-the-art methods. The code is available at github.com/HITCSC/T-ESKF.
Read the full abstract ▾
Robotics
Large language model-based task planning for service robots: A review
Shaohan Bian, Ying Zhang, Guohui Tian et al. (+4 more)
With the rapid advancement of large
language models (LLMs) and robotics, service
robots are increasingly becoming an integral part of daily life, offering a wide range of services in complex environments. To deliver these services intelligently and efficiently, robust and accurate task planning capabilities are essential. This paper presents a comprehensive overview of the integration of LLMs into service robotics, with a particular focus on their role in enhancing robotic task planning. First, the development and foundational techniques of LLMs, including pre-training, fine-tuning,
retrieval-augmented generation (RAG), and
prompt engineering, are reviewed. We then explore the application of LLMs as the cognitive core-`
brain'-of service
robots, discussing how LLMs contribute to improved autonomy and decision-making. Furthermore, recent advancements in LLM-driven task planning across various input modalities are analyzed, including text, visual, audio, and
multimodal inputs. Finally, we summarize key challenges and limitations in current research and propose future directions to advance the task planning capabilities of service
robots in complex, unstructured domestic environments. This review aims to serve as a valuable reference for researchers and practitioners in the fields of artificial intelligence and robotics.
Read the full abstract ▾
All authors Shaohan Bian, Ying Zhang, Guohui Tian, Zhiqiang Miao, Edmond Q. Wu, Simon X. Yang, Changchun Hua
Robotics
Transferable Deep Reinforcement Learning for Cross-Domain Navigation: from Farmland to the Moon
Shreya Santra, Thomas Robbins and Kazuya Yoshida
Autonomous
navigation in unstructured environments is essential for field and planetary robotics, where
robots must efficiently reach goals while avoiding obstacles under uncertain conditions. Conventional algorithmic approaches often require extensive environment-specific tuning, limiting scalability to new domains. Deep Reinforcement Learning (DRL) provides a data-driven alternative, allowing
robots to acquire
navigation strategies through direct interactions with their environment. This work investigates the feasibility of DRL policy generalization across visually and topographically distinct simulated domains, where policies are trained in terrestrial settings and validated in a zero-shot manner in extraterrestrial environments. A 3D simulation of an agricultural rover is developed and trained using Proximal Policy Optimization (PPO) to achieve goal-directed
navigation and obstacle avoidance in farmland settings. The learned policy is then evaluated in a lunar-like simulated environment to assess transfer performance. The results indicate that policies trained under terrestrial conditions retain a high level of effectiveness, achieving close to 50% success in lunar simulations without the need for additional training and fine-tuning. This underscores the potential of cross-domain DRL-based policy transfer as a promising approach to developing adaptable and efficient autonomous
navigation for future planetary exploration missions, with the added benefit of minimizing retraining costs.
Read the full abstract ▾
Robotics · Cross-listed from eess.SY
Payload trajectory tracking control for aerial transportation systems with cable length online optimization
Hai Yu, Zhichao Yang, Wei He et al. (+3 more)
Cable-suspended aerial transportation systems are employed extensively across various industries. The capability to flexibly adjust the relative position between the multirotor and the payload has spurred growing interest in the system equipped with variable-length cable, promising broader application potential. Compared to systems with fixed-length cables, introducing the variable-length cable adds a new degree of freedom. However, it also results in increased nonlinearity and more complex dynamic coupling among the multirotor, the cable and the payload, posing significant challenges in control design. This paper introduces a backstepping control strategy tailored for aerial transportation systems with variable-length cable, designed to precisely track the payload trajectory while dynamically adjusting cable length. Then, a cable length generator has been developed that achieves online optimization of the cable length while satisfying state constraints, thus balancing the multirotor's motion and cable length changes without the need for manual trajectory planning. The asymptotic stability of the closed-loop system is guaranteed through Lyapunov techniques and the growth restriction condition. Finally, simulation results confirm the efficacy of the proposed method in managing trajectory tracking and cable length adjustments effectively.
Read the full abstract ▾
All authors Hai Yu, Zhichao Yang, Wei He, Jianda Han, Yongchun Fang, Xiao Liang
Robotics
Precise Time Delay Measurement and Compensation for Tightly Coupled Underwater SINS/piUSBL Navigation
Jin Huang, Yingqiang Wang, Haoda Li et al. (+3 more)
In multisensor systems, time synchronization is particularly challenging for
underwater integrated
navigation systems (INSs) incorporating acoustic positioning, where time delays can significantly degrade accuracy when measurement and fusion epochs are misaligned. This article introduces a tightly coupled
navigation framework that integrates a passive inverted ultrashort baseline (piUSBL) acoustic positioning system, a strapdown inertial
navigation system (SINS), and a depth gauge under precise time synchronization. The framework fuses piUSBL azimuth and slant range with depth measurements, avoiding poor vertical-angle observability in planar arrays. By combining synchronized timing with acoustic signal processing, the proposed method transforms delay from an unobservable error into a measurable parameter, enabling explicit quantification of both acoustic propagation and system processing delays. Field experiments demonstrate that the proposed approach reduces position RMSE by 44.02% and maximum error (MAXERR) by 40.79% compared to the uncompensated baseline while achieving further RMSE reductions of 37.66% and 35.82% in horizontal directions relative to filter-based delay compensation. The results confirm that explicit delay measurement outperforms filter-based estimation though instantaneous performance remains sensitive to acoustic signal quality, emphasizing the need for robust signal processing alongside accurate time synchronization in latency-sensitive multisensor systems.
Read the full abstract ▾
All authors Jin Huang, Yingqiang Wang, Haoda Li, Zichen Liu, Zhikun Wang, Ying Chen
Robotics
Deep Active Inference with Diffusion Policy and Multiple Timescale World Model for Real-World Exploration and Navigation
Riko Yokozawa, Kentaro Fujii, Yuta Nomura et al. (+1 more)
Autonomous robotic
navigation in real-world environments requires exploration to acquire environmental information as well as goal-directed
navigation in order to reach specified targets. Active inference (AIF) based on the free-energy principle provides a unified framework for these behaviors by minimizing the expected free energy (EFE), thereby combining epistemic and extrinsic values. To realize this practically, we propose a deep AIF framework that integrates a
diffusion policy as the policy model and a multiple timescale recurrent state-space model (MTRSSM) as the world model. The
diffusion policy generates diverse candidate actions while the MTRSSM predicts their long-horizon consequences through latent imagination, enabling action selection that minimizes EFE. Real-world
navigation experiments, including baseline comparisons, component ablations, and robustness evaluations, demonstrated that our framework achieved higher success rates and fewer collisions, particularly in exploration-demanding scenarios. These results highlight how AIF based on EFE minimization can unify exploration and goal-directed
navigation in real-world robotic settings.
Read the full abstract ▾
All authors Riko Yokozawa, Kentaro Fujii, Yuta Nomura, Shingo Murata
Robotics
Optimal Dimensioning of Elastic-Link Manipulators regarding Lifetime Estimation
Klaus Zauner, Hubert Gattringer and Andreas Mueller
Resourceful operation and design of
robots is key for sustainable industrial automation. This will be enabled by lightweight design along with time and energy optimal control of robotic manipulators. Design and control of such systems is intertwined as the control must take into account inherent mechanical compliance while the design must accommodate the dynamic requirements demanded by the control. As basis for such design optimization, a method for estimating the lifetime of elastic link robotic manipulators is presented. This is applied to the geometry optimization of flexible serial manipulators performing pick-and-place operations, where the optimization objective is a combination of overall weight and vibration amplitudes. The lifetime estimation draws from a fatigue analysis combining the rainflow counting
algorithm and the method of critical cutting plane. Tresca hypothesis is used to formulate an equivalent stress, and linear damage accumulation is assumed. The final
robot geometry is selected from a Pareto front as a tradeoff of lifetime and vibration characteristic. The method is illustrated for a three degrees of freedom articulated robotic manipulator.
Read the full abstract ▾
Robotics
Workspace Registration and Collision Detection for Industrial Robotics Applications
Klaus Zauner, Josef El Dib, Hubert Gattringer et al. (+1 more)
Motion planning for robotic manipulators relies on precise knowledge of the environment in order to be able to define restricted areas and to take collision objects into account. To capture the workspace, point clouds of the environment are acquired using various sensors. The collision objects are identified by region growing
segmentation and VCCS
algorithm. Subsequently the point clusters are approximated. The aim of the present paper is to compare different sensors, to illustrate the process from detection to the finished collision environment and to detect collisions between the
robot and this environment.
Read the full abstract ▾
All authors Klaus Zauner, Josef El Dib, Hubert Gattringer, Andreas Mueller
Robotics
If They Disagree, Will You Conform? Exploring the Role of Robots' Value Awareness in a Decision-Making Task
Giulia Pusceddu, Giulio Antonio Abbo, Francesco Rea et al. (+2 more)
This study investigates whether the opinions of robotic
agents can influence human decision-making when
robots display value awareness (i.e., the capability of understanding human preferences and prioritizing them in decision-making). We designed an experiment in which participants interacted with two Furhat
robots - one programmed to be Value-Aware and the other Non-Value-Aware - during a labeling task for images representing human values. Results indicate that participants distinguished the Value-Aware
robot from the Non-Value-Aware one. Although their explicit choices did not indicate a clear preference for one
robot over the other, participants directed their gaze more toward the Value-Aware
robot. Additionally, the Value-Aware
robot was perceived as more loyal, suggesting that value awareness in a social
robot may enhance its perceived commitment to the group. Finally, when both
robots disagreed with the participant, conformity occurred in about one out of four trials, and participants took longer to confirm their responses, suggesting that two
robots expressing dissent may introduce hesitation in decision-making. On one hand, this highlights the potential risk that
robots, if misused, could manipulate users for unethical purposes. On the other hand, it reinforces the idea that social
robots could encourage reflection in ambiguous situations and help users avoid scams.
Read the full abstract ▾
All authors Giulia Pusceddu, Giulio Antonio Abbo, Francesco Rea, Tony Belpaeme, Alessandra Sciutti
Robotics
TARC: Time-Adaptive Robotic Control
Arnav Sukhija, Lenart Treven, Jin Cheng et al. (+3 more)
Most robotic systems rely on fixed-frequency discrete-time controllers, creating a trade-off between the efficiency of low-frequency control and the responsiveness of high-frequency feedback. As a result, systems typically default to high control rates for robustness, at the cost of wasted inference and unnecessary actuation. Addressing this, we introduce Time-Adaptive Robotic Control (TARC), a reinforcement learning framework in which the policy jointly predicts a control action and its duration of application. TARC learns temporally extended actions by optimizing task performance under soft or hard constraints on the number of control switches, enabling adaptive modulation of control rates. We evaluate TARC on two robotic hardware platforms: a high-speed RC car and the Unitree Go1 quadruped, and on a
vision-
language action model in simulation, where each query incurs a costly
transformer forward pass. Across all settings, TARC matches the performance of high-frequency discrete-time controllers while operating at less than half their control frequency. Unlike fixed-rate controllers, TARC adapts its control frequency online, allocating high-frequency feedback only when required.
Read the full abstract ▾
All authors Arnav Sukhija, Lenart Treven, Jin Cheng, Florian Dörfler, Stelian Coros, Andreas Krause
Robotics
Combining High Level Scheduling and Low Level Control to Manage Fleets of Mobile Robots
Sabino Francesco Roselli, Ze Zhang and Knut Åkesson
The deployment of mobile
robots for material handling in industrial environments requires scalable coordination of large fleets in dynamic settings. This paper presents a two-layer framework that combines high-level scheduling with low-level control. Tasks are assigned and scheduled using the compositional
algorithm ComSat, which generates time-parameterized routes for each
robot. These schedules are then used by a distributed Model Predictive Control (MPC) system in real time to compute local reference trajectories, accounting for static and dynamic obstacles. The approach ensures safe, collision-free operation, and supports rapid rescheduling in response to disruptions such as
robot failures or environmental changes. We evaluate the method in simulated 2D environments with varying road capacities and traffic conditions, demonstrating high task completion rates and robust behavior even under congestion. The modular structure of the framework allows for computational tractability and flexibility, making it suitable for deployment in complex, real-world industrial scenarios.
Read the full abstract ▾
Robotics
Reliable Robotic Task Execution in the Face of Anomalies
Bharath Santhanam, Alex Mitrevski, Santosh Thoduka et al. (+2 more)
Learned
robot policies have consistently been shown to be versatile, but they typically have no built-in mechanism for handling the complexity of open environments, making them prone to execution failures; this implies that deploying policies without the ability to recognise and react to failures may lead to unreliable and unsafe
robot behaviour. In this paper, we present a framework that couples a learned policy with a method to detect visual anomalies during policy deployment and to perform recovery behaviours when necessary, thereby aiming to prevent failures. Specifically, we train an anomaly detection model using data collected during nominal executions of a trained policy. This model is then integrated into the online policy execution process, so that deviations from the nominal execution can trigger a three-level sequential recovery process that consists of (i) pausing the execution temporarily, (ii) performing a local perturbation of the
robot's state, and (iii) resetting the
robot to a safe state by sampling from a learned execution success model. We verify our proposed method in two different scenarios: (i) a door handle reaching task with a Kinova Gen3 arm using a policy trained in simulation and transferred to the real
robot, and (ii) an object placing task with a UFactory xArm 6 using a general-purpose policy model. Our results show that integrating policy execution with anomaly detection and recovery increases the execution success rate in environments with various anomalies, such as trajectory deviations and
adversarial human interventions.
Read the full abstract ▾
All authors Bharath Santhanam, Alex Mitrevski, Santosh Thoduka, Sebastian Houben, Teena Hassan
Robotics
OmniDexGrasp: Generalizable Dexterous Grasping via Foundation Model and Force Feedback
Yi-Lin Wei, Zhexi Luo, Yuhao Lin et al. (+4 more)
Enabling
robots to dexterously grasp and manipulate objects based on human commands is a promising direction in robotics. However, existing approaches are challenging to generalize across diverse objects or tasks due to the limited scale of semantic dexterous grasp
datasets. Foundation models offer a new way to enhance generalization, yet directly leveraging them to generate feasible robotic actions remains challenging due to the gap between abstract model knowledge and physical
robot execution. To address these challenges, we propose OmniDexGrasp, a generalizable framework that achieves omni-capabilities in user prompting, dexterous embodiment, and grasping tasks by combining foundation models with the transfer and control strategies. OmniDexGrasp integrates three key modules: (i) foundation models are used to enhance generalization by generating human grasp images supporting omni-capability of user
prompt and task; (ii) a human-image-to-
robot-action transfer strategy converts human demonstrations into executable
robot actions, enabling omni dexterous embodiment; (iii) force-aware adaptive grasp strategy ensures robust and stable grasp execution. Experiments in simulation and on real
robots validate the effectiveness of OmniDexGrasp on diverse user
prompts, grasp task and dexterous hands, and further results show its extensibility to dexterous manipulation tasks.
Read the full abstract ▾
All authors Yi-Lin Wei, Zhexi Luo, Yuhao Lin, Mu Lin, Zhizhao Liang, Shuoyu Chen, Wei-Shi Zheng