Peer Revenue Past citations are no guarantee of future returns

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Computer Science Robotics

cs.RO Watch Robotics

Friday 13 February

Robotics

TRAIN: Terrain-Aware Interactive Navigation for Quadruped Robots via Reinforcement Learning

Wei Zhu, Irfan Tito Kurniawan, Ye Zhao et al. (+1 more)

This study introduces TRAIN: TeRrain-Aware Interactive Navigation, a deep reinforcement learning (DRL)-based framework for quadrupedal navigation across unstructured terrain and human-populated environments. Conventional quadrupedal navigation typically separates motion planning from locomotion control, neglecting whole-body constraints and terrain awareness. On the other hand, end-to-end methods are more integrated but require high-frequency sensing, which is often noisy and computationally costly. In addition, most existing approaches assume static environments, limiting their use in human-populated settings. To address these limitations, we propose a two-stage training framework with three DRL pipelines. (1) TRAIN-Loco employs an asymmetric actor-critic (AC) model for quadrupedal locomotion, enabling agile traversal of uneven terrains without explicit terrain or contact observations. (2) TRAIN-Nav applies a symmetric AC framework for navigation with interactive pedestrians, directly mapping transformed LiDAR data to ego-agent actions under differential-drive kinematics. (3) A unified pipeline, TRAIN, integrates TRAIN-Loc and TRAIN-Nav, supporting terrain-aware quadrupedal navigation in uneven and interactive environments. Comprehensive benchmarks against quadrupedal locomotion and robot navigation baselines demonstrate the effectiveness of TRAIN. Hardware experiments further confirm its potential for sim-to-real transfer.
Read the full abstract

All authors Wei Zhu, Irfan Tito Kurniawan, Ye Zhao, Mitsuhiro Hayashibe

Robotics

Constrained PSO Six-Parameter Fuzzy PID Tuning Method for Balanced Optimization of Depth Tracking Performance in Underwater Vehicles

Yanxi Ding and Tingyue Jia

Depth control of underwater vehicles in engineering applications must simultaneously satisfy requirements for rapid tracking, low overshoot, and actuator constraints. Traditional fuzzy PID tuning often relies on empirical methods, making it difficult to achieve a stable and reproducible equilibrium solution between performance enhancement and control cost. This paper proposes a constrained particle swarm optimization (PSO) method for tuning six-parameter fuzzy PID controllers. By adjusting the benchmark PID parameters alongside the fuzzy controller's input quantization factor and output proportional gain, it achieves synergistic optimization of the overall tuning strength and dynamic response characteristics of the fuzzy PID system. To ensure engineering feasibility of the optimization results, a time-weighted absolute error integral, adjustment time, relative overshoot control energy, and saturation occupancy rate are introduced. Control energy constraints are applied to construct a constraint-driven comprehensive evaluation system, suppressing pseudo-improvements achieved solely by increasing control inputs. Simulation results demonstrate that, while maintaining consistent control energy and saturation levels, the proposed method significantly enhances deep tracking performance: the time-weighted absolute error integral decreases from 0.2631 to 0.1473, the settling time shortens from 2.301 s to 1.613 s, and the relative overshoot reduces from 0.1494 to 0.01839. Control energy varied from 7980 to 7935, satisfying the energy constraint, while saturation occupancy decreased from 0.004 to 0.003. These results validate the effectiveness and engineering significance of the proposed constrained six-parameter joint tuning strategy for depth control in underwater vehicle navigation scenarios.
Read the full abstract

Robotics

ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Training

Rushuai Yang, Hecheng Wang, Zhichao Wu et al. (+11 more)

We study how to improve large foundation vision-language-action (VLA) systems through human-in-the-loop reinforcement learning (RL) in real-world environments. A key challenge is learning reliable value functions from heterogeneous real-world experience, as value estimation provides the primary learning signal for VLA training. In practice, replay buffers contain trajectories collected from historical policies, online rollouts, demonstrations, and intermittent human interventions. Because replay buffers mix trajectories generated by different behaviors, the observed returns can be mismatched with the quality of the current policy. Prior VLA post-training methods often rely on progress-style value signals, which reflect the average quality of historical behaviors, leading to mismatched learning signals for the current policy. In this paper, we propose ALOE, an off-policy evaluation framework whose value function directly evaluates current-policy behavior for each iteration. Specifically, ALOE combines chunked temporal-difference bootstrapping and conservative value aggregation to perform stable current-policy evaluation, then uses these estimates for advantage-weighted policy improvement. This design improves credit assignment to critical action chunks under sparse rewards and supports stable policy improvement. We evaluate ALOE on four real-world manipulation tasks encompassing long-horizon and high-precision scenarios: smartphone packing, laundry folding, multi-object sorting, and phone assembly. Across all tasks, ALOE outperforms other VLA post-training methods, highlighting the benefit of off-policy value estimates for real-world VLA post-training. Videos are available at our project website https://rooshy-yang.github.io/aloe.
Read the full abstract

All authors Rushuai Yang, Hecheng Wang, Zhichao Wu, Chiming Liu, Xiaohan Yan, Xuan Du, Shuoyu Yue, Chuheng Zhang, Yunlong Wang, Yongcheng Liu, Lizhe Qi, Yi Chen, Wei Shan, Maoqing Yao

Robotics

SignScene: Visual Sign Grounding for Mapless Navigation

Nicky Zimmerman, Joel Loo, Benjamin Koh et al. (+2 more)

Navigational signs enable humans to navigate unfamiliar environments without maps. This work studies how robots can similarly exploit signs for mapless navigation in the open world. A central challenge lies in interpreting signs: real-world signs are diverse and complex, and their abstract semantic contents need to be grounded in the local 3D scene. We formalize this as sign grounding, the problem of mapping semantic instructions on signs to corresponding scene elements and navigational actions. Recent Vision-Language Models (VLMs) offer the semantic common-sense and reasoning capabilities required for this task, but are sensitive to how spatial information is represented. We propose SignScene, a sign-centric spatial-semantic representation that captures navigation-relevant scene elements and sign information, and presents them to VLMs in a form conducive to effective reasoning. We evaluate our grounding approach on a dataset of 114 queries collected across nine diverse environment types, achieving 88% grounding accuracy and significantly outperforming baselines. Finally, we demonstrate that it enables real-world mapless navigation on a Spot robot using only signs.
Read the full abstract

All authors Nicky Zimmerman, Joel Loo, Benjamin Koh, Zishuo Wang, David Hsu

Robotics

Xiaomi-Robotics-0: An Open-Sourced Vision-Language-Action Model with Real-Time Execution

Rui Cai, Jun Guo, Xinze He et al. (+20 more)

In this report, we introduce Xiaomi-Robotics-0, an advanced vision-language-action (VLA) model optimized for high performance and fast and smooth real-time execution. The key to our method lies in a carefully designed training recipe and deployment strategy. Xiaomi-Robotics-0 is first pre-trained on large-scale cross-embodiment robot trajectories and vision-language data, endowing it with broad and generalizable action-generation capabilities while avoiding catastrophic forgetting of the visual-semantic knowledge of the underlying pre-trained VLM. During post-training, we propose several techniques for training the VLA model for asynchronous execution to address the inference latency during real-robot rollouts. During deployment, we carefully align the timesteps of consecutive predicted action chunks to ensure continuous and seamless real-time rollouts. We evaluate Xiaomi-Robotics-0 extensively in simulation benchmarks and on two challenging real-robot tasks that require precise and dexterous bimanual manipulation. Results show that our method achieves state-of-the-art performance across all simulation benchmarks. Moreover, Xiaomi-Robotics-0 can roll out fast and smoothly on real robots using a consumer-grade GPU, achieving high success rates and throughput on both real-robot tasks. To facilitate future research, code and model checkpoints are open-sourced at https://xiaomi-robotics-0.github.io
Read the full abstract

All authors Rui Cai, Jun Guo, Xinze He, Piaopiao Jin, Jie Li, Bingxuan Lin, Futeng Liu, Wei Liu, Fei Ma, Kun Ma, Feng Qiu, Heng Qu, Yifei Su, Qiao Sun, Dong Wang, Donghao Wang, Yunhong Wang, Rujie Wu, Diyun Xiang, Yu Yang, Hangjun Ye, Yuan Zhang, Quanyun Zhou

Robotics

PMG: Parameterized Motion Generator for Human-like Locomotion Control

Chenxi Han, Yuheng Min, Zihao Huang et al. (+4 more)

Recent advances in data-driven reinforcement learning and motion tracking have substantially improved humanoid locomotion, yet critical practical challenges remain. In particular, while low-level motion tracking and trajectory-following controllers are mature, whole-body reference-guided methods are difficult to adapt to higher-level command interfaces and diverse task contexts: they require large, high-quality datasets, are brittle across speed and pose regimes, and are sensitive to robot-specific calibration. To address these limitations, we propose the Parameterized Motion Generator (PMG), a real-time motion generator grounded in an analysis of human motion structure that synthesizes reference trajectories using only a compact set of parameterized motion data together with high-dimensional control commands. Combined with an imitation-learning pipeline and an optimization-based sim-to-real motor parameter identification module, we validate the complete approach on our humanoid prototype ZERITH Z1 and show that, within a single integrated system, PMG produces natural, human-like locomotion, responds precisely to high-dimensional control inputs-including VR-based teleoperation-and enables efficient, verifiable sim-to-real transfer. Together, these results establish a practical, experimentally validated pathway toward natural and deployable humanoid control. Website: https://pmg-icra26.github.io/
Read the full abstract

All authors Chenxi Han, Yuheng Min, Zihao Huang, Ao Hong, Hang Liu, Yi Cheng, Houde Liu

Robotics · Cross-listed from cs.LG

Dual-Granularity Contrastive Reward via Generated Episodic Guidance for Efficient Embodied RL

Xin Liu, Yixuan Li, Yuhui Chen et al. (+3 more)

Designing suitable rewards poses a significant challenge in reinforcement learning (RL), especially for embodied manipulation. Trajectory success rewards are suitable for human judges or model fitting, but the sparsity severely limits RL sample efficiency. While recent methods have effectively improved RL via dense rewards, they rely heavily on high-quality human-annotated data or abundant expert supervision. To tackle these issues, this paper proposes Dual-granularity contrastive reward via generated Episodic Guidance (DEG), a novel framework to seek sample-efficient dense rewards without requiring human annotations or extensive supervision. Leveraging the prior knowledge of large video generation models, DEG only needs a small number of expert videos for domain adaptation to generate dedicated task guidance for each RL episode. Then, the proposed dual-granularity reward that balances coarse-grained exploration and fine-grained matching, will guide the agent to efficiently approximate the generated guidance video sequentially in the contrastive self-supervised latent space, and finally complete the target task. Extensive experiments on 18 diverse tasks across both simulation and real-world settings show that DEG can not only serve as an efficient exploration stimulus to help the agent quickly discover sparse success rewards, but also guide effective RL and stable policy convergence independently.
Read the full abstract

All authors Xin Liu, Yixuan Li, Yuhui Chen, Yuxing Qin, Haoran Li, Dongbin Zhao

Robotics

Real-to-Sim for Highly Cluttered Environments via Physics-Consistent Inter-Object Reasoning

Tianyi Xiang, Jiahang Cao, Sikai Guo et al. (+3 more)

Reconstructing physically valid 3D scenes from single-view observations is a prerequisite for bridging the gap between visual perception and robotic control. However, in scenarios requiring precise contact reasoning, such as robotic manipulation in highly cluttered environments, geometric fidelity alone is insufficient. Standard perception pipelines often neglect physical constraints, resulting in invalid states, e.g., floating objects or severe inter-penetration, rendering downstream simulation unreliable. To address these limitations, we propose a novel physics-constrained Real-to-Sim pipeline that reconstructs physically consistent 3D scenes from single-view RGB-D data. Central to our approach is a differentiable optimization pipeline that explicitly models spatial dependencies via a contact graph, jointly refining object poses and physical properties through differentiable rigid-body simulation. Extensive evaluations in both simulation and real-world settings demonstrate that our reconstructed scenes achieve high physical fidelity and faithfully replicate real-world contact dynamics, enabling stable and reliable contact-rich manipulation.
Read the full abstract

All authors Tianyi Xiang, Jiahang Cao, Sikai Guo, Guoyang Zhao, Andrew F. Luo, Jun Ma

Robotics

Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models

Liangzhi Shi, Shuaihang Chen, Feng Gao et al. (+8 more)

Simulation offers a scalable and low-cost way to enrich vision-language-action (VLA) training, reducing reliance on expensive real-robot demonstrations. However, most sim-real co-training methods rely on supervised fine-tuning (SFT), which treats simulation as a static source of demonstrations and does not exploit large-scale closed-loop interaction. Consequently, real-world gains and generalization are often limited. In this paper, we propose an RL-based sim-real Co-training (RL-Co) framework that leverages interactive simulation while preserving real-world capabilities. Our method follows a generic two-stage design: we first warm-start the policy with SFT on a mixture of real and simulated demonstrations, then fine-tune it with reinforcement learning in simulation while adding an auxiliary supervised loss on real-world data to anchor the policy and mitigate catastrophic forgetting. We evaluate our framework on four real-world tabletop manipulation tasks using two representative VLA architectures, OpenVLA and $\pi_{0.5}$, and observe consistent improvements over real-only fine-tuning and SFT-based co-training, including +24% real-world success on OpenVLA and +20% on $\pi_{0.5}$. Beyond higher success rates, RL co-training yields stronger generalization to unseen task variations and substantially improved real-world data efficiency, providing a practical and scalable pathway for leveraging simulation to enhance real-robot deployment.
Read the full abstract

All authors Liangzhi Shi, Shuaihang Chen, Feng Gao, Yinuo Chen, Kang Chen, Tonghe Zhang, Hongzhi Zang, Jiakai Zhou, Weinan Zhang, Chao Yu, Yu Wang

Robotics · Cross-listed from eess.SY

When Environments Shift: Safe Planning with Generative Priors and Robust Conformal Prediction

Kaizer Rahaman, Jyotirmoy V. Deshmukh, Ashish R. Hota et al. (+1 more)

Autonomous systems operate in environments that may change over time. An example is the control of a self-driving vehicle among pedestrians and human-controlled vehicles whose behavior may change based on factors such as traffic density, road visibility, and social norms. Therefore, the environment encountered during deployment rarely mirrors the environment and data encountered during training – a phenomenon known as distribution shift – which can undermine the safety of autonomous systems. Conformal prediction (CP) has recently been used along with data from the training environment to provide prediction regions that capture the behavior of the environment with a desired probability. When embedded within a model predictive controller (MPC), one can provide probabilistic safety guarantees, but only when the deployment and training environments coincide. Once a distribution shift occurs, these guarantees collapse. We propose a planning framework that is robust under distribution shifts by: (i) assuming that the underlying data distribution of the environment is parameterized by a nuisance parameter, i.e., an observable, interpretable quantity such as traffic density, (ii) training a conditional diffusion model that captures distribution shifts as a function of the nuisance parameter, (iii) observing the nuisance parameter online and generating cheap, synthetic data from the diffusion model for the observed nuisance parameter, and (iv) designing an MPC that embeds CP regions constructed from such synthetic data. Importantly, we account for discrepancies between the underlying data distribution and the diffusion model by using robust CP. Thus, the plans computed using robust CP enjoy probabilistic safety guarantees, in contrast with plans obtained from a single, static set of training data. We empirically demonstrate safety under diverse distribution shifts in the ORCA simulator.
Read the full abstract

All authors Kaizer Rahaman, Jyotirmoy V. Deshmukh, Ashish R. Hota, Lars Lindemann

Robotics

PISHYAR: A Socially Intelligent Smart Cane for Indoor Social Navigation and Multimodal Human-Robot Interaction for Visually Impaired People

Mahdi Haghighat Joo, Maryam Karimi Jafari and Alireza Taheri

This paper presents PISHYAR, a socially intelligent smart cane designed by our group to combine socially aware navigation with multimodal human-AI interaction to support both physical mobility and interactive assistance. The system consists of two components: (1) a social navigation framework implemented on a Raspberry Pi 5 that integrates real-time RGB-D perception using an OAK-D Lite camera, YOLOv8-based object detection, COMPOSER-based collective activity recognition, D* Lite dynamic path planning, and haptic feedback via vibration motors for tasks such as locating a vacant seat; and (2) an agentic multimodal LLM-VLM interaction framework that integrates speech recognition, vision language models, large language models, and text-to-speech, with dynamic routing between voice-only and vision-only modes to enable natural voice-based communication, scene description, and object localization from visual input. The system is evaluated through a combination of simulation-based tests, real-world field experiments, and user-centered studies. Results from simulated and real indoor environments demonstrate reliable obstacle avoidance and socially compliant navigation, achieving an overall system accuracy of approximately 80% under different social conditions. Group activity recognition further shows robust performance across diverse crowd scenarios. In addition, a preliminary exploratory user study with eight visually impaired and low-vision participants evaluates the agentic interaction framework through structured tasks and a UTAUT-based questionnaire reveals high acceptance and positive perceptions of usability, trust, and perceived sociability during our experiments. The results highlight the potential of PISHYAR as a multimodal assistive mobility aid that extends beyond navigation to provide socially interactive support for such users.
Read the full abstract

Robotics

Hemispherical Angular Power Mapping of Installed mmWave Radar Modules Under Realistic Deployment Constraints

Maaz Qureshi, Mohammad Omid Bagheri, William Melek et al. (+1 more)

Characterizing the angular radiation behavior of installed millimeter-wave (mmWave) radar modules is increasingly important in practical sensing platforms, where packaging, mounting hardware, and nearby structures can significantly alter the effective emission profile. However, once a device is embedded in its host environment, conventional chamber- and turntable-based antenna measurements are often impractical. This paper presents a hemispherical angular received-power mapping methodology for in-situ EM validation of installed mmWave modules under realistic deployment constraints. The approach samples the accessible half-space around a stationary device-under-test by placing a calibrated receiving probe at prescribed (phi, theta, r) locations using geometry-consistent positioning and quasi-static acquisition. Amplitude-only received-power is recorded using standard RF instrumentation to generate hemispherical angular power maps that capture installation-dependent radiation characteristics. Proof-of-concept measurements on a 60-GHz radar module demonstrate repeatable hemi-spherical mapping with angular trends in good agreement with full-wave simulation, supporting practical on-site characterization of embedded mmWave transmitters.
Read the full abstract

All authors Maaz Qureshi, Mohammad Omid Bagheri, William Melek, George Shaker

Robotics

Eva-Tracker: ESDF-update-free, Visibility-aware Planning with Target Reacquisition for Robust Aerial Tracking

Yue Lin, Yang Liu, Dong Wang et al. (+1 more)

The Euclidean Signed Distance Field (ESDF) is widely used in visibility evaluation to prevent occlusions and collisions during tracking. However, frequent ESDF updates introduce considerable computational overhead. To address this issue, we propose Eva-Tracker, a visibility-aware trajectory planning framework for aerial tracking that eliminates ESDF updates and incorporates a recovery-capable path generation method for target reacquisition. First, we design a target trajectory prediction method and a visibility-aware initial path generation algorithm that maintain an appropriate observation distance, avoid occlusions, and enable rapid replanning to reacquire the target when it is lost. Then, we propose the Field of View ESDF (FoV-ESDF), a precomputed ESDF tailored to the tracker's field of view, enabling rapid visibility evaluation without requiring updates. Finally, we optimize the trajectory using differentiable FoV-ESDF-based objectives to ensure continuous visibility throughout the tracking process. Extensive simulations and real-world experiments demonstrate that our approach delivers more robust tracking results with lower computational effort than existing state-of-the-art methods. The source code is available at https://github.com/Yue-0/Eva-Tracker.
Read the full abstract

All authors Yue Lin, Yang Liu, Dong Wang, Huchuan Lu

Robotics · Cross-listed from cs.CV

Self-Supervised JEPA-based World Models for LiDAR Occupancy Completion and Forecasting

Haoran Zhu and Anna Choromanska

Autonomous driving, as an agent operating in the physical world, requires the fundamental capability to build world models that capture how the environment evolves spatiotemporally in order to support long-term planning. At the same time, scalability demands learning such models in a self-supervised manner; joint-embedding predictive architecture (JEPA) enables learning world models via leveraging large volumes of unlabeled data without relying on expensive human annotations. In this paper, we propose AD-LiST-JEPA, a self-supervised world model for autonomous driving that predicts future spatiotemporal evolution from LiDAR data using a JEPA framework. We evaluate the quality of the learned representations through a downstream LiDAR-based occupancy completion and forecasting (OCF) task, which jointly assesses perception and prediction. Proof of concept experiments show better OCF performance with pretrained encoder after JEPA-based world model learning.
Read the full abstract

Robotics

CRAFT: Adapting VLA Models to Contact-rich Manipulation via Force-aware Curriculum Fine-tuning

Yike Zhang, Yaonan Wang, Xinxin Sun et al. (+6 more)

Vision-Language-Action (VLA) models have shown a strong capability in enabling robots to execute general instructions, yet they struggle with contact-rich manipulation tasks, where success requires precise alignment, stable contact maintenance, and effective handling of deformable objects. A fundamental challenge arises from the imbalance between high-entropy vision and language inputs and low-entropy but critical force signals, which often leads to over-reliance on perception and unstable control. To address this, we introduce CRAFT, a force-aware curriculum fine-tuning framework that integrates a variational information bottleneck module to regulate vision and language embeddings during early training. This curriculum strategy encourages the model to prioritize force signals initially, before progressively restoring access to the full multimodal information. To enable force-aware learning, we further design a homologous leader-follower teleoperation system that collects synchronized vision, language, and force data across diverse contact-rich tasks. Real-world experiments demonstrate that CRAFT consistently improves task success, generalizes to unseen objects and novel task variations, and adapts effectively across diverse VLA architectures, enabling robust and generalizable contact-rich manipulation.
Read the full abstract

All authors Yike Zhang, Yaonan Wang, Xinxin Sun, Kaizhen Huang, Zhiyuan Xu, Junjie Ji, Zhengping Che, Jian Tang, Jingtao Sun

Robotics

Monocular Reconstruction of Neural Tactile Fields

Pavan Mantripragada, Siddhanth Deshmukh, Eadom Dessalene et al. (+2 more)

Robots operating in the real world must plan through environments that deform, yield, and reconfigure under contact, requiring interaction-aware 3D representations that extend beyond static geometric occupancy. To address this, we introduce neural tactile fields, a novel 3D representation that maps spatial locations to the expected tactile response upon contact. Our model predicts these neural tactile fields from a single monocular RGB image – the first method to do so. When integrated with off-the-shelf path planners, neural tactile fields enable robots to generate paths that avoid high-resistance objects while deliberately routing through low-resistance regions (e.g. foliage), rather than treating all occupied space as equally impassable. Empirically, our learning framework improves volumetric 3D reconstruction by $85.8\%$ and surface reconstruction by $26.7\%$ compared to state-of-the-art monocular 3D reconstruction methods (LRM and Direct3D).
Read the full abstract

All authors Pavan Mantripragada, Siddhanth Deshmukh, Eadom Dessalene, Manas Desai, Yiannis Aloimonos

Robotics

Composable Model-Free RL for Navigation with Input-Affine Systems

Xinhuan Sang, Abdelrahman Abdelgawad and Roberto Tron

As autonomous robots move into complex, dynamic real-world environments, they must learn to navigate safely in real time, yet anticipating all possible behaviors is infeasible. We propose a composable, model-free reinforcement learning method that learns a value function and an optimal policy for each individual environment element (e.g., goal or obstacle) and composes them online to achieve goal reaching and collision avoidance. Assuming unknown nonlinear dynamics that evolve in continuous time and are input-affine, we derive a continuous-time Hamilton-Jacobi-Bellman (HJB) equation for the value function and show that the corresponding advantage function is quadratic in the action and optimal policy. Based on this structure, we introduce a model-free actor-critic algorithm that learns policies and value functions for static or moving obstacles using gradient descent. We then compose multiple reach/avoid models via a quadratically constrained quadratic program (QCQP), yielding formal obstacle-avoidance guarantees in terms of value-function level sets, providing a model-free alternative to CLF/CBF-based controllers. Simulations demonstrate improved performance over a PPO baseline applied to a discrete-time approximation.
Read the full abstract

Robotics

Gradient-Enhanced Partitioned Gaussian Processes for Real-Time Quadrotor Dynamics Modeling

Xinhuan Sang, Adam Rozman, Sheryl Grace et al. (+1 more)

We present a quadrotor dynamics Gaussian Process (GP) with gradient information that achieves real-time inference via state-space partitioning and approximation, and that includes aerodynamic effects using data from mid-fidelity potential flow simulations. While traditional GP-based approaches provide reliable Bayesian predictions with uncertainty quantification, they are computationally expensive and thus unsuitable for real-time simulations. To address this challenge, we integrate gradient information to improve accuracy and introduce a novel partitioning and approximation strategy to reduce online computational cost. In particular, for the latter, we associate a local GP with each non-overlapping region; by splitting the training data into local near and far subsets, and by using Schur complements, we show that a large part of the matrix inversions required for inference can be performed offline, enabling real-time inference at frequencies above 30 Hz on standard desktop hardware. To generate a training dataset that captures aerodynamic effects, such as rotor-rotor interactions and apparent wind direction, we use the CHARM code, which is a mid-fidelity aerodynamic solver. It is applied to the SUI Endurance quadrotor to predict force and torque, along with noise at three specified locations. The derivative information is obtained via finite differences. Experimental results demonstrate that the proposed partitioned GP with gradient conditioning achieves higher accuracy than standard partitioned GPs without gradient information, while greatly reducing computational time. This framework provides an efficient foundation for real-time aerodynamic prediction and control algorithms in complex and unsteady environments.
Read the full abstract

All authors Xinhuan Sang, Adam Rozman, Sheryl Grace, Roberto Tron

Thursday 12 February

Robotics

Your Robot Will Feel You Now: Empathy in Robots and Embodied Agents

Angelica Lim and Ö. Nilay Yalçin

The fields of human-robot interaction (HRI) and embodied conversational agents (ECAs) have long studied how empathy could be implemented in machines. One of the major drivers has been the goal of giving multimodal social and emotional intelligence to these artificially intelligent agents, which interact with people through facial expressions, body, gesture, and speech. What empathic behaviors and models have these fields implemented by mimicking human and animal behavior? In what ways have they explored creating machine-specific analogies? This chapter aims to review the knowledge from these studies, towards applying the lessons learned to today's ubiquitous, language-based agents such as ChatGPT.
Read the full abstract

Robotics · Cross-listed from cs.CY

Empirical Study of Gaze Behavior in Children and Young Adults Using Deep Neural Networks and Robot Implementation: A Comparative Analysis of Social Situations

Ramtin Tabatabaei, Milad Hosseini, Ali Mohajerzarrinkelk et al. (+2 more)

In a preliminary exploratory study, our goal was to train deep neural network models to mimic children's and/or adults' gaze behavior in certain social situations to reach this objective. Additionally, we aim to identify potential differences in gaze behavior between these two age groups based on our participants' gaze data. Furthermore, we aimed to assess the practical effectiveness of our adult and children models by deploying them on a Nao robot in real-life settings. To achieve this, we first created two video clips, one animation and one live-action, to depict some social situations. Using an eye-tracking device, we collected eye-tracking data from 24 participants, including 12 children and 12 adults. Then, we utilized deep neural networks, specifically LSTM and Transformer Networks, to analyze and model the gaze patterns of each group of participants. Our results indicate that when the models attempted to predict people's locations (in the next frame), they had an accuracy in the range of 62%-70% with one attempt, which increased by ~20% when attempted twice (i.e. the two highest-ranked predicted labels as outputs). As expected, the result underscores that gaze behavior is not a wholly unique phenomenon. We obtained feedback from 57 new participants to evaluate the robot's functionality. These participants were asked to watch two videos of the robot's performance in each mode and then complete a comprehensive questionnaire. The questionnaire results indicate that the participants expressed satisfaction with the robot's interaction, including its attention, intelligence, and responsiveness to human actions. However, they did not perceive the robot as a social companion comparable to a human. This exploratory study tries to address/show potentials of the social acceptance of robots based on human nonverbal behavioral cues for future research.
Read the full abstract

All authors Ramtin Tabatabaei, Milad Hosseini, Ali Mohajerzarrinkelk, Ali F. Meghdari, Alireza Taheri

Robotics

SceneVGGT: VGGT-based online 3D semantic SLAM for indoor scene understanding and navigation

Anna Gelencsér-Horváth, Gergely Dinya, Dorka Boglárka Erős et al. (+3 more)

We present SceneVGGT, a spatio-temporal 3D scene understanding framework that combines SLAM with semantic mapping for autonomous and assistive navigation. Built on VGGT, our method scales to long video streams via a sliding-window pipeline. We align local submaps using camera-pose transformations, enabling memory- and speed-efficient mapping while preserving geometric consistency. Semantics are lifted from 2D instance masks to 3D objects using the VGGT tracking head, maintaining temporally coherent identities for change detection. As a proof of concept, object locations are projected onto an estimated floor plane for assistive navigation. The pipeline's GPU memory usage remains under 17 GB, irrespectively of the length of the input sequence and achieves competitive point-cloud performance on the ScanNet++ benchmark. Overall, SceneVGGT ensures robust semantic identification and is fast enough to support interactive assistive navigation with audio feedback.
Read the full abstract

All authors Anna Gelencsér-Horváth, Gergely Dinya, Dorka Boglárka Erős, Péter Halász, Islam Muhammad Muqsit, Kristóf Karacs

Robotics · Cross-listed from cs.CV

Visual Foresight for Robotic Stow: A Diffusion-Based World Model from Sparse Snapshots

Lijun Zhang, Nikhil Chacko, Petter Nilsson et al. (+8 more)

Automated warehouses execute millions of stow operations, where robots place objects into storage bins. For these systems it is valuable to anticipate how a bin will look from the current observations and the planned stow behavior before real execution. We propose FOREST, a stow-intent-conditioned world model that represents bin states as item-aligned instance masks and uses a latent diffusion transformer to predict the post-stow configuration from the observed context. Our evaluation shows that FOREST substantially improves the geometric agreement between predicted and true post-stow layouts compared with heuristic baselines. We further evaluate the predicted post-stow layouts in two downstream tasks, in which replacing the real post-stow masks with FOREST predictions causes only modest performance loss in load-quality assessment and multi-stow reasoning, indicating that our model can provide useful foresight signals for warehouse planning.
Read the full abstract

All authors Lijun Zhang, Nikhil Chacko, Petter Nilsson, Ruinian Xu, Shantanu Thakar, Bai Lou, Harpreet Sawhney, Zhebin Zhang, Mudit Agrawal, Bhavana Chandrashekhar, Aaron Parness

Robotics

An Autonomous, End-to-End, Convex-Based Framework for Close-Range Rendezvous Trajectory Design and Guidance with Hardware Testbed Validation

Minduli C. Wijayatunga, Julian Guinane, Nathan D. Wallace et al. (+1 more)

Autonomous satellite servicing missions must execute close-range rendezvous under stringent safety and operational constraints while remaining computationally tractable for onboard use and robust to uncertainty in sensing, actuation, and dynamics. This paper presents CORTEX (Convex Optimization for Rendezvous Trajectory Execution), an autonomous, perception-enabled, real-time trajectory design and guidance framework for close-range rendezvous. CORTEX integrates a deep-learning perception pipeline with convex-optimisation-based trajectory design and guidance, including reference regeneration and abort-to-safe-orbit logic to recover from large deviations caused by sensor faults and engine failures. CORTEX is validated in high-fidelity software simulation and hardware-in-the-loop experiments. The software pipeline (Basilisk) models high-fidelity relative dynamics, realistic thruster execution, perception, and attitude control. Hardware testing uses (i) an optical navigation testbed to assess perception-to-estimation performance and (ii) a planar air-bearing testbed to evaluate the end-to-end guidance loop under representative actuation and subsystem effects. A Monte-Carlo campaign in simulation includes initial-state uncertainty, thrust-magnitude errors, and missed-thrust events; under the strongest case investigated, CORTEX achieves terminal docking errors of $36.85 \pm 44.46$ mm in relative position and $1.25 \pm 2.26$ mm/s in relative velocity. On the planar air-bearing testbed, 18 cases are executed (10 nominal; 8 off-nominal requiring recomputation and/or abort due to simulated engine failure and sensor malfunctions), yielding terminal errors of $8.09 \pm 5.29$ mm in position and $2.23 \pm 1.72$ mm/s in velocity.
Read the full abstract

All authors Minduli C. Wijayatunga, Julian Guinane, Nathan D. Wallace, Xiaofeng Wu

Robotics

Control Barrier Functions with Audio Risk Awareness for Robot Safe Navigation on Construction Sites

Johannes Mootz and Reza Akhavian

Construction automation increasingly requires autonomous mobile robots, yet robust autonomy remains challenging on construction sites. These environments are dynamic and often visually occluded, which complicates perception and navigation. In this context, valuable information from audio sources remains underutilized in most autonomy stacks. This work presents a control barrier function (CBF)-based safety filter that provides safety guarantees for obstacle avoidance while adapting safety margins during navigation using an audio-derived risk cue. The proposed framework augments the CBF with a lightweight, real-time jackhammer detector based on signal envelope and periodicity. Its output serves as an exogenous risk that is directly enforced in the controller by modulating the barrier function. The approach is evaluated in simulation with two CBF formulations (circular and goal-aligned elliptical) with a unicycle robot navigating a cluttered construction environment. Results show that the CBF safety filter eliminates safety violations across all trials while reaching the target in 40.2% (circular) vs. 76.5% (elliptical), as the elliptical formulation better avoids deadlock. This integration of audio perception into a CBF-based controller demonstrates a pathway toward richer multimodal safety reasoning in autonomous robots for safety-critical and dynamic environments.
Read the full abstract

Robotics

MiDAS: A Multimodal Data Acquisition System and Dataset for Robot-Assisted Minimally Invasive Surgery

Keshara Weerasinghe, Seyed Hamid Reza Roodabeh, Andrew Hawkins et al. (+3 more)

Background: Robot-assisted minimally invasive surgery (RMIS) research increasingly relies on multimodal data, yet access to proprietary robot telemetry remains a major barrier. We introduce MiDAS, an open-source, platform-agnostic system enabling time-synchronized, non-invasive multimodal data acquisition across surgical robotic platforms. Methods: MiDAS integrates electromagnetic and RGB-D hand tracking, foot pedal sensing, and surgical video capturing without requiring proprietary robot interfaces. We validated MiDAS on the open-source Raven-II and the clinical da Vinci Xi by collecting multimodal datasets of peg transfer and hernia repair suturing tasks performed by surgical residents. Correlation analysis and downstream gesture recognition experiments were conducted. Results: External hand and foot sensing closely approximated internal robot kinematics and non-invasive motion signals achieved gesture recognition performance comparable to proprietary telemetry. Conclusion: MiDAS enables reproducible multimodal RMIS data collection and is released with annotated datasets, including the first multimodal dataset capturing hernia repair suturing on high-fidelity simulation models.
Read the full abstract

All authors Keshara Weerasinghe, Seyed Hamid Reza Roodabeh, Andrew Hawkins, Zhaomeng Zhang, Zachary Schrader, Homa Alemzadeh

Robotics

Self-Refining Vision Language Model for Robotic Failure Detection and Reasoning

Carl Qi, Xiaojie Wang, Silong Yong et al. (+6 more)

Reasoning about failures is crucial for building reliable and trustworthy robotic systems. Prior approaches either treat failure reasoning as a closed-set classification problem or assume access to ample human annotations. Failures in the real world are typically subtle, combinatorial, and difficult to enumerate, whereas rich reasoning labels are expensive to acquire. We address this problem by introducing ARMOR: Adaptive Round-based Multi-task mOdel for Robotic failure detection and reasoning. We formulate detection and reasoning as a multi-task self-refinement process, where the model iteratively predicts detection outcomes and natural language reasoning conditioned on past outputs. During training, ARMOR learns from heterogeneous supervision - large-scale sparse binary labels and small-scale rich reasoning annotations - optimized via a combination of offline and online imitation learning. At inference time, ARMOR generates multiple refinement trajectories and selects the most confident prediction via a self-certainty metric. Experiments across diverse environments show that ARMOR achieves state-of-the-art performance by improving over the previous approaches by up to 30% on failure detection rate and up to 100% in reasoning measured through LLM fuzzy match score, demonstrating robustness to heterogeneous supervision and open-ended reasoning beyond predefined failure modes. We provide additional visualizations on our website: https://sites.google.com/utexas.edu/armor
Read the full abstract

All authors Carl Qi, Xiaojie Wang, Silong Yong, Stephen Sheng, Huitan Mao, Sriram Srinivasan, Manikantan Nambi, Amy Zhang, Yesh Dattatreya

Robotics

Zero-Shot Adaptation to Robot Structural Damage via Natural Language-Informed Kinodynamics Modeling

Anuj Pokhrel, Aniket Datar, Mohammad Nazeri et al. (+2 more)

High-performance autonomous mobile robots endure significant mechanical stress during in-the-wild operations, e.g., driving at high speeds or over rugged terrain. Although these platforms are engineered to withstand such conditions, mechanical degradation is inevitable. Structural damage manifests as consistent and notable changes in kinodynamic behavior compared to a healthy vehicle. Given the heterogeneous nature of structural failures, quantifying various damages to inform kinodynamics is challenging. We posit that natural language can describe and thus capture this variety of damages. Therefore, we propose Zero-shot Language Informed Kinodynamics (ZLIK), which employs self-supervised learning to ground semantic information of damage descriptions in kinodynamic behaviors to learn a forward kinodynamics model in a data-driven manner. Using the high-fidelity soft-body physics simulator BeamNG.tech, we collect data from a variety of structurally compromised vehicles. Our learned model achieves zero-shot adaptation to different damages with up to 81% reduction in kinodynamics error and generalizes across the sim-to-real and full-to-1/10$^{\text{th}}$ scale gaps.
Read the full abstract

All authors Anuj Pokhrel, Aniket Datar, Mohammad Nazeri, Francesco Cancelliere, Xuesu Xiao

Robotics

Predicting Dynamic Map States from Limited Field-of-View Sensor Data

Knut Peterson and David Han

When autonomous systems are deployed in real-world scenarios, sensors are often subject to limited field-of-view (FOV) constraints, either naturally through system design, or through unexpected occlusions or sensor failures. In conditions where a large FOV is unavailable, it is important to be able to infer information about the environment and predict the state of nearby surroundings based on available data to maintain safe and accurate operation. In this work, we explore the effectiveness of deep learning for dynamic map state prediction based on limited FOV time series data. We show that by representing dynamic sensor data in a simple single-image format that captures both spatial and temporal information, we can effectively use a wide variety of existing image-to-image learning models to predict map states with high accuracy in a diverse set of sensing scenarios.
Read the full abstract

Robotics

LongNav-R1: Horizon-Adaptive Multi-Turn RL for Long-Horizon VLA Navigation

Yue Hu, Avery Xi, Qixin Xiao et al. (+4 more)

This paper develops LongNav-R1, an end-to-end multi-turn reinforcement learning (RL) framework designed to optimize Visual-Language-Action (VLA) models for long-horizon navigation. Unlike existing single-turn paradigm, LongNav-R1 reformulates the navigation decision process as a continuous multi-turn conversation between the VLA policy and the embodied environment. This multi-turn RL framework offers two distinct advantages: i) it enables the agent to reason about the causal effects of historical interactions and sequential future outcomes; and ii) it allows the model to learn directly from online interactions, fostering diverse trajectory generation and avoiding the behavioral rigidity often imposed by human demonstrations. Furthermore, we introduce Horizon-Adaptive Policy Optimization. This mechanism explicitly accounts for varying horizon lengths during advantage estimation, facilitating accurate temporal credit assignment over extended sequences. Consequently, the agent develops diverse navigation behaviors and resists collapse during long-horizon tasks. Experiments on object navigation benchmarks validate the framework's efficacy: With 4,000 rollout trajectories, LongNav-R1 boosts the Qwen3-VL-2B success rate from 64.3% to 73.0%. These results demonstrate superior sample efficiency and significantly outperform state-of-the-art methods. The model's generalizability and robustness are further validated by its zero-shot performance in long-horizon real-world navigation settings. All source code is open-sourced at https://github.com/UMich-CURLY/LongNav-R1.
Read the full abstract

All authors Yue Hu, Avery Xi, Qixin Xiao, Seth Isaacson, Henry X. Liu, Ram Vasudevan, Maani Ghaffari

Robotics

Schur-MI: Fast Mutual Information for Robotic Information Gathering

Kalvik Jakkala, Jason O'Kane and Srinivas Akella

Mutual information (MI) is a principled and widely used objective for robotic information gathering (RIG), providing strong theoretical guarantees for sensor placement (SP) and informative path planning (IPP). However, its high computational cost - dominated by repeated log-determinant evaluations - has limited its use in real-time planning. This paper presents Schur-MI, a Gaussian process (GP) MI formulation that (i) leverages the iterative structure of RIG to precompute and reuse expensive intermediate quantities across planning steps, and (ii) uses a Schur-complement factorization to avoid large determinant computations. Together, these methods reduce the per-evaluation cost of MI from $\mathcal{O}(|\mathcal{V}|^3)$ to $\mathcal{O}(|\mathcal{A}|^3)$, where $\mathcal{V}$ and $\mathcal{A}$ denote the candidate and selected sensing locations, respectively. Experiments on real-world bathymetry datasets show that Schur-MI achieves up to a $12.7\times$ speedup over the standard MI formulation. Field trials with an autonomous surface vehicle (ASV) performing adaptive IPP further demonstrate the method's practicality. By making MI computation tractable for online planning, Schur-MI helps bridge the gap between information-theoretic objectives and real-time robotic exploration. Our code is available at: www.sgp-tools.com
Read the full abstract