Robotics
GPUSimBench: Towards Scalable and Reliable GPU-Accelerated Simulators in Embodied AI
Huzhenyu Zhang, Shenghai Yuan, Wenrui Yan et al. (+4 more)
Data-driven
embodied AI is rapidly transitioning into a paradigm that scales training through massively parallel simulation, where GPU-accelerated simulators serve as the foundational data infrastructure. However, as computational throughput scales, the underlying trade-offs between parallel efficiency, physical fidelity, and execution determinism remain largely unexamined, hindering the development of reliable
robot learning. In this paper, we expose the hidden limits of mainstream GPU-based robotic simulators (e.g., Isaac Lab, Genesis) by introducing GPUSimBench, which focuses on scalability, physical consistency, and computational determinism. First, GPUSimBench establishes a physical grounding evaluation with a controlled inclined-plane task, quantifying the distributional
alignment between simulated dynamics and their real-world counterparts. Second, we
benchmark parallel scalability by measuring throughput and
memory footprints across scaling environment counts. Crucially, beyond standard performance metrics, we unveil and quantify the inherent non-determinism introduced by GPU-batched execution, characterized by significant run-to-run and inter-environment variability even under identical initial conditions. Finally, we identify four empirical regimes of stochasticity within current simulator stacks, highlighting that unbounded scaling can compromise reproducibility without explicit constraints.
Read the full abstract ▾
All authors Huzhenyu Zhang, Shenghai Yuan, Wenrui Yan, Li Ma, Hengjie Li, Jingcheng Pang, Dmitry Yudin
Robotics
Power from Potential: A Survey of Electrostatic Actuators for Haptics
Ahad M. Rauf, Ran Zhou, Eric Acome et al. (+5 more)
As haptic interfaces integrate more seamlessly into wearables and everyday environments, they increasingly require actuators that are soft, thin, silent, and energy efficient. However, conventional motors and temperature-responsive
polymers often struggle to deliver these properties due to their bulky form factors and high power consumption. High-Voltage Electrostatic Actuators (HVEAs), which generate force by applying an electric field to localized charge concentrations using high voltages and ultra-low currents, have recently emerged as a compelling alternative due to their fast, silent, and low-power operation within highly customizable and compliant form factors. This paper presents a focused review of HVEAs for haptics, examining four major classes: electrostatic switchable adhesives, dielectric elastomer actuators, soft electrohydraulic actuators, and electrokinetic pumps. For each class, we describe their mechanisms that enable haptic output; characterize their bandwidths, force densities, and spatial scalability; and evaluate their versatility for rendering cutaneous and kinesthetic feedback across wearable and world-grounded interfaces. Through this cross-technology analysis, we identify common design constraints and emerging strategies for improving ergonomics, streamlining fabrication, and integrating self-sensing. We conclude by outlining where HVEAs are uniquely positioned to advance haptic interaction and highlighting key research directions needed to translate these technologies into practical systems.
Read the full abstract ▾
All authors Ahad M. Rauf, Ran Zhou, Eric Acome, Madeline Balaam, Sean Follmer, Teng Han, Craig Shultz, Daniel Leithinger
Robotics
IMR: Iterative Mode-World Weighted Regression for Multi-Agent Trajectory Prediction
Honglin Wang, Shiyao Pan and Yun-Fu Liu
Multi-
agent motion prediction is essential for automated vehicles to understand the intentions of surrounding vehicles. However, previous prediction-based and anchor-based methods have limitations in mode diversity and prediction accuracy, respectively. These limitations may cause inadequate safety assessments and behavioral deviations in automated vehicles. To address this issue, a mode-world weighted regression loss is proposed to bridge the gap between these features. Specifically, this approach mitigates mode collapse while simultaneously improving world ranking and top-1 confidence. Furthermore, the proposed iterative decoder improves prediction accuracy by recurrently and segmentally generating trajectories. Experimental results show the proposed method ranks first in the Argoverse 2 multi-
agent motion
forecasting benchmark against other methods.
Read the full abstract ▾
Robotics
Uncertainty-Aware Velocity Correction for Proprioceptive Vehicle Localization using Evidential Mamba
Abinav Kalyanasundaram, Karthikeyan Chandra Sekaran, Wolfgang Utschick et al. (+1 more)
Reliable localization in GNSS-denied environments remains a fundamental challenge for intelligent vehicles, as inertial
navigation systems accumulate unbounded drift without external correction. Existing approaches provide drift correction through dedicated infrastructure, expensive external sensors, or complex multi-sensor fusion, each introducing practical deployment barriers. We propose Evidential Velocity Correction using Mamba (EVC-Mamba), a learning-based architecture that transforms onboard vehicle sensor data into a virtual velocity sensor for IMU drift correction without additional hardware. A Mamba-based selective state space model captures the temporal dynamics of vehicle motion, while evidential deep learning with a Normal-Inverse-Gamma distribution provides principled uncertainty quantification. The resulting uncertainty-aware velocity estimate is incorporated as a virtual correction measurement into an Error-State Extended Kalman Filter to reduce position drift. Evaluation on real-world vehicle data demonstrates that inertial
navigation using the proposed velocity correction achieves localization accuracy within 10% of a dedicated external velocity sensor across different outage durations. The proposed architecture supports real-time onboard deployment at 40 Hz on edge hardware, enabling reliable localization during prolonged GNSS outages.
Read the full abstract ▾
All authors Abinav Kalyanasundaram, Karthikeyan Chandra Sekaran, Wolfgang Utschick, Michael Botsch
Robotics
Efficient Transfer Learning of Robot Dynamic Models Using Morphological Similarity
Pavlo Kupyn, Yuya Hamamatsu, Roza Gkliva et al. (+2 more)
This study proposes a neural network-based transfer learning framework for modeling the dynamics of soft, fin-actuated
underwater robots. We focus on morphologically similar
robots that differ in scale and hydrodynamic properties. A model trained on data from a larger
robot (source domain) is adapted to a smaller one (target domain) with limited labeled data. To enable label-efficient transfer, we develop an autoencoder-based domain adaptation approach that learns a shared latent representation aligning the dynamics of both
robots. Experiments on two real
underwater robots show that the proposed method enables accurate state estimation of the body-frame velocities on a target platform without labeled data, highlighting its potential for efficient cross-
robot dynamics transfer among morphologically similar platforms.
Read the full abstract ▾
All authors Pavlo Kupyn, Yuya Hamamatsu, Roza Gkliva, Asko Ristolainen, Maarja Kruusmaa
Robotics
Physics-Regularized Machine Learning for Proprioceptive Vehicle Localization Using Onboard Sensors
Abinav Kalyanasundaram, Karthikeyan Chandra Sekaran, Wolfgang Utschick et al. (+1 more)
Accurate and robust localization is essential for autonomous mobility systems in real-world environments. While fusing Inertial Measurement Unit (IMU) data with satellite-based correction signals provides precise vehicle pose estimates, performance degrades substantially during outages. Recent studies indicate that Machine Learning (ML) can improve IMU-based proprioceptive localization, highlighting untapped potential for onboard sensors readily available in production vehicles. This paper introduces Physics-Regularized Machine Learning for Localization (PRML2), a hybrid framework that combines the complementary strengths of Kalman filtering and data-driven learning to estimate vehicle pose directly from onboard sensors. A key aspect of PRML2 is its physics-regularized learning, enabled by end-to-end training of an ML model through a differentiable Kalman filter. This improves consistency with vehicle motion models, thereby enhancing both localization accuracy and generalization across driving conditions. We evaluate the performance limits of ML-enhanced onboard odometry on a publicly available
dataset and show that PRML2 achieves superior localization accuracy and demonstrates real-time capability. This work also introduces a novel
dataset to support vehicle localization research under low-friction conditions. The proposed framework provides a robust and cost-effective solution for vehicle localization under degraded sensing conditions by integrating learning with physics-based priors.
Read the full abstract ▾
All authors Abinav Kalyanasundaram, Karthikeyan Chandra Sekaran, Wolfgang Utschick, Michael Botsch
Robotics
Dynamic Evaluation of Classical and Control-Aware Optimal Trajectory Planning in Robot Manipulators
Bhanuka Dayawansa and Rohan Munasinghe
Trajectory planning strongly influences tracking accuracy, actuator demand, and overall execution behavior in robotic manipulators. Classical planners such as cubic, quintic, and trapezoidal profiles are widely used for their simplicity and smoothness, yet they remain purely kinematic and ignore system dynamics and control effort during trajectory generation. As a result, nominally smooth trajectories can lead to inefficient nonlinear execution and increased corrective control action. This paper presents a control-aware optimal trajectory planning framework that explicitly incorporates manipulator dynamics and actuator effort within a finite-horizon formulation. A midpoint linearization strategy is introduced to improve approximation accuracy for large point-to-point motions. In contrast to prior comparisons, the proposed approach enables fair, isolated evaluation of trajectory generation effects under identical closed-loop nonlinear execution conditions. To this end, a unified evaluation framework is developed in which all planners are executed under identical nonlinear dynamics, controller structure, and actuator constraints. Simulations on a nonlinear simplified UR5 manipulator show that the proposed approach consistently reduces tracking error, corrective torque, and closed-loop execution cost compared to classical methods, achieving substantial reductions in actuator effort and execution cost across all evaluated scenarios, demonstrating that kinematic smoothness alone does not ensure dynamically efficient execution.
Read the full abstract ▾
Robotics
GEM-Occ: From Visual Geometry Evidence to Embodied Semantic Occupancy Memory
Hu Zhu, Bohan Li, Xianda Guo et al. (+8 more)
Embodied agents exploring indoor environments require reliable semantic occupancy
memory that persists across observations and revisits. Building such
memory is challenging because each observation provides incomplete and uncertain
geometric and semantic evidence. We introduce GEM-Occ, a Gaussian Evidence
Memory framework that consolidates evidence accumulated over time into persistent semantic occupancy
memory. Local predictions are converted into occupied semantic Gaussians and free-space ray evidence. Confidence- and visibility-aware causal updates integrate supporting observations, suppress occupancy contradicted by observed free space, and preserve previously observed structures through occlusion. A hierarchical
memory organization supports continued mapping and efficient queries across connected indoor spaces. To evaluate this capability, we introduce HIOcc, a unified
benchmark for
embodied semantic occupancy
memory. HIOcc establishes a shared semantic label space and evaluation framework spanning local prediction, room-level online mapping, and building-level mapping, while accommodating perspective and panoramic observations. Experiments on HIOcc demonstrate that GEM-Occ outperforms existing methods, enabling accurate semantic occupancy prediction and consistent online mapping across spatial scales with efficient
memory usage and fast occupancy queries.
Read the full abstract ▾
All authors Hu Zhu, Bohan Li, Xianda Guo, Yanlun Peng, Hongsi Liu, Baorui Peng, Xiaofeng Wang, Mingqi Yuan, Xin Jin, Wenjun Zeng, Chang Wen Chen
Robotics · Cross-listed from cs.SE
Guojun Chen, Alex Schott and Lin Zhong
Large
language models (LLMs) can plan behavior for
embodied agents from natural
language, but treating the LLM as a request/response oracle on the critical path is fundamentally at odds with real-time control and concurrent goals. We argue for an operating-system-style runtime for
embodied agents, and instantiate this idea in an early prototype, TypeGo. TypeGo structures LLM-based planning as asynchronous loops at multiple timescales that overlap with execution, and manages the
agent's physical body like an OS manages hardware: the Skill Kernel arbitrates typed physical subsystems among concurrent per-task processes, a scheduler preempts them and resumes or replaces each by source, and speculative skill streaming hides LLM latency behind ongoing motion, while a fast first-action path yields visible feedback within a second. Users program behavior through natural
language prescriptions that TypeGo dispatches to the LLM-based planners or compiles into low-latency interrupt handlers. Our prototype of Kalos, a Unitree Go2 quadruped, provides preliminary evidence for the design: in our current task suite, it cuts per-step delay by 50% over step-by-step planning and time-to-first-action by 73% over monolithic planning, while admitting concurrent tasks at low scheduling overhead.
Read the full abstract ▾
Robotics
Learning 4D Geometric Priors for Inference-Efficient World Action Models
Jianjun Zhang, Jian Zhu, Taiyi Su et al. (+4 more)
World Action Models (WAMs) have shown strong potential for robotic manipulation by jointly modeling visual future dynamics and executable action sequences. However, existing
video-action co-training methods primarily optimize appearance-oriented
video latents, which may insufficiently capture the temporally evolving geometry required for precise manipulation. We propose MECo-WAM, a Multi-Expert Co-Training World Action Model that injects action-relevant 4D
geometric priors into
video-action representations while preserving the original lightweight inference graph. During training, MECo-WAM combines
video and action experts with a lightweight 4D expert supervised by relational targets from a frozen VGGT encoder. Asymmetric expert visibility prevents non-causal shortcuts from auxiliary geometry to action generation. To transfer
geometric knowledge into the deployed
video-action pathway, we introduce decayed 4D read-mask
attention, which provides restricted current-frame
geometric guidance early in training and progressively removes this dependency. We further propose action-aware temporal
geometric distillation, which aligns within-frame
geometric relations and their temporal
evolution while emphasizing visual regions most relevant to
robot actions. At deployment, all auxiliary 4D components are removed. Experiments on LIBERO (98.2%), RoboTwin 2.0 (92.6%), and challenging real-world manipulation tasks show that MECo-WAM improves manipulation performance without increasing inference cost.
Read the full abstract ▾
All authors Jianjun Zhang, Jian Zhu, Taiyi Su, Chong Ma, Zitai Huang, Yi Xu, Hanli Wang
Robotics · Cross-listed from cs.AI
Governable Individuals: An Identity Layer for Embodied Agents That Keep Learning
Xue Qin, Simin Luan, Cong Yang et al. (+1 more)
Embodied artificial intelligence is moving from deployable models to persistent
agents that learn in the field, acquire skills and migrate across bodies. Governing such a system means governing an individual, not a model, and existing proposals (
agent identifiers, activity logs, guardrails) do not survive an
agent that keeps rewriting itself. We propose the governable individual: an
agent whose competence may change without bound, but whose authority,
memory schema, embodiment rights and capability roster can widen only through signed lifecycle transitions that update a public identity commitment. In our tests, neither learned judgement nor behavioural testing was sufficient to carry this on its own; the load-bearing layer must be architectural. We describe the abstraction, a runtime mechanism that realizes it, and the open problems in between.
Read the full abstract ▾
All authors Xue Qin, Simin Luan, Cong Yang, Zhijun Li
Robotics · Cross-listed from cs.CV
From Fixed to Free Cameras: Calibration-Free View-Robust Vision-Language-Action Model
Wenhao Li, Xueying Jiang, Quanhao Qian et al. (+4 more)
Real-world
robot deployment rarely maintains the training-stage camera setup, where cameras often experience repositioning or remounting depending on actual scenarios. Existing view-robust
Vision-
Language-Action (VLA) policies tolerate such camera variations only when the camera extrinsics are explicitly provided, making them fragile and hard to use especially when view robustness is critical. We argue that the policy should not be told where the camera is, but rather figure it out by itself. To this end, we introduce Camera-Centric VLA (CamVLA), a new VLA model that decouples manipulation controls from camera geometry by predicting (i) a camera-centric end-effector action expressed in the local camera frame, and (ii) a 6-DoF hand-eye
matrix relating cameras to the
robot base. A deterministic
geometric transformation composes the two predictions into a
robot base-frame action. This disentangles how I should move in pose-independent camera-centric action generation from where I am looking from in camera-perspective
geometric grounding. The resulting policy is calibration-free, depth-free, and single-view, requiring only a single
monocular RGB image as the visual observation and task instruction at deployment. Evaluations in both simulation and real-world
robot data show that CamVLA consistently improves success rates across diverse unseen viewpoints. Project page: https://alibaba-damo-academy.github.io/CamVLA/.
Read the full abstract ▾
All authors Wenhao Li, Xueying Jiang, Quanhao Qian, Deli Zhao, Shijian Lu, Gongjie Zhang, Ran Xu
Robotics · Cross-listed from cs.AI
LLM-as-a-Verifier: A General-Purpose Verification Framework
Jacky Kwok, Shulu Li, Pranav Atreya et al. (+6 more)
Scaling pre-training, post-training, and test-time compute have become the central paradigms for improving the capabilities of LLMs. In this work, we identify verification, the ability to determine the correctness of a solution, as a new scaling axis. To unlock this and demonstrate its effectiveness, we introduce LLM-as-a-Verifier, a general-purpose verification framework that provides fine-grained feedback for agentic tasks without requiring additional training. Unlike standard LM judges that
prompt LLMs to produce discrete scores for candidate solutions, LLM-as-a-Verifier computes the expectation over the distribution of scoring
token logits to generate continuous scores. This probabilistic formulation enables verification to scale along multiple dimensions: (1) score granularity, (2) repeated evaluation, and (3) criteria decomposition. In particular, we show that scaling the scoring granularity leads to better separation between positive and negative solutions, resulting in more calibrated comparisons. Moreover, scaling repeated evaluation and criteria decomposition consistently lead to additional gains in verification accuracy through variance and complexity reduction. We further introduce a cost-efficient ranking
algorithm for selecting the best solution among candidates using the verifier's continuous scores. LLM-as-a-Verifier achieves state-of-the-art performance on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%). Beyond verification, the fine-grained signals from LLM-as-a-Verifier can also serve as a proxy for estimating task progress. We build an extension for Claude Code, enabling developers to monitor and improve their own agentic systems. Finally, we show that LLM-as-a-Verifier can provide dense feedback for RL, improving the sample efficiency of SAC and GRPO on robotics and mathematical
reasoning benchmarks.
Read the full abstract ▾
All authors Jacky Kwok, Shulu Li, Pranav Atreya, Yuejiang Liu, Yixing Jiang, Chelsea Finn, Marco Pavone, Ion Stoica, Azalia Mirhoseini
Robotics
Deform360: A Massive Multi-view Visuotactile Dataset for Deformable World Models
Hongyu Li, Wanjia Fu, Xiaoyan Cong et al. (+11 more)
Predicting object dynamics (i.e., world modeling) is a fundamental challenge for robotic manipulation, and modeling deformable objects presents a particularly difficult case due to their high-dimensional state spaces and complex material properties. While current world models approach this through two distinct paradigms: learning the dynamics over the 2D pixel space or more explicit 3D
geometric space. A systematic understanding of their relative strengths and limitations remains elusive due to the lack of diverse, large-scale real-world data. To address this, we present Deform360, a large-scale visuotactile
dataset featuring 198 daily-life objects, 1,980 interaction sequences, and over 215 hours of observations from 41 surround-view cameras and bimanual tactile grippers to capture both global motion and contact-induced local deformations. Leveraging a novel markerless visuotactile 3D tracking pipeline to extract dense geometry and motion, we systematically evaluate current state-of-the-art world models, comparing 2D
video models against 3D particle models. Finally, we provide a preliminary demonstration indicating the real-world applicability of our
dataset by performing
robot planning tasks on deformable objects. Our analysis reveals key insights into the trade-offs between structural priors and scalability, providing a solid
benchmark for future research in generalizable deformable object-centric world modeling. Project website: https://deform360.lhy.xyz
Read the full abstract ▾
All authors Hongyu Li, Wanjia Fu, Xiaoyan Cong, Zekun Li, Binghao Huang, Hanxiao Jiang, Xintong He, Yiqing Liang, Rao Fu, Tao Lu, Srinath Sridhar, Kevin A. Smith, George Konidaris, Yunzhu Li
Robotics
Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation
Jiaqi Peng, Xiqian Yu, Delin Feng et al. (+10 more)
While recent
Vision-
Language-Action (VLA) models show promise toward generalist manipulation policies, they struggle with long-horizon tasks due to their Markovian nature-relying solely on current observations. Hierarchical dual-system methods address this but suffer from a gap between high-level planning semantics and low-level execution kinematics. We introduce Cortex, a bidirectionally aligned
embodied agent framework with a customized planning interface that conveys executable and tractable subtask plans from high-level VLM to low-level VLA. Specifically, we standardize manipulation subtasks into 32 canonical skill primitives and inject tractability principles, such as representative object attributes and improved trajectory reachability, into the data generation pipeline. This enables automatic annotation of over 4k hours of open-source
video data and generation of 30 hours of simulation data. We further devise an event-balanced sampling strategy to construct training data for fine-tuning the framework to better handle planning ambiguity during subtask transitions, enhanced by carefully designed harness engineering from task contexts to skill constraints during inference. Both open-loop VLM and closed-loop system evaluations demonstrate Cortex's efficacy, e.g., it outperforms monolithic baselines by 3.1% on Libero-long and 4.1% on RoboTwin. Notably, Cortex's generalist VLM enables zero-shot completion of unseen real-world long-horizon tasks, such as multi-stage chemistry experiments, by simply combining with a fine-tuned VLA-a capability infeasible through VLA fine-tuning alone.
Read the full abstract ▾
All authors Jiaqi Peng, Xiqian Yu, Delin Feng, Yuqiang Yang, Wenzhe Cai, Jing Xiong, Ganlin Yang, Jinliang Zheng, Jiafei Cao, Xueyuan Wei, Jiangmiao Pang, Yuan Shen, Tai Wang
Robotics
GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks
Kaiyuan Chen, Shuangyu Xie, Letian Fu et al. (+21 more)
For
robots to work reliably in commercial and industrial applications, can recent advances in agentic coding systems combine interpretable
robot programming with the open-world adaptability of model-free policies? We focus on "Variational Automation" (VA), a class of tasks that have larger variations in object geometry and pose than fixed automation. Model-free policies often struggle to close the reliability gap for VA tasks, which must be executed persistently and reliably in commercial and industrial applications. Motivated by prior work on Task and Motion Planning (TAMP) and the
Robot Operating System (ROS), we introduce Graph-as-Policy (GaP), a multi-
agent coding harness that generates directed computation graphs with perception, planning, and control nodes from a Modular Open
Robot Skill Library (MORSL). GaP then generates an internal simulation environment to rehearse task instances with different graphs in parallel to iteratively refine the graph structure and parameters to improve success rates and throughput. Evaluation with 8 new open VA task
benchmarks, 4 in-simulation and 4 in real-world, suggests that GaP can achieve success rates that significantly outperform baselines. Details, code, and data can be found online: https://graph-
robots.github.io/gap
Read the full abstract ▾
All authors Kaiyuan Chen, Shuangyu Xie, Letian Fu, Justin Yu, William Pacini, Sandeep Bajamahal, Hudson Kim, Jaimyn Drake, Daehwa Kim, Haoru Xue, Jonathan Francis, Christian Juette, Peter Schaldenbrand, Muhammet Yunus Seker, Ruwan Wickramarachchi, Uksang Yoo, Guanzhi Wang, Adithyavairavan Murali, Balakumar Sundaralingam, S. Shankar Sastry, Spencer Huang, Yuke Zhu, Linxi "Jim" Fan, Ken Goldberg
Robotics · Cross-listed from cs.CV
Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis
Xianhao Chen, Jiarui Hu, Yuanbo Yang et al. (+5 more)
Open-vocabulary 3D scene understanding aims to segment 3D scenes beyond predefined categories by transferring semantic knowledge from
vision-
language models. Existing methods have advanced this task by lifting
language-aligned 2D features into 3D, yet they often rely on context-independent semantic representations, leaving object relationships underexplored for contextual refinement. We propose RelGraphOV, a relationship-aware framework that uses 3D scene graphs to enhance open-vocabulary 3D understanding. Our method constructs relational scene graphs from multi-view observations by leveraging
vision-
language reasoning to infer object relationships and prune geometrically implausible connections, without manual relationship annotations. To aggregate relational context while avoiding feature interference, we introduce an Adaptive Gated Dual-Stream Contextual GAT that separates dense
geometric features and semantic CLIP
embeddings, performs edge-guided message passing, and adaptively fuses complementary semantics. A hierarchical contrastive objective further promotes instance-level consistency and category-level discrimination. Experiments on ScanNetV2, ScanNet200, ScanNet$++$, and Replica demonstrate strong performance and generalization ability. Project Page: https://cxavireh.github.io/relgraphov-projectpage
Read the full abstract ▾
All authors Xianhao Chen, Jiarui Hu, Yuanbo Yang, Xiyu Zhang, Tengyue Wang, Hujun Bao, Guofeng Zhang, Zhaopeng Cui
Robotics
Socially-Aware Autonomous Doorway Traversal and Payload Delivery for Emergency Assistance
Andrew Snowdy, Ananya Trivedi, Sarvesh Prajapati et al. (+2 more)
In this work, we focus on the scenario of a
robot-assisted emergency evacuation. We consider two capabilities relevant to such a setting. The first is opening doors ahead of the people being evacuated, so that their path toward an exit stays clear. The second is retrieving rescue equipment and delivering it to the emergency responders carrying out the evacuation. From a systems perspective, this involves several tasks at once. The
robot must locate ADA-compliant door buttons and the rescue equipment it needs to retrieve. Additionally, it must remain aware of the people around it and adapt its behavior to them, so that it supports the evacuation rather than getting in the way. We address these demands with a behavior tree at the core of our framework. This structure is chosen for its ability to select high-level tasks based on environmental triggers, and to extend to new situations as they arise. We evaluate the system in 105 trials on the Toyota Human Support
Robot, across five hardware and three simulation scenarios. These trials capture the decisions the
robot must make in this setting: whether to press a door button, yield to a nearby person, walk through a door someone else is holding, or first retrieve rescue equipment before traversing the door. Overall, the system completes 97 of the 105 trials successfully. These results suggest our framework provides a practical basis for robotic assistance in broader emergency response tasks. Code and
video demonstrations are available at https://github.com/AndrewSnowdy/hsr_mm_control.
Read the full abstract ▾
All authors Andrew Snowdy, Ananya Trivedi, Sarvesh Prajapati, Lorena Maria Genua, Taskin Padir
Robotics
GelNeuro: A Sensing-Computing Integrated Neuromorphic Tactile System for Texture Recognition
Luoyang Bian, Xinpan Meng, Zhenghua Ma et al. (+2 more)
Neuromorphic visuo-tactile sensing offers a promising paradigm for low-latency and low-power robotic perception. However, existing systems still rely heavily on a host computer for event readout, preprocessing, or relaying prior to chip inference. This paper presents GelNeuro, a fully integrated sensing-computing visuo-tactile system that directly pairs a GelSight Mini-based optical tactile front end with the Speck2f neuromorphic system-on-chip (SoC). Contact-induced marker motions are captured as dynamic
vision sensor (DVS) events and routed through the on-chip network to a spiking convolutional neural network (SCNN) classifier. To mitigate accuracy degradation during 8-bit deployment, a hardware-aware weight clamping strategy is introduced. Evaluated on a 15-class natural texture recognition task, hardware-in-the-loop testing on the physical chip achieves a 96.3% accuracy within an 80 ms inference window. Notably, the system consumes only 19.6 mW of board-level active power-over three orders of magnitude lower than conventional CPU/GPU baselines on the same
benchmark. GelNeuro also exhibits robust generalization across unseen contact depths, demonstrating the viability of direct sensor-to-chip tactile recognition on edge neuromorphic hardware.
Read the full abstract ▾
All authors Luoyang Bian, Xinpan Meng, Zhenghua Ma, Houcheng Li, Long Cheng
Robotics
VLM-CASE: Vision-Language Model Enabled Context-Adaptive Safety Envelopes for Anticipatory Safe Autonomous Driving
Tianjia Yang, Ke Li, Ruwen Qin et al. (+1 more)
Adverse driving conditions, such as bad weather, remain a principal barrier to autonomous driving because they degrade two things at once: what the vehicle can perceive and what it can physically do. Human drivers cope by anticipation,
reasoning about the scene and re-budgeting speed, following distance, and steering before grip or sight is lost, whereas current autonomous driving systems at best react after the fact. This paper proposes VLM-CASE, a framework that gives an autonomous vehicle this anticipatory capacity while keeping its motion bounded by a formal safety model at all times. A
vision-
language model (VLM), fine-tuned with low-rank adaptation (LoRA), reasons about the scene from the front-camera image and reports the road surface and visibility conditions. This output parametrizes a context-adaptive safety envelope (CASE), derived from physical limits and the guarantees of responsibility-sensitive safety, that couples braking and steering through a shared friction budget. A model predictive controller then drives freely within the envelope, while the VLM runs asynchronously so it never blocks the real-time control loop. We validate the framework in closed-loop CARLA simulation on tasks that demand both lateral and longitudinal control, across a range of weather, road-surface, and lighting conditions. The resulting controller, VLM-CASE-MPC, completes all trials, outperforming a conventional MPC baseline and a state-of-the-art VLM-integrated controller. Ablations confirm that the gains come from context adaptation, with the friction and visibility adaptations proving complementary. Furthermore, the framework is controller-agnostic and pairs with almost any low-level controller, offering a promising direction for safe autonomous driving. The
dataset and supplementary materials for VLM-CASE are available at https://github.com/ytj254/VLM-CASE.
Read the full abstract ▾
All authors Tianjia Yang, Ke Li, Ruwen Qin, Xianbiao Hu
Robotics
Geometry-Aware Visual Odometry for Bronchoscopic Navigation via High-Gain Observer Fusion
Mohammadreza Kasaei, Francis Xiatian Zhang, Feng Li et al. (+3 more)
Navigational bronchoscopy is critical for pulmonary interventions, yet current platforms depend heavily on pre-operative CT or external sensors, limiting their use in critical care and resource-constrained settings.
Vision-only
navigation offers a scalable alternative, but conventional visual odometry (VO) struggles with texture-poor airway images, specularities, and the vanishing-point singularities of tubular anatomy, leading to frequent tracking failures and drift. We present a geometry-aware VO framework that explicitly leverages vanishing-point cues from airway lumens. Detected lumens are back-projected to 3D rays, whose weighted fusion yields a stable forward heading even when parallax cues are absent. This heading, together with looming-based velocity estimates, is fused with noisy VO outputs using a bespoke high-gain observer that enforces airway-following priors and rejects drift. We validate the method on ex-vivo mechanically ventilated human lungs with electromagnetic tracking ground truth. Compared to state-of-the-art pipelines (ORB-SLAM2, LoFTR-VO, DPVO), our approach reduces absolute trajectory error by more than 50% and achieves the lowest relative pose error across all test sequences.
Read the full abstract ▾
All authors Mohammadreza Kasaei, Francis Xiatian Zhang, Feng Li, Farshid Alambeigi, Kevin Dhaliwal, Mohsen Khadem
Robotics · Cross-listed from cs.CV
Green for Go, Red for No: Visual Grounding via Semantic Segmentation for VLA Navigation Policies
Adrian Szvoren, Dimitrios Kanoulas and Nilufer Tuptuk
Vision-
language-action (VLA) models enable
robot navigation from natural
language and visual goals, but remain susceptible to perceptual distractions and ambiguous scene interpretations. This paper presents the first empirical evaluation of visual grounding for VLA
navigation policies. We propose a real-time
segmentation-based grounding method that highlights traversable areas in green and non-traversable areas in red using SegFormer. Two variants are evaluated: observation-only
segmentation and joint observation-goal augmentation. Using OmniVLA on the Grand Tour
dataset, we show that visual grounding reduces the mean waypoint error by 27-44% at the farthest waypoint, depending on the instruction length. The benefits are greater for long instructions than for short instructions, and grounding provides little improvement for image goals. Normalized error analysis indicates that grounding primarily acts as a trajectory length regularizer, reducing the predicted path length by 30% without improving per-unit-distance
reasoning. Our results indicate that visual grounding offers a simple, computationally inexpensive method to improve VLA
navigation without model retraining, although it cannot compensate for missing training signals in out-of-distribution instructions.
Read the full abstract ▾
Robotics · Cross-listed from cs.HC
Toward Personalized Social Robots for Child Well-being: Data Requirement Principles from a Recommender-System Perspective
Jin Huang, Eric Nichols, Fethiye Irmak Dogan et al. (+1 more)
Social
robots are increasingly deployed in clinical settings to support the well-being of children, where effective support must be personalized to each child. Personalization, choosing the
robot action best suited to each child, can be framed as a recommendation problem, and a recently proposed recommender-system framework for social
robots offers a principled approach through user profiling, ranking, and responsible computing. Instantiating it, however, is blocked not by the model but by the data, which is hard to gather. A child's state shifts within and across visits, so no fixed description of the user holds. Within a session, the few signals of whether the
robot's actions helped are weak and indirect. Across sessions, children are rarely seen more than once, and anonymization breaks the identity needed to link visits. Because care cannot be randomized, existing data is observational, biased toward whatever was already done. Each is a familiar recommender-system problem, and we propose four data principles in response: an integrated profile, effectiveness signals, linkable coverage, and an exposure record logged at collection time. We identify which of these principles each capability requires, and frame them as concrete guidelines for data collection.
Read the full abstract ▾
All authors Jin Huang, Eric Nichols, Fethiye Irmak Dogan, Hatice Gunes
Robotics
ECO: Incremental Ego-Centric Octree Update for Point Streams
Jaemin Yu, Seongyoon Jeong, Kang-Wook Chon et al. (+1 more)
Constructing octrees for mobile
robots that process continuous point streams in real time poses significant computational and
memory challenges. Standard global structures often suffer from high latency and unbalanced tree growth. We introduce the Ego-Centric Octree (ECO), a spatial data structure that acts as a 3D sliding window, dynamically bounding the mapping space to the
robot's immediate surroundings. ECO uses an efficient incremental update
algorithm that categorizes the environment into shift-out, shift-in, and overlap regions, eliminating redundant global coordinate transformations. Evaluations on the KITTI
benchmark demonstrate that ECO reduces update times by up to 25.60% (24.87% on average) compared to full static reconstruction and by up to 67.52% (54.60% on average) compared to a bounded incremental baseline. Furthermore, ECO substantially lowers the total system latency of downstream tasks, running up to 34.17% faster than full reconstruction in voxel-map generation. In dynamic scenes, ECO naturally retains a short-term temporal
memory of moving objects, providing useful temporal context while keeping update cost bounded and the tree balanced for real-time spatial perception.
Read the full abstract ▾
All authors Jaemin Yu, Seongyoon Jeong, Kang-Wook Chon, Duksu Kim
Robotics
InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action for Compositional Generalization
Haoxiang Ma, Junhao Cai, Xiaoxu Xu et al. (+26 more)
Unified models for
robot manipulation aim to equip one policy with both the semantic priors of pretrained VLMs and the physical dynamics learned through future prediction. In practice, existing designs tend to erode the semantics of the pretrained backbone, suffer interference among heterogeneous objectives, and learn future prediction from scratch in pixel space, leaving the dynamics priors of pretrained
video generators unexploited. We present InternVLA-A1.5, which builds the policy on a native VLM backbone that keeps training on VQA and subtask prediction, and attaches a lightweight unified expert for continuous action generation. Future prediction is recast as a latent-querying problem, where a small set of learnable foresight
tokens condenses the task-relevant future into a compact latent code under the supervision of a frozen pretrained
video generation model, so the policy inherits world-model dynamics priors without ever learning pixel-level generation. The
video branch is discarded at inference, keeping real-time control. Pretrained on 1.2M
robot episodes and 3M
multimodal samples, InternVLA-A1.5 achieves the best overall results on all six simulation
benchmarks. In the real world, the preserved semantics deliver the strongest compositional generalization on held-out instruction bindings, and the two designs together sustain long-horizon execution.
Read the full abstract ▾
All authors Haoxiang Ma, Junhao Cai, Xiaoxu Xu, Hao Li, Yuyin Yang, Yang Tian, Jiafei Cao, Hongrui Zhu, Zherui Qiu, Zhaxizhuoma, Yuqiang Yang, Jiaqi Peng, Xueyuan Wei, Yangkun Zhu, Jiahao Jiang, Xing Gao, Hanqing Wang, Feng Yuan, Kailin Li, Xueyue Zhu, Tai Wang, Yan Ding, Jiangmiao Pang, Jia Zeng, Jingjing Zhang, Bowen Zhou, Yao Mu, Chunhua Shen, Weinan Zhang
Robotics
Designing Touch for Trauma-Informed Social Robots: A Design Space for Direct and Indirect Actuation
Madeleine Rischer and Benedikt Bußmann
Touch is a fundamental communication modality in human-
robot interaction and may support grounding, emotional regulation, and stress reduction in therapeutic contexts. However, designing touch-based interactions for individuals with post-traumatic stress disorder (PTSD) requires careful consideration of trauma-informed care (TIC) principles, including safety, transparency, autonomy, and trigger avoidance. This paper investigates how touch actuation in social
robots can be designed to align with trauma-informed care. We distinguish between direct touch, involving physical contact between a user and a
robot, and indirect touch, mediated through artifacts such as wearables or smart textiles. Building on this distinction, we propose a design space comprising three dimensions: Actuation Modality, Objective of the Actuation, and Intended Effect of the Actuation. We analyze these dimensions through the lens of trauma-informed care and derive key design considerations for touch-based interventions. The resulting approach provides a conceptual foundation for developing trauma-sensitive social
robots that support individuals with PTSD through touch-based interactions.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
Qantara: Bridge-Flow Training for Multi-Paradigm JEPA Inference
Ruslan Rakhimov, George Bredis, Yuriy Maksyuta et al. (+1 more)
Joint-
Embedding Predictive Architectures (JEPAs) underpin a growing family of latent world models for control from raw pixels. Published JEPA world models for pixel-based control (PLDM, DINO-WM, LeWM, V-JEPA-2-AC) each commit at training time to a single inference paradigm, trajectory optimisation in a learned dynamics model, so a checkpoint trained to plan cannot be read off as a policy without retraining. Deferring that choice to inference would let the constraints that bind at deployment, per-step latency and whether a goal observation can be supplied, pick the path. We present Qantara, an end-to-end JEPA of about 21M parameters whose joint training objective pairs a Brownian-bridge interpolant between consecutive clean latents on the state axis with noise-to-data flow matching on the action axis. One checkpoint serves three inference paradigms without retraining: latent planning, behaviour-cloning action sampling, and a
video-inverse composition that predicts the next latent without action conditioning, then recovers the action bridging the two. Training samples the two noise levels along the edges of the (action-noise, state-noise) square, one edge per inference dispatch, so each path is trained where it is queried. Replacing that sampler with uniform sampling over the square at matched compute drops Push-T planning from 90.1 to 53.3 success rate (SR). On the LeWM control suite at a matched 10-epoch budget, Qantara averages 91.2 SR over the four environments and reaches 93.7 on OGBench-Cube, +7.7 over DINO-WM and +17.0 over LeWM, each measured against the stronger of that baseline's published score and our retrain of it. From the same weights the two goal-blind paths, which receive no goal image, reach 82.1-83.2 SR on Push-T and 70.8-72.8 on Cube at roughly 15-65x lower inference cost, and fall to near-noise on Reacher, whose released training corpus comes from a random policy.
Read the full abstract ▾
All authors Ruslan Rakhimov, George Bredis, Yuriy Maksyuta, Daniil Gavrilov
Robotics
Multi-Robot Open Adaptive Teaming Across Unseen Environments, Partners, and Scales
Yang Li, Feng Xue, Fan Mo et al. (+6 more)
Deploying
robot teams in the real world requires simultaneous adaptation to unseen environments, unknown partners, and varying team sizes, yet existing approaches often address these challenges in isolation under the closed-world assumption of fixed teammates. We formalize this as open adaptive multi-
robot teaming and propose a hypergraphic-form game formulation that captures team-level cooperative relationships beyond pairwise interactions, providing a principled foundation for coordination structure inference when team composition changes dynamically within episodes. Unlike graph neural network architectures, this is a game-theoretic construct for modeling strategic interactions and payoff structures among
agents. Building on this formulation, we develop the Hypergraphic Open-ended Learning
Algorithm (HOLA), which progressively expands partner and environment diversity during training rather than optimizing for fixed configurations. Evaluated on cooperative pursuit with multi-drone and multi-quadruped platforms, HOLA outperforms all baselines across all three adaptability dimensions. Learned policies transfer directly to physical hardware without fine-tuning, with successful deployments on Crazyflie and Zsibot L1 platforms confirming robust real-world coordination in novel environments with unseen teammates.
Read the full abstract ▾
All authors Yang Li, Feng Xue, Fan Mo, Yunhao Liu, Jianhong Wang, Ying Wen, Qingrui Zhang, Shaoshuai Mou, Wei Pan
Robotics · Cross-listed from cs.SE
Real-World Perturbation Testing of Autonomous Driving Systems
Stefano Carlo Lambertenghi, Matthias Weil and Andrea Stocco
Autonomous Driving Systems (ADS) must operate reliably under diverse conditions, yet representative data for rare or adverse scenarios is difficult to obtain. Perturbation-based testing is widely used to assess robustness, but most studies focus on offline
datasets or simulation, leaving open questions about how such results translate to real-world driving. We present a large-scale study of 72 camera and
LiDAR perturbations, evaluated across three testing modalities: offline model-level analysis, hardware-in-the-loop execution, and closed-loop system-level testing on a full-scale autonomous vehicle. The study covers both an end-to-end
vision-based driving model and a modular
LiDAR-based perception and planning stack. Our results reveal a clear gap between testing levels. For camera-based systems, perturbations with limited offline impact can still induce unstable control and failures in real-world driving. For
LiDAR-based systems, degradation is more consistent at the perception level but weakly predictive of system-level failures. Across both modalities, model-level metrics alone are insufficient to identify the most harmful perturbations. We further show that real-time feasibility is a key constraint in real-world testing, and that robustness observations obtained from recorded data do not consistently transfer to closed-loop behavior on a physical vehicle, highlighting the importance of complementary real-world, system-level evaluation.
Read the full abstract ▾