Robotics
Implementation of Tightly-Coupled SLAM Fusion of GPS, IMU, and LiDAR for Autonomous Vehicles
Amr O. Elmehrath, Farah Khaled, Rana Nahas et al. (+1 more)
Autonomous vehicles depend entirely on Simultaneous Localization and Mapping (SLAM) to navigate safely in unknown environments. However, relying on a single sensory modality introduces critical failure points:
LiDAR systems degrade in featureless corridors, Inertial Measurement Units (IMUs) accumulate mathematical drift, and GPS drops frequently in urban canyons. This paper presents the implementation of a tightly-coupled SLAM fusion architecture that integrates a Velodyne 3D
LiDAR, a high-frequency IMU, and GPS to achieve continuous spatial awareness. Utilizing a phased development methodology, we establish a 2D baseline to validate hardware synchronization and transform geometries before upgrading to a full 3D architecture driven by FAST-LIO2. This advanced approach uses an Iterated Error-State Kalman Filter (IESKF) to process dense 3D
laser points alongside continuous inertial data, eliminating motion blur at high speeds. To eradicate long-term drift, a GTSAM pose-graph optimization back-end executes multi-modal loop closures. Evaluated across simulated environments and physical deployments, the results demonstrate that tightly-coupled 3D fusion effectively overcomes individual sensor blind spots to generate highly detailed point clouds, providing the foundational High-Definition (HD) maps required for advanced downstream autonomous planners.
Read the full abstract ▾
All authors Amr O. Elmehrath, Farah Khaled, Rana Nahas, Catherine M. Elias
Robotics
LLM-Guided Transformation of Non-Critical Driving Scenes into Safety-Critical Scenarios Using Augmented Reality
Noura Fady, Farah Khaled and Catherine M. Elias
Testing Autonomous Driving Systems (ADS) requires realistic safety-critical scenarios, but collecting such data from real-world driving is costly and unsafe. This paper presents an automated pipeline that transforms safe driving scenes into safety-critical scenarios by combining computer
vision, Large
Language Models (LLMs), and Augmented Reality (AR). The system detects and tracks road users, extracts safety features including distance, velocity, motion direction, and Time-to-Collision (TTC), and assesses scene criticality. Safe scenes are modified by an LLM, which generates realistic collision-inducing objects and behaviors that are integrated into the original scene using AR. The proposed pipeline was evaluated on the nuScenes
dataset, achieving 97.52% safety classification accuracy and successfully generating realistic scenarios such as pedestrian crossings, rear overtaking vehicles, and sudden-stop events. The results demonstrate an effective and flexible approach for automated generation of safety-critical scenarios to support the testing and validation of autonomous driving systems.
Read the full abstract ▾
Robotics
Generalizing deep reinforcement learning across cable-driven parallel robot configurations with actuator-level policies
Abir Bouaouda, Mohamed Boutayeb, François Charpillet et al. (+2 more)
Cable-driven parallel
robots (CDPRs) present diverse configurations and complex control challenges, which can be addressed by deep reinforcement learning (DRL) by learning their nonlinear dynamics. However, DRL methods often require extensive training time, and the resulting policies do not generalize well to different
robot configurations or varying numbers of actuators. In this article, we introduce a novel DRL approach for controlling CDPRs that does not depend on the specific
robot configuration. Our method trains an actuator-level policy that controls each motor to achieve its target cable length, in contrast to conventional DRL approaches that learn to control the entire
robot to reach a desired end-effector position. To the best of our knowledge, this is the first work to apply DRL to control CDPRs using an actuator-level policy. This approach offers two main advantages: (i) a single shared policy can be applied to any CDPR configuration, regardless of actuator count, and (ii) reliance on inverse kinematics, avoiding the more challenging forward kinematics problem. Training is performed in simulation, and the learned policy is successfully transferred to a real CDPR. Experimental results show that the actuator-level policy (ALP) surpasses traditional reinforcement learning methods in both robustness and precision. We further control a real 8-motor CDPR with 3D motion using a policy trained on a simulated 4-motor planar CDPR operating in 2D. This illustrates that the proposed method is applicable to any CDPR configuration, independent of actuator number or placement.
Read the full abstract ▾
All authors Abir Bouaouda, Mohamed Boutayeb, François Charpillet, Dominique Martinez, Rémi Pannequin
Robotics
Action Chunk Scheduling for Batched Robot Policy Serving
Rohan Bansal, David He, Nadun Ranawaka Arachchige et al. (+4 more)
Deploying
robot foundation models at scale is the next step towards realizing the potential of general-purpose
robots. However,
Vision-
Language-Action (VLA) and other foundation models are computationally demanding, and on-device compute is constrained by power and space. In this paper, we introduce the problem of serving a
robot policy to multiple
robots from a remote GPU and formulate it as a scheduling problem. We build Armory, a serving system validated on fleets of both simulated and real
robots. Our experiments show that naive scheduling heuristics perform well when all
robots are the same, but fall short when
robots consume action chunks at different rates, uncovering a mismatch between conventional batching methods and the closed-loop requirements of
robot policy execution. To address this, we propose a scheduling
algorithm that accounts for this heterogeneity and improves overall system throughput by up to $18\%$ in real-world experiments. Additional details are available at https://gatech-rl2.github.io/actionchunkscheduling.
Read the full abstract ▾
All authors Rohan Bansal, David He, Nadun Ranawaka Arachchige, Zhenyang Chen, Soobum Kim, Kexin Rong, Danfei Xu
Robotics
Belief-Space Perception Routing under Coupled Sensor Faults and Compute Contention
Sparsh Roy, Vihan Aggarwal and Davin Yin
A
robot that has to see and react on a fixed clock runs into two problems at once. Its cameras degrade in rain, mud, fog, and darkness. And the single onboard processor it runs on is shared with planning and control, so the compute left over for perception moves around from second to second. Most systems model the two separately. We present a perception router that tracks probabilistic estimates of sensor-fault state and compute- contention state, couples them with a noisy-OR term, and uses the coupled estimate to pick one of four detector configurations (YOLO11x/n at 1280 or 640 px) so that the frame finishes before its deadline. Where the two stressors co-occur, the coupled policy cuts the deadline-miss rate by 1.1 to 9.4 percentage points against a policy that treats them independently. The interval excludes zero in five of six conditions, the pooled effect over 10 sequences and 6 conditions has sign-test p = 0.001, and every uncoupled control and the fault-free trajectory sit at exactly 0.0 pp. Routing costs tens of microseconds per frame. We then asked whether the coupling the method exploits arises on its own. Across eight real RADIATE adverse-weather sequences and three workload proxies independent of the fault signal, after Benjamini-Hochberg correction and a replication run, none of 24 tests found it. We report that null and scope the routing result as a proof of mechanism. Whether such coupling occurs in the field is still open, and the released evaluation pipeline lets a deployment settle it on its own traces.
Read the full abstract ▾
Robotics
Towards General Language-Conditioned Latent Safety Filters
Ihab Tabbara, Yuxuan Yang and Hussein Sibai
Robot policies are becoming increasingly general, with
vision-
language-action (VLA) models enabling a single policy to execute diverse tasks specified in natural
language. Safe deployment, however, requires adapting not only to new tasks but also to varying safety requirements across users, environments, and applications. Existing safety filters remain largely constraint-specific and thus must be redesigned or relearned when safety requirements change. In this paper, we investigate
language-conditioned safety filtering, in which a Hamilton-Jacobi safety actor and critic are conditioned on
language-specified constraints. We evaluate this formulation across pick-and-place, table-wiping, and block-stacking tasks in the
vision-based setting, examining its ability to enforce
language-specified constraints and transfer to unseen constraint instances within the evaluated constraint families. Our experiments provide evidence that
language-conditioned safety filters reduce constraint violations and exhibit partial transfer to unseen constraint instances.
Read the full abstract ▾
Robotics · Cross-listed from cs.AI
RF-HOI: Recognize Human-Object Interaction with Radio Frequency Signals
Lihao Wang, Linlu Gao, Jiacan Yu et al. (+5 more)
Recognizing Human-Object Interactions (HOI) is essential for intelligent systems, underpinning applications in virtual and augmented reality,
embodied AI, and assistive robotics. However,
vision-based HOI methods face challenges in
privacy concerns and poor light conditions. In this work, we introduce RF-HOI, the first framework that only uses
radio frequency (RF) signals for HOI recognition. A key challenge of RF-HOI is that single-modality RF sensing is insufficient to recognize both actions and the objects being interacted with. RF-HOI addresses this through a novel modality fusion that combines mmWave radar and RFID, enabling simultaneous action recognition and target identification. Another challenge is limited training data across diverse setups, which impairs the generalizability of the recognition model. To overcome this, we develop a simulator that synthesizes
multimodal RF data for diverse HOIs at scale, allowing us to fine-tune with only a small amount of real-world data. Experiment results show that RF-HOI outperforms all baselines, approaching
vision model performance, and that our diverse synthetic training data can significantly boost our system's performance on real-world scenarios. These results highlight the potential of
multimodal RF sensing for robust and
privacy-preserving HOI recognition as well as the effectiveness of our RF data synthesis.
Read the full abstract ▾
All authors Lihao Wang, Linlu Gao, Jiacan Yu, Yanyu Lin, Yifan Yin, Jianxin Wang, Tianmin Shu, Renjie Zhao
Robotics
Hybrid Attention Estimation Pipeline for Adaptive HRI Using an Expressive Robotic Head
Pablo Moraes, Monica Rodriguez, Christopher Peters et al. (+4 more)
This paper presents an applied case study on hybrid visual
attention estimation for human-
robot interaction using an expressive robotic head based on the InMoov ecosystem. The proposed pipeline combines a fast
geometric perception layer with an independent semantic perception layer based on a
vision-
language model. The
geometric layer provides high-frequency face and head-pose information for temporal regulation, while the semantic layer receives only raw
egocentric camera frames and produces contextual
attention labels related to
attention toward the
robot, phone use, or
attention elsewhere. These signals are integrated through a finite state machine that regulates adaptive interaction behavior, including activation, waiting, interaction resumption, and return to rest. The system was evaluated with 10 participants across 40 trials covering baseline and adaptive interaction conditions. Results show reliable interaction start across all trials, consistent pause behavior in the adaptive distraction condition, and non-redundant semantic information between the
geometric and semantic outputs.
Read the full abstract ▾
All authors Pablo Moraes, Monica Rodriguez, Christopher Peters, Hiago Sodre, Tobias Doernbach, Bruna Guterres, Ricardo Grando
Robotics
Localization in Spatiotemporal Fields via Environmental PDEs
Jose Fuentes, Abdullah Al Redwan Newaz, Ana Cavalcanti et al. (+1 more)
This paper proposes a localization framework that uses spatiotemporal fields governed by partial differential equations (PDEs) as localization signatures. Two PDE classes are considered: the shallow water equations, which describe free-surface flows in coastal and riverine environments, and the advection-
diffusion equation, which models the transport and mixing of scalar quantities such as temperature, salinity, and dissolved oxygen. A numerical PDE solver provides predicted fields over the domain, and multiple field channels are fused as
multimodal measurements to improve localization accuracy. We formulate the problem within a Rao-Blackwellized particle filter (RBPF) that partitions the vehicle state into a nonlinear component sampled by particles and a linear sensor bias component tracked analytically via per-particle Kalman filters. This factorization reduces the required number of particles compared to a standard particle filter while accounting for realistic sensor drift. Simulation studies on both PDE scenarios show that the RBPF consistently outperforms a standard particle filter in terms of final position error and Root Mean Square Error (RMSE) across varying particle counts. Field experiments with an autonomous surface vehicle measuring salinity, temperature, and dissolved oxygen validate that PDE-governed environmental fields provide sufficient spatial variability for practical localization. Related experimental
videos are available at https://localization-environmental-pdes.github.io/.
Read the full abstract ▾
All authors Jose Fuentes, Abdullah Al Redwan Newaz, Ana Cavalcanti, Leonardo Bobadilla
Robotics
Compliant Sphere Lattice Contact: Distributed Contact Modeling for Sphere-Based Robot Representations
Nataliya Nechyporenko, Ava Abderezaei and Alessandro Roncone
Contact planning in robotics requires models that are both computationally efficient and physically accurate. Sphere-based
robot representations satisfy the first requirement by enabling fast collision checking and differentiable geometry, but sacrifice physical accuracy by relying on point contact which cannot capture contact patch area, pressure distributions, rotational stiffness, or frictional moments. We introduce Compliant Sphere
Lattice Contact (CSLC), a distributed contact model that operates natively on sphere representations by modeling the
robot interface as a compliant
lattice of surface spheres connected through anchor and lateral springs. When pressed against an object, the
lattice deforms to produce a spatially distributed contact patch that improves the physical accuracy of sphere-based contact. We validate CSLC across two independent solvers and show preliminary results demonstrating contact patch formation and improved grasp stability.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Latent-Centroid Steering: Single-Pass Classifier-Free Guidance for Command-Aligned Autonomous Driving
Meibo Hu, Jiamian Wang, Pichao Wang et al. (+1 more)
Vision-
language models (VLMs) have recently emerged as a promising paradigm for end-to-end autonomous driving, enabling
agents to map
multimodal inputs and high-level
navigation instructions directly to executable trajectories. However, in practice, these models exhibit a persistent command-following gap: predicted trajectories often show weak sensitivity to
navigation commands, resulting in incorrect behavior at critical decision points. We identify this issue as a form of conditional policy collapse, where regression-based training under
multimodal trajectory distributions encourages the model to rely on dominant visual priors while marginalizing the
language-conditioned signal. To address this issue, we introduce a principled formulation of classifier-free guidance (CFG) for regression-based
vision-
language driving. We show that CFG can be interpreted as isolating the instruction-induced residual in the action space by contrasting conditional and unconditional predictions, thereby explicitly amplifying the effect of the
navigation command at inference time. However, a standard two-pass CFG introduces prohibitive latency for real-time control and produces noisy instance-level guidance directions. Building on a mean-shift interpretation of CFG, we propose Latent-Centroid Steering (LCS), a single-pass guidance mechanism that replaces instance-level residuals with class-level latent shifts. By projecting conditional representations toward precomputed command-specific centroids, LCS performs class-level latent steering based on cluster geometry that is both more stable and computationally efficient. We demonstrate that LCS reduces inference latency by approximately 50% while achieving stronger command adherence and improved driving performance on both closed-loop (Bench2Drive) and open-loop (nuScenes)
benchmarks. Code are available at https://github.com/codingmlinprocess/LCS.
Read the full abstract ▾
All authors Meibo Hu, Jiamian Wang, Pichao Wang, Zhiqiang Tao
Robotics
Developing Combined Manipulation and Locomotion Skills with Interaction Representation and Skill Composition
Fanxing Meng and Jing Xiao
This paper addresses how to enable a
humanoid robot to learn motion policies based on developmental principles and combine policies to create more sophisticated and useful behaviors. Specifically, we present an approach to (1) learning a whole-body reaching and grasping policy and (2) combining it and a standing-up and walking policy to compose a more complex policy of manipulation and locomotion: grasping, standing up, and walking. In (1), our method draws inspiration from harmonic analysis and adopts cubic harmonics as weights to represent the hand-object spatial relationship via spatial convolution. Utilizing an intra-episode finger joint decoupling curriculum based on developmental principles, a
robot can autonomously learn a generalizable grasping policy without relying on external
datasets or pretrained models. In (2), our method combines the grasping policy with a separately learned getting-up policy by providing both policies with their respective observation vectors and using hand-object interaction scores to determine when each policy should control which
robot joints. Our results show a 93% zero-shot success rate for grasping unseen objects and a 96-100% success rate for standing up while holding the object. Our work also demonstrates that combining different policies is only effective if each policy learning happens on the same whole
humanoid body even if a policy (such as for locomotion) does not seem to need all the body parts (such as fingers).
Read the full abstract ▾
Robotics
Motion Planning for Mobile Manipulators Navigating Doorways via Model Predictive Control
Kasra Sinaei, Kasun Weerakoon, Christopher Bradley et al. (+2 more)
Navigating doorways is a fundamental capability for mobile manipulators operating in human environments, requiring coordinated motion between the mobile base and manipulator arm. This paper presents a motion planning framework that generates dynamically feasible and collision-free trajectories for autonomously opening and traversing both push and pull doors. The proposed method formulates the
robot and door as a coupled dynamical system within a nonlinear Model Predictive Control (MPC) optimization framework. Manipulation feasibility is enforced through a penalty-based constraint, avoiding explicit arm kinematic modeling in the planner. Simulations and a hardware experiment demonstrate that the approach successfully plans feasible trajectories for door traversal.
Read the full abstract ▾
All authors Kasra Sinaei, Kasun Weerakoon, Christopher Bradley, Seyed Abolfazl Fakoorian, Donald Ebeigbe
Robotics
Track-Guided Hierarchical Reinforcement Learning for Autonomous Vehicle Drifting with Minimum-Lap-Time Planning
Sheng Zhao, Bolin Zhao, Xiaodong Wu et al. (+1 more)
In Formula 1, drivers optimize racing lines within tire grip limits to minimize lap times; however, in rally racing, drivers intentionally break traction to drift on loose surfaces. This maneuver rapidly aligns the vehicle for corner exits, ultimately reducing lap time. Autonomously executing such maneuvers formulates a complex dual-objective control problem: stabilizing highly nonlinear drift dynamics while strictly minimizing lap time. Addressing this challenge motivates the development of advanced Minimum-Lap-Time (MLT) drift control architectures. This paper proposes a planning-control framework specifically designed for MLT drifting scenario. First, we formulate an optimal control problem to generate a MLT drift planning trajectory, which is used as prior data to train a deep reinforcement learning drift controller. Given that drifting involves extremely large sideslip angles and is therefore challenging to learn directly, a Track-guided Reinforcement Learning (TgRL) drift control method is proposed to enable progressive training in a step-by-step manner, from drift control policy, to drift corner policy, and finally to a comprehensive drift race policy. The reward function incorporates both an instant reward term and an end reward term derived from the Minimum-Lap-Time objective. Simulation results demonstrate that the proposed framework enables the
agent to learn a drift racing policy that not only ensures vehicle motion control performance but also effectively reduces lap time.
Read the full abstract ▾
All authors Sheng Zhao, Bolin Zhao, Xiaodong Wu, Chen Lv
Robotics
Diagnosing Compositional Generalization in Sequential Robot Tasks
Yixiao Wang, Cheng-En Wu, Lingfeng Sun et al. (+5 more)
Sequential
robot manipulation requires policies to execute novel combinations of familiar instruction components. However, collecting demonstrations for all possible instruction tuples is combinatorially expensive, while sparsely covered
datasets often fail under out-of-distribution recombination. This paper studies compositional generalization through the lens of instruction-space coverage. We decompose the generalization gap into three sources: marginal instruction shift, instruction-compositional shift, and context–action shift. This decomposition allows us to diagnose when sparse training coverage is sufficient, and what structure the training set must preserve for reliable action prediction. Our results show that exhaustive tuple enumeration is unnecessary: a structured subset can recover strong out-of-distribution performance when it covers action-relevant dependencies. We further find that sparse training often fails due to instruction steering rather than missing low-level skills; finetuning with only one demonstration per task improves OOD success from \(0.4\%\) to \(54.7\%\). For semantically dependent tasks, effective coverage must capture relational structure rather than only factor diversity. These findings suggest that efficient
robot data collection should prioritize dependency coverage in instruction space over exhaustive task expansion. Additional results, code, and
dataset are available on the \href{https://yixiaowang7.github.io/Diagnosing_Compositional_Generalization_
Robot_Page/}{project website}.
Read the full abstract ▾
All authors Yixiao Wang, Cheng-En Wu, Lingfeng Sun, Pengcheng Wang, Xiang Ji, Boyuan Liang, Guojian Zhan, Masayoshi Tomizuka
Robotics
Bootstrapping Self-Supervised Learning of Binary Classification Using Error Bounds: A Case Study on a Robotic Insertion Task
Zebin Duan, Norbert Krüger, Juan Heredia et al. (+2 more)
Flexible manufacturing requires rapid deployment of solutions and minimal setup time to remain competitive. An essential attribute is the ability to control error levels, as failures can range from minor performance degradation to severe equipment damage. However, conventional deployment often involves extensive setup, data collection, model training or parameter tuning, and system testing, resulting in significant delays that hinder commercial feasibility. We propose a data engine which gathers data and improves its performance while executing the task. The data engine consists of two classifiers, a fast model prediction and expensive verification. First, a model prediction is performed and based on the confidence level of the prediction, the expensive verification can be used. By adjusting the confidence level, users can control the level of tolerable error. Our method is implemented on a real-world robotic insertion task, which uses force data for the model prediction. The system applies UMAP dimensionality reduction and uses Wilson-Score to compute the confidence bounds of the prediction. Results demonstrate the ability to learn and reduce the need for expensive verifications over time, while staying within the set error-rate. The results highlight the potential of confidence bounds in self-improving models to enhance reliability in robotic classification task.
Read the full abstract ▾
All authors Zebin Duan, Norbert Krüger, Juan Heredia, Thorbjørn Mosekjær Iversen, Frederik Hagelskjær
Robotics
Balancing of Humanoid with Object Mass: Trade-off Analyses and Lifting Control
Hyunjong Song, William Z. Peng and Joo H. Kim
The demand for
humanoid loco-manipulation tasks with an object has recently increased, and most existing control approaches for stability in such tasks rely on heuristics or machine-learning techniques. This study rigorously analyzes and exploits the dynamic effects of the object mass on balance stability. By formulating the object mass parameters in the whole-body dynamics with distributed contact wrenches and centers of pressure at the stance contacts, their nonlinear effects on the system momenta and constraints are quantified. The dynamic models and constraints are incorporated into the construction of the balanced state basin/boundary (BSB), a partition of the center-of-mass state space for a biped system to maintain balance in its desired contacts. The implications of the BSB for prediction and control are highlighted using a
humanoid robot and an analytically tractable reduced-order mechanism. The BSBs under different conditions of base of support, actuation capacity, and pose provide systematic analyses of the effects of object mass on the balancing capability of a system. In particular, the trade-off relationships between momentum regulation and limiting factors in balancing are characterized, introducing two key quantities of the object: the critical mass, at which the system's balancing capability is maximum, and the transition mass, which activates different limiting factors. In addition, sufficient conditions for imposing balanced states on a trajectory are established and implemented with BSBs as explicit threshold constraints in the whole-body trajectory optimization for stable object-lifting control of the
humanoid, demonstrating the lift-and-hold and lift-and-release tasks with distinct mass properties in simulations and experiments.
Read the full abstract ▾
Robotics
RayViT: Ray-Conditioned Visual Representations for Viewpoint-Robust Imitation Learning
Qian Wang, Longrui Chen, Peiran Sun et al. (+8 more)
Visual imitation learning enables
robots to acquire visuomotor skills directly from images, yet RGB observations lack explicit
geometric cues, making learned policies brittle to camera perturbations. To address this, we propose Ray-conditioned
Vision Transformer Encoder (RayViT), a lightweight architecture that injects camera geometry into pretrained ViT backbones. RayViT represents camera geometry as a Plücker ray map, patchifies it into ray features, and uses gated cross-
attention to produce a ray-conditioned class
token. These ray features are added as dense positional
embeddings, while the ray class
token replaces the original ViT class
token to provide a geometry-aware summary representation. We combine this approach with an auxiliary cosine similarity loss to consistently improve the performance and robustness for geometry-aware
tokens. Experiments on sim- and real-
robot tasks demonstrate that RayViT improves robustness by approximately 13 percentage points under camera perturbations in multi-task RoboCasa
benchmark and by 1.78 average completed stages in real-world multi-task success rate compared to baselines.
Read the full abstract ▾
All authors Qian Wang, Longrui Chen, Peiran Sun, Aleksandar Taranovic, Niklas Freymuth, Ge Li, Weiran Liao, C. F. Maximilian Nagy, Yucheng Tan, Tao Chen, Gerhard Neumann
Robotics
WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning
Senyu Fei, Xiaopeng Yu, Siyin Wang et al. (+3 more)
Reinforcement learning (RL) post-training of
Vision-
Language-Action (VLA) models has shown strong promise for robotic manipulation. Among RL methods, critic-based approaches rely on a value
estimator that predominantly operates on single-frame observations or single-frame VLM backbone latents, which is a fundamental mismatch with the partially observable nature of
robot control. A naive approach to incorporate observation history into the critic incurs exponential complexity with high-dimensional visual space, and still fails because pure scalar-return regression provides insufficient supervision for learning cross-temporal dynamics. We identify the root cause as a state approximation problem: without an explicit world modeling objective, the critic's representation cannot capture the temporal structure needed for accurate value estimation. To address this, we propose the World Critic Model (WCM), built on a lightweight LeJEPA architecture; WCM jointly predicts future latent state and estimates values, such that the critic's representation is explicitly trained to capture temporal dynamics rather than merely regress scalar returns. WCM integrates seamlessly into both on-policy and off-policy training pipelines and is compatible with state-of-the-art VLA backbones including Pi0, Pi0.5, and OpenVLA-OFT. Extensive experiments on 149 tasks across four
benchmarks demonstrate that WCM consistently achieves state-of-the-art performance in both in-distribution and out-of-distribution settings, with particularly strong generalization gains. We further validate WCM on seven real-world manipulation tasks using OpenVLA-OFT and Pi0.5 with off-policy RL, confirming stable deployment across diverse settings.
Read the full abstract ▾
All authors Senyu Fei, Xiaopeng Yu, Siyin Wang, Xianzhong Zhao, Jingjing Gong, Xipeng Qiu
Robotics
HAM-VLN: Harnessing Hierarchical Agentic Memory for Zero-Shot Vision-and-Language Navigation
An Liu, Bingxi Liu, Hongyu Ding et al. (+6 more)
Vision-and-
language navigation (VLN) enables
robots to follow instructions in previously unseen environments. Recently, a training-free paradigm has emerged: the
robot queries a
multimodal LLM to understand its observations and plan the next action. However, long-horizon
navigation based on either image streams or dense map inevitably introduces a growing
memory and
reasoning bottleneck. We present HAM-VLN, a decision-coupled,
agent-authored
memory that equips the
robot with a persistent, depth-grounded world graph. In the same model call used to select the next action, HAM-VLN also records semantic and reflective information—including room type, objects,
navigation progress, and failure notes. Recent waypoints remain verbatim within a bounded window, while older history re-enters the context only through
retrieval scored by relevance, recency, and salience, together with one-hop topological expansion. This design requires no additional LLM calls beyond the per-waypoint decision. Compared to previous methods, HAM-VLN not only improves various
navigation metrics but also reduces the context length by more than 65%. Specifically, HAM-VLN achieves 61.0% Success Rate (SR) on VLN-CE R2R, 52.7% SR on VLN-CE RxR, and 79.7% SR on HM3D-v2 ObjectNav without any training.
Read the full abstract ▾
All authors An Liu, Bingxi Liu, Hongyu Ding, Yixuan Jiang, Yaran Chen, Fulin Tang, Cong Leng, Hong Zhang, Jian Cheng
Robotics
FibVLA: An Efficient Temporal Vision-Language-Action Model with Fibonacci Sampling
Li Lin, Wujun Xu, Weiwei Meng et al. (+3 more)
Vision-
language-action models (VLAs), which leverage the cognition of
multimodal information to infer physical-world actions, provide a generalized solution for
embodied AI applications. Conventional VLAs usually concentrate on current digital cognition. While some efforts are made to enhance VLAs'
reasoning capabilities by capturing temporal information, encoding the long-context history causes an efficiency-decreasing issue. To reconcile the conflict between capturing temporal information and maintaining inference efficiency in VLAs, this paper introduces FibVLA, an efficient framework featuring temporal perception of long-context history. Specifically, we leverage logarithmic hindsight sampling to both proprioceptive states and visual frames to capture long-term temporal dependencies with minimal redundancy. For the action expert, we introduce the flow matching to produce action distributions, and the Fibonacci recurrent inference strategy to generate long-range planning steps based on real-time closed-loop feedback. Experiments demonstrate that FibVLA significantly improves action smoothness and success rates without retraining large-scale visual encoders. Efficiency analysis demonstrates superior real-time responsiveness compared to
video-based baselines in real-world evaluations.
Read the full abstract ▾
All authors Li Lin, Wujun Xu, Weiwei Meng, Kaiwen Xia, Kang Hao Cheong, Shuai Wang
Robotics
Safe Vision Language Action Models via Barrier Enhanced Flow Matching
Kasra Sinaei, Hung-Chieh Wu and Donald Ebeigbe
This article presents a modular inference framework that integrates Flow Matching generative models with formal Control Barrier Function (CBF) safety guarantees. Unlike existing methods that apply external safety filters to a model's final output, our approach modifies the Flow Matching denoising process within the model to inherently generate safe trajectories. By employing a smooth Log-Sum-Exponential aggregate barrier, we enforce safety over entire action chunks. This aggregate barrier ensures a minimal increase in computational overhead and does not alter the semantic intent of the model. We show that, within the proposed framework, the 2-Wasserstein distance between the generated distribution and the target distribution remains bounded. Our method eliminates the need for safety-specific
datasets or costly model retraining, providing a versatile solution for safe inference. We validate the approach on two robotic manipulation platforms and a 2D
navigation benchmark, verifying that our framework achieves reliable safety without degrading the success rate of the model.
Read the full abstract ▾
Robotics
TransGraspNet: Physically and Geometrically Consistent Manipulation of Transparent Labware
Hailing Hu, Mingyi Zhu, Yiquan An et al. (+3 more)
Manipulating transparent laboratory glassware that contains
liquid is inherently safety-critical: even small
geometric errors can cause unstable grasps and hazardous spillage. Although recent progress has been made in transparent object perception and robotic grasping, most existing systems optimize detection, depth reconstruction, and grasp planning independently, which leads to cross-stage inconsistency imperfect boundaries induce depth bleeding, distorted surfaces corrupt normal estimation, and task agnostic grasp scoring yields tilted or off-center grasps that fail under dynamic motion. In this paper, we propose TransGraspNet, a geometry physics consistent framework that explicitly enforces consistency from perception to execution through three coupled principles: boundary consistency to produce structurally reliable object contours as downstream priors, surface consistency to preserve
geometric fidelity and surface normal accuracy during depth reconstruction, and physics consistency to refine grasp selection with centroid
alignment and wrench-space stability for upright and dynamically robust manipulation. We evaluate TransGraspNet on public
benchmarks, a dedicated transparent glassware
dataset, and a real robotic platform. The results show improved boundary quality and surface normal fidelity, and demonstrate strong task-level performance in cluttered transparent scenes. Most importantly, the proposed system achieves reliable real-world operation, including high grasp success rates in clutter and zero spillage during high speed
liquid transport, highlighting the effectiveness of our method.
Read the full abstract ▾
All authors Hailing Hu, Mingyi Zhu, Yiquan An, Yifei Tian, Tianyou Zuo, Lifeng Zhou
Robotics · Cross-listed from cs.AI
LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback
Manith Adikari, Bei Peng, Samuele Vinanzi et al. (+1 more)
Reinforcement Learning (RL) systems are typically trained using a single, well-specified scalar reward function. However, real-world decision-making tasks often involve multiple, competing objectives, such as performance versus efficiency, where ground-truth reward functions are difficult to specify or inaccessible. While Multi-Objective RL (MORL) addresses such trade-offs by modeling rewards as vectors, existing approaches typically assume access to a well-specified reward function for each objective, inheriting the same challenges faced by single-objective RL. Meanwhile, Preference-based RL (PbRL) has shown great potential in solving complex tasks without access to a pre-defined reward function through reward learning from human feedback, yet has largely been studied in single-objective settings. In this work, we bridge this gap with LEMUR: Learning to Align with Multi-Objective Reinforcement Learning with Preference feedback, a novel framework where an
agent interactively learns from the preferences of multiple humans to learn optimal multi-objective policies. Our approach jointly learns policies and multiple objective-specific reward models from human feedback, enabling
agents to effectively balance competing objectives during learning. We evaluate LEMUR on a variety of
benchmark multi-objective tasks, and empirical results demonstrate its superior performance over baseline methods. Our method presents a promising direction for solving multi-objective decision-making tasks without pre-defined reward functions.
Read the full abstract ▾
All authors Manith Adikari, Bei Peng, Samuele Vinanzi, Angelo Cangelosi
Robotics
STAGE: STyle-controllable Action GEneration for personalized autonomous driving
Zihao Liu, Xing Liu, Yizhai Zhang et al. (+1 more)
Driving style refers to the behavioral preferences that drivers maintain during driving, shaped by their diverse experiences, habits, and needs, and is typically reflected in varying levels of aggressiveness. If humans choose to use autonomous driving systems, they would expect the driving style of the systems to closely resemble their own habit. However, this is challenging for current industrial autonomous driving systems. To address this, we developed a style controllable action generation method, STAGE, for driving tasks. Its training process is based on imitation learning, incorporating both style value and latent value action modality encoding. Preference learning is then used to identify the user's driving style as a continuous, monotonic style value. And to reduce the cost of human involvement in the preference training process, we also developed a set of rules to compare driving style in data pairs. Then, during inference, the user inputs the style value to control the generated action patterns, dynamically meeting the user's expectations. Using the STAGE method, we verified that the style-controlled action generation results in several typical road scenarios significantly align with human expectations. Furthermore, through comparisons between the STAGE method and various other approaches, we reveal the unique functionalities of STAGE, including its style controllability, style continuity, driving style
alignment capability and driving safety. The code for this work is available at: https://github.com/CarlDegio/STAGE
Read the full abstract ▾
All authors Zihao Liu, Xing Liu, Yizhai Zhang, Panfeng Huang
Robotics
Homotopy-Aware Corridor Generation without Predefined Reference Paths
Haoze Dong, Minghan Li, Meng Guo et al. (+1 more)
Generating safe corridors is essential for collision-free robotic motion planning, yet most existing methods rely on predefined reference paths, which bias corridor geometry and implicitly limit the homotopy classes that can be explored. We propose a reference-path-free corridor generation framework on graphs of convex sets (GCS) that constructs corridors directly as sequences of convex sets, allowing corridor structure to emerge from the free-space representation rather than from a guiding path. To reason about similarity among corridors, we extend visibility-based deformation from paths to convex-set sequences, enabling the fusion of topologically redundant corridors while preserving distinct alternatives. To overcome the limited adaptability of existing GCS methods based on static global decompositions, we further develop an adaptive multi-scale GCS, in which a sampling-based fine-scale graph supports localized updates and a visibility-based coarse-scale graph enables compact global exploration. The two levels maintain topological consistency, allowing incremental updates without full graph reconstruction under environmental uncertainty. Numerical experiments characterize GCS construction, corridor generation, homotopy-aware exploration, and local updates, showing efficient graph construction, stable trajectory-level performance, and shorter-duration homotopy-aware trajectories than existing baselines. Hardware experiments on ground and aerial
robots, including deployment with onboard localization, further validate the framework under translated and previously unknown obstacles.
Read the full abstract ▾
All authors Haoze Dong, Minghan Li, Meng Guo, Zhongkui Li
Robotics
Tri-Space Operational Control of Redundant Multilink and Hybrid Cable-Driven Parallel Robots Using an Iterative-Learning based Reactive Approach
Dipankar Bhattacharya, Yin Pok Chan, Siqi Shang et al. (+3 more)
Cable-Driven Parallel
Robots (CDPRs) are a type of parallel mechanism in which cables are used as actuators. Due to the two levels of redundancy and numerous constraints within the CDPR actuation, joint and operational spaces (together known as the tri-space), tracking a given trajectory in the operational space while satisfying constraints in tri-space simultaneously is challenging. To the best of the authors' knowledge, there does not exist any tri-space control framework, which is robust, effective, and directly applicable to several architectures of redundantly actuated CDPRs. This paper proposes a tri-space control framework that combines Reactive Control (RC) and Iterative-Learning Control (ILC) to perform repetitive tasks in the operational space. The framework allows the tracking of operational space trajectories online with feasible cable forces, while avoiding undesirable situations such as cable-link interference, joint interference, and loss of manipulability. On the other hand, by finding an optimal parameter in the null space using a novel parameterization of a null space vector, the performance can be improved through ILC when the task is repeatedly executed. Simulation and hardware results on various Multilink Cable-Driven
Robot (MCDRs) and Hybrid Cable-Driven
Robots (HCDRs) show that the proposed tri-space control framework can be conveniently and effectively applied to the real-time control of different CDPRs.
Read the full abstract ▾
All authors Dipankar Bhattacharya, Yin Pok Chan, Siqi Shang, Yuen Shan Chan, Ying Tan, Darwin Lau
Robotics
Temporal Policy: History-Initialized Action Generation for Robotic Learning from Demonstration
Dylan Miller and Martin Jagersand
By relying on independent couplings from uninformative Gaussian priors, standard
diffusion and flow matching models are forced to learn complex, high-cost vector fields to reach the physical action space. Generative models excel at capturing
multimodal behaviors for robotic Learning from Demonstration (LfD), but often suffer from high inference cost. This paper introduces Temporal Policy, a generative framework based on
stochastic interpolants that formulates action generation as a temporally coupled transport problem. By initializing the generative flow at the
robot's recent history, we explicitly couple past states to future action sequences. This data-dependent coupling reduces transport cost and produces straight vector fields. We validate Temporal Policy across visuomotor simulation
benchmarks and on a physical Barrett WAM 2x 7DoF teleoperation platform. Our approach reduces transport costs by nearly an order of magnitude compared to noise-initialized baselines, achieving a 19.1 ms inference latency on a single NVIDIA RTX 4080. Crucially, these
geometric and computational efficiencies are achieved while matching the success rates of state-of-the-art baselines. This simplified transport geometry bypasses the computational bottleneck of independent Gaussian priors, helping enable high-frequency, closed-loop control. The code is publicly available at https://github.com/dmiller12/TemporalPolicy.
Read the full abstract ▾