Robotics
Vision-Augmented On-Track System Identification for Autonomous Racing via Attention-Based Priors and Iterative Neural Correction
Zhiping Wu, Cheng Hu, Yiqin Wang et al. (+2 more)
Operating autonomous vehicles at the absolute limits of handling requires precise, real-time identification of highly non-linear tire dynamics. However, traditional online optimization methods suffer from "cold-start" initialization failures and struggle to model high-frequency transient dynamics. To address these bottlenecks, this paper proposes a novel
vision-augmented, iterative system identification framework. First, a lightweight CNN (MobileNetV3) translates visual road textures into a continuous heuristic friction prior, providing a robust "warm-start" for parameter optimization. Next, a S4 model captures complex temporal dynamic residuals, circumventing the
memory and latency limitations of traditional MLPs and RNNs. Finally, a derivative-free Nelder-Mead
algorithm iteratively extracts physically interpretable Pacejka tire parameters via a hybrid virtual simulation. Co-simulation in CarSim demonstrates that the lightweight
vision backbone reduces friction estimation error by 76.1 using 85 fewer FLOPs, accelerating cold-start convergence by 71.4. Furthermore, the S4-augmented framework improves parameter extraction accuracy and decreases lateral force RMSE by over 60 by effectively capturing complex vehicle dynamics, demonstrating superior performance compared to conventional neural architectures.
Read the full abstract ▾
All authors Zhiping Wu, Cheng Hu, Yiqin Wang, Lei Xie, Hongye Su
Robotics · Cross-listed from cs.LG
SPAARS: Safer RL Policy Alignment through Abstract Exploration and Refined Exploitation of Action Space
Swaminathan S K and Aritra Hazra
Offline-to-online reinforcement learning (RL) offers a promising paradigm for robotics by pre-training policies on safe, offline demonstrations and fine-tuning them via online interaction. However, a fundamental challenge remains: how to safely explore online without deviating from the behavioral support of the offline data? While recent methods leverage conditional variational autoencoders (CVAEs) to bound exploration within a latent space, they inherently suffer from an exploitation gap – a performance ceiling imposed by the decoder's reconstruction loss. We introduce SPAARS, a curriculum learning framework that initially constrains exploration to the low-dimensional latent
manifold for sample-efficient, safe behavioral improvement, then seamlessly transfers control to the raw action space, bypassing the decoder bottleneck. SPAARS has two instantiations: the CVAE-based variant requires only unordered (s,a) pairs and no trajectory
segmentation; SPAARS-SUPE pairs SPAARS with OPAL temporal skill pretraining for stronger exploration structure at the cost of requiring trajectory chunks. We prove an upper bound on the exploitation gap using the Performance Difference Lemma, establish that latent-space policy gradients achieve provable variance reduction over raw-space exploration, and show that concurrent behavioral cloning during the latent phase directly controls curriculum transition stability. Empirically, SPAARS-SUPE achieves 0.825 normalized return on kitchen-mixed-v0 versus 0.75 for SUPE, with 5x better sample efficiency; standalone SPAARS achieves 92.7 and 102.9 normalized return on hopper-medium-v2 and walker2d-medium-v2 respectively, surpassing IQL baselines of 66.3 and 78.3 respectively, confirming the utility of the unordered-pair CVAE instantiation.
Read the full abstract ▾
Robotics
NLiPsCalib: An Efficient Calibration Framework for High-Fidelity 3D Reconstruction of Curved Visuotactile Sensors
Xuhao Qin, Feiyu Zhao, Yatao Leng et al. (+2 more)
Recent advances in visuotactile sensors increasingly employ biomimetic curved surfaces to enhance sensorimotor capabilities. Although such curved visuotactile sensors enable more conformal object contact, their perceptual quality is often degraded by non-uniform illumination, which reduces reconstruction accuracy and typically necessitates calibration. Existing calibration methods commonly rely on customized indenters and specialized devices to collect large-scale photometric data, but these processes are expensive and labor-intensive. To overcome these calibration challenges, we present NLiPsCalib, a physics-consistent and efficient calibration framework for curved visuotactile sensors. NLiPsCalib integrates controllable near-field light sources and leverages Near-Light Photometric Stereo (NLiPs) to estimate contact geometry, simplifying calibration to just a few simple contacts with everyday objects. We further introduce NLiPsTac, a controllable-light-source tactile sensor developed to validate our framework. Experimental results demonstrate that our approach enables high-fidelity 3D reconstruction across diverse curved form factors with a simple calibration procedure. We emphasize that our approach lowers the barrier to developing customized visuotactile sensors of diverse geometries, thereby making visuotactile sensing more accessible to the broader community.
Read the full abstract ▾
All authors Xuhao Qin, Feiyu Zhao, Yatao Leng, Runze Hu, Chenxi Xiao
Robotics
CORAL: Scalable Multi-Task Robot Learning via LoRA Experts
Yuankai Luo, Woping Chen, Tong Liang et al. (+1 more)
Deploying
Vision-
Language-Action (VLA) models in real-world robotics exposes a core multi-task learning challenge: reconciling task interference in multi-task robotic learning. When multiple tasks are jointly fine-tuned in a single stage, gradients from different tasks can conflict, causing negative transfer and reducing per-task performance. Yet maintaining a separate full checkpoint per task is often storage- and deployment-prohibitive. To address this dilemma, we present CORAL, a backbone- and embodiment-agnostic framework designed primarily to mitigate multi-task interference while remaining naturally extensible to a continuous stream of new tasks. CORAL freezes a single pre-trained VLA backbone and attaches one lightweight Low-Rank Adaptation (LoRA) expert per task; at runtime, a dynamic inference engine (the CORAL Manager) routes
language instructions to the appropriate expert and swaps experts on the fly with zero inference overhead. This strict parameter isolation avoids complex gating networks and prevents parameter-level cross-task interference by construction; as an added capability, it also enables sequentially introducing new tasks without parameter overwriting caused by catastrophic forgetting. We validate CORAL on a real-world Galaxea R1 dual-arm mobile manipulator and three simulation
benchmarks (LIBERO, WidowX, Google
Robot), where CORAL overcomes fine-grained instructional ambiguity and substantially outperforms joint training, yielding a practical and scalable system for lifelong multi-task
robot learning. Website: https://frontierrobo.github.io/CORAL
Read the full abstract ▾
All authors Yuankai Luo, Woping Chen, Tong Liang, Zhenguo Li
Robotics
See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation
Tingjun Dai, Mingfei Han, Tingwen Du et al. (+6 more)
Measurement of task progress through explicit, actionable milestones is critical for robust robotic manipulation. This progress awareness enables a model to ground its current task status, anticipate verifiable intermediate states, and detect and recover from failures when progress stalls. To embody this capability, we introduce See, Plan, Rewind (SPR), a progress-aware
vision-
language-action framework that dynamically grounds
language instructions into a sequence of spatial subgoals. SPR operates through a continuous core cycle, Seeing the current state and upcoming milestone, Planning a trajectory towards the next 2D waypoint, and Rewinding to a recoverable state upon failure by monitoring progress against the expected sequence. This closed-loop approach enables robust error correction without requiring additional training data or auxiliary models. Extensive experiments demonstrate the framework's effectiveness, generalization and robustness: SPR outperforms the MolmoAct baseline by 5% on the LIBERO
benchmark. On the challenging LIBERO-Plus
benchmark with unseen instructions and initial states, SPR achieves state-of-the-art robustness with the smallest performance drop, surpassing OpenVLA-OFT and UniVLA, demonstrating superior out-of-distribution robustness.
Read the full abstract ▾
All authors Tingjun Dai, Mingfei Han, Tingwen Du, Zhiheng Liu, Zihao Zhang, Zhihui Li, Salman Khan, Jun Yu, Xiaojun Chang
Robotics · Cross-listed from cs.CV
Mingfei Han, Haihong Hao, Liang Ma et al. (+6 more)
Vision-and-
Language Navigation (VLN) has long been constrained by the limited diversity and scalability of simulator-curated
datasets, which fail to capture the complexity of real-world environments. To overcome this limitation, we introduce a large-scale
video-instruction framework derived from web-based room tour
videos, enabling
agents to learn from natural human walking demonstrations in diverse, realistic indoor settings. Unlike existing
datasets, our framework integrates both open-ended description-enriched trajectories and action-enriched trajectories reconstructed in 3D, providing richer spatial and semantic supervision. A key extension in this work is the incorporation of implicit geometry representations, which extract spatial cues directly from RGB frames without requiring fragile 3D reconstruction. This approach substantially improves data utilization, alleviates reconstruction failures, and unlocks large portions of previously unusable
video data. Comprehensive experiments across multiple VLN
benchmarks (CVDN, SOON, R2R, and REVERIE) demonstrate that our method not only sets new state-of-the-art performance but also enables the development of robust zero-shot
navigation agents. By bridging large-scale web
videos with implicit spatial
reasoning, this work advances
embodied navigation towards more scalable, generalizable, and real-world applicable solutions.
Read the full abstract ▾
All authors Mingfei Han, Haihong Hao, Liang Ma, Kamila Zhumakhanova, Ekaterina Radionova, Jingyi Zhang, Xiaojun Chang, Xiaodan Liang, Ivan Laptev
Robotics · Cross-listed from cs.CV
RAE-NWM: Navigation World Model in Dense Visual Representation Space
Mingkun Zhang, Wangtian Shen, Fan Zhang et al. (+3 more)
Visual
navigation requires
agents to reach goals in complex environments through perception and planning. World models address this task by simulating action-conditioned state transitions to predict future observations. Current
navigation world models typically learn state
evolution under actions within the compressed latent space of a Variational Autoencoder, where spatial compression often discards fine-grained structural information and hinders precise control. To better understand the propagation characteristics of different representations, we conduct a linear dynamics probe and observe that dense DINOv2 features exhibit stronger linear predictability for action-conditioned transitions. Motivated by this observation, we propose the Representation Autoencoder-based
Navigation World Model (RAE-NWM), which models
navigation dynamics in a dense visual representation space. We employ a Conditional
Diffusion Transformer with Decoupled
Diffusion Transformer head (CDiT-DH) to model continuous transitions, and introduce a separate time-driven gating module for dynamics conditioning to regulate action injection strength during generation. Extensive evaluations show that modeling sequential rollouts in this space improves structural stability and action accuracy, benefiting downstream planning and
navigation.
Read the full abstract ▾
All authors Mingkun Zhang, Wangtian Shen, Fan Zhang, Haijian Qin, Zihao Pei, Ziyang Meng
Robotics
MO-Playground: Massively Parallelized Multi-Objective Reinforcement Learning for Robotics
Neil Janwani, Ellen Novoseller, Vernon J. Lawhern et al. (+1 more)
Multi-objective reinforcement learning (MORL) is a powerful tool to learn Pareto-optimal policy families across conflicting objectives. However, unlike traditional RL
algorithms, existing MORL
algorithms do not effectively leverage large-scale parallelization to concurrently simulate thousands of environments, thus facing vastly increased computation time. Ultimately, this has limited the application of MORL towards complex multi-objective robotics problems. To address these challenges, we present 1) MORLAX, a new GPU-native, fast MORL
algorithm, and 2) MO-Playground, a pip-installable playground of GPU-accelerated multi-objective environments. Together, MORLAX and MO-Playground approximate Pareto sets within minutes, offering 26-271x speed-ups compared to legacy CPU-based approaches and up to 19x speed-ups over prior GPU-based approaches whilst learning superior Pareto front hypervolumes. We demonstrate MO-Playground's versatility by implementing a custom BRUCE
humanoid robot environment and learning Pareto-optimal locomotion policies across 6 practical objectives in simulation, such as smoothness, efficiency and arm swinging. Our toolbox is available at https://dynamicmobility.github.io/moplayground/.
Read the full abstract ▾
All authors Neil Janwani, Ellen Novoseller, Vernon J. Lawhern, Maegan Tucker
Robotics
TRIP-Bag: A Portable Teleoperation System for Plug-and-Play Robotic Arms and Leaders
Noboru Myers, Sankalp Yamsani, Obin Kwon et al. (+1 more)
Large scale, diverse demonstration data for manipulation tasks remains a major challenge in learning-based
robot policies. Existing in-the-wild data collection approaches often rely on
vision-based pose estimation of hand-held grippers or gloves, which introduces an embodiment gap between the collection platform and the target
robot. Teleoperation systems eliminate the embodiment gap, but are typically impractical to deploy outside the laboratory environment. We propose TRIP-Bag (Teleoperation, Recording, Intelligence in a Portable Bag), a portable, puppeteer-style teleoperation system fully contained within a commercial suitcase, as a practical solution for collecting high-fidelity manipulation data across varied settings. With a setup time of under five minutes and direct joint-to-joint teleoperation, TRIP-Bag enables rapid and reliable data collection in any environment. We validated TRIP-Bag's usability through experiments with non-expert users, showing that the system is intuitive and easy to operate. Furthermore, we confirmed the quality of the collected data by training
benchmark manipulation policies, demonstrating its value as a practical resource for
robot learning.
Read the full abstract ▾
All authors Noboru Myers, Sankalp Yamsani, Obin Kwon, Joohyung Kim
Robotics
Embodied Human Simulation for Quantitative Design and Analysis of Interactive Robotics
Chenhui Zuo, Jinhao Xu, Michael Qian Vergnolle et al. (+1 more)
Physical interactive robotics, ranging from wearable devices to collaborative
humanoid robots, require close coordination between mechanical design and control. However, evaluating interactive dynamics is challenging due to complex human biomechanics and motor responses. Traditional experiments rely on indirect metrics without measuring human internal states, such as muscle forces or joint loads. To address this issue, we develop a scalable simulation-based framework for the quantitative analysis of physical human-
robot interaction. At its core is a full-body musculoskeletal model serving as a predictive surrogate for the human dynamical system. Driven by a reinforcement learning controller, it generates adaptive, physiologically grounded motor behaviors. We employ a sequential training pipeline where the pre-trained human motion control policy acts as a consistent evaluator, making large-scale design space exploration computationally tractable. By simulating the coupled human-
robot system, the framework provides access to internal biomechanical metrics, offering a systematic way to concurrently co-optimize a
robot's structural parameters and control policy. We demonstrate its capability in optimizing human-exoskeleton interactions, showing improved joint
alignment and reduced contact forces. This work establishes
embodied human simulation as a scalable paradigm for interactive robotics design.
Read the full abstract ▾
All authors Chenhui Zuo, Jinhao Xu, Michael Qian Vergnolle, Yanan Sui
Robotics
WESPR: Wind-adaptive Energy-Efficient Safe Perception & Planning for Robust Flight with Quadrotors
Khuzema Habib, Pranav Deshakulkarni Manjunath, Kasra Torshizi et al. (+2 more)
Local wind conditions strongly influence drone performance: headwinds increase flight time, crosswinds and wind shear hinder agility in cluttered spaces, while tailwinds reduce travel time. Although adaptive controllers can mitigate
turbulence, they remain unaware of the surrounding geometry that generates it, preventing proactive avoidance. Existing methods that model how wind interacts with the environment typically rely on computationally expensive fluid dynamics simulations, limiting real-time adaptation to new environments and conditions. To bridge this gap, we present WESPR, a fast framework that predicts how environmental geometry affects local wind conditions, enabling proactive path planning and control adaptation. Our lightweight pipeline integrates
geometric perception and local weather data to estimate wind fields, compute cost-efficient paths, and adjust control strategies, all within 10 seconds. We validate WESPR on a Crazyflie drone navigating turbulent obstacle courses. Our results show a 35.6% reduction on average in maximum trajectory deviation and a 24.6% improvement in stability compared to a wind-agnostic adaptive controller.
Read the full abstract ▾
All authors Khuzema Habib, Pranav Deshakulkarni Manjunath, Kasra Torshizi, Pratap Tokekar, Troi Williams
Robotics
Robust Spatiotemporal Motion Planning for Multi-Agent Autonomous Racing via Topological Gap Identification and Accelerated MPC
Mingyi Zhang, Cheng Hu, Yiqin Wang et al. (+3 more)
High-speed multi-
agent autonomous racing demands robust spatiotemporal planning and precise control under strict computational limits. Current methods often oversimplify interactions or abandon strict kinematic constraints. We resolve this by proposing a Topological Gap Identification and Accelerated MPC framework. By predicting opponent behaviors via SGPs, our method constructs dynamic occupancy corridors to robustly select optimal overtaking gaps. We ensure strict kinematic feasibility using a Linear Time-Varying MPC powered by a customized Pseudo-Transient Continuation (PTC) solver for high-frequency execution. Experimental results on the F1TENTH platform show that our method significantly outperforms state-of-the-art baselines: it reduces total maneuver time by 51.6% in sequential scenarios, consistently maintains an overtaking success rate exceeding 81% in dense bottlenecks, and lowers average computational latency by 20.3%, pushing the boundaries of safe and high-speed autonomous racing.
Read the full abstract ▾
All authors Mingyi Zhang, Cheng Hu, Yiqin Wang, Haotong Qin, Hongye Su, Lei Xie
Robotics
STONE Dataset: A Scalable Multi-Modal Surround-View 3D Traversability Dataset for Off-Road Robot Navigation
Konyul Park, Daehun Kim, Jiyong Oh et al. (+7 more)
Reliable off-road
navigation requires accurate estimation of traversable regions and robust perception under diverse terrain and sensing conditions. However, existing
datasets lack both scalability and multi-modality, which limits progress in 3D traversability prediction. In this work, we introduce STONE, a large-scale multi-modal
dataset for off-road
navigation. STONE provides (1) trajectory-guided 3D traversability maps generated by a fully automated, annotation-free pipeline, and (2) comprehensive surround-view sensing with synchronized 128-channel
LiDAR, six RGB cameras, and three 4D imaging radars. The
dataset covers a wide range of environments and conditions, including day and night, grasslands, farmlands, construction sites, and lakes. Our auto-labeling pipeline reconstructs dense terrain surfaces from
LiDAR scans, extracts
geometric attributes such as slope, elevation, and roughness, and assigns traversability labels beyond the
robot's trajectory using a Mahalanobis-distance-based criterion. This design enables scalable, geometry-aware ground-truth construction without manual annotation. Finally, we establish a
benchmark for voxel-level 3D traversability prediction and provide strong baselines under both single-modal and multi-modal settings. STONE is available at: https://konyul.github.io/STONE-
dataset/
Read the full abstract ▾
All authors Konyul Park, Daehun Kim, Jiyong Oh, Seunghoon Yu, Junseo Park, Jaehyun Park, Hongjae Shin, Hyungchan Cho, Jungho Kim, Jun Won Choi
Robotics
ZeroWBC: Learning Natural Whole-Body Humanoid Interaction from Human Egocentric Data
Haoran Yang, Jiacheng Bao, Yucheng Xin et al. (+5 more)
Achieving versatile and natural whole-body
humanoid interaction control remains challenging due to the high cost of whole-body teleoperation data. We present ZeroWBC, a teleoperation-free framework that learns
humanoid whole-body interaction from human
egocentric videos paired with synchronized whole-body motion and text annotations. ZeroWBC adopts a generation-then-tracking formulation to tackle the static scene whole-body interaction control problem. Given an initial
egocentric image and a
language instruction, a fine-tuned
Vision-
Language Model generates future human whole-body motion
tokens, which are decoded into continuous motions and retargeted to the
humanoid. The resulting reference motions, together with root and key body-part trajectories, are then executed by a general interactive motion tracking policy. To improve interaction performance, we introduce an interaction-oriented tracking reward that prioritizes global root and key body-part trajectory
alignment while preserving natural whole-body motion. Experiments on the Unitree G1
humanoid robot show that ZeroWBC enables diverse scene-aware behaviors without
robot teleoperation demonstrations. These results suggest a scalable paradigm for learning natural
humanoid whole-body interaction from human
egocentric data.
Read the full abstract ▾
All authors Haoran Yang, Jiacheng Bao, Yucheng Xin, Haoming Song, Yuyang Tian, Bin Zhao, Dong Wang, Xuelong Li
Robotics
SPAN-Nav: Generalized Spatial Awareness for Versatile Embodied Navigation
Jiahang Liu, Tianyu Xu, Jiawei Chen et al. (+9 more)
Recent
embodied navigation approaches leveraging
Vision-
Language Models (VLMs) demonstrate strong generalization in versatile
Vision-
Language Navigation (VLN). However, reliable path planning in complex environments remains challenging due to insufficient spatial awareness. In this work, we introduce SPAN-Nav, an end-to-end foundation model designed to infuse
embodied navigation with universal 3D spatial awareness using RGB
video streams. SPAN-Nav extracts spatial priors across diverse scenes through an occupancy prediction task on extensive indoor and outdoor environments. To mitigate the computational burden, we introduce a compact representation for spatial priors, finding that a single
token is sufficient to encapsulate the coarse-grained cues essential for
navigation tasks. Furthermore, inspired by the Chain-of-Thought (CoT) mechanism, SPAN-Nav utilizes this single spatial
token to explicitly inject spatial cues into action
reasoning through an end-to end framework. Leveraging multi-task co-training, SPAN-Nav captures task-adaptive cues from generalized spatial priors, enabling robust spatial awareness to generalize even to the task lacking explicit spatial supervision. To support comprehensive spatial learning, we present a massive
dataset of 4.2 million occupancy annotations that covers both indoor and outdoor scenes across multi-type
navigation tasks. SPAN-Nav achieves state-of-the-art performance across three
benchmarks spanning diverse scenarios and varied
navigation tasks. Finally, real-world experiments validate the robust generalization and practical reliability of our approach across complex physical scenarios.
Read the full abstract ▾
All authors Jiahang Liu, Tianyu Xu, Jiawei Chen, Lu Yue, Jiazhao Zhang, Zhiyong Wang, Minghan Li, Qisheng Zhao, Anqi Li, Qi Su, Zhizheng Zhang, He Wang
Robotics
Walking on Rough Terrain with Any Number of Legs
Zhuoyang Chen, Xinyuan Wang and Shai Revzen
Robotics would gain by replicating the remarkable agility of arthropods in navigating complex environments. Here we consider the control of multi-legged systems which have 6 or more legs. Current multi-legged control strategies in
robots include large black-box machine learning models, Central Pattern Generator (CPG) networks, and open-loop feed-forward control with stability arising from mechanics. Here we present a multi-legged control architecture for rough terrain using a segmental
robot with 3 actuators for every 2 legs, which we validated in simulation for
robots with 6 to 16 legs. Segments have identical state machines, and each segment also receives input from the segment in front of it. Our design bridges the gap between WalkNet-like event cascade controllers and CPG-based controllers: it tightly couples to the ground when contact is present, but produces fictive locomotion when ground contact is missing. The approach may be useful as an adaptive and computationally lightweight controller for multi-legged
robots, and as a baseline capability for scaffolding the learning of machine learning controllers.
Read the full abstract ▾
Robotics
DexHiL: A Human-in-the-Loop Framework for Vision-Language-Action Model Post-Training in Dexterous Manipulation
Yifan Han, Zhongxi Chen, Yuxuan Zhao et al. (+5 more)
While
Vision-
Language-Action (VLA) models have demonstrated promising generalization capabilities in robotic manipulation, deploying them on specific and complex downstream tasks still demands effective post-training. In parallel, Human-in-the-Loop (HiL) learning has proven to be a powerful mechanism for refining
robot policies. However, extending this paradigm to dexterous manipulation remains challenging: multi-finger control is high-dimensional, contact-intensive, and exhibits execution distributions that differ markedly from standard arm motions, leaving existing dexterous VLA systems limited in reliability and adaptability. We present DexHiL, the first integrated arm-hand human-in-the-loop framework for dexterous VLA models, enabling coordinated interventions over the arm and the dexterous hand within a single system. DexHiL introduces an intervention-aware data sampling strategy that prioritizes corrective segments for post-training, alongside a lightweight teleoperation interface that supports instantaneous human corrections during execution. Real-
robot experiments demonstrate that DexHiL serves as an effective post-training framework, yielding a substantial performance leap, outperforming standard offline-only fine-tuning baselines by an average of 25% in success rates across distinct tasks. Project page: https://chenzhongxi-sjtu.github.io/dexhil/
Read the full abstract ▾
All authors Yifan Han, Zhongxi Chen, Yuxuan Zhao, Congsheng Xu, Yanming Shao, Yichuan Peng, Yao Mu, Wenzhao Lian
Robotics
PM-Nav: Priori-Map Guided Embodied Navigation in Functional Buildings
Jiang Gao, Xiangyu Dong, Haozhou Li et al. (+3 more)
Existing
language-driven
embodied navigation paradigms face challenges in functional buildings (FBs) with highly similar features, as they lack the ability to effectively utilize priori spatial knowledge. To tackle this issue, we propose a Priori-Map Guided
Embodied Navigation (PM-Nav), wherein environmental maps are transformed into
navigation-friendly semantic priori-maps, a hierarchical chain-of-thought
prompt template with an annotation priori-map is designed to enable precise path planning, and a multi-model collaborative action output mechanism is built to accomplish positioning decisions and execution control for
navigation planning. Comprehensive tests using a home-made FB
dataset show that the PM-Nav obtains average improvements of 511% and 1175%, and 650% and 400% over the SG-Nav and the InstructNav in simulation and real-world, respectively. These tremendous boosts elucidate the great potential of using the PM-Nav as a backbone
navigation framework for FBs.
Read the full abstract ▾
All authors Jiang Gao, Xiangyu Dong, Haozhou Li, Haoran Zhao, Yaoming Zhou, Xiaoguang Ma
Robotics
Latent World Models for Automated Driving: A Unified Taxonomy, Evaluation Framework, and Open Challenges
Rongxiang Zeng and Yongqi Dong
Emerging generative world models and
vision-
language-action (VLA) systems are rapidly reshaping automated driving by enabling scalable simulation, long-horizon
forecasting, and capability-rich decision making. Across these directions, latent representations serve as the central computational substrate: they compress high-dimensional multi-sensor observations, enable temporally coherent rollouts, and provide interfaces for planning,
reasoning, and controllable generation. This paper proposes a unifying latent-space framework that synthesizes recent progress in world models for automated driving. The framework organizes the design space by the target and form of latent representations (latent worlds, latent actions, latent generators; continuous states, discrete
tokens, and hybrids) and by structural priors for geometry,
topology, and semantics. Building on this taxonomy, the paper articulates five cross-cutting internal mechanics (i.e, structural isomorphism, long-horizon temporal stability, semantic and
reasoning alignment, value-aligned objectives and post-training, as well as adaptive computation and deliberation) and connects these design choices to robustness, generalization, and deployability. The work also proposes concrete evaluation prescriptions, including a closed-loop metric suite and a resource-aware deliberation cost, designed to reduce the open-loop / closed-loop mismatch. Finally, the paper identifies actionable research directions toward advancing latent world model for decision-ready, verifiable, and resource-efficient automated driving.
Read the full abstract ▾
Robotics
Provably Safe Trajectory Generation for Manipulators Under Motion and Environmental Uncertainties
Fei Meng, Zijiang Yang, Xinyu Mao et al. (+2 more)
Robot manipulators operating in uncertain and non-convex environments present significant challenges for safe and optimal motion planning. Existing methods often struggle to provide efficient and formally certified collision risk guarantees, particularly when dealing with complex geometries and non-Gaussian uncertainties. This article proposes a novel risk-bounded motion planning framework to address this unmet need. Our approach integrates a rigid manipulator deep
stochastic Koopman operator (RM-DeSKO) model to robustly predict the
robot's state distribution under motion uncertainty. We then introduce an efficient, hierarchical verification method that combines parallelizable physics simulations with sum-of-squares (SOS) programming as a filter for fine-grained, formal certification of collision risk. This method is embedded within a Model Predictive Path Integral (MPPI) controller that uniquely utilizes binary collision information from SOS decomposition to improve its policy. The effectiveness of the proposed framework is validated on two typical
robot manipulators through extensive simulations and real-world experiments, including a challenging human-
robot collaboration scenario, demonstrating sim-to-real transfer of the learned model and its ability to generate safe and efficient trajectories in complex, uncertain settings.
Read the full abstract ▾
All authors Fei Meng, Zijiang Yang, Xinyu Mao, Haobo Liang, Max Q.-H. Meng
Robotics · Cross-listed from cs.CV
GST-VLA: Structured Gaussian Spatial Tokens for 3D Depth-Aware Vision-Language-Action Models
Md Selim Sarowar, Omer Tariq and Sungho Kim
VLA models encode visual observations as 2D patch
tokens with no intrinsic
geometric structure. We introduce GST-VLA with two contributions. First, the Gaussian Spatial Tokenizer (GST) converts frozen dense depth and frozen semantic patch features into $N_g{=}128$ anisotropic 3D Gaussian primitives, each parameterized by a metric residual mean $\mu \in \mathbb{R}^3$, log-scale covariance $\log \sigma \in \mathbb{R}^3$, and learned opacity $\alpha \in (0,1)$. The covariance eigenstructure encodes local surface orientation, and opacity provides per-primitive
geometric confidence, both inaccessible from scalar depth. Spatial
attention pooling with learned queries concentrates the fixed
token budget on geometrically salient regions rather than distributing uniformly. Second, 3D Depth-Aware Chain-of-Thought (DA-CoT)
reasoning supervises four structured intermediate spatial thoughts, covering 3D object grounding, grasp affordance contact geometry, pairwise metric distances, and coarse SE(3) waypoints, as explicit generation targets in the training loss. A cross-
attention sublayer at every VLM
transformer block provides direct access to the raw 256-primitive Gaussian field during DA-CoT generation. A 300M-parameter flow-matching action expert with mixture-of-experts feedforward sublayers decodes 7-DoF delta action chunks via conditional ODE integration, conditioned on both VLM hidden states and DA-CoT outputs through dual cross-
attention. Trained with composite $\mathcal{L}_\mathrm{flow} + \mathcal{L}_\mathrm{CoT} + \mathcal{L}_\mathrm{depth}$ across three progressive stages, GST-VLA achieves 96.4% on LIBERO (+2.0%), and 80.2% on SimplerEnv (+5.4%). Ablations isolate the contribution of each GST component, each DA-CoT thought, and each training stage, confirming independent and synergistic gains concentrated on precision demanding tasks.
Read the full abstract ▾
Robotics
High-Slip-Ratio Control for Peak Tire-Road Friction Estimation Using Automated Vehicles
Zhaohui Liang, Hang Zhou, Heye Huanh et al. (+1 more)
Accurate estimation of the tire-road friction coefficient (TRFC) is critical for ensuring safe vehicle control, especially under adverse road conditions. However, most existing methods rely on naturalistic driving data from regular vehicles, which typically operate under mild acceleration and braking. As a result, the data provide insufficient slip excitation and offer limited observability of the peak TRFC. This paper presents a high-slip-ratio control framework that enables automated vehicles (AVs) to actively excite the peak friction region during empty-haul operations while maintaining operational safety. A simplified Magic Formula tire model is adopted to represent nonlinear slip-force dynamics and is locally fitted using repeated high-slip measurements. To support safe execution in car-following scenarios, we formulate a constrained optimal control strategy that balances slip excitation, trajectory tracking, and collision avoidance. In parallel, a binning-based statistical projection method is introduced to robustly estimate peak TRFC under noise and local sparsity. The framework is validated through both closed-loop simulations and real-vehicle experiments, demonstrating its accuracy, safety, and feasibility for scalable, cost-effective roadway friction screening.
Read the full abstract ▾
All authors Zhaohui Liang, Hang Zhou, Heye Huanh, Xiaopeng Li
Robotics
3D UAV Trajectory Estimation and Classification from Internet Videos via Language Model
Haoxiang Lei, Daotong Wang, Shenghai Yuan et al. (+1 more)
Reliable 3D trajectory estimation of unmanned aerial vehicles (
UAVs) is a fundamental requirement for anti-
UAV systems, yet the acquisition of large-scale and accurately annotated trajectory data remains prohibitively expensive. In this work, we present a novel framework that derives
UAV 3D trajectories and category information directly from Internet-scale
UAV videos, without relying on manual annotations. First,
language-driven data acquisition is employed to autonomously discover and collect
UAV-related
videos, while
vision-
language reasoning progressively filters task-relevant segments. Second, a training-free cross-modal label generation module is introduced to infer 3D trajectory hypotheses and
UAV type cues. Third, a physics-informed refinement process is designed to impose temporal smoothness and kinematic consistency on the estimated trajectories. The resulting
video clips and trajectory annotations can be readily utilized for downstream anti-
UAV tasks. To assess effectiveness and generalization, we conduct zero-shot transfer experiments on a public, well-annotated 3D
UAV benchmark. Results reveal a clear data scaling behavior: as the amount of online
video data increases, zero-shot transfer performance on the target
dataset improves consistently, without any target-domain training. The proposed method closely approaches the current state-of-the-art, highlighting its robustness and applicability to real-world anti-
UAV scenarios. Code and
datasets will be released upon acceptance.
Read the full abstract ▾
All authors Haoxiang Lei, Daotong Wang, Shenghai Yuan, Jianbo Su
Robotics
Quality over Quantity: Demonstration Curation via Influence Functions for Data-Centric Robot Learning
Haeone Lee, Taywon Min, Junsu Kim et al. (+4 more)
Learning from demonstrations has emerged as a promising paradigm for end-to-end
robot control, particularly when scaled to diverse and large
datasets. However, the quality of demonstration data, often collected through human teleoperation, remains a critical bottleneck for effective data-driven
robot learning. Human errors, operational constraints, and teleoperator variability introduce noise and suboptimal behaviors, making data curation essential yet largely manual and heuristic-driven. In this work, we propose Quality over Quantity (QoQ), a grounded and systematic approach to identifying high-quality data by defining data quality as the contribution of each training sample to reducing loss on validation demonstrations. To efficiently estimate this contribution, we leverage influence functions, which quantify the impact of individual training samples on model performance. We further introduce two key techniques to adapt influence functions for
robot demonstrations: (i) using maximum influence across validation samples to capture the most relevant state-action pairs, and (ii) aggregating influence scores of state-action pairs within the same trajectory to reduce noise and improve data coverage. Experiments in both simulated and real-world settings show that QoQ consistently improves policy performances over prior data selection methods.
Read the full abstract ▾
All authors Haeone Lee, Taywon Min, Junsu Kim, Sinjae Kang, Fangchen Liu, Lerrel Pinto, Kimin Lee
Robotics
Cutting the Cord: System Architecture for Low-Cost, GPU-Accelerated Bimanual Mobile Manipulation
Artemis Shaw, Chen Liu, Justin Costa et al. (+5 more)
We present a bimanual mobile manipulator built on the open-source XLeRobot with integrated onboard compute for less than \$1300. Key contributions include: (1) optimized mechanical design maximizing stiffness-to-weight ratio, (2) a Tri-Bus power
topology isolating compute from motor-induced voltage transients, and (3) embedded autonomy using NVIDIA Jetson Orin Nano for untethered operation. The platform enables teleoperation, autonomous SLAM
navigation, and
vision-based manipulation without external dependencies, providing a low-cost alternative for research and education in robotics and
robot learning.
Read the full abstract ▾
All authors Artemis Shaw, Chen Liu, Justin Costa, Rane Gray, Alina Skowronek, Kevin Diaz, Nam Bui, Nikolaus Correll
Robotics
Beyond Amplitude: Channel State Information Phase-Aware Deep Fusion for Robotic Activity Recognition
Rojin Zandi, Hojjat Salehinejad and Milad Siami
Wi-Fi Channel State Information (CSI) has emerged as a promising non-line-of-sight sensing modality for human and robotic activity recognition. However, prior work has predominantly relied on CSI amplitude while underutilizing phase information, particularly in robotic arm activity recognition. In this paper, we present GateFusion-Bidirectional Long Short-Term
Memory network (GF-BiLSTM) for WiFi sensing in robotic activity recognition. GF-BiLSTM is a two-stream gated fusion network that encodes amplitude and phase separately and adaptively integrates per-time features through a learned gating mechanism. We systematically evaluate state-of-the-art deep learning models under a Leave-One-Velocity-Out (LOVO) protocol across four input configurations: amplitude only, phase only, amplitude + unwrapped phase, and amplitude + sanitized phase. Experimental results demonstrate that incorporating phase alongside amplitude consistently improves recognition accuracy and cross-speed robustness, with GF-BiLSTM achieving the best performance. To the best of our knowledge, this work provides the first systematic exploration of CSI phase for robotic activity recognition, establishing its critical role in Wi-Fi-based sensing.
Read the full abstract ▾
Robotics
ImpedanceDiffusion: Diffusion-Based Global Path Planning for UAV Swarm Navigation with Generative Impedance Control
Faryal Batool, Yasheerah Yaqoot, Muhammad Ahsan Mustafa et al. (+3 more)
Safe swarm
navigation in cluttered indoor environment requires long-horizon planning, reactive obstacle avoidance, and adaptive compliance. We propose ImpedanceDiffusion, a hierarchical framework that leverages image-conditioned
diffusion-based global path planning with Artificial Potential Field (APF) tracking and semantic-aware variable impedance control for aerial drone swarms. The
diffusion model generates
geometric global trajectories directly from RGB images without explicit map construction. These trajectories are tracked by an APF-based reactive layer, while a VLM-RAG module performs semantic obstacle classification with 90%
retrieval accuracy to adapt impedance parameters for mixed obstacle environments during execution. Two
diffusion planners are evaluated: (i) a top-view long-horizon planner using single-pass inference and (ii) a first-person-view (FPV) short-horizon planner deployed via a two-stage inference pipeline. Both planners achieve a 100% trajectory generation rate across twenty static and dynamic experimental configurations and are validated via zero-shot sim-to-real deployment on Crazyflie 2.1 drones through the hierarchical APF-impedance control stack. The top-view planner produces smoother trajectories that yield conservative tracking speeds of 1.0-1.2 m/s near hard obstacles and 0.6-1.0 m/s near soft obstacles. In contrast, the FPV planner generates trajectories with greater local clearance and typically higher speeds, reaching 1.4-2.0 m/s near hard obstacles and up to 1.6 m/s near soft obstacles. Across 20 experimental configurations (100 total runs), the framework achieved a 92% success rate while maintaining stable impedance-based formation control with bounded oscillations and no in-flight collisions, demonstrating reliable and adaptive swarm
navigation in cluttered indoor environments.
Read the full abstract ▾
All authors Faryal Batool, Yasheerah Yaqoot, Muhammad Ahsan Mustafa, Roohan Ahmed Khan, Aleksey Fedoseev, Dzmitry Tsetserukou