Robotics
Legs Over Arms: On the Predictive Value of Lower-Body Pose for Human Trajectory Prediction from Egocentric Robot Perception
Nhat Le, Daeun Song and Xuesu Xiao
Predicting human trajectory is crucial for social
robot navigation in crowded environments. While most existing approaches treat human as point mass, we present a study on multi-
agent trajectory prediction that leverages different human skeletal features for improved forecast accuracy. In particular, we systematically evaluate the predictive utility of 2D and 3D skeletal keypoints and derived biomechanical cues as additional inputs. Through a comprehensive study on the JRDB
dataset and another new
dataset for social
navigation with 360-degree panoramic
videos, we find that focusing on lower-body 3D keypoints yields a 13% reduction in Average Displacement Error and augmenting 3D keypoint inputs with corresponding biomechanical cues provides a further 1-4% improvement. Notably, the performance gain persists when using 2D keypoint inputs extracted from equirectangular panoramic images, indicating that
monocular surround
vision can capture informative cues for motion
forecasting. Our finding that
robots can forecast human movement efficiently by watching their legs provides actionable insights for designing sensing capabilities for social
robot navigation.
Read the full abstract ▾
Robotics
TwinRL: Digital Twin-Driven Reinforcement Learning for Real-World Robotic Manipulation
Qinwen Xu, Jiaming Liu, Rui Zhou et al. (+11 more)
Despite strong generalization capabilities,
Vision-
Language-Action (VLA) models remain constrained by the high cost of expert demonstrations and limited real-world interaction. While online reinforcement learning (RL) has shown promise, its application to real-world VLA manipulation is hindered by low exploration efficiency and restricted exploration coverage. Through systematic real-world experiments, we observe that the effective exploration space of online RL is largely constrained by the trajectory distribution induced during supervised fine-tuning (SFT). Motivated by this observation, we propose TwinRL, a digital twin-real-world collaborative post-training framework that expands and guides RL exploration for VLA models through three stages: SFT warm-up, twin RL warm-up, and real-world RL. TwinRL first reconstructs a high-fidelity digital twin from smartphone-captured scenes. During the SFT stage, we introduce an exploration space expansion strategy that expands the support of the trajectory distribution beyond real demonstrations, reshaping the exploration space for more effective RL. Rather than treating the twin as a data augmentation tool, we propose a twin RL warm-up strategy that enables it to act as an exploration guide for real-world RL. Specifically, TwinRL performs efficient parallel RL in the digital twin to generate interactive trajectories that populate the replay buffer and stabilize subsequent real-world RL learning. This process also identifies failure-prone yet informative configurations, enabling targeted human-in-the-loop rollouts to further improve on-
robot efficiency. Across four tasks, TwinRL achieves near-100% success in both in-distribution and out-of-distribution regions, delivering over 30% faster convergence than prior real-world RL methods with only 20 minutes of on-
robot interaction.
Read the full abstract ▾
All authors Qinwen Xu, Jiaming Liu, Rui Zhou, Shaojun Shi, Nuowei Han, Zhuoyang Liu, Chenyang Gu, Shuo Gu, Yang Yue, Gao Huang, Wenzhao Zheng, Sirui Han, Peng Jia, Shanghang Zhang
Robotics
$\chi_{0}$: Resource-Aware Robust Manipulation via Taming Distributional Inconsistencies
Checheng Yu, Chonghao Sima, Gangcheng Jiang et al. (+14 more)
High-reliability long-horizon robotic manipulation has traditionally relied on large-scale data and compute to understand complex real-world dynamics. However, we identify that the primary bottleneck to real-world robustness is not resource scale alone, but the distributional shift among the human demonstration distribution, the inductive bias learned by the policy, and the test-time execution distribution – a systematic inconsistency that causes compounding errors in multi-stage tasks. To mitigate these inconsistencies, we propose $\chi_{0}$, a resource-efficient framework with effective modules designated to achieve production-level robustness in robotic manipulation. Our approach builds off three technical pillars: (i) Model Arithmetic, a weight-space merging strategy that efficiently soaks up diverse distributions of different demonstrations, varying from object appearance to state variations; (ii) Stage Advantage, a stage-aware advantage
estimator that provides stable, dense progress signals, overcoming the numerical instability of prior non-stage approaches; and (iii) Train-Deploy
Alignment, which bridges the distribution gap via spatio-temporal augmentation, heuristic DAgger corrections, and temporal chunk-wise smoothing. $\chi_{0}$ enables two sets of dual-arm
robots to collaboratively orchestrate long-horizon garment manipulation, spanning tasks from flattening, folding, to hanging different clothes. Our method exhibits high-reliability autonomy; we are able to run the system from arbitrary initial state for consecutive 24 hours non-stop. Experiments validate that $\chi_{0}$ surpasses the state-of-the-art $\pi_{0.5}$ in success rate by nearly 250%, with only 20-hour data and 8 A100 GPUs. Code, data and models will be released to facilitate the community.
Read the full abstract ▾
All authors Checheng Yu, Chonghao Sima, Gangcheng Jiang, Hai Zhang, Haoguang Mai, Hongyang Li, Huijie Wang, Jin Chen, Kaiyang Wu, Li Chen, Lirui Zhao, Modi Shi, Ping Luo, Qingwen Bu, Shijia Peng, Tianyu Li, Yibo Yuan
Robotics
Robustness Is a Function, Not a Number: A Factorized Comprehensive Study of OOD Robustness in Vision-Based Driving
Amir Mallak and Alaa Maalouf
Out of distribution (OOD) robustness in autonomous driving is often reduced to a single number, hiding what breaks a policy. We decompose environments along five axes: scene (rural/urban), season, weather, time (day/night), and
agent mix; and measure performance under controlled $k$-factor perturbations ($k \in \{0,1,2,3\}$). Using closed loop control in VISTA, we
benchmark FC, CNN, and ViT policies, train compact ViT heads on frozen foundation-model (FM) features, and vary ID support in scale, diversity, and temporal context. (1) ViT policies are markedly more OOD-robust than comparably sized CNN/FC, and FM features yield state-of-the-art success at a latency cost. (2) Naive temporal inputs (multi-frame) do not beat the best single-frame baseline. (3) The largest single factor drops are rural $\rightarrow$ urban and day $\rightarrow$ night ($\sim 31\%$ each); actor swaps $\sim 10\%$, moderate rain $\sim 7\%$; season shifts can be drastic, and combining a time flip with other changes further degrades performance. (4) FM-feature policies stay above $85\%$ under three simultaneous changes; non-FM single-frame policies take a large first-shift hit, and all no-FM models fall below $50\%$ by three changes. (5) Interactions are non-additive: some pairings partially offset, whereas season-time combinations are especially harmful. (6) Training on winter/snow is most robust to single-factor shifts, while a rural+summer baseline gives the best overall OOD performance. (7) Scaling traces/views improves robustness ($+11.8$ points from $5$ to $14$ traces), yet targeted exposure to hard conditions can substitute for scale. (8) Using multiple ID environments broadens coverage and strengthens weak cases (urban OOD $60.6\% \rightarrow 70.1\%$) with a small ID drop; single-ID preserves peak performance but in a narrow domain. These results yield actionable design rules for OOD-robust driving policies.
Read the full abstract ▾
Robotics
Contact-Anchored Policies: Contact Conditioning Creates Strong Robot Utility Models
Zichen Jeff Cui, Omar Rayyan, Haritheja Etukuru et al. (+16 more)
The prevalent paradigm in
robot learning attempts to generalize across environments, embodiments, and tasks with
language prompts at runtime. A fundamental tension limits this approach:
language is often too abstract to guide the concrete physical understanding required for robust manipulation. In this work, we introduce Contact-Anchored Policies (CAP), which replace
language conditioning with points of physical contact in space. Simultaneously, we structure CAP as a library of modular utility models rather than a monolithic generalist policy. This factorization allows us to implement a real-to-sim iteration cycle: we build EgoGym, a lightweight simulation
benchmark, to rapidly identify failure modes and refine our models and
datasets prior to real-world deployment. We show that by conditioning on contact and iterating via simulation, CAP generalizes to novel environments and embodiments out of the box on three fundamental manipulation skills while using only 23 hours of demonstration data, and outperforms large, state-of-the-art VLAs in zero-shot evaluations by 56%. All model checkpoints, codebase, hardware, simulation, and
datasets will be open-sourced. Project page: https://cap-policy.github.io/
Read the full abstract ▾
All authors Zichen Jeff Cui, Omar Rayyan, Haritheja Etukuru, Bowen Tan, Zavier Andrianarivo, Zicheng Teng, Yihang Zhou, Krish Mehta, Nicholas Wojno, Kevin Yuanbo Wu, Manan H Anjaria, Ziyuan Wu, Manrong Mao, Guangxun Zhang, Binit Shah, Yejin Kim, Soumith Chintala, Lerrel Pinto, Nur Muhammad Mahi Shafiullah
Robotics
Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstruction
Hongyi Chen, Tony Dong, Tiancheng Wu et al. (+7 more)
Multi-finger robotic hand manipulation and grasping are challenging due to the high-dimensional action space and the difficulty of acquiring large-scale training data. Existing approaches largely rely on human teleoperation with wearable devices or specialized sensing equipment to capture hand-object interactions, which limits scalability. In this work, we propose VIDEOMANIP, a device-free framework that learns dexterous manipulation directly from RGB human
videos. Leveraging recent advances in computer
vision, VIDEOMANIP reconstructs explicit 3D
robot-object trajectories from
monocular videos by estimating human hand poses, object meshes, and retargets the reconstructed human motions to robotic hands for manipulation learning. To make the reconstructed
robot data suitable for dexterous manipulation training, we introduce hand-object contact optimization with interaction-centric grasp modeling, as well as a demonstration synthesis strategy that generates diverse training trajectories from a single
video, enabling generalizable policy learning without additional
robot demonstrations. In simulation, the learned grasping model achieves a 70.25% success rate across 20 diverse objects using the Inspire Hand. In the real world, manipulation policies trained from RGB
videos achieve an average 62.86% success rate across seven tasks using the LEAP Hand, outperforming retargeting-based methods by 15.87%. Project
videos are available at videomanip.github.io.
Read the full abstract ▾
All authors Hongyi Chen, Tony Dong, Tiancheng Wu, Liquan Wang, Yash Jangir, Yaru Niu, Yufei Ye, Homanga Bharadhwaj, Zackory Erickson, Jeffrey Ichnowski
Robotics
From Obstacles to Etiquette: Robot Social Navigation with VLM-Informed Path Selection
Zilin Fang, Anxing Xiao, David Hsu et al. (+1 more)
Navigating socially in human environments requires more than satisfying
geometric constraints, as collision-free paths may still interfere with ongoing activities or conflict with social norms. Addressing this challenge calls for analyzing interactions between
agents and incorporating common-sense
reasoning into planning. This paper presents a social
robot navigation framework that integrates
geometric planning with contextual social
reasoning. The system first extracts obstacles and human dynamics to generate geometrically feasible candidate paths, then leverages a fine-tuned
vision-
language model (VLM) to evaluate these paths, informed by contextually grounded social expectations, selecting a socially optimized path for the controller. This task-specific VLM distills social
reasoning from large foundation models into a smaller and efficient model, allowing the framework to perform real-time adaptation in diverse human-
robot interaction contexts. Experiments in four social
navigation contexts demonstrate that our method achieves the best overall performance with the lowest personal space violation duration, the minimal pedestrian-facing time, and no social zone intrusions. Project page: https://path-etiquette.github.io
Read the full abstract ▾
All authors Zilin Fang, Anxing Xiao, David Hsu, Gim Hee Lee
Robotics
CLUE: Crossmodal disambiguation via Language-vision Understanding with attEntion
Mouad Abrini and Mohamed Chetouani
With the increasing integration of
robots into daily life, human-
robot interaction has become more complex and multifaceted. A critical component of this interaction is Interactive Visual Grounding (IVG), through which
robots must interpret human intentions and resolve ambiguity. Existing IVG models generally lack a mechanism to determine when to ask clarification questions, as they implicitly rely on their learned representations. CLUE addresses this gap by converting the VLM's cross-modal
attention into an explicit, spatially grounded signal for deciding when to ask. We extract text to image
attention maps and pass them to a lightweight CNN to detect referential ambiguity, while a LoRA fine-tuned decoder conducts the dialog and emits grounding location
tokens. We train on a real-world interactive
dataset for IVG, and a mixed ambiguity set for the detector. With InViG-only supervision, our model surpasses a state-of-the-art method while using parameter-efficient fine-tuning. Similarly, the ambiguity detector outperforms prior baselines. Overall, CLUE turns the internal cross-modal
attention of a VLM into an explicit, spatially grounded signal for deciding when to ask. The data and code are publicly available at: mouadabrini.github.io/clue
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Models
Yu Shang, Zhuohang Li, Yiding Ma et al. (+18 more)
While world models have emerged as a cornerstone of
embodied intelligence by enabling
agents to reason about environmental dynamics through action-conditioned prediction, their evaluation remains fragmented. Current evaluation of
embodied world models has largely focused on perceptual fidelity (e.g.,
video generation quality), overlooking the functional utility of these models in downstream decision-making tasks. In this work, we introduce WorldArena, a unified
benchmark designed to systematically evaluate
embodied world models across both perceptual and functional dimensions. WorldArena assesses models through three dimensions:
video perception quality, measured with 16 metrics across six sub-dimensions;
embodied task functionality, which evaluates world models as data engines, policy evaluators, and action planners integrating with subjective human evaluation. Furthermore, we propose EWMScore, a holistic metric integrating multi-dimensional performance into a single interpretable index. Through extensive experiments on 14 representative models, we reveal a significant perception-functionality gap, showing that high visual quality does not necessarily translate into strong
embodied task capability. WorldArena
benchmark with the public leaderboard is released at https://world-arena.ai, providing a framework for tracking progress toward truly functional world models in
embodied AI.
Read the full abstract ▾
All authors Yu Shang, Zhuohang Li, Yiding Ma, Weikang Su, Xin Jin, Ziyou Wang, Lei Jin, Xin Zhang, Yinzhou Tang, Haisheng Su, Chen Gao, Wei Wu, Xihui Liu, Dhruv Shah, Zhaoxiang Zhang, Zhibo Chen, Jun Zhu, Yonghong Tian, Tat-Seng Chua, Wenwu Zhu, Yong Li
Robotics
Reduced-order Control and Geometric Structure of Learned Lagrangian Latent Dynamics
Katharina Friedl, Noémie Jaquier, Seungyeon Kim et al. (+2 more)
Model-based controllers can offer strong guarantees on stability and convergence by relying on physically accurate dynamic models. However, these are rarely available for high-dimensional mechanical systems such as deformable objects or soft
robots. While neural architectures can learn to approximate complex dynamics, they are either limited to low-dimensional systems or provide only limited formal control guarantees due to a lack of embedded physical structure. This paper introduces a latent control framework based on learned structure-preserving reduced-order dynamics for high-dimensional Lagrangian systems. We derive a reduced tracking law for fully actuated systems and adopt a Riemannian perspective on projection-based model-order reduction to study the resulting latent and projected closed-loop dynamics. By quantifying the sources of modeling error, we derive interpretable conditions for stability and convergence. We extend the proposed controller and analysis to underactuated systems by introducing learned actuation patterns. Experimental results on simulated and real-world systems validate our theoretical investigation and the accuracy of our controllers.
Read the full abstract ▾
All authors Katharina Friedl, Noémie Jaquier, Seungyeon Kim, Jens Lundell, Danica Kragic
Robotics · Cross-listed from cs.CV
Modeling 3D Pedestrian-Vehicle Interactions for Vehicle-Conditioned Pose Forecasting
Guangxun Zhu, Xuan Liu, Nicolas Pugeault et al. (+2 more)
Accurately predicting pedestrian motion is crucial for safe and reliable autonomous driving in complex urban environments. In this work, we present a 3D vehicle-conditioned pedestrian pose
forecasting framework that explicitly incorporates surrounding vehicle information. To support this, we enhance the Waymo-3DSkelMo
dataset with aligned 3D vehicle bounding boxes, enabling realistic modeling of multi-
agent pedestrian-vehicle interactions. We introduce a sampling scheme to categorize scenes by pedestrian and vehicle count, facilitating training across varying interaction complexities. Our proposed network adapts the TBIFormer architecture with a dedicated vehicle encoder and pedestrian-vehicle interaction cross-
attention module to fuse pedestrian and vehicle features, allowing predictions to be conditioned on both historical pedestrian motion and surrounding vehicles. Extensive experiments demonstrate substantial improvements in
forecasting accuracy and validate different approaches for modeling pedestrian-vehicle interactions, highlighting the importance of vehicle-aware 3D pose prediction for autonomous driving. Code is available at: https://github.com/GuangxunZhu/VehCondPose3D
Read the full abstract ▾
All authors Guangxun Zhu, Xuan Liu, Nicolas Pugeault, Chongfeng Wei, Edmond S. L. Ho
Robotics
Finite-Time Teleoperation of Euler-Lagrange Systems via Energy-Shaping
Lazaro F. Torres, Carlos I. Aldana, Emmanuel Nuño et al. (+1 more)
This paper proposes a family of finite-time controllers for the bilateral teleoperation of fully actuated nonlinear Euler-Lagrange systems. Based on the energy-shaping framework and under the standard assumption of passive interactions with the human and the environment, the controllers ensure that the position error and velocities globally converge to zero in the absence of time delays. In this case, the closed-loop system admits a homogeneous approximation of negative degree, and thus the control objective is achieved in finite-time. The proposed controllers are simple, continuous-time proportional-plus-damping-injection schemes, validated through both simulation and experimental results.
Read the full abstract ▾
All authors Lazaro F. Torres, Carlos I. Aldana, Emmanuel Nuño, Emmanuel Cruz-Zavala
Robotics · Cross-listed from cs.AR
karl. – A Research Vehicle for Automated and Connected Driving
Jean-Pierre Busch, Lukas Ostendorf, Guido Linden et al. (+5 more)
As highly automated driving is transitioning from single-vehicle closed-access testing to commercial deployments of public ride-hailing in selected areas (e.g., Waymo), automated driving and connected cooperative intelligent transport systems (C-ITS) remain active fields of research. Even though simulation is omnipresent in the development and validation life cycle of automated and connected driving technology, the complex nature of public road traffic and software that masters it still requires real-world integration and testing with actual vehicles. Dedicated vehicles for research and development allow testing and validation of software and hardware components under real-world conditions early on. They also enable collecting and publishing real-world
datasets that let others conduct research without vehicle access, and support early demonstration of futuristic use cases. In this paper, we present karl., our new research vehicle for automated and connected driving. Apart from major corporations, few institutions worldwide have access to their own L4-capable research vehicles, restricting their ability to carry out independent research. This paper aims to help bridge that gap by sharing the
reasoning, design choices, and technical details that went into making karl. a flexible and powerful platform for research, engineering, and validation in the context of automated and connected driving. More impressions of karl. are available at https://karl.ac.
Read the full abstract ▾
All authors Jean-Pierre Busch, Lukas Ostendorf, Guido Linden, Lennart Reiher, Till Beemelmanns, Bastian Lampe, Timo Woopen, Lutz Eckstein
Robotics
Multi-Staged Framework for Safety Analysis of Offloaded Services in Distributed Intelligent Transportation Systems
Robin Dehler, Oliver Schumann, Jona Ruof et al. (+1 more)
The integration of service-oriented architectures (SOA) with function offloading for distributed, intelligent transportation systems (ITS) offers the opportunity for connected autonomous vehicles (CAVs) to extend their locally available services. One major goal of offloading a subset of functions in the processing chain of a CAV to remote devices is to reduce the overall computational complexity on the CAV. The extension of using remote services, however, requires careful safety analysis, since the remotely created data are corrupted more easily, e.g., through an attacker on the remote device or by intercepting the wireless transmission. To tackle this problem, we first analyze the concept of SOA for distributed environments. From this, we derive a safety framework that validates the reliability of remote services and the data received locally. Since it is possible for the autonomous driving task to offload multiple different services, we propose a specific multi-staged framework for safety analysis dependent on the service composition of local and remote services. For efficiency reasons, we directly include the multi-staged framework for safety analysis in our service-oriented function offloading framework (SOFOF) that we have proposed in earlier work. The evaluation compares the performance of the extended framework considering computational complexity, with energy savings being a major motivation for function offloading, and its capability to detect data from corrupted remote services.
Read the full abstract ▾
All authors Robin Dehler, Oliver Schumann, Jona Ruof, Michael Buchholz
Robotics
A Generic Service-Oriented Function Offloading Framework for Connected Automated Vehicles
Robin Dehler and Michael Buchholz
Function offloading is a promising solution to address limitations concerning computational capacity and available energy of Connected Automated Vehicles~(CAVs) or other autonomous
robots by distributing computational tasks between local and remote computing devices in form of distributed services. This paper presents a generic function offloading framework that can be used to offload an arbitrary set of computational tasks with a focus on autonomous driving. To provide flexibility, the function offloading framework is designed to incorporate different offloading decision making
algorithms and quality of service~(QoS) requirements that can be adjusted to different scenarios or the objectives of the CAVs. With a focus on the applicability, we propose an efficient location-based approach, where the decision whether tasks are processed locally or remotely depends on the location of the CAV. We apply the proposed framework on the use case of service-oriented trajectory planning, where we offload the trajectory planning task of CAVs to a Multi-Access Edge Computing~(MEC) server. The evaluation is conducted in both simulation and real-world application. It demonstrates the potential of the function offloading framework to guarantee the QoS for trajectory planning while improving the computational efficiency of the CAVs. Moreover, the simulation results also show the adaptability of the framework to diverse scenarios involving simultaneous offloading requests from multiple CAVs.
Read the full abstract ▾
Robotics
GaussianCaR: Gaussian Splatting for Efficient Camera-Radar Fusion
Santiago Montiel-Marín, Miguel Antunes-García, Fabio Sánchez-García et al. (+3 more)
Robust and accurate perception of dynamic objects and map elements is crucial for autonomous vehicles performing safe
navigation in complex traffic scenarios. While
vision-only methods have become the de facto standard due to their technical advances, they can benefit from effective and cost-efficient fusion with radar measurements. In this work, we advance fusion methods by repurposing Gaussian Splatting as an efficient universal view
transformer that bridges the view disparity gap, mapping both image pixels and radar points into a common Bird's-Eye View (BEV) representation. Our main contribution is GaussianCaR, an end-to-end network for BEV
segmentation that, unlike prior BEV fusion methods, leverages Gaussian Splatting to map raw sensor information into latent features for efficient camera-radar fusion. Our architecture combines multi-scale fusion with a
transformer decoder to efficiently extract BEV features. Experimental results demonstrate that our approach achieves performance on par with, or even surpassing, the state of the art on BEV
segmentation tasks (57.3%, 82.9%, and 50.1% IoU for vehicles, roads, and lane dividers) on the nuScenes
dataset, while maintaining a 3.2x faster inference runtime. Code and project page are available online.
Read the full abstract ▾
All authors Santiago Montiel-Marín, Miguel Antunes-García, Fabio Sánchez-García, Angel Llamazares, Holger Caesar, Luis M. Bergasa
Robotics
Mind the Gap: Rethinking I/O Design for Contact-Rich Visuomotor Policy Learning
Cuijie Xu, Shurui Zheng, Zihao Su et al. (+7 more)
Contact-rich teleoperation logs expose a policy I/O design choice: demonstrations may contain the
robot execution (E), leader command (C), or both. These signals are not interchangeable: E2E may discard contact-generating command offsets, whereas E2C preserves these offsets but omits the
robot's execution response. We propose Dual-State Conditioning (EC2C), which conditions on both E and C while predicting future C, exposing command-execution mismatch as a cue for contact, latency, payload, and operator compensation; in quasi-static contact, this cue is often force-correlated. On a low-cost setup without force, tactile, or motor-current policy input, EC2C outperforms E2E and a strong E2C baseline across several real-world contact-rich, force-sensitive, and dynamic tasks. These results support EC2C as a practical default I/O setting for contact-rich imitation learning. We further formulate latency-adaptive inpainting as a temporal extension of this I/O choice for action-chunking policies, and discuss when long histories help dynamic inference or introduce causal confounding.
Read the full abstract ▾
All authors Cuijie Xu, Shurui Zheng, Zihao Su, Zhongchen Jian, Yuanfan Xu, Tinghao Yi, Xudong Zhang, Jian Wang, Yu Wang, Jinchen Yu
Robotics
High-Speed Vision-Based Flight in Clutter with Safety-Shielded Reinforcement Learning
Jiarui Zhang, Chengyong Lei, Chengjiang Dai et al. (+4 more)
Quadrotor unmanned aerial vehicles (
UAVs) are increasingly deployed in complex missions that demand reliable autonomous
navigation and robust obstacle avoidance. However, traditional modular pipelines often incur cumulative latency, whereas purely reinforcement learning (RL) approaches typically provide limited formal safety guarantees. To bridge this gap, we propose an end-to-end RL framework augmented with model-based safety mechanisms. We incorporate physical priors in both training and deployment. During training, we design a physics-informed reward structure that provides global navigational guidance. During deployment, we integrate a real-time safety filter that projects the policy outputs onto a provably safe set to enforce strict collision-avoidance constraints. This hybrid architecture reconciles high-speed flight with robust safety assurances.
Benchmark evaluations demonstrate that our method outperforms both traditional planners and recent end-to-end obstacle avoidance approaches based on differentiable physics. Extensive experiments demonstrate strong generalization, enabling reliable high-speed
navigation in dense clutter and challenging outdoor forest environments at velocities up to 7.5 m/s}.
Read the full abstract ▾
All authors Jiarui Zhang, Chengyong Lei, Chengjiang Dai, Kenghou Hoi, Lijie Wang, Zhichao Han, Fei Gao
Robotics
Mimic Intent, Not Just Trajectories
Renming Huang, Chendong Zeng, Wenjing Tang et al. (+3 more)
While imitation learning (IL) has achieved impressive success in dexterous manipulation through generative modeling and pretraining, state-of-the-art approaches like
Vision-
Language-Action (VLA) models still struggle with adaptation to environmental changes and skill transfer. We argue this stems from mimicking raw trajectories without understanding the underlying intent. To address this, we propose explicitly disentangling behavior intent from execution details in end-2-end IL: Mimic Intent, Not just Trajectories(MINT). We achieve this via multi-scale frequency-space tokenization, which enforces a spectral decomposition of action chunk representation. We learn action
tokens with a multi-scale coarse-to-fine structure, and force the coarsest
token to capture low-frequency global structure and finer
tokens to encode high-frequency details. This yields an abstract Intent
token that facilitates planning and transfer, and multi-scale Execution
tokens that enable precise adaptation to environmental dynamics. Building on this hierarchy, our policy generates trajectories through next-scale autoregression, performing progressive intent-to-execution
reasoning, thus boosting learning efficiency and generalization. Crucially, this disentanglement enables one-shot transfer of skills, by simply injecting the Intent
token from a demonstration into the autoregressive generation process. Experiments on several manipulation
benchmarks and on a real
robot demonstrate state-of-the-art success rates, superior inference efficiency, robust generalization against disturbances, and effective one-shot transfer.
Read the full abstract ▾
All authors Renming Huang, Chendong Zeng, Wenjing Tang, Jintian Cai, Cewu Lu, Panpan Cai
Robotics
A Precise Real-Time Force-Aware Grasping System for Robust Aerial Manipulation
Kenghou Hoi, Yuze Wu, Annan Ding et al. (+4 more)
Aerial manipulation requires force-aware capabilities to enable safe and effective grasping and physical interaction. Previous works often rely on heavy, expensive force sensors unsuitable for typical quadrotor platforms, or perform grasping without force feedback, risking damage to fragile objects. To address these limitations, we propose a novel force-aware grasping framework incorporating six low-cost, sensitive skin-like tactile sensors. We introduce a
magnetic-based tactile sensing module that provides high-precision three-dimensional force measurements. We eliminate geomagnetic interference through a reference
Hall sensor and simplify the calibration process compared to previous work. The proposed framework enables precise force-aware grasping control, allowing safe manipulation of fragile objects and real-time weight measurement of grasped items. The system is validated through comprehensive real-world experiments, including balloon grasping, dynamic load variation tests, and ablation studies, demonstrating its effectiveness in various aerial manipulation scenarios. Our approach achieves fully onboard operation without external motion capture systems, significantly enhancing the practicality of force-sensitive aerial manipulation. The supplementary
video is available at: https://www.youtube.com/watch?v=mbcZkrJEf1I.
Read the full abstract ▾
All authors Kenghou Hoi, Yuze Wu, Annan Ding, Junjie Wang, Anke Zhao, Chengqian Zhang, Fei Gao
Robotics
MOSAIC: Bridging the Sim-to-Real Gap in Generalist Humanoid Motion Tracking and Teleoperation with Rapid Residual Adaptation
Zhenguo Sun, Bo-Sheng Huang, Yibo Peng et al. (+9 more)
Generalist
humanoid motion trackers have recently achieved strong simulation metrics by scaling data and training, yet often remain brittle on hardware during sustained teleoperation due to interface- and dynamics-induced errors. We present MOSAIC, an open-source, full-stack system for
humanoid motion tracking and whole-body teleoperation across multiple interfaces. MOSAIC first learns a teleoperation-oriented general motion tracker via RL on a multi-source motion bank with adaptive resampling and rewards that emphasize world-frame motion consistency, which is critical for mobile teleoperation. To bridge the sim-to-real interface gap without sacrificing generality, MOSAIC then performs rapid residual adaptation: an interface-specific policy is trained using minimal interface-specific data, and then distilled into the general tracker through an additive residual module, outperforming naive fine-tuning or continual learning. We validate MOSAIC with systematic ablations, out-of-distribution benchmarking, and real-
robot experiments demonstrating robust offline motion replay and online long-horizon teleoperation under realistic latency and noise. Project page: baai-
humanoid.github.io/MOSAIC.
Read the full abstract ▾
All authors Zhenguo Sun, Bo-Sheng Huang, Yibo Peng, Xukun Li, Jingyu Ma, Yu Sun, Zhe Li, Haojun Jiang, Biao Gao, Zhenshan Bing, Xinlong Wang, Alois Knoll
Robotics
Head-to-Head autonomous racing at the limits of handling in the A2RL challenge
Simon Hoffmann, Simon Sagmeister, Tobias Betz et al. (+17 more)
Autonomous racing presents a complex challenge involving multi-
agent interactions between vehicles operating at the limit of performance and dynamics. As such, it provides a valuable research and testing environment for advancing autonomous driving technology and improving road safety. This article presents the
algorithms and deployment strategies developed by the TUM Autonomous Motorsport team for the inaugural Abu Dhabi Autonomous Racing League (A2RL). We showcase how our software emulates human driving behavior, pushing the limits of vehicle handling and multi-vehicle interactions to win the A2RL. Finally, we highlight the key enablers of our success and share our most significant learnings.
Read the full abstract ▾
All authors Simon Hoffmann, Simon Sagmeister, Tobias Betz, Joscha Bongard, Sascha Büttner, Dominic Ebner, Daniel Esser, Georg Jank, Sven Goblirsch, Alexander Langmann, Maximilian Leitenstern, Levent Ögretmen, Phillip Pitschi, Ann-Kathrin Schwehn, Cornelius Schröder, Marcel Weinmann, Frederik Werner, Boris Lohmann, Johannes Betz, Markus Lienkamp
Robotics
Combining Constrained Sampling and Reinforcement Learning for Manipulation
Marc Toussaint, Cornelius V. Braun, Armand Jordana et al. (+5 more)
Training manipulation policies in contact-rich settings is a core challenge in robotics. While Reinforcement Learning (RL) has demonstrated its strength in such settings, it may struggle to sufficiently explore and discover complex manipulation strategies. To address this, we combine two basic ideas: First, designing appropriate reset strategies (the start state distribution of episodes) has shown promise in improving RL exploration and effectiveness. Second, while model-based approaches to finding trajectories through manipulation are hard, recent work showed that model-based approaches to sampling states on constrained
manifolds can be highly efficient. Based on these observations, we propose a novel state sampler that enables efficient goal-conditioned RL in complex contact-rich manipulation tasks. Our sampler explicitly takes into account the structure of contact in order to provide a rich covering of diverse contact modes. By combining constrained sampling resets with projected interpolation and curriculum learning, our novel approach outperforms RL without constrained sampling and alternative reset methods, and effectively trains universal, prehensile and non-prehensile manipulation policies in contact-rich whole-body and in-hand settings. See the project page https://www.user.tu-berlin.de/mtoussai/26-CSRL/ for
videos and full source code access.
Read the full abstract ▾
All authors Marc Toussaint, Cornelius V. Braun, Armand Jordana, Sayantan Auddy, Eckart Cobo-Briesewitz, Denis Shcherba, Tilman Burghoff, Justin Carpentier
Robotics
UniPlan: Vision-Language Task Planning for Mobile Manipulation with Unified PDDL Formulation
Haoming Ye, Yunxiao Xiao, Cewu Lu et al. (+1 more)
Integration of VLM
reasoning with symbolic planning has proven to be a promising approach to real-world
robot task planning. Existing work like UniDomain effectively learns symbolic manipulation domains from real-world demonstrations, described in Planning Domain Definition
Language (PDDL), and has successfully applied them to real-world tasks. These domains, however, are restricted to tabletop manipulation. We propose UniPlan, a
vision-
language task planning system for long-horizon mobile-manipulation in large-scale indoor environments, that unifies scene
topology, visuals, and
robot capabilities into a holistic PDDL representation. UniPlan programmatically extends learned tabletop domains from UniDomain to support
navigation, door traversal, and bimanual coordination. It operates on a visual-topological map, comprising
navigation landmarks anchored with scene images. Given a
language instruction, UniPlan retrieves task-relevant nodes from the map and uses a VLM to ground the anchored image into task-relevant objects and their PDDL states; next, it reconnects these nodes to a compressed, densely-connected topological map, also represented in PDDL, with connectivity and costs derived from the original map; Finally, a mobile-manipulation plan is generated using off-the-shelf PDDL solvers. Evaluated on human-raised tasks in a large-scale map with real-world imagery, UniPlan significantly outperforms VLM and LLM+PDDL planning in success rate, plan quality, and computational efficiency.
Read the full abstract ▾
All authors Haoming Ye, Yunxiao Xiao, Cewu Lu, Panpan Cai
Robotics
Characteristics, Management, and Utilization of Muscles in Musculoskeletal Humanoids: Empirical Study on Kengoro and Musashi
Kento Kawaharazuka, Kei Okada and Masayuki Inaba
Various musculoskeletal
humanoids have been developed so far, and numerous studies on control mechanisms have been conducted to leverage the advantages of their biomimetic bodies. However, there has not been sufficient and unified discussion on the diverse properties inherent in these musculoskeletal structures, nor on how to manage and utilize them. Therefore, this study categorizes and analyzes the characteristics of muscles, as well as their management and utilization methods, based on the various research conducted on the musculoskeletal
humanoids we have developed, Kengoro and Musashi. We classify the features of the musculoskeletal structure into five properties: Redundancy, Independency, Anisotropy, Variable Moment Arm, and Nonlinear Elasticity. We then organize the diverse advantages and disadvantages of musculoskeletal
humanoids that arise from the combination of these properties. In particular, we discuss body schema learning and reflex control, along with muscle grouping and body schema adaptation. Also, we describe the implementation of movements through an integrated system and discuss future challenges and prospects.
Read the full abstract ▾
Robotics
Reliability-aware Execution Gating for Near-field and Off-axis Vision-guided Robotic Alignment
Ning Hu, Senhao Cao and Maochen Li
Vision-guided robotic systems are increasingly deployed in precision
alignment tasks that require reliable execution under near-field and off-axis configurations. While recent advances in pose estimation have significantly improved numerical accuracy, practical robotic systems still suffer from frequent execution failures even when pose estimates appear accurate. This gap suggests that pose accuracy alone is insufficient to guarantee execution-level reliability. In this paper, we reveal that such failures arise from a deterministic
geometric error amplification mechanism, in which small pose estimation errors are magnified through system structure and motion execution, leading to unstable or failed
alignment. Rather than modifying pose estimation
algorithms, we propose a Reliability-aware Execution Gating mechanism that operates at the execution level. The proposed approach evaluates
geometric consistency and configuration risk before execution, and selectively rejects or scales high-risk pose updates. We validate the proposed method on a real UR5 robotic platform performing single-step visual
alignment tasks under varying camera-target distances and off-axis configurations. Experimental results demonstrate that the proposed execution gating significantly improves task success rates, reduces execution variance, and suppresses tail-risk behavior, while leaving average pose accuracy largely unchanged. Importantly, the proposed mechanism is
estimator-agnostic and can be readily integrated with both classical geometry-based and learning-based pose estimation pipelines. These results highlight the importance of execution-level reliability modeling and provide a practical solution for improving robustness in near-field
vision-guided robotic systems.
Read the full abstract ▾
Robotics
UAV-Supported Maritime Search System: Experience from Valun Bay Field Trials
Stefan Ivić, Luka Lanča, Karlo Jakac et al. (+5 more)
This paper presents the integration of flow field reconstruction, dynamic probabilistic modeling, search control, and machine
vision detection in a system for autonomous maritime search operations. Field experiments conducted in Valun Bay (Cres Island, Croatia) involved real-time drifter data acquisition, surrogate flow model fitting based on computational fluid dynamics and numerical optimization, advanced multi-
UAV search control and
vision sensing, as well as deep learning-based object detection. The results demonstrate that a tightly coupled approach enables reliable detection of floating targets under realistic uncertainties and complex environmental conditions, providing concrete insights for future autonomous maritime search and rescue applications.
Read the full abstract ▾
All authors Stefan Ivić, Luka Lanča, Karlo Jakac, Ante Sikirica, Stella Dumenčić, Matej Mališa, Zvonimir Mrle, Bojan Crnković
Robotics
Post-Collision Trajectory Restoration for a Single-track Ackermann Vehicle using Heuristic Steering and Tractive Force Functions
Samsaptak Ghosh, M. Felix Orlando and Sohom Chakrabarty
Post-collision trajectory restoration is a safety-critical capability for autonomous vehicles, as impact-induced lateral motion and yaw transients can rapidly drive the vehicle away from the intended path. This paper proposes a structured heuristic recovery control law that jointly commands steering and tractive force for a generalized single-track Ackermann vehicle model. The formulation explicitly accounts for time-varying longitudinal velocity in the lateral-yaw dynamics and retains nonlinear steering-coupled interaction terms that are commonly simplified in the literature. Unlike approaches that assume constant longitudinal speed, the proposed design targets the transient post-impact regime where speed variations and nonlinear coupling significantly influence recovery. The method is evaluated in simulation on the proposed generalized single-track model and a standard 3DOF single-track reference model in MATLAB, demonstrating consistent post-collision restoration behaviour across representative initial post-impact conditions.
Read the full abstract ▾
Robotics
SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarios
Tian Gao, Celine Tan, Catherine Glossop et al. (+8 more)
A fundamental challenge in autonomous driving is the integration of high-level, semantic
reasoning for long-tail events with low-level, reactive control for robust driving. While large
vision-
language models (VLMs) trained on web-scale data offer powerful common-sense
reasoning, they lack the grounded experience necessary for safe vehicle control. We posit that an effective autonomous
agent should leverage the world knowledge of VLMs to guide a steerable driving policy toward robust control in driving scenarios. To this end, we propose SteerVLA, which leverages the
reasoning capabilities of VLMs to produce fine-grained
language instructions that steer a
vision-
language-action (VLA) driving policy. Key to our method is this rich
language interface between the high-level VLM and low-level VLA, which allows the high-level policy to more effectively ground its
reasoning in the control outputs of the low-level policy. To provide fine-grained
language supervision aligned with vehicle control, we leverage a VLM to augment existing driving data with detailed
language annotations, which we find to be essential for effective
reasoning and steerability. We evaluate SteerVLA on a challenging closed-loop
benchmark, where it outperforms state-of-the-art methods by 4.77 points in overall driving score and by 8.04 points on a long-tail subset. The project website is available at: https://steervla.github.io/.
Read the full abstract ▾
All authors Tian Gao, Celine Tan, Catherine Glossop, Timothy Gao, Jiankai Sun, Kyle Stachowicz, Shirley Wu, Oier Mees, Dorsa Sadigh, Sergey Levine, Chelsea Finn
Robotics
Bi-Adapt: Few-shot Bimanual Adaptation for Novel Categories of 3D Objects via Semantic Correspondence
Jinxian Zhou, Ruihai Wu, Yiwei Liu et al. (+5 more)
Bimanual manipulation is imperative yet challenging for
robots to execute complex tasks, requiring coordinated collaboration between two arms. However, existing methods for bimanual manipulation often rely on costly data collection and training, struggling to generalize to unseen objects in novel categories efficiently. In this paper, we present Bi-Adapt, a novel framework designed for efficient generalization for bimanual manipulation via semantic correspondence. Bi-Adapt achieves cross-category affordance mapping by leveraging the strong capability of
vision foundation models. Fine-tuning with restricted data on novel categories, Bi-Adapt exhibits notable generalization to out-of-category objects in a zero-shot manner. Extensive experiments conducted in both simulation and real-world environments validate the effectiveness of our approach and demonstrate its high efficiency, achieving a high success rate on different
benchmark tasks across novel categories with limited data. Project website: https://biadapt-project.github.io/
Read the full abstract ▾
All authors Jinxian Zhou, Ruihai Wu, Yiwei Liu, Yiwen Hou, Xunzhe Zhou, Checheng Yu, Licheng Zhong, Lin Shao