Robotics
Patch Policy: Efficient Embodied Control via Dense Visual Representations
Gaoyue Zhou, Zichen Jeff Cui, Ada Langford et al. (+3 more)
Pretrained dense visual features from
Vision Transformers (ViTs) are powerful yet have been underutilized in
robot learning. Modern
robot policies either compress each observation into a single global
token, or rely on visual backbones trained from scratch, sacrificing both fine-grained spatial detail and the benefits of large-scale visual pre-training. While there exist policies that do operate on dense patch features like large
vision-
language-action models (VLAs), they tend to be heavy and slow, inheriting the full cost of a billion-parameter
vision-
language model (VLM) backbone. We close this gap with Patch Policy, a minimal architectural extension that enables
transformer-based policies to consume dense pre-trained patch
tokens directly without the computational overhead of a full VLM. At its core is a block-causal
attention mask that preserves the temporal causality of standard policies while letting the model attend over many patch
tokens per observation, alongside other state information. Patch Policy is lightweight, fast, and highly effective. Across four simulated and three real-world environment suites, our method achieves a 40% relative improvement over policies using state-of-the-art global-pooled representations. Furthermore, it surpasses fine-tuned OpenVLA-OFT by 18% while using roughly 0.7% of the parameters. We believe Patch Policy provides a pipeline for the robotics community to readily leverage continuing progress in visual representation learning, without sacrificing the training efficiency or inference speed required for high-frequency, reactive control.
Videos can be viewed at https://patch-policy.github.io
Read the full abstract ▾
All authors Gaoyue Zhou, Zichen Jeff Cui, Ada Langford, Bowen Tan, Yann LeCun, Lerrel Pinto
Robotics
FM-VLA: Force-based Memory for Vision-Language-Action Models in Contact-Rich Manipulation
Ruicheng Li, Qixiu Li, Ruichun Ma et al. (+8 more)
Vision-
language-action (VLA) models have achieved impressive generalization in robotic manipulation, and recent
memory-augmented VLAs have relaxed the Markovian assumption by conditioning on past images or
language summaries.
Vision-based
memory approaches address this by conditioning on sampled past image frames, but they are computationally expensive and fundamentally limited when temporal events are visually ambiguous, e.g., pushing a button multiple times with small movements. We propose FM-VLA, a VLA model with force-based
memory, enabling temporal context
reasoning for non-Markovian, contact-rich manipulation. We encode force histories into compact force
memory tokens with a variational autoencoder (VAE) pretrained with force time series reconstruction. By projecting force latent representations and short state history as additional conditioning
tokens to the action expert module, we enable VLAs to leverage accumulated contact event history to guide manipulation. We evaluate FM-VLA on three
memory-dependent tasks, including finding a hidden block, pressing a button, and wiping a dish for a specific number of times. Our lightweight force
memory achieves over 80% success rate with minimal inference overhead, significantly outperforming baseline approaches. Project page: https://qft-333.github.io/FM-VLA-Page/
Read the full abstract ▾
All authors Ruicheng Li, Qixiu Li, Ruichun Ma, Yu Deng, Lin Luo, Zhiying Du, Jianfeng Xiang, Huizhi Liang, Ruicheng Wang, Jiaolong Yang, Baining Guo
Robotics
Optimization of sim-to-real transfer in the humanoid robot NICO
Juraj Gavura and Igor Farkaš
Robotic grasping requires accurate coordination between visual perception, object localization, inverse kinematics, and hand control. However, when movements planned in simulation are executed on a physical
robot, the sim-to-real gap can cause small positioning errors that prevent successful grasping. In our previous work, we introduced a low-cost haptic calibration method that improved 2D reaching accuracy of the
humanoid robot NICO. In this paper, we extend this approach from reaching to tabletop object grasping by adding YOLO-based object and hand detection, stereo
vision-based localization using the
robot's built-in low-resolution fisheye cameras, and task-specific corrections for grasp execution. Together, these components form a novel calibration-based grasping pipeline that does not require RGB-D cameras, motion capture, or external tracking systems. We also implemented a visual feedback model that aligns the
robot hand with the detected object before grasping. Our results show that the fully nonlinear calibration model achieved the best performance inside the calibrated area, while the visual feedback model achieved the highest overall grasping success across the full tabletop workspace.
Read the full abstract ▾
Robotics
Learning Adaptive Safety Margins for Visual Navigation
Junyi Hu, Shuaihang Yuan, Geeta Chandra Raju Bethala et al. (+2 more)
Robots in cluttered indoor spaces often fail not because they cannot generate collision-free paths, but because a fixed safety margin is mis-calibrated: conservative margins cause detours and timeouts, while permissive margins lead to near-boundary shortcuts under perception bias.
Diffusion-based planners propose diverse trajectory candidates from
egocentric RGB-D, yet reliable selection remains the bottleneck. We propose a context-conditioned safety critic that learns an adaptive clearance preference for ranking
diffusion proposals, decomposed into three complementary terms: (i) a safety term with a clearance-budget penalty and a control-barrier-function residual for waypoint- and transition-wise safety, (ii) an efficiency term combining a smoothness penalty with a safety-gated detour-ratio penalty that avoids detours without incentivizing risky shortcuts, and (iii) a distance-constraint matching term that anchors the learned budget to realized ESDF clearances to prevent margin collapse. We train the critic with privileged ESDF geometry in simulation and distill it into a perception-only selector via a two-stage teacher-student procedure. On PointGoal
navigation in HM3D and MP3D, including cross-
dataset transfer, our method achieves the highest success rate (SR) and success weighted by path length (SPL) among strong
diffusion, optimization, and RL baselines. Trained purely in simulation, it transfers to a Unitree G1
humanoid and navigates cluttered indoor scenes without task-specific tuning.
Read the full abstract ▾
All authors Junyi Hu, Shuaihang Yuan, Geeta Chandra Raju Bethala, Anthony Tzes, Yi Fang
Robotics
Imitation of Arm Gestures by the Semi-Humanoid Robot NICO
Anastasiya Ihnatovich and Igor Farkaš
Seamless human-
robot interaction (HRI) requires a number of perceptual and motor abilities from the
robot, one of them being the imitation of human gestures.
Humanoid robots have an advantage in HRI thanks to their anthropomorphic features. In this work, we develop a system for imitation of human arm gestures by the semi-
humanoid robot NICO based on analytical geometry and a pretrained MediaPipe pose-estimation model. For each input RGB frame, 3D coordinates of relevant human body landmarks, including arm joints and hand keypoints, are obtained using the MediaPipe framework. Joint angles are then computed from these coordinates using derived
geometric relations. Finally, the computed angles are properly mapped to NICO's motor configuration and executed in a predefined motion sequence. Preliminary experiments on several representative arm gestures with six participants of different height indicate that the proposed method can produce meaningful imitative motions from
monocular RGB input only, while also highlighting limitations in more complex poses and wrist-related movements.
Read the full abstract ▾
Robotics
World Translation: Minimizing Sim-to-Real Gap with Backward Dynamics Extraction and Unpaired Domain Translation
Xinchen Yao, Leixin Chang and Hua Chen
The gap between simulation and reality remains a fundamental challenge in deploying simulation-trained robotic policies in the real world. Real-to-sim methods narrow this gap from the real side, learning transition dynamics from real data to build a more realistic digital world. Learned dynamics models are their dominant instance. Such methods, however, face a partial observability problem: the same observation may branch to different transitions due to unobservable factors. Existing methods assume these factors can be recovered from observation history. However, this may fail whenever observation history is uninformative, such as a sudden contact event with no prior warning. To address this limitation, we propose World Translation, which exploits a complementary strength of simulators and learned dynamics. Simulators are deterministic but physically imperfect, while learned models are accurate but underdetermined under partial observability. Rather than predicting transitions forward from history, we extract the unobservable dynamics information backward from an observed transition, then translate this feature across simulation and reality as an unpaired domain-translation problem that preserves dynamics content while transferring domain style. Experiments across
humanoid, quadruped, and manipulator platforms show that our method achieves more accurate dynamics modeling than baselines, with the largest gains when unobservable factors cannot be recovered from observation history. Real-
robot deployment on Go2 quadruped confirms improved policy transfer.
Read the full abstract ▾
Robotics
Isaac Sim-to-Real: Reinforcement Learning based Locomotion for Quadrupeds
Jordan Dowdy and Jean Chagas Vaz
Learning-based approaches to locomotion have risen in popularity in recent years, showing the capability for complex legged locomotion and whole-body control. Reinforcement learning (RL), the primary learning-based approach for locomotion, often utilizes a high-performance simulation tool, providing a controlled and efficient training and development environment. However, policies that perform well in simulation frequently encounter unexpected challenges when deployed on a physical system, known as the sim-to-real gap. This work presents a robust RL locomotion framework capable of whole-body control. The proposed RL framework utilizes Nvidia's new set of simulation tools, Isaac Sim, and its companion RL framework, Isaac Lab, for training, achieving a zero-shot sim-to-real policy. The performance of our policy is validated on physical hardware using the Unitree Go1, with experimental results showing similar velocity tracking performance to the quadruped's integrated controller, with a greater ability to recover from large disturbances, and achieve linear velocities of 2.0 m/s and angular velocities of 1.8 rad/s.
Read the full abstract ▾
Robotics
Importance Sampling and PCA for Finding Failures in Commercial Autonomous Vehicles
Hailey Warner, Duncan Eddy, Shreya Parjan et al. (+6 more)
Methods for discovering rare failures in autonomous systems have so far been demonstrated almost exclusively in simulations with simple, academic driving stacks, leaving open whether they generalize to the more robust planners used in commercial systems. We address this gap by applying two rare-event discovery
algorithms to a commercial autonomous trucking stack. Adaptive stress testing (AST) uses reinforcement learning to search for the most likely noise trajectories leading to a simulated collision, while
diffusion-based failure sampling (DiFS) trains a denoising
diffusion model to sample a diverse set of failures. We show that both
algorithms find simulated collisions during merge and cut-in maneuvers where traditional Monte Carlo simulation does not. To make these failures actionable, we introduce a statistical analysis based on principal component analysis (PCA) that classifies failures into common modes and identifies the timesteps that most influence the outcome. We cluster the principal components and invert the PCA transform to recover generalized noise trajectories, and show that these trajectories reproduce failures in identical and similar scenarios. This provides a path from failure discovery to systematic diagnosis of perception-level flaws.
Read the full abstract ▾
All authors Hailey Warner, Duncan Eddy, Shreya Parjan, Caroline Cahilly, Harrison Delecki, Matthias Kleinstauber, Chaitanya Shinde, Jerry Lopez, Mykel J. Kochenderfer
Robotics
Technical Design Review of Duke Robotics Club's Oogway & Crush: AUVs for RoboSub 2026
Patrick Zheng, Saagar Arya, Hung Le et al. (+16 more)
The Duke Robotics Club presents Oogway and Crush, our AUVs for RoboSub 2026. This year's strategy expands on our previously narrowed scope, targeting all four of RoboSub's design goals for the first time: movement,
vision, manipulation, and acoustic tracking. This expansion is based on sustained reliability investment across all three subsystems. Mechanically, Crush gained two additional thrusters and a CFD-optimized case, providing pitch stability. Electrically, we addressed accumulated failure points by repairing unreliable connections and upgraded thruster control hardware. We also redesigned our acoustics system, adding a new custom PCB with higher-order filters, significantly improving pinger detection reliability. On the software side, improvements to state estimation, sonar-based object detection,
vision-driven task planning, and IVC enable more capable and coordinated autonomous runs. Paired with investments in our testing infrastructure to maximize our limited pool time, we can now attempt a broader set of tasks while maintaining the reliability our competition strategy demands.
Read the full abstract ▾
All authors Patrick Zheng, Saagar Arya, Hung Le, Mathew Chu, Nathanael Ren, Niko Weaver, Isabella Chen, Jill Wang, Raine Cheng, Siddharth Kini, Avrick Altmann, Srinath Iyer, Ivan Chen, Ian Suh, Parker Jones, Pierson Jones, Sebastian deSouza, Suhaani Sriram, Suvas Aggarwal
Robotics
UniETP: Unifying Environments for Generalizable Embodied Task Planning
Peiran Xu, Jiaqi Zheng, Ziyou Wang et al. (+1 more)
This paper focuses on the problem of
Embodied Task Planning, where an
agent is required to execute a sequence of atomic actions within an interactive environment to complete a user-specified task. Though a variety of simulators and
datasets have previously been built for this task, these efforts are largely isolated, with each using its own observation format, action type, and task domain. This fragmentation complicates comprehensive model evaluation and hinders the scalability of training data. As an effort towards generalizable
embodied planning, we propose UniETP, a unified interface integrating four commonly-used simulators (AI2-THOR, VirtualHome, Habitat, BEHAVIOR). UniETP is characterized by both standardization and diversity. On one hand, it formalizes all the simulators into a consistent observation and action space, and builds an evaluation system to support complicated task goal. On the other hand, it enhances task diversity and complexity across dimensions like task logic, instance grounding, and instruction understanding, constructing a new
dataset with varied levels of difficulty in an automatic manner. Extensive experiments on the proposed
benchmark are conducted to evaluate the
embodied planning capabilities of recent models and analyze the performance bottlenecks. Codes and data will be available at https://github.com/woyut/UniETP .
Read the full abstract ▾
All authors Peiran Xu, Jiaqi Zheng, Ziyou Wang, Yadong Mu
Robotics
RoboHarness: Memory-Driven Orchestration of Heterogeneous Robot Policies for Long-Horizon Planning
Jinbang Huang, Zhiyuan Li, Yuanzhao Hu et al. (+6 more)
Long-horizon robotic tasks require a breadth of capabilities beyond what any single existing
robot control policy can reliably provide. Combining heterogeneous policies with complementary strengths offers a promising solution, but introduces two key challenges: uncertain capability boundaries and distribution mismatches during policy handoffs. These challenges remain largely unaddressed by existing planning methods, which typically assume homogeneous, predefined skills with fixed applicability. We propose RoboHarness, a unified framework that encapsulates independently developed heterogeneous policies, including
vision-
language-action models (VLAs), world-action models (WAMs), reinforcement learning (RL) policies, and task and motion planners (TAMP), as reusable agentic skills. RoboHarness integrates understanding,
memory, and
evolution skills to reason about policy capabilities and support capability-aware task decomposition and policy routing. To mitigate distribution mismatches during policy handoffs, we introduce
Memory Bridge, a plug-in policy-chaining mechanism that enables reliable transitions between heterogeneous policies without joint retraining. Extensive experiments across five public
benchmarks, 500 customized tasks across 10 classes, and 135 real-
robot trials demonstrate substantial gains in long-horizon and
memory-dependent tasks, as well as robustness to out-of-distribution conditions.
Read the full abstract ▾
All authors Jinbang Huang, Zhiyuan Li, Yuanzhao Hu, Ran Qi, Yixin Xiao, Mark Coates, Tongtong Cao, Zhanguang Zhang, Yingxue Zhang
Robotics
Closing the Loop in Humanoid VLA: Persistent 3D Object Tokens for Verifiable Loco-Manipulation
Peng Ren, Haoyang Ge, Jiang Zhao et al. (+4 more)
Vision-
language-action policies are a promising foundation for general
robot control, but long-horizon
humanoid loco-manipulation requires the
robot to treat task objects as persistent physical entities across movement, contact, occlusion, and recovery. We study this problem as object-state divergence: the object state used to condition a whole-body action can differ from the state used to decide whether the action achieved the intended physical relation. We propose Persistent Object Tokenization (POT), which maintains role-indexed 3D object records from RGB-D observations and converts them into object
tokens for a whole-body action expert. Instantiated as POT-VLA, the same object records condition action generation and support
geometric predicate checks, yielding a closed-loop execution system in which object state is both actionable and verifiable. On a Unitree G1, POT-VLA improves a matched direct GR00T-N1.7 baseline from 39/80 to 71/80 successes over eight real-world task families. In an external Being-0-aligned reference, POT-VLA achieves 44/50 successes on aligned service tasks, compared with the 37/50 success reported by the Being-0 paper. The largest gains occur on tasks requiring maintained 3D relations, suggesting that persistent object-centered state is a useful abstraction for verifiable
humanoid VLA execution.
Read the full abstract ▾
All authors Peng Ren, Haoyang Ge, Jiang Zhao, Cong Huang, Yukun Shi, Pei Chi, Kai Chen
Robotics · Cross-listed from cs.IR
Remote Awareness of Seafloor Images Collected by AUVs over Low-Bandwidth Communication Links
Adrian Bodenmann, Cailei Liang, Miquel Massot-Campos et al. (+8 more)
This paper introduces a method for real-time processing and transmission of autonomous
underwater vehicle (AUV) imagery over low-bandwidth communication links. It leverages artificial intelligence (AI) techniques to identify a set of images that best represent an entire
dataset, or automatically finds the most similar images to a given query image for transmission to operators. Combined with metadata of a larger set of images, compressed versions of the selected images can be transmitted over satellite communication links or
underwater modems, and provide operators on shore with information about the type of imagery the AUV is collecting while it is still deployed. Data from three deployments off the coast of the UK and in Gran Canaria using different AUVs and imaging systems demonstrate the method in the field. It achieved an almost 400,000-fold reduction in data volume compared to the raw data size, enabling transmission of data summaries of a 2-hour 47-minute-long mapping mission in just over 34 minutes over low-bandwidth satellite communication.
Read the full abstract ▾
All authors Adrian Bodenmann, Cailei Liang, Miquel Massot-Campos, Samuel Simmons, Alexander B. Phillips, Alberto Consensi, Matthew Kingsland, Rashiid Sherif, Stan Brown, Adam Riese, Blair Thornton
Robotics
Distilling Global Traversability Priors for Image-based Affordance Prediction in Off-road Environments
Matthew Sivaprakasam, Samuel Triest, Micah Nye et al. (+5 more)
Standard methods for autonomous
navigation in unstructured terrain are prone to myopic behaviors in long-horizon scenarios. The use of metric maps built from
LiDAR or cameras provides necessary local geometry and semantic information but is strictly limited by depth sensing range. By discarding data beyond the mapping horizon
robots suffer from suboptimal, short-sighted decisions. To recover this lost information, we focus on extracting long-range traversability-aware frontiers directly from first-person-view (FPV) images. By leveraging satellite imagery, we compute the set of feasible
navigation paths for a
dataset of image/pose pairs and use them to supervise our network, reducing the need for extensive human demonstration data. We demonstrate that this approach improves performance in long-range off-road
navigation over existing methods by more than 10% in various offline
benchmarks and reduces the number of human interventions incurred in a set of real-world experiments. More details can be found at https://theairlab.org/ss_frontiers_iros .
Read the full abstract ▾
All authors Matthew Sivaprakasam, Samuel Triest, Micah Nye, Deegan Atha, Shehryar Khattak, David Fan, Wenshan Wang, Sebastian Scherer
Robotics
RynnBrain 1.1: Towards More Capable and Generalizable Embodied Foundation Model
Kehan Li, Bohan Hou, Minghao Zhu et al. (+28 more)
We present RynnBrain 1.1, a family of
embodied foundation models spanning 2B, 9B, and 122B-A10B scales. Trained with a unified spatio-temporal and physically grounded framework, RynnBrain 1.1 supports
embodied perception, spatial
reasoning, localization, and planning. Compared with RynnBrain 1.0, it further introduces contact-point prediction across the model family and native 3D grounding for the 2B and 9B models, yielding representations and outputs that are more directly aligned with
robot manipulation. We also develop RynnBrain-VLA with a unified cross-embodiment action space and embodiment-specific masking, and deploy it on Unitree G1, Astribot-S1, and Tianji-Wuji. RynnBrain 1.1 achieves strong results on
embodied cognition, localization, and 3D grounding, with the 122B-A10B model outperforming all evaluated proprietary and open-source models on VSI-Bench, MMSI, and RefSpatial-Bench. Real-
robot experiments show that RynnBrain-initialized policies outperform Qwen-based and representative generalist VLAs, while joint multi-task and multi-embodiment training improves process scores and success rates over per-task training.
Read the full abstract ▾
All authors Kehan Li, Bohan Hou, Minghao Zhu, Tianyi Zhang, Zesen Cheng, Zhikai Wang, Sicong Leng, Xin Li, Xiao Lin, Biying Yao, Minghua Zeng, Jiangpin Liu, Ronghao Dang, Jiayan Guo, Siteng Huang, Haoyu Zhao, Heng Ping, Yaxi Zhao, Tong Zhao, Kexiang Wang, Tong Lu, Shengke Xue, Jiahao Tang, Yulei Wang, Zejing Wang, Jianwei Gao, Shijian Lu, Chengju Liu, Jianfei Yang, Mingxiu Chen, Deli Zhao
Robotics
MEVION: Low-Cost Open-Source Data Collection System for Powerful and High-Speed Dual-Arm Manipulation
Kento Kawaharazuka, Yoshiki Obinata, Hirokazu Ishida et al. (+6 more)
The global competition for developing robotic foundation models is intensifying. Among the data collection systems used for dual-arm
robots, ALOHA is representative of being low-cost and open-source, and is widely adopted by researchers as a de facto standard. However, due to its limited ability to generate high forces and speeds, it is difficult to handle heavy objects or perform fast manipulations. To address this, we developed MEVION, a low-cost and open-source dual-arm
robot data collection system capable of generating greater force and speed. All parts of this
robot can be sourced through e-commerce, and by extensively utilizing sheet metal welding, its large body structure is constructed with a small number of components at low cost, while also simplifying assembly. MEVION is equipped with four 6-DoF arms with parallel grippers. Each arm weighs 7.0 kg and has a maximum torque of 60 Nm, and the entire system can be constructed for about USD 14,000. The elbow joint adopts a closed-link mechanism similar to those used in quadruped
robots, which reduces the distal mass and enables higher force and speed output at the end-effector. We demonstrate that MEVION enables data collection for object manipulation tasks not previously possible and supports imitation learning-based motion generation. All hardware and software of this work are included in the Supplementary Materials or https://github.com/haraduka/mevion.
Read the full abstract ▾
All authors Kento Kawaharazuka, Yoshiki Obinata, Hirokazu Ishida, Jihoon Oh, Temma Suzuki, Shintaro Inoue, Keita Yoneda, Ayumu Iwata, Kei Okada
Robotics
Does Robust VIO Need More Learning? Geometry-Verified Visual Measurements under Distribution Shift
Yangyang Ning, Shu Liang, Quanbo Ge et al. (+3 more)
Learning is increasingly introduced into visual-inertial odometry (VIO), ranging from learned feature front-ends to learning-dominant motion and geometry estimation. However, learning more of the pipeline does not necessarily improve robustness when deployment conditions differ from the training distribution. This work asks whether robust VIO under distribution shift truly requires deeper learned estimation, or whether learning can be confined to visual measurement generation. We propose a minimal-learning stereo VIO framework in which SEA-RAFT is used only to propose dense stereo correspondences and predict their uncertainty, while temporal tracking,
geometric verification, and state estimation remain explicit. Dense flow is sampled at sparse feature locations, filtered using predicted uncertainty and stereo epipolar consistency, and incorporated into a sliding-window stereo-inertial
estimator through uncertainty-weighted reprojection factors. The same uncertainty is further propagated through stereo triangulation for downstream anisotropic 3D Gaussian mapping. Experiments on EuRoC, VIODE, and 4Seasons demonstrate accurate and stable estimation under motion blur, dynamic scenes, illumination changes, and large indoor-to-outdoor distribution shifts. Ablations show that learned flow alone is insufficient: the gains arise from combining learned correspondence proposals with
geometric verification and uncertainty-aware weighting. These results suggest that, for OOD-robust VIO, carefully integrated learned visual measurements can be more effective than learning a larger fraction of the estimation pipeline. Code and configs for the
benchmark will be open-source upon acceptance. A supplementary
video is available at https://drive.google.com/file/d/1EVRhOkhanmNXHbQS1Vr80FoEIAYOYOV2/view
Read the full abstract ▾
All authors Yangyang Ning, Shu Liang, Quanbo Ge, Tianchen Deng, Yuhua Qi, Shenghai Yuan
Robotics · Cross-listed from cs.CR
RT-SHCUA: Real-Time Self-Hosted Computer-Use Agent for UAV Control
Di Lu, Bo Zhang, Xiyuan Li et al. (+5 more)
Natural-
language control offers a promising interface for unmanned aerial vehicles (
UAVs), but directly applying self-hosted computer-use
agents (SHCUAs) to
UAV control introduces a structural mismatch. SHCUAs are designed for interactive host-side tool use, where delayed
agent iterations are often acceptable.
UAV control, however, is coupled with continuously changing physical states, strict timing constraints, safety risks, and security accountability. A stale, unauthorized, or tampered
agent decision may therefore lead to unsafe or untraceable vehicle behavior. This paper proposes a real-time and security-oriented restructuring of SHCUA-based
UAV control. Instead of allowing an SHCUA to directly issue flight commands, we transform its outputs into contract-bound
UAV skill invocations with explicit timing, state, authority, fallback, and evidence semantics. Based on this abstraction, we design an architecture that separates semantic
reasoning from onboard execution and security/safety enforcement. Slow cloud or edge
reasoning is used for mission understanding, while onboard components validate and dispatch only timely, authorized, and state-consistent skills. Security-critical enforcement points can be protected by TEE-style or microcontroller isolation mechanisms without moving the full
language agent or high-frequency flight-control loop into trusted components. Prototype evaluation shows that RT-SHCUA maintains bounded task-level responsiveness while supporting degraded handling, trusted admission, and auditable evidence preservation for SHCUA-mediated
UAV actions.
Read the full abstract ▾
All authors Di Lu, Bo Zhang, Xiyuan Li, Yongzhi Liao, Xuewen Dong, Yulong Shen, Zhiquan Liu, Jianfeng Ma
Robotics
UMCP: A Unified Multi-Task Collaborative Perception Network for Luggage Trolley Pose Estimation
Zhirui Sun, Zhihao Jiang, Yao Wang et al. (+2 more)
In robotic autonomous luggage trolley collection,
robots must continuously localize scattered luggage trolleys in cluttered and dynamic environments. This requires the
vision system to achieve both high accuracy and real-time performance. However, existing visual perception approaches for luggage trolleys often rely on cascaded multi-model inference, leading to increased inference latency and high deployment costs. To address these limitations, this article presents a unified multi-task collaborative perception network (UMCP) that simultaneously performs luggage trolley detection, keypoint detection and orientation estimation. Based on the YOLOv12 architecture, keypoint features are fused with orientation features and then fed into an orientation feature enhancement module (OFEM), thereby improving orientation estimation accuracy. In addition, circular probability distribution modeling with a Kullback-Leibler (KL) divergence loss is adopted to enhance orientation estimation accuracy further. Experimental results demonstrate that the proposed method achieves competitive overall accuracy while substantially reducing model complexity and computational cost compared with existing methods. A website about this work is available at https://sites.google.com/view/
robot-umcp.
Read the full abstract ▾
All authors Zhirui Sun, Zhihao Jiang, Yao Wang, Jianwei Peng, Jiankun Wang
Robotics · Cross-listed from cs.MA
Value-Aware Prediction for Robust Multi-Agent Coordination Under Communication Loss
Kemal Devrim Kafadar, Eren Özaltun, Mahmud Efnan Şanl{\i} et al. (+4 more)
Robust multi-
agent coordination relies heavily on inter-
agent communication, which is frequently disrupted by physical and environmental constraints in real-world deployments. To maintain operation during these intermittent communication failures,
agents can employ internal prediction models to estimate missing shared state information. However, predictors trained with standard reconstruction objectives treat all transitions equally. In a Reinforcement Learning context, this forces the model to waste capacity learning
stochastic exploration noise and the outdated dynamics of suboptimal policies. In this paper, we propose a value-aware extension of Multi-
Agent Observation Sharing under Communication Dropout (MARO) to patch communication gaps; we refer to this method as Value-Aware MARO. By dynamically weighting the predictor's loss function using advantage estimates derived from the underlying actor-critic architecture, our objective explicitly couples the predictor's learning process to the policy's
evolution. This formulation focuses the model's capacity on the intentional, high-return dynamics actively reinforced by the
agents. We evaluate our framework on several tasks within the Multi-
Agent Particle Environment under varying communication reliability levels. Experimental results demonstrate that our approach maintains performance under declining communication reliability, particularly below 40%. While our method performs comparably in tasks where the baseline already maintains high coordination, our value-aware weighting effectively prevents the performance collapse observed in the standard predictor during high-attrition scenarios. In these environments, our method achieves an average improvement in mean returns of more than 20% and reduces performance variance by a mean of 64.7% compared to the standard unweighted baseline.
Read the full abstract ▾
All authors Kemal Devrim Kafadar, Eren Özaltun, Mahmud Efnan Şanl{\i}, Feyza Orak, Emirhan Gazi, Kubilay Kağan Kömürcü, Naz{\i}m Kemal Üre
Robotics
Manifold-Guided Motion Planning for Tight Assemblies
Dror Livnat, Michael M. Bilevich, Michal Kleinbort et al. (+1 more)
Motion planning for rigid-body assembly poses a fundamental challenge in robotics due to tight
geometric constraints. In such scenarios, feasible motions often require passing through (near-)zero clearance configurations in which the parts are tightly constrained by contact. In this work, we introduce Critical-
Manifold Guided RRT (CMG-RRT), a sampling-based planner designed specifically for tight assembly problems. Our key observation is that in tight assemblies, valid solution paths lie on or near a critical
manifold: the subset of configuration space consisting of poses with at least one contact point between parts. CMG-RRT guides exploration by adaptively biasing sampling toward neighborhoods of the critical
manifold using a hierarchical subdivision of the configuration space. We prove that CMG-RRT is probabilistically complete under standard clearance assumptions. Empirical evaluation on challenging rotational assembly
benchmarks demonstrates a 100% success rate across all tested instances, including, to the best of our knowledge, the first fully automatic solution of the Elk disentanglement puzzle. Our open source software is available through our project page: https://www.cgl.cs.tau.ac.il/projects/tight-assembly-planning.
Read the full abstract ▾
All authors Dror Livnat, Michael M. Bilevich, Michal Kleinbort, Dan Halperin
Robotics
Leveraging Two Robotic Arms for Tight Assembly Performance Gains
Dror Livnat, Yuval Lavi, Michael M. Bilevich et al. (+1 more)
We provide a novel end-to-end framework for the execution of an assembly operation by two robotic arms, given the digital CAD models of the parts and their desired relative placement in their assembled state. We analyze and demonstrate the advantages of using two robotic arms simultaneously in tight assembly operations, compared to single-arm systems. Our method is implemented in both simulation and using physical
robots. It provides theoretical guarantees on execution time and trajectory accuracy, supported by empirical evidence. In particular, we show that coordinated movement of two arms reduces average execution time by more than 50% compared to using a single arm only, produces higher-quality trajectories, and accelerates the search for valid
robot placements. Furthermore, we establish bounds on the required dimensions of the robotic
cell. Our open source software together with real-life
video demonstrations are available in our project page.
Read the full abstract ▾
All authors Dror Livnat, Yuval Lavi, Michael M. Bilevich, Dan Halperin
Robotics
ConceptTree: Bringing Semantic Transparency to Black-Box Decision Making for Robotic Manipulation
Yongyan Wen, Feifan Liu, Jinyi Chen et al. (+3 more)
Establishing interpretable decision-making processes in long-horizon robotic manipulation is critical for enabling reliable human oversight and intervention. However, existing approaches to robotic manipulation largely treat skill selection as opaque mappings from observations to actions, offering limited transparency into how decisions are formed. In this work, we propose ConceptTree, a framework that reframes high-level manipulation skill selection as
reasoning over human-interpretable concepts, representing high-level policies as a sequence of concept-level predicates over visual observations. Rather than relying on implicit latent representations, our method learns a normalized concept space grounded in visual inputs, over which a decision tree is trained to predict high-level skills. This formulation yields a transparent decision process that is both traceable and intervenable, enabling direct inspection and modification of policy behavior. We evaluate our approach on a set of real-world robotic manipulation tasks with increasing complexity. Experimental results show that ConceptTree consistently outperforms existing concept-based baselines, particularly in complex, long-horizon scenarios. Furthermore, we provide qualitative case studies showing that our model supports fine-grained intervention by modifying individual concepts, enabling targeted correction of decision errors without retraining.
Read the full abstract ▾
All authors Yongyan Wen, Feifan Liu, Jinyi Chen, Bo An, Peng Liu, Siyuan Li
Robotics
Lifelong Localization in Dynamic Indoor Environments Combining Odometry with Sparse Distance Sampling
Michael M. Bilevich, Tomer Buber and Dan Halperin
Localization is a key task in
robot navigation, and many techniques exist for it. In many plausible scenarios, a
robot might face unforeseen, dynamic obstacles, rendering any pre-determined map inaccurate for localization. In this work, we propose a robust lifelong localization framework in dynamic planar indoor environments, using the
robot's odometry and sparse distance sampling. We demonstrate how distance samples can be used to provide a robust prior on the
robot's location. This technique can solve the kidnapped
robot problem in real time, up to
symmetries. Based on insights from real-world recorded data, we also account for dynamic obstacles. We then fuse this prior, over time, with the odometry to converge to the
robot's location. A central property of our method is that it provably converges to the
robot's ground truth pose even in large indoor environments when the environment is static. We further show that this guarantee also holds in dynamic environments, as long as the nature of those changes has been correctly learned. We demonstrate the effectiveness of our approach in different real-world indoor environments. In particular, we achieve a localization comparable to SLAM with merely a few (sixteen) distance samples, as opposed to the full
LiDAR range. Sufficing with only sparse distance sampling is advantageous in terms of sensor cost,
privacy, storage space, and transmission bandwidth.
Read the full abstract ▾
Robotics
Receiver-Centered Robot-to-Human Handover with Grasp-Aware Object Orientation
Federico Biagi, Dario Onfiani, Simone Silenzi et al. (+1 more)
Collaborative
robots are increasingly sharing workspaces with human operators, making tool handover a frequent and safety-critical micro-interaction. However, traditional static handovers often lead to awkward grasps when handling asymmetric industrial tools. This paper presents a receiver-centered voice-driven adaptive handover system for mechanical tools, built on a Franka cobot. Using an LLM for intention recognition and MediaPipe for real-time 3D hand tracking, the framework dynamically adjusts the end-effector's orientation to present tools in an ergonomically optimal, handle-first pose. A within-subjects study compared this adaptive approach with an object-agnostic static baseline. The results demonstrate that the adaptive system reduces the grasp delay for asymmetric tools, improving the fluency of the interaction. Furthermore, the adaptive strategy improved specific trust-related perceptions, particularly motion predictability and perceived task simplicity.
Read the full abstract ▾
All authors Federico Biagi, Dario Onfiani, Simone Silenzi, Luigi Biagiotti
Robotics
Task-Space Constrained Stochastic Trajectory Optimization for Time-Optimal Forestry Crane Motion Planning
Marc-Philip Ecker, Christoph Fröhlich, Bernhard Bischof et al. (+2 more)
Efficient, collision-free, and time-optimal motion planning is a fundamental requirement for autonomous forestry cranes operating under hydraulic pump-flow constraints. The Via-Point-based
Stochastic Trajectory Optimization (VP-STO)
algorithm has demonstrated near-time-optimal hybrid motion planning in this domain, but requires a fixed terminal joint configuration specified prior to optimization. For kinematically redundant manipulators such as forestry cranes, this pre-commitment to a single inverse kinematics solution restricts the planner's ability to exploit redundancy, particularly under the nonlinear, globally coupled pump-flow constraint where admissible joint velocities depend on their combined hydraulic demand. This paper presents TSC-VP-STO, a task-space-constrained extension of VP-STO that replaces the strict terminal joint-space constraint with a task-space constraint, jointly optimizing the trajectory and the redundant degrees of freedom of the terminal configuration. This enables the planner to adapt end configurations to the environment-dependent motion and hydraulic flow allocation, yielding more balanced pump utilization and shorter trajectory durations. We formalize the approach through a configuration space decomposition and derive a concrete reachability constraint for the forestry crane kinematics. Experimental evaluations across multiple planning targets and via-point configurations demonstrates a reduction on trajectory durations by 12-15% on average and improved pump-flow utilization compared to the baseline VP-STO. The practical applicability of TSC-VP-STO is validated through real-world deployment on a forestry crane, including a full log-loading cycle.
Read the full abstract ▾
All authors Marc-Philip Ecker, Christoph Fröhlich, Bernhard Bischof, Wolfgang Kemmetmüller, Tobias Glück
Robotics
A2RL V\textsubscript{max}: The A2RL autonomous racing dataset for long-range, high-speed perception and multi-vehicle interaction
Marvin Klemp, Dominic Ebner, Cornelius Schröder et al. (+15 more)
In autonomous driving development, a perception
dataset is crucial, as it provides fundamental data for training, testing, and validating
algorithms for an autonomous vehicle's
multimodal perception systems. So far, most research has concentrated on providing
datasets for well-structured urban environments. This work introduces the A2RL V\textsubscript{max} open-source
dataset, specifically designed for perception tasks in high-speed autonomous driving and multi-vehicle interaction. The
dataset was captured during the 2024 Abu Dhabi Autonomous Racing League (A2RL), held at the Yas Marina F1
Circuit, with participation from all competing teams. It contains diverse scenarios, including single-vehicle data at varying speeds, multi-vehicle sessions, and the full final four-vehicle race. The
dataset contains almost 30,000 professionally annotated
LiDAR point clouds, along with RADAR point clouds. In particular, it is the first large-scale
dataset in autonomous racing to feature professionally annotated
LiDAR point clouds, enabling deep learning-based perception research. The data is provided in a developer-friendly format, enabling easy implementation and evaluation in future research. We provide implementation and evaluation for off-the-shelf 3D detection and tracking methods. Although baseline methods show promising results for both 3D detection and tracking, specialized methods are required to address the unique challenges of high-speed autonomous driving. For a detailed description of the
dataset, please visit the \href{https://tum-avs.github.io/A2RL_
Dataset_website/}{A2RL V\textsubscript{max}
Dataset Website}
Read the full abstract ▾
All authors Marvin Klemp, Dominic Ebner, Cornelius Schröder, Davide Malvezzi, László Turányi, Riccardo Donati, Ilia Schminik, Xia Ning, Yanxin Zhou, Matthew Flagg, Christoph Stiller, Markus Lienkamp, Marko Bertogna, Gergely Bári, Andreas Birk, Ren Jin, Chen Lv, Johannes Betz
Robotics
Reasoning as a Double-Edged Sword: Architecture and Cross-Stage Robustness in Vision-Language-Action Models
Tuan Duong Trinh, Naveed Akhtar and Basim Azam
Does adding a
reasoning step make a
Vision-
Language-Action (VLA) model more robust to perturbation? Intuitively, a policy that reasons before acting should absorb a perturbed input better than one that maps observations directly to actions. We test this premise head-on across three models that span the
reasoning spectrum (no
reasoning, a text chain-of-thought, and a latent iterative loop), perturbing each at the
vision,
reasoning, and action stages on LIBERO and SimplerEnv. Two questions organize the study: does the
reasoning design shift robustness, and can the
reasoning be read back at runtime as a safety signal? We find that the latent-iterative model is by far the least robust: under both
stochastic noise and white-box perturbation its task success collapses, while the other two hold. This fragility is structural rather than cumulative: varying the
reasoning depth at inference barely moves it.
Reasoning outputs can in principle be monitored, but the monitors fail under fair tests. A plan–action consistency probe that looks near-perfect under naive evaluation falls to chance under adaptive attack. Under matched-FPR calibration, fusing it with an action-anomaly probe never lifts defended success above undefended. Scoped to these output-level behavioral probes under white-box
vision-stage attack, this ceiling is a precondition that any viable defense must first satisfy.
Read the full abstract ▾
Robotics
Polar Coordinate-based Differential Evolution for Moving Target Search Using Vision Sensor on Unmanned Aerial Vehicles
Thu Hang Khuat, Duy-Nam Bui, Thuy Ngan Duong et al. (+1 more)
In search and rescue operations, there is a period known as the "golden time" during which the probability of finding the target alive is highest. The objective of this work is to propose a new search
algorithm for unmanned aerial vehicles (
UAVs) with a focus on improving the detection probability and execution time. We approach this problem by first modeling target dynamics as a Markov process and the detection likelihood as a function of image quality and the observer's
vision. We then employ
Bayesian theory to derive a fitness function representing the probability distribution of the target's location over the search area. Finally, we introduce a new
algorithm named polar coordinate-based differential
evolution (PDE) to generate a
UAV search path that maximizes this fitness function. The PDE
algorithm utilizes polar coordinates to incorporate kinematic constraints and maneuver properties of the
UAV, allowing for better exploration of the solution space. A series of simulations and comparative analyses have been conducted to evaluate the performance of the proposed
algorithm. Experiments involving a real
UAV have also been conducted. Results demonstrate that the PDE
algorithm outperforms state-of-the-art
algorithms in terms of detection probability and execution time across diverse search scenarios while remaining practical for real-world applications. The source code of the
algorithm is available at https://github.com/thuhangkhuat/PDE_target_search.
Read the full abstract ▾
All authors Thu Hang Khuat, Duy-Nam Bui, Thuy Ngan Duong, Manh Duong Phung
Robotics
From Sign Language Generation to Humanoid Execution: Vision-Language Guided Retargeting with Collision Mitigation
Nabeela Khan, Bowen Wu, Runwu Shi et al. (+5 more)
Recent sign
language generation (SLG) systems increasingly output dense 3D body representations, which better preserve full-body kinematics and geometry for downstream embodiment on
humanoid robots. However, these generated motions frequently exhibit self-intersections such as hand-hand and hand-torso penetration. While such artifacts may be tolerated in offline rendering, they become critical in
humanoid execution as they lead to infeasible inverse-kinematics (IK) solutions, collisions, and unstable retargeted trajectories. We present a system-level framework that bridges SLG outputs to
humanoid joint-space execution via two components. First, we introduce a volumetric SMPL-X collision-mitigation module that projects generated signing motions toward physically plausible configurations while minimally deviating from the original trajectory. Second, we propose a
vision-
language-guided retargeting
algorithm built on an IK backbone: a VLM serves as a visual critic over rendered
humanoid motion, identifies embodiment-specific failure modes, and triggers targeted task-space corrections. Our results highlight collision handling and perception-guided refinement as key missing components for reliable
humanoid signing.
Read the full abstract ▾
All authors Nabeela Khan, Bowen Wu, Runwu Shi, Benjamin Yen, Takeshi Ashizawa, Carlos Toshinori Ishi, Takashi Minato, Kazuhiro Nakadai