Robotics
Reinforcement Learning with Inner-loop Dynamics Estimator for Aerial Manipulation under Uncertainty
Shivansh Pratap Singh, Samaksh Ujjwal, Ishita Chaudhary et al. (+3 more)
Aerial manipulators enable physical interaction in hard-to-reach environments; however, the combined problem of direct whole-body aerial manipulation under rapid arm motion, payload changes, and related unknown dynamic uncertainty remains a largely unsolved problem. We present a hierarchical control framework that combines Reinforcement Learning (RL) with an inner-loop dynamics
estimator to address this problem. The RL outer loop maps desired 6-degrees-of-freedom (DOF) end-effector targets to coordinated whole-body commands, enabling direct task-driven control without relying on a fully accurate coupled dynamic model in the policy layer. An inner loop then tracks these commands while compensating for transient inertial shifts and uncertainty during execution via a dynamics
estimator scheme without requiring system model knowledge. We validate the proposed approach on a custom quadrotor equipped with a 3-DoF manipulator through hardware experiments under varying payload conditions. Compared with RL+PID and RL+INDI+PID baselines, the proposed method reduces end-effector tracking error and improves task success rate across the tested hardware conditions. These results show that combining learned whole-body coordination with
estimator-based low-level compensation improves the precision and robustness of aerial manipulation under changing operating conditions.
Read the full abstract ▾
All authors Shivansh Pratap Singh, Samaksh Ujjwal, Ishita Chaudhary, V R Vasudevan, Rishabh Dev Yadav, Spandan Roy
Robotics
WaveSync: Constrained Wavefront Optimization for Synchronized Co-Speech Gestures in Humanoid Robots
Thang Tran Viet, Thanh Nguyen Canh, Gia Huy Uong et al. (+4 more)
Expressive co-
speech gestures are crucial for natural human–
robot interaction, yet generating them on physical
humanoid robots remains challenging because, unlike virtual avatars,
robots must synchronize gestures with
speech under strict kinematic and actuator constraints. We present WaveSync, a hybrid framework in which a Large
Language Model decomposes dialogue responses into structured semantic schemas and assigns per-word importance weights, forming a continuous Semantic Importance Wave. Gesture trajectories are shaped through Dynamic Movement Primitives to ensure kinematic feasibility while enhancing expressiveness. A Wavefront Optimization stage aligns gesture stroke peaks with
speech emphasis peaks and resolves residual temporal conflicts through gesture-duration compression and forward propagation. Experimental evaluation across five dialogue scenarios demonstrates effective gesture–
speech alignment and favorable performance in both objective and subjective evaluations. The results further show that the key components of WaveSync contribute to producing gestures that are expressive, semantically grounded, and kinematically feasible. The code, resources, and
videos are available at \href{https://github.com/pairs-lab/WaveSync}{WaveSync}.
Read the full abstract ▾
All authors Thang Tran Viet, Thanh Nguyen Canh, Gia Huy Uong, Phuc Van Dinh, Tan Viet Tuyen Nguyen, Xiem HoangVan, Nak Young Chong
Robotics
Steering Generative Reinforcement Learning into Stable Robotic Controller
Yixuan Wang, Shutong Ding, Ke Hu et al. (+3 more)
Diffusion and flow-based generative policies provide a powerful policy class for reinforcement learning by inducing rich
stochastic exploration through iterative action generation. However, the stochasticity of
diffusion policies is not suitable for stable and precise control in high-dimensional robotic systems, where small action variations can accumulate into inconsistent motion and reduced robustness. To address this issue, we propose SteerGenPO, a latent-space reinforcement learning framework that steers a trained generative policy into a robust deterministic robotic controller. The key idea is to replace
stochastic latent sampling of the trained generative policy with a learned latent actor that predicts a state-dependent latent input for the generative policies. This separates exploration and control:
stochastic generative sampling provides diverse action proposals during policy learning, while deterministic latent steering provides stable and adaptive control at deployment. We evaluate SteerGenPO on six Isaac Lab
benchmarks and a Unitree G1 locomotion task. The results show SteerGenPO improves over both classical RL and generative RL baselines, while its deterministic latent steering produces more stable inference-time behaviors and more reliable command responses.
Read the full abstract ▾
All authors Yixuan Wang, Shutong Ding, Ke Hu, Tianxiang Gui, Jingya Wang, Ye Shi
Robotics
Automated Digital Twin Construction for Highway Scenarios Using LiDAR Point Clouds and OpenStreetMap
Yongqi Zhao, Dong Bi, Paul Kovacevic et al. (+4 more)
Accurate road environment modeling is fundamental to the simulation and validation of automated driving systems. However, constructing road maps in standardized formats such as ASAM OpenDRIVE from real-world sensor data remains a time-consuming and costly process. Mobile mapping
LiDAR captures accurate lane-level geometry but is confined to the driven corridor, while OpenStreetMap (OSM) provides broad road network
topology but lacks
geometric precision at the lane level. To address this, an automated workflow is proposed to fuse
LiDAR point clouds with OSM data to generate georeferenced ASAM OpenDRIVE maps of highway environments, requiring minimal manual intervention. The pipeline reconstructs mainline roads from
LiDAR-derived measurements and infers ramp geometry and
topology from the OSM road graph, enabling complete highway interchange modeling without full sensor coverage. Experiments demonstrate a mean lateral RMSE of 0.740 m, and the generated maps are directly usable in mainstream simulation platforms including IPG CarMaker and Esmini. These results validate the effectiveness of combining measurement-derived geometry with map-derived
topology for automated OpenDRIVE digital twin generation. The project code is available at https://github.com/ftgTUGraz/opendrive-digital-twin-generator
Read the full abstract ▾
All authors Yongqi Zhao, Dong Bi, Paul Kovacevic, Tomislav Mihalj, Martin Schabauer, Johannes Betz, Arno Eichberger
Robotics · Cross-listed from cs.CV
PROSE: Training-Free Egocentric Scene Registration with Vision-Language Models
Zhiang Chen, Nahyuk Lee, Boyang Sun et al. (+4 more)
Registering two captures of the same indoor space taken at different times underpins persistent spatial
memory for
robots and AR systems, yet the realistic version of this task is
egocentric and its most scalable form is RGB-only. Head-mounted cameras yield blurry, fast-moving, partially overlapping views from which dense geometry is hard to recover. Classical registration leans on exactly the clean point clouds this setting lacks, while learned scene-graph methods require a pre-built or annotated graph and a trained matcher that we find brittle under
egocentric data. We take a different route, using a pretrained
vision-
language model as the source of both scene understanding and cross-scan matching. Our method, PROSE (Prompted Scene rEgistration), lifts each RGB sequence into an object-level 3D scene graph using off-the-shelf foundation models for geometry,
segmentation, and
language, then
prompts the same VLM to match object instances across the two RGB sequences. To make this matching tractable and reliable, we leverage object heights as a prior and verify each proposed match with a paired same/different query, then solve for the rigid transform by hypothesizing a candidate per matched object and selecting the one with the strongest
geometric consensus. PROSE adds no learned parameters and requires no depth sensor, training, or annotated graph. On the
egocentric Aria Digital Twin and Aria Everyday Activities
benchmarks, it outperforms both
geometric and learned scene-graph baselines in registration accuracy, on ground-truth and RGB-reconstructed point clouds alike, and the scene graph it produces transfers directly to downstream tasks.
Read the full abstract ▾
All authors Zhiang Chen, Nahyuk Lee, Boyang Sun, Taein Kwon, Marc Pollefeys, Zuria Bauer, Sunghwan Hong
Robotics
Elastic ODYN: Differentiable Optimization for Infeasible Control and Learning in Robotics
Aristotelis Papatheodorou, Jose Rojas, Ioannis Havoutis et al. (+1 more)
Robotic systems routinely encounter conflicting objectives, modeling errors, and degenerate contact conditions that render quadratic programs (QPs) infeasible. Yet most optimization solvers and differentiable QP layers assume feasibility, leading to numerical failures, unstable gradients, or solver breakdown when constraints cannot be simultaneously satisfied. We present Elastic ODYN, a primal-dual non-interior-point QP solver that handles infeasibility through smooth squared-$\ell_2$ elastic relaxations. The formulation remains well posed under ill-conditioning and degeneracy, supports warm starting, and converges to closest-to-feasible solutions, with lightweight refinement recovering physically meaningful dual variables. Building on this framework, we develop Elastic ODYNLayer, a differentiable QP layer with stable gradients under infeasibility, and Elastic OdynSQP, an SQP method that resolves inconsistent subproblems and intrinsically infeasible optimal control tasks through selective constraint elasticity. Across
benchmark QPs, singular contact mechanics, differentiable parameter identification, and quadrupedal and
humanoid trajectory optimization, Elastic ODYN outperforms state-of-the-art elastic QP solvers in robustness, warm-start performance, and convergence reliability, enabling optimization, simulation, control, and learning beyond standard feasibility assumptions.
Read the full abstract ▾
All authors Aristotelis Papatheodorou, Jose Rojas, Ioannis Havoutis, Carlos Mastalli
Robotics · Cross-listed from cs.AI
ROSA-RL: Uncertainty-Aware Roundabout Optimized Speed Advisory with Reinforcement Learning
Anna-Lena Schlamp, Jeremias Gerner, Klaus Bogenberger et al. (+2 more)
Roundabouts challenge automated driving in mixed traffic, as heterogeneous and non-deterministic human behavior, unknown driving intentions, and high interaction complexity create uncertainty about whether the conflict zone will be blocked or available at the moment of entry. We present ROSA-RL – uncertainty-aware Roundabout Optimized Speed Advisory with Reinforcement Learning. It enables safe and efficient roundabout entry for automated and human-driven vehicles in mixed traffic through probabilistic conflict
forecasting. A
Transformer-based model predicts conflict zone occupancy over a five-second horizon, capturing multi-
agent interactions to anticipate upcoming conflicts and available gaps. The prediction outputs encode uncertainty in future motion and intent, and augment the state of a classical RL framework, enabling uncertainty-aware speed coordination. Evaluated in simulations grounded in real-world data, ROSA-RL can effectively handle uncertainty and outperform a comparable model-based baseline, closing the gap to an ideal setting assuming fully known occupancy while improving traffic efficiency and safety. The source code of this work is available under: github.com/urbanAIthi/ROSA-RL.
Read the full abstract ▾
All authors Anna-Lena Schlamp, Jeremias Gerner, Klaus Bogenberger, Werner Huber, Stefanie Schmidtner
Robotics
ADAPT: Analytical Disturbance-Aware Policy Training for Humanoid Locomotion
Bofan Lyu, Jindou Jia, Kuangji Zuo et al. (+7 more)
Humanoids deployed in human-centered environments must handle force-interactive tasks, where external contacts introduce unexpected disturbances that disrupt locomotion accuracy and stability. Existing learning-based approaches rely on broad domain randomization, task-specific force objectives, or learning-based force
estimators from motion history, each of which compromises accuracy, task transferability, or out-of-distribution (OOD) robustness. We present Analytical Disturbance-Aware Policy Training (ADAPT), a framework that equips
humanoid policies with a physically grounded disturbance observer. The core of ADAPT is an analytical whole-body disturbance observer that estimates residual force/torque online with the accessible
robot dynamics, without requiring force/torque sensors. Fed directly into the policy, the estimated disturbances give the
humanoid an explicit, physics-derived sense of external force/torque that can generalize across diverse unseen scenes. Experiments on a Unitree G1
humanoid show that ADAPT achieves accurate disturbance prediction and stronger robustness than a proprioception-only baseline under torso perturbations, standing pushes, and asymmetric hand payloads, with improved velocity tracking even on OOD disturbances. Moreover, ADAPT enables penalizing inferred disturbances at lower-body joints to encourage lighter locomotion.
Read the full abstract ▾
All authors Bofan Lyu, Jindou Jia, Kuangji Zuo, Yanshuo Lu, Shijia Han, Gen Li, Boyu Ma, Jingliang Li, Geng Li, Jianfei Yang
Robotics · Cross-listed from cs.LG
Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learning
Swaminathan S K, Damiya Gondha, Theyanesh Eswaramoorthy Rajahkrishnan et al. (+1 more)
Hamilton-Jacobi-Bellman theory implies that the optimal goal-conditioned action depends on the goal only through the gradient of the goal-reaching distance at the current state, yet standard online GCRL still conditions the actor on the raw goal – a signal that is geometrically uninformative when the goal is far from the data distribution. We propose Direction-Conditioned Policies (DCP), a fully online method that decomposes goal-reaching into two components sharing one InfoNCE representation $\psi$: a subgoal-scoring step that selects a visited state $z_t$ aligned with the final goal $g$ in $\psi_g$, and a direction-conditioned actor that consumes the unit direction $d_t$ and magnitude $r_t$ from $\psi(s_t)$ to $\psi(z_t)$. The two components train jointly, factor cleanly at deployment (subgoal scoring is removed, while direction conditioning remains with $g$ in place of $z_t$), and admit independent modification at the same $(d_t,r_t)$ interface. We prove three results. First, direction sufficiency under HJB: the optimal action under control-affine dynamics depends on the goal only through the value gradient. Second, a quantitative bound showing that, under mild conditions on the learned representation and assuming the scoring rule returns an on-path $z_t$, the actor's conditioning input at training and at deployment coincide up to representation error and geodesic slack. Third, a controllable-subspace characterization of when directional conditioning fails. Across nine environments, DCP improves over Contrastive RL on most final metrics, with the largest gains on manipulation and obstacle-interaction tasks; a qualitative analysis of the learned $\psi$-distance landscape shows the contrastive representation behaves as an online quasimetric encoding environment
topology, and the single failure case (AntSoccer) localizes to a learned-gradient pathology that the theory anticipates.
Read the full abstract ▾
All authors Swaminathan S K, Damiya Gondha, Theyanesh Eswaramoorthy Rajahkrishnan, Aritra Hazra
Robotics
Agile Fall Recovery for Quadrotors with Bidirectional Thrust via Reinforcement Learning
Anke Zhao, Yuhang Zhong, Kenghou Hoi et al. (+5 more)
Autonomous fall recovery is a critical capability for quadrotors operating in real-world environments, where collisions or failures may leave the vehicle resting on the ground in an arbitrary attitude. This problem is challenging because recovery must be achieved under limited onboard sensing, in constrained free space, with ground contact, and in the presence of unknown disturbances. In this letter, we present an RL-based framework for autonomous fall recovery of a quadrotor from arbitrary ground attitudes to stable hover using only lightweight onboard sensors. To address severe partial observability and intermittent sensor invalidity, we train a recurrent policy within an asymmetric actor–critic architecture, leveraging an Incremental Nonlinear Dynamic Inversion (INDI) controller to track the policy output. Combined with high-fidelity simulations of motor response and optical flow, the overall training framework significantly reduces the sim-to-real gap. Simulation ablation studies validate the importance of the main design choices, while real-world experiments demonstrate zero-shot transfer and robust recovery under different initial attitudes, wind disturbances, and additional payloads. These results demonstrate that agile quadrotor fall recovery can be achieved without explicit state estimation using only limited and unreliable onboard sensing.
Read the full abstract ▾
All authors Anke Zhao, Yuhang Zhong, Kenghou Hoi, Junyu Mou, Junjie Wang, Lijie Wang, Jialiang Hou, Fei Gao
Robotics
APEX: Adaptive Policy Execution for Precise Manipulation
Mengfei Zhao, Chenxi Jiang, Tuo An et al. (+2 more)
Modern imitation learning methods, including visuomotor and
Vision-
Language-Action (VLA) policies, typically output high-level action references that are executed by low-level controllers. However, the absence of higher-order reference signals, together with the policy's lack of awareness of the underlying low-level control dynamics during training, inevitably induces an execution gap. As a result, realized actions deviate systematically from policy-commanded ones, with a critical impact on precision-sensitive manipulation. Prior work either modifies the policy architecture or the low-level controller, both requiring intrusive changes to the pretrained policy or packaged controller. This raises a natural question: when the policy and controller are both treated as inaccessible black boxes, can we bridge the execution gap? We propose Adaptive Policy Execution (APEX), a plug-and-play framework inserted between the policy and the controller that reconstructs a dynamically feasible reference from policy outputs and adapts at test-time according to low-level state feedback, with a provable convergence guarantee. Extensive empirical studies show that APEX reduces controller-induced tracking error by 41.2% on demonstration replay and improves manipulation success by 4.8–25.8 percentage points across four visuomotor and VLA policy classes.
Read the full abstract ▾
All authors Mengfei Zhao, Chenxi Jiang, Tuo An, Jindou Jia, Jianfei Yang
Robotics
HATS: A Human-Agent Teleoperation System for Multi-Arm Data Collection
Zesen Lin, Jian-Jian Jiang, Haoming Cen et al. (+3 more)
Many real-world manipulation scenarios, such as handling complex collaborative tasks and dealing with large workspaces, require coordination of more than two robotic arms. Consequently, an effective multi-arm teleoperation system is required to collect demonstrations for training coordinated multi-arm manipulation policies. However, existing teleoperation frameworks mainly focus on single-operator or multi-operator setups, facing a practical trade-off between the cognitive load placed on a single operator and the coordination cost incurred by multiple operators. To address this problem, we introduce HATS, a human-
agent teleoperation system that enables a single human operator, assisted by an MLLM-based
agent, to collect data for multi-arm manipulation tasks. Our system decouples the control space: two primary arms are directly teleoperated by the human, while two assistive arms are controlled by a training-free
agent that handles sub-tasks. In addition, the human operator can use voice commands to prevent collisions and correct assistive arm behaviors during execution. Extensive evaluations demonstrate that HATS achieves data collection efficiency and success rates comparable to expert dual-human teams. Moreover, downstream policy evaluations demonstrate the efficacy and quality of the data collected through HATS.
Read the full abstract ▾
All authors Zesen Lin, Jian-Jian Jiang, Haoming Cen, Xiao-Ming Wu, Dandan Zhang, Wei-Shi Zheng
Robotics
Robots that Collaborate: Sequential Asymmetric Imitation for Learning Coupled Robot Policies
Yincong Chen, Ranpeng Qiu, Zihao Li et al. (+3 more)
Collaborative mobile manipulation requires
robots to coordinate with a partially observed partner while physically interacting through shared objects. This is difficult because failures often arise not from poor local skills, but from mistimed waiting, yielding, pulling, releasing, or repositioning. We study this problem with two bimanual mobile manipulators coupled through rigid and deformable objects. We propose Sequential Asymmetric Imitation (SAI), a single-teleoperator curriculum for learning coupled multi-
robot behaviors without synchronized dual-operator demonstrations or explicit inter-
robot communication. SAI trains
Robot A from unilateral demonstrations with a compliant human partner, trains
Robot B against the deployed
Robot A policy, and then refines
Robot A using sparse interventions near coordination failures. This staged process exposes the policies to increasingly realistic partner behaviors, including delay, phase mismatch,insufficient yielding, and interaction conflict. Across real-world dual-
robot manipulation tasks, SAI improves task success, phase synchronization, and partner-contingent yielding over independent imitation and curriculum-ablation baselines. These results suggest that physically coupled collaboration can be learned through the structure of the imitation curriculum, rather than through synchronized multi-operator demonstrations or explicit coordination mechanisms. More
videos on project page:http://cyc0429.github.io/sai-project-page/
Read the full abstract ▾
All authors Yincong Chen, Ranpeng Qiu, Zihao Li, Yanan Zhou, Guoqiang Ren, Weiming Zhi
Robotics
HOLO-MPPI: Multi-Scenario Motion Planning via Hierarchical Policy Optimization
Youngjae Min, Jovin D'sa, Faizan M. Tariq et al. (+3 more)
Robots deployed in the real world must plan motions across diverse scenarios without per-scenario retuning. End-to-end reinforcement learning (RL) can generalize across scenarios but often becomes brittle under distribution shift, reward misspecification, and
stochastic interactions. Model predictive path integral (MPPI) control enables strong real-time refinement without gradients, but its performance depends on a well-shaped sampling prior, while manually designing the priors does not scale to multi-scenario deployment. We present HOLO-MPPI (High-level Offline, Low-level Online MPPI), a multi-scenario motion planning framework that combines high-level policy learning with low-level
stochastic optimal control. Offline, we learn a high-level policy that proposes scenario-robust plans in an abstract action space, with a learned world model for online rollout. Online, the policy serves as a data-driven prior generator that parameterizes MPPI's sampling distribution conditioned on the current observation and goal. MPPI then optimizes low-level control sequences around this prior in real time to adapt to local disturbances. We instantiate HOLO-MPPI in autonomous driving by designing an effective high-level action space and tailored model architectures. Our evaluation across diverse driving scenarios shows that HOLO-MPPI improves upon MPPI and end-to-end RL baselines while maintaining real-time control.
Read the full abstract ▾
All authors Youngjae Min, Jovin D'sa, Faizan M. Tariq, David Isele, Navid Azizan, Sangjae Bae
Robotics · Cross-listed from cs.CV
MVOFormer: Flow-Semantic Transformer for Robust Monocular Visual Odometry
Jituo Li, Shunwang Sun, Jialu Zhang et al. (+5 more)
Monocular visual odometry (MVO) is foundational to autonomous
navigation and robotic localization. However, existing learning-based MVO approaches often struggle with either a lack of interpretable, complementary features or overly complex multi-stage architectures. These limitations inherently restrict their robustness and cross-domain generalization. In this work, we propose MVOFormer, a novel
transformer framework for robust
monocular visual odometry. Our architecture features a Flow-Semantic Dual Branch Encoder that synergizes dense
geometric motion cues with object-centric semantic priors, explicitly distinguishing static structures from dynamic distractors. These representations are then fused by an Iterative
Multimodal Decoder, enabling coarse-to-fine pose refinement while dynamically suppressing
attention on unreliable regions. Extensive evaluations demonstrate that, without any target-domain fine-tuning, MVOFormer achieves superior zero-shot generalization and robustness, significantly outperforming prior learning-based frame-to-frame methods across diverse
benchmarks including TartanAir, KITTI, TUM-RGBD, and ETH3D-SLAM.
Read the full abstract ▾
All authors Jituo Li, Shunwang Sun, Jialu Zhang, Xinqi Liu, Jinyao Hu, Zhicheng Lu, Sajad Saeedi, Guodong Lu
Robotics · Cross-listed from cs.CV
Decoupled Object-Centric Video Understanding for Generating Robotic Manipulation Commands
Thanh Nguyen Canh, Thanh-Tuan Tran, Haolan Zhang et al. (+3 more)
Translating
video demonstrations into executable
robot commands remains challenging because existing methods often fail to identify which objects are functionally involved in the demonstrated action. As a result, they may generate commands that are linguistically plausible but operationally ambiguous. We propose an object-centric
video understanding framework that decouples action recognition from object identification to generate precise, grammar-free manipulation commands. Our approach integrates Temporal Shift Modules (TSM) for efficient spatio-temporal action classification with a novel Object Selection
algorithm that identifies task-relevant objects through trajectory-based role classification, blur detection, and overlap minimization. The selected objects are then processed by
Vision-
Language Models (VLMs) for robust category recognition and zero-shot generalization. Evaluated on a modified Something-Something V2
dataset, our method achieves 86.79% action classification accuracy and BLEU-4 scores of 0.337 on standard objects and 0.261 on novel objects. These results improve over the strongest task-specific baseline by 80.2% and 143.9%, respectively. Larger gains are observed in METEOR and CIDEr, reaching 157.9% and 171.7% on novel objects. Across all semantic metrics, our approach consistently outperforms task-specific methods and remains competitive with, or surpasses, large general-purpose VLMs while retaining a modular, object-centric design.
Read the full abstract ▾
All authors Thanh Nguyen Canh, Thanh-Tuan Tran, Haolan Zhang, Ziyan Gao, Xiem HoangVan, Nak Young Chong
Robotics · Cross-listed from cs.CR
A Formal Resilience Framework for Cyber-Physical Embodied Systems under Device-Level Cyberattacks
Alberto Giaretta
In cyber-physical systems (CPSs), fault tolerance is traditionally achieved by analysing sensor and actuator outputs, detecting progressive drift or sudden failures, and initiating suitable tolerance mechanisms. Reasonable under general failure models, this approach fails to capture nuanced disruptions caused by cyberattacks, which may employ subtle strategies. This is particularly critical in
embodied CPSs, where computational and physical devices not only have an active role in task completion, but also in embodiment preservation (that is, maintaining the system's physical integrity). To prevent structural physical damage,
embodied CPSs require a framework that enables proactive response to cyberattacks. This paper proposes a formal dependability framework that incorporates IDS information into resilience evaluation predicates, enabling assessment of tolerance to disruption and degradation. The framework supports structured
reasoning about how cyberattacks affect task execution and embodiment preservation, and whether mitigation strategies must be deployed. Analytical examples demonstrate its analytical capability and soundness, establishing a theoretical foundation for dependable and secure
embodied CPSs.
Read the full abstract ▾
Robotics
RHO: Your Coding Agent is Secretly a Roboticist
Karim Elmaaroufi, Justin Svegliato, Sarunas Kalade et al. (+3 more)
Code-as-Policies (CaP) has shown that large
language models (LLMs) can write code to solve robotics tasks by composing perception, planning, and control primitives. Recent CaP systems, however, rely on multi-turn code-generation loops at test time, which is often infeasible for real-time
robot control. We introduce Robotics Harness Optimization (RHO), a novel paradigm in which tool-enabled coding
agents, at training time, propose and search for interpretable, neurosymbolic multi-file policy repositories (Repositories-as-Policies) that compose these primitives rather than a single
prompt, function, or file. RHO searches with reflective feedback from environment reward and execution rather than teleoperation demonstrations. It generalizes to perturbed pick-and-place settings like LIBERO-PRO, where OpenVLA scores 0.0% and $\pi_{0.5}$ averages 12.83%. Using the same low-level primitives, RHO reaches a 45.0% success rate, 2.5x higher than the strongest multi-turn agentic system, and 3.5x higher than $\pi_{0.5}$. On Robosuite, RHO sets a new state-of-the-art of 70.0%, exceeding the prior multi-turn record of 68.29% using single-turn execution with no corrective LLM code edits at deployment. When an LLM is used in the control loop, as on RAI's O3DE
benchmark, RHO optimizes the deployed
agent's multi-file harness of
prompts, tools, and control code, improving held-out success from 23.5% to 44.3% with 20% less wall-clock time and 27% fewer tool calls.
Read the full abstract ▾
All authors Karim Elmaaroufi, Justin Svegliato, Sarunas Kalade, Graham Schelle, Sanjit A. Seshia, Matei Zaharia
Robotics
Training and Evaluating Diffusion Policies with Long Context Lengths
Abhinav Agarwal, Adam Wei, Taylan Kargin et al. (+6 more)
Imitation learning has enabled highly-dexterous robotic manipulation from RGB observations. Policies trained with these methods, however, typically condition
robot actions on only a short history of observations. These policies cannot solve tasks that require
memory and can get stuck repeatedly executing the same failing motions. In this work, we first
benchmark policy performance as context length is incrementally increased from short to long, across a spectrum of tasks with varying local stability and
memory requirements, and in multiple data regimes. To our knowledge, this is the first study to investigate context length for
Diffusion Policies at this level of detail. Our results challenge prior claims: naively scaling context length is not as brittle as advertised in literature. With an appropriate conditioning method and denoising backbone (UNet+Cross-
Attention), single-task policies achieve high success rates on many tasks in the usual data regime even with naive scaling. Next, we propose a training
algorithm to jointly train policies at multiple context lengths, further reducing the sample complexity of long-context learning. Finally, we apply our findings to re-evaluate some previously proposed solutions to long-context imitation learning.
Read the full abstract ▾
All authors Abhinav Agarwal, Adam Wei, Taylan Kargin, Michael Zeng, Cole Becker, Arif Kerem Dayi, Pablo Parrilo, Asuman Ozdaglar, Russ Tedrake
Robotics
V2P-Manip: Learning Dexterous Manipulation from Monocular Human Videos
Kaihan Chen, Yanming Shao, Haifeng Ji et al. (+2 more)
Achieving autonomous robotic dexterous manipulation requires precise, human-like action sequences at scale. As a scalable supplement to costly teleoperation data, extracting trajectories with both visual fidelity and physical plausibility from
monocular videos represents a promising frontier in
embodied AI. To this end, we introduce V2P-Manip, an efficient framework designed to learn dexterous manipulation policies directly from human demonstration
videos. We establish an efficient, integrated pipeline encompassing 3D asset acquisition, trajectory estimation, and dexterous policy learning. To bridge the gap between visual perception and physical constraints, we introduce a two-stage refinement process to enforce spatial
alignment and physical consistency. Evaluations on the TACO and OakInk
benchmarks demonstrate that our approach significantly outperforms previous methods in pose accuracy, adaptability to unstructured environments, and training efficiency. Ultimately, experimental results confirm an average success rate of over 75% across multiple synthetic manipulation tasks and validate the adaptability of the extracted manipulation priors across diverse dexterous hand embodiments.
Read the full abstract ▾
All authors Kaihan Chen, Yanming Shao, Haifeng Ji, Xiaokang Yang, Yao Mu
Robotics
An Augmented Reality Brain-Robot Interface for Generalist Robot Arm Manipulation
Shangkai Zhang, Rousslan Fernand Julien Dossa, Luca Nunziante et al. (+2 more)
The integration of augmented reality (AR) and
EEG-based
brain-computer interfaces (BCIs) offers a promising path for enabling intuitive control of
robots for assistive purposes. However, existing AR
brain-
robot interface (BRI) systems are often constrained to task-specific structures, limiting their utility in real-world environments. We present an AR BRI designed for generalist
robot arm manipulation that combines gaze-based object selection with motor imagery action control. Our system uses eye-tracking for intuitive object targeting and context-aware visual overlays ("Place" and "Use") to guide the user through tasks within a shared autonomy framework. We evaluated the interface through a feasibility study with 18 healthy participants performing three multi-step activities of daily living: drinking, using a drawer, and operating an oven. Our results demonstrate that this interaction paradigm enables effective sequential task execution and high user engagement, achieving a "Good" usability rating (SUS > 70). These findings support the feasibility of the proposed interaction paradigm for complex BCI-driven robotic assistance, and motivate future evaluation with the intended target population. Project website: https://ar-bri-manip.github.io/.
Read the full abstract ▾
All authors Shangkai Zhang, Rousslan Fernand Julien Dossa, Luca Nunziante, Marina Di Vincenzo, Kai Arulkumaran
Robotics
SemGeoNav:A Safety-Guided Visual Navigation Approach with Semantic Reasoning and Geometric Planning
Yu Liu, Zongyang Chen, Yan Guo et al. (+2 more)
Learning-based visual
navigation has enhanced semantic goal-reaching capabilities. However, due to their black-box nature, purely end-to-end models often lack explicit
geometric constraints, leading to unpredictable and unreliable obstacle avoidance in open environments. Conversely, traditional
geometric planners ensure safety but struggle with high-dimensional visual targets. To address these limitations, we propose SemGeoNav, a novel hierarchical visual
navigation framework.It tightly integrates the high-level semantic
reasoning of end-to-end models with the reliable local planning ability of geometry-based methods, achieving robust image-based
navigation while significantly improving obstacle avoidance. Furthermore, we introduce a temporal trajectory smoothing mechanism to ensure continuous and stable
robot motion. We evaluated SemGeoNav on a Unitree Go2 quadruped
robot in real-world environments. The results demonstrate that SemGeoNav outperforms existing representative methods, including ViNT and NoMaD, achieving higher success rates and shorter
navigation times.
Read the full abstract ▾
All authors Yu Liu, Zongyang Chen, Yan Guo, Chao Liu, Xianfei Pan
Robotics
ART-Glove: Articulated Tactile Glove for Contact-Grounded Dexterous Interaction Capture
Changyi Lin and Ding Zhao
We present ART-Glove, an articulated tactile glove designed to capture contact-grounded dexterous demonstrations while preserving human dexterity. ART-Glove makes hand-side contact geometry explicit with 16 rigid functional surfaces covering the fingers, thumb, and palm. Twenty-two anatomically aligned joints connect these surfaces and allow them to follow human hand motion during dexterous manipulation. Encoder-based sensing tracks surface motion, while dense piezoresistive tactile sensing records contact over the same surfaces. The complete system captures synchronized 22-DoF joint measurements and 2048-taxel tactile measurements at 120 Hz. We evaluate ART-Glove across experiments on motion freedom, joint sensing, tactile sensing, and contact-rich interaction capture, demonstrating its ability to preserve human dexterity while recording contact-grounded information that can support downstream dexterous
robot learning.
Read the full abstract ▾
Robotics
Is Your Trajectory Displacement Safe in Long-tail?
Qiao Sun, Weicheng Zheng, Yixin Huang et al. (+1 more)
Long-tail scenarios remain a major bottleneck for autonomous driving evaluation, even as
datasets grow by orders of magnitude. Existing evaluation pipelines are rarely human-aligned, safety-aware, verifiable, and explainable at the same time: closed-loop metrics often saturate among strong planners, while unstructured human ratings can be noisy without a carefully designed protocol. We formulate planning evaluation as additional-threat detection: given a planner trajectory and an expert reference, does the planner's displacement introduce new unsafe driving behavior? We propose FluidTest, an evaluation pipeline with three components: a pairwise WebUI protocol for reliable human annotation; a taxonomy of 32 semantic threats with evidence-grounded decision graphs; and a three-
agent verification system with reflection for precision and auditability. Experiments on the WOD-E2E
dataset show that FluidTest produces consistent labels among trained annotators and identifies additional threats in 65% of Poutine trajectories and 51% of RAP trajectories. These results show that state-of-the-art planners can still exhibit substantial safety-relevant failures despite high Rater Feedback Scores (RFS) and low Average Displacement Error (ADE). Additional details, guidance, and code are available at https://fluidtest.web.app.
Read the full abstract ▾
All authors Qiao Sun, Weicheng Zheng, Yixin Huang, Hang Zhao
Robotics · Cross-listed from cs.LG
FlowMPC: Improving Flow Matching policies with World Models
Chandon Hamel
Flow Matching (FM) is a powerful approach for behavior cloning in
multimodal action spaces [Jiang et al., 2025], but because it is not trained to directly maximize expected return, there is still room to improve how FM policies act at test time. This work investigates whether a learned world model can improve FM policies by enabling Model Predictive Path Integral (MPPI) planning over candidate action sequences proposed by the policy. Building on TD-MPC2 [Hansen et al., 2024], I introduce FlowMPC, a framework that combines an imitation-learned FM policy with a learned world model for test-time planning in ManiSkill manipulation tasks [Tao et al., 2025]. Across PickCube and PickSingleYCB, adding the world model improved performance over the FM policy alone, with especially clear gains in end-of-episode success. These results suggest that world-model-based planning can effectively complement flow-based imitation policies without modifying the FM training objective.
Read the full abstract ▾
Robotics
TopoRetarget: Interaction-Preserving Retargeting for Dexterous Manipulation
Jielin Wu, Shenzhe Yao, Guanqi He et al. (+6 more)
Human hand-object demonstrations provide dense reference motions for training dexterous manipulation reinforcement learning (RL) policies through reference tracking. However, to use such demonstrations for RL policy learning, retargeting must preserve hand pose and task-relevant hand-object contact structure. Otherwise, contact and feasibility artifacts can degrade downstream RL policy performance. We introduce TopoRetarget, an interaction-preserving retargeting framework that uses a single set of parameters across diverse retargeting conditions while maintaining task-relevant hand-object interaction and adapting human demonstrations to dexterous
robot hands. The method constructs a sparse interaction graph over hand and object keypoints and optimizes distance-weighted
Laplacian deformation with directional consistency, kinematic constraints, and penetration handling. Evaluations show that the generated references improve both interaction fidelity and policy learning: TopoRetarget achieves the best contact precision and
alignment over all baselines on the ContactPose
Dataset, improves Pen-
Spin training success by 40.6 percentage points over the existing baseline methods, and enables zero-shot transfer to Wuji Hand hardware on cube reorientation and pen spinning.
Read the full abstract ▾
All authors Jielin Wu, Shenzhe Yao, Guanqi He, Xiaohan Liu, Zhaoqing Zeng, Xiangrui Jiang, Han Yang, Wentao Zhang, Hang Zhao
Robotics
PolyMerge: Compressing 3D Gaussian Splats with Polytope Coverings for Provably Safe Resource-Constrained Navigation
Jihoon Hong, Chih-Yuan Chiu, Sara Fridovich-Keil et al. (+1 more)
Obstacle avoidance is essential for safe
navigation and motion planning. Recent radiance field reconstruction methods enable object detection and modeling with high fidelity, but remain too
memory- and compute-intensive for on-board perception-based path planning. To address these limitations, we propose PolyMerge to convert a large, photorealistic 3D Gaussian Splatting (3DGS) model of a scene into a lightweight representation of convex polytopes whose union provably over-approximates all obstacles in the original 3DGS model. PolyMerge tunes the polytope count to trade off conservativeness and compute cost, and integrates with control barrier functions (CBFs) to plan collision-free paths. We showcase PolyMerge in simulation and hardware experiments on a Crazyflie drone, which uses PolyMerge to compute and follow safe trajectories in real time under severe onboard compute constraints, outperforming baselines in speed while guaranteeing safety. For our code and
videos, visit https://athlon76.github.io/PolyMerge-website/.
Read the full abstract ▾
All authors Jihoon Hong, Chih-Yuan Chiu, Sara Fridovich-Keil, Glen Chou
Robotics
ATHENA: Accelerated Multi-Task Heterogeneous Influence Functions for Robot Data Curation
Tao Xu, Jiaxin Wang, Runhao Zhang et al. (+7 more)
In
robot imitation learning, influence functions provide a principled approach to quantify each demonstration's effect on
robot task outcomes, yet scaling them to billion-parameter
Vision-
Language-Action (VLA) models is limited by computational and multitask bottlenecks. To this end, we propose ATHENA, an influence function framework tailored for multitask VLA data curation at a billion-parameter scale. Concretely, it leverages the Kronecker structure of linear-layer gradients to reduce projection cost, and approximates dense Hessian inversion with a rank-r Random Truncated Approximation, achieving about a 313.4x speedup in influence computation. Furthermore, ATHENA formulates global and local interactive influence to balance data curation across 50 jointly trained tasks. Extensive evaluations on RoboTwin 2.0 and real-
robot deployment, covering 9.34 and 6.90 hours of demonstrations, respectively, show that ATHENA matches or exceeds full-data joint fine-tuning using only 50% of demonstrations in simulation and 66.7% of data across six real-
robot tasks. Overall, ATHENA demonstrates its effectiveness for data curation in billion-parameter multitask VLA fine-tuning.
Read the full abstract ▾
All authors Tao Xu, Jiaxin Wang, Runhao Zhang, Jiayi Guan, Xianchao Zeng, Weixi Song, Xinyu Zhou, Zhetao Chen, Guang Chen, Yong-Lu Li
Robotics · Cross-listed from cs.CV
EgoPhys: Learning Generalizable Physics Models of Deformable Objects from Egocentric Video
Hyunjin Kim, Ri-Zhao Qiu, Guangqi Jiang et al. (+1 more)
Humans naturally understand object physics through everyday interactions, but faithfully predicting complex deformable dynamics, such as elastic materials and fabrics, remains a major challenge for computer
vision and robotics. We present EgoPhys, a framework that constructs deformable physical digital twins from
egocentric RGB-only
video using generalizable priors. EgoPhys overcomes the limitations of existing methods to enable controllable deformable digital twin generation from
egocentric videos by distilling per-object inverse-physics solutions into a compact codebook, enabling prediction of dense spring stiffness fields for unseen objects without per-spring test-time optimization. Trained with generalizable priors from diverse
egocentric interactions, EgoPhys outperforms baselines in reconstruction, future prediction, and zero-shot generalization. To support training and evaluation, we curate an
egocentric interaction
dataset covering diverse deformable objects, scenes, and manipulation styles. We deploy EgoPhys on a real xArm6
robot, demonstrating that a digital twin initialized from a single
egocentric human play
video can serve as an internal world representation to aid in deformable-object planning, highlighting
egocentric RGB observations as a scalable path toward real-to-sim pipelines.
Read the full abstract ▾
All authors Hyunjin Kim, Ri-Zhao Qiu, Guangqi Jiang, Xiaolong Wang
Robotics
Scaling Short-Term Memory of Visuomotor Policies for Long-Horizon Tasks
Rutav Shah, Rajat Kumar Jenamani, Xiaohan Zhang et al. (+5 more)
Many robotic tasks require short-term
memory, whether it's retrieving an object that's no longer visible or turning off an appliance after a set period. Yet, most visuomotor policies trained via imitation learning rely only on immediate sensory input without using past experiences to guide decisions. We present PRISM, a
transformer-based architecture for visuomotor policies to effectively use short-term
memory via two key components: (i) gated
attention, which filters retrieved information to suppress irrelevant details, improving performance by reducing the spurious correlations between the history and current action prediction, (ii) a hierarchical architecture that first compresses local information into compact
tokens and then integrates them to capture temporally extended dependencies, improving its compute and
memory footprint. Together, these mechanisms enable us to scale short-term
memory in visuomotor policies for up to two minutes. To systematically evaluate
memory in visuomotor control, we introduce ReMemBench – a
benchmark of eight diverse household manipulation tasks spanning four categories of short-term
memory – designed to foster general
memory mechanisms rather than siloed, task-specific solutions. PRISM consistently outperforms prior works, including recurrent architectures,
transformers, and their variants – achieving an absolute improvement of 5%–12% over the strongest baseline. On the RoboCasa and LIBERO
benchmarks, it achieves absolute improvements of 11%–15% over its no-
memory variant and fine-tuned
Vision-
Language-Action baselines such as GR00T-N1-3B and OpenVLA, despite not leveraging any large-scale pretraining. Together, PRISM and ReMemBench establish a foundation for developing and evaluating short-term
memory-augmented visuomotor policies that scale to long-horizon tasks. Additional materials are available at https://shahrutav.github.io/short-term-
memory
Read the full abstract ▾
All authors Rutav Shah, Rajat Kumar Jenamani, Xiaohan Zhang, Lingfeng Sun, Roberto Martín-Martín, Yuke Zhu, Deva Ramanan, Karl Schmeckpeper