Robotics
HM3D-M3: A Benchmark for Multi-Agent, Multi-Modal, Multi-Object Goal Navigation
Hongyu Ding, Kanyu Xu, Mingjia Chen et al. (+5 more)
Rapid progress in AI
agents and robotics makes it a matter of the near future that humans are served by a team of
robots. Such a team will receive several goals at once, each specified in whatever form is convenient for the human, such as a category name, a photo, or a spoken description. Existing
benchmarks study multi-
agent, multi-modal, and multi-object goal navigation in isolation, and a
benchmark that places all three in one episode is missing. To fill this gap, we introduce HM3D-M3. A team of K
agents starts at distributed locations in a photorealistic indoor scene and searches for an unordered set of M object goals, each given as an open-vocabulary category, an instance image, or an instance description. HM3D-M3 provides 1,000 validation episodes over 3,926 goals from 379 open-vocabulary categories, 100K observation-assignment training samples, and two team-level metrics, goal-level SR and makespan-normalized SPL. We further contribute two baselines. HiGA is a trained hierarchical goal assigner on top of a frozen open-vocabulary navigation policy, and LaViRA-M3 is a training-free team of VLM
agents with shared mapping and progress analysis. Adapted multi-
robot and zero-shot methods reach 16–27% SR, while our baselines reach 38–41% with complementary strengths and distinct failure modes. Furthermore, we deploy LaViRA-M3 on a heterogeneous three-
robot team. Project page: https://robo-lavira.github.io/hm3d-m3/
Read the full abstract ▾
All authors Hongyu Ding, Kanyu Xu, Mingjia Chen, Zixuan Chen, Jieqi Shi, Jing Huo, Yifan Zhang, Yang Gao
Robotics
FoveaVLA: Foveation Actions Replace the Wrist Camera in VLAs
Bingao Chen, Lihan Zha, C. Karen Liu et al. (+1 more)
We present FoveaVLA, a
Vision-
Language-Action (VLA) model that replaces the wrist camera with learned foveation actions. Wrist cameras provide close-up detail but restrict the view to the end-effector's surroundings. External cameras see the whole scene, but the policy receives them at low resolution, so the fine detail needed for manipulation is lost. FoveaVLA recovers this detail by predicting, alongside each action chunk, the interaction region to observe at the next policy query. The policy then receives crops of these regions, taken from the original-resolution images, together with the full views. We build FoveaVLA on $\pi_{0.5}$ in two configurations, Single and Dual, using one or two external stereo cameras with full and foveal RGB and depth inputs, and post-train both on DROID with RoboInter interaction annotations. Across four real-world task suites, FoveaVLA-Single achieves 70.8% success with one external RGB-D view, and FoveaVLA-Dual achieves 77.1%, versus 52.1% for the wrist-equipped $\pi_{0.5}$. Both match the baseline on plain pick-and-place and improve success with distractors, cloth manipulation, and far-away objects.
Read the full abstract ▾
All authors Bingao Chen, Lihan Zha, C. Karen Liu, Guy Tevet
Robotics
MorphAct: Robot Policies Morphing with the Worlds They Act In
Ziyan Liu, Jiting Liu, Weihong Zhong et al. (+7 more)
Progress toward general intelligence in the physical world requires scaling
robot learning across diverse environments and interactions. However, existing
robot policies use shared parameters that remain fixed across interaction contexts. When different operations favor competing parameter adjustments, fitting their mixture can leave individual operations insufficiently adapted. Our atomic-task diagnostics reveal competing update directions in both
vision-
language and action parameters, with source-improving updates increasing loss on other tasks. This motivates MorphAct, a contextual parameterization framework that dynamically adjusts
robot policy parameters as interaction contexts change. Two coupled generators produce context-dependent low-rank updates for the
vision-
language and action modules. Their coupling follows the policy's computational dependency:
vision-
language updates shape the representations that condition action-side parameter generation. Both generators are trained jointly through the native action objective, without operation labels or expert-adapter targets. MorphAct improves performance across three VLA backbones on LIBERO and Meta-World, with gains of up to 16.30%. In real-world evaluation, it achieves gains of up to 14% over $\pi_{0.5}$. Ablations support the benefits of contextual parameter generation and representation coupling.
Read the full abstract ▾
All authors Ziyan Liu, Jiting Liu, Weihong Zhong, Wenda Yu, Yeqiu Chen, Jason Li, Xiaojiang Peng, Shuyuan Hu, Kai Wang, Bo Zhao
Robotics · Cross-listed from cs.CV
Why 3D Reconstruction Breaks Down on Under-Ice AUV Video, and How to Predict It
Ana-Ştefania Kogălniceanu, Klára Orbán, Ashwin Nedungadi et al. (+2 more)
Reconstruction of 3D structures using under-ice AUV
video footage has been inaccurate, although it is not certain whether this is due to shortcomings in the methods or an inherent deficiency of the capture system. We test four different models for reconstruction on the same Antarctic under-ice AUV
video (Icefin, Kamb Ice Stream): classical feature matching SfM, differentiable photometric splatting, volumetric depth fusion, and a pose-free feed-forward foundation model. All four fail, each in a distinct way: near-total registration yielding almost no structure, geometrically broken output despite healthy photometric metrics, a persistent coverage artifact, and unstable cross-chunk scale. We show that all four are consistent with the same capture regime, a small camera-baseline-to-scene-depth ratio. Following our analysis, we propose a detection system that flags this type of failure in minutes, rather than hours of wasted compute.
Read the full abstract ▾
All authors Ana-Ştefania Kogălniceanu, Klára Orbán, Ashwin Nedungadi, Stefan Lüdtke, Kuderna-Iulian Benţa
Robotics
Complementary Future Token Pruning for World Action Models
Baekseung Kim, Junhoo Lee, Sohyeon Hong et al. (+5 more)
World action models (WAMs) jointly predict future visual states and
robot actions, but processing dense future-frame
tokens incurs substantial inference latency. We propose CF-Prune, a training-free framework for
pruning future-frame
tokens in pretrained WAMs. Our key insight is that future visual predictions support action generation and therefore need not preserve every visual detail required for high-fidelity
video synthesis. CF-Prune combines instruction and action
attention to identify task-relevant
tokens during the initial denoising forward pass, even when future-frame and action representations remain highly noisy. An
entropy-based phase identification mechanism further retains dense computation during precision phases to mitigate
pruning-induced performance degradation. Experiments on RoboLab, RoboCasa, and real-world manipulation tasks demonstrate consistent inference speedups. At a 35%
pruning ratio, CF-Prune achieves average speedups of 1.19-1.27x while improving task success rates. At a 50%
pruning ratio, it achieves 1.29-1.33x average speedups on both simulation
benchmarks, with overall success rates within 1.0 percentage point of the baseline.
Read the full abstract ▾
All authors Baekseung Kim, Junhoo Lee, Sohyeon Hong, Suhyun Jeon, Minkyu Kim, Seungyeon Kim, Donghoon Han, Nojun Kwak
Robotics
MUCO: A Mani-Unit-Structured Manipulation Framework for Compositional Long-Horizon Robot Tasks
Xiangyi Wei, Hanzhe Yao, Jingwen Shang et al. (+3 more)
Learning and executing long-horizon
robot manipulation as complete task trajectories raises data-collection costs, lim?its capability reuse, and causes local failures to cascade into later task stages. We introduce manipulation units (MUs) as reusable operations, each defined by a local goal, an initi?ation condition, and observable success and failure criteria. MUs are learned and verified individually. During execution, consecutive MUs remain coupled through shared environ?ment dynamics. We formulate MU-Structured Manipulation (MUSM) and realize it through MU-Centric Orchestration (MUCO). Given
language and
vision, MUCO constructs an MU sequence, matches each MU to compatible policy models in a frozen capability registry, and dispatches the model with the highest calibrated success rate. Visual verification checks MU outcomes against success and failure criteria and governs transitions, enabling retry or fallback to another model from the current state without re-executing verified MUs. Canonical keys enable checkpoint reuse for equivalent MUs across tasks without new demonstrations or updates. We evaluate MUCO on RLBench, RoboTwin 2.0, and five real-world tasks. MUCO achieves higher complete-task success and verified progress, stronger failure containment, and lower error propagation than baselines. Results demonstrate cross-task MU reuse and het?erogeneous policy composition without joint training, shared backbones, or common action representations.
Read the full abstract ▾
All authors Xiangyi Wei, Hanzhe Yao, Jingwen Shang, Zhenhong Wu, Yang Li, Changbo Wang
Robotics
Active Perception of Upper-Limb Functional Mobility through Human-Robot Handovers
Francesco Semeraro, Marta Lagomarsino, Alberto Ranavolo et al. (+1 more)
The reintegration of people in the workplace after an upper-limb injury is often a long process. Collaborative
robots and humanoids operating in shared workspaces must adapt their behavior to the physical capabilities of their human partners. Achieving this requires
robots to autonomously estimate task-relevant human mobility during interaction, without relying on external clinical assessments. In this work, we formulate the estimation of upper-limb functional mobility as an active perception problem. We model task-available upper-limb functional ranges of motion through impairment coefficients that parameterize reductions of healthy joint limits, and integrate this representation into a human model used by the
robot during interaction. Based on this formulation, we propose an online active perception pipeline in which the
robot actively probes the user through repeated handover requests while progressively updating a
Bayesian belief over the impairment coefficients. This enables the
robot to continuously refine its internal model of the user's joint-level capabilities through interaction rather than passive observation. The proposed framework was evaluated through simulation studies and a proof-of-concept physical validation with a robotic platform and enacted human impairments, demonstrating that the pipeline can accurately recover the impairment coefficients during real human-
robot handovers.
Read the full abstract ▾
All authors Francesco Semeraro, Marta Lagomarsino, Alberto Ranavolo, Arash Ajoudani
Robotics
FOMI: A Force-Feedback Manipulation Interface for Teaching Contact-Rich Manipulation
Yishu Shen, Yiming Luo, Ruilin Wang et al. (+7 more)
Robot-free collection with handheld interfaces enables scalable acquisition of human demonstrations for policy learning. However, existing handheld interfaces often provide limited intuitive force feedback, making contact-rich demonstration collection challenging for human operators. To address this, we present FOMI, a force-feedback manipulation interface with a scissor-shaped mechanism, which transmits object interaction forces directly to the operator's hands through actuation links without force sensors. This intuitive force feedback supports real-time action adjustment during
robot-free collection of contact-rich demonstrations. Around FOMI, we build a demonstration collection system that integrates sensor calibration, data acquisition, and synchronization for
robot-free collection. By further mapping pose changes and gripper opening of FOMI to
robot end-effector poses and gripper commands, the system also supports teleoperation. Across four contact-rich tasks, FOMI improves collection efficiency by approximately 29% and 18% over the best baselines in
robot-free collection and teleoperation, respectively. Demonstrations collected with FOMI in both modes enable policies to perform challenging contact-rich manipulation tasks with high success rates.
Read the full abstract ▾
All authors Yishu Shen, Yiming Luo, Ruilin Wang, Bowen Yang, Shuai Yang, Fangyi Xu, Yujun Shen, Fu Zhang, Boyu Zhou, Yinghao Xu
Robotics
What Matters for World Modeling in Vision-Language-Action Models?
Qisen Ma, Yixiang Chen, Yuan Xu et al. (+10 more)
Integrating world models into
Vision-
Language-Action models (VLAs) has emerged as a promising direction, enabling policies to anticipate future dynamics beyond reactive observation-to-action mapping. However, existing approaches vary simultaneously in their world representations, integration paradigms, and policy backbones, making it difficult to determine what truly matters for effective world model integration. To disentangle these factors, we categorize existing efforts into four representative paradigms, namely Implicit Future
Alignment, Explicit Future Query, MoT Future Expert, and External World Predictor. We instantiate each within a unified framework and systematically evaluate these paradigms with diverse world representations and policy backbones across in-domain, out-of-domain, long-horizon, dynamic, few-shot, and real-world settings. Our study yields four main findings: 1) world representation effectiveness is paradigm-dependent, whereas 2D semantic representations offer robust, cross-paradigm gains; 2) no single paradigm is universally optimal, yet the decoupled MoT Future Expert offers competitive overall performance, particularly in OOD robustness; 3) world modeling consistently benefits policies built on standard VLMs, while decoupled designs exhibit greater compatibility with pretrained VLAs; and 4) world modeling improves performance over action-only baselines across multiple settings, with notable gains on long-horizon and dynamic tasks. Project page and
video demonstrations are available at https://dylanmqs.github.io/WM-in-VLAs/.
Read the full abstract ▾
All authors Qisen Ma, Yixiang Chen, Yuan Xu, Peiyan Li, Jiabing Yang, Kai Wang, Jianhua Yang, Jianlou Si, Jun Huang, Jing Liu, Nianfeng Liu, Yan Huang, Liang Wang
Robotics
Learning from Demonstrations meets Symbolic Control - A Supervisory Control Architecture for Formal Certification of Learned Motion
Ashok Kumar, Mahathi Anand, Angela Schoellig et al. (+1 more)
Dynamical system (DS)-based learning from demonstration generates flexible
robot motions, but existing guarantees rely on certificate generation to obtain formal guarantees of safety and goal convergence. We propose a symbolic supervisory control architecture that automatically generates a fail-safe backup plan, certifying safety, reachability, and actuator feasibility directly from the
robot's dynamics, independent of the learned DS. A nominal controller tracks the learned DS, while the supervisory controller monitors safety and target convergence, providing non-conservative intervention when the nominal controller violates necessary constraints. Experimental results include simulations and real
robot deployment with various 2D and 3D
datasets, including handwriting
datasets, validating the effectiveness of our approach in providing formal guarantees with learned motion even in complex, dynamic environments.
Read the full abstract ▾
All authors Ashok Kumar, Mahathi Anand, Angela Schoellig, Abdalla Swikir
Robotics
Design and Characterization of a Unified Continuum Robot Actuation Platform
Yunti Xu, Aedan Mangan, Miheer Potdar et al. (+2 more)
Continuum
robots enable dexterous access to confined spaces in applications such as minimally invasive surgery and industrial inspection. However, their actuation systems are often custom-designed for specific
robot architectures and tasks, limiting reuse across publications, institutions, and applications. This paper presents a unified actuation platform that decouples actuation hardware and software from continuum
robot design. The platform provides modular actuation across continuum
robot classes, scales to a large number of degrees of freedom (DOFs), and integrates with conventional serial manipulators for hybrid architectures. We validate the platform on an open-sourced surgical robotic system and experimentally characterize the actuation modules, achieving sub-millimeter translational accuracy and less than $5^\circ$ rotational error over ranges of $30$~mm and $180^\circ$, respectively. We further demonstrate the platform in two surgical use cases: teleoperation of a continuum camera in a constrained environment and bimanual manipulation using $2.53$~mm concentric-tube-
robot-based forceps alongside a conventional wristed instrument. All hardware designs and ROS-based software are open-sourced to facilitate reuse and further research in continuum and surgical robotics.
Read the full abstract ▾
All authors Yunti Xu, Aedan Mangan, Miheer Potdar, Alex Lu, Tania K. Morimoto
Robotics
Refine Actions If Needed: Adaptive Action Chunk Refinement for Robot Manipulation
Hanlin Liu, Zichen Zhou, Jiacheng Zhang et al. (+6 more)
Computationally lightweight manipulation policies predict executable actions with low inference latency, but prediction errors can limit task success. Iterative generative policies can achieve higher task success rates, but repeated network evaluations increase inference latency. Refining action proposals from computationally lightweight policies can combine these strengths, but the amount of refinement needed varies across states. We introduce RAIN (Refine Actions If Needed), a framework that predicts action errors at different refinement levels to decide whether and how much to refine each executable proposal, before performing any refinement. We train an ensemble to predict these errors by comparing action proposals and their refined versions with expert actions from offline demonstrations, without requiring explicit labels of refinement needs. Calibration balances uncertainty, computation, and temporal consistency. We evaluate RAIN on eight ManiSkill tasks and four real-
robot tasks. In simulation, our policy achieves 66.6% success, approaching the best evaluated
diffusion-policy success rate with 5.71 times lower policy latency. On the real
robot, success increases from 55.0% with Action Chunking with
Transformers (ACT) to 72.5%, with comparable average successful completion time. Matched controls show that dynamic routing maintains the strongest observed fixed-refinement success level with 26.6% fewer refiner evaluations. Project page: https://rain-policy.github.io/.
Read the full abstract ▾
All authors Hanlin Liu, Zichen Zhou, Jiacheng Zhang, Liuyuan Na, Jin Jin, Guangyan Chen, Yufeng Yue, Gen Li, Fangqiang Ding
Robotics
Mixed-Reality Feedback Control: Towards Blending of Control and Digital Twinning
Melisa Soydan and Mario Gleirscher
Technologies, such as co-simulation, in particular, digital twins (DTs) are used for comprehensive and reliable prototyping, verification and validation (V&V) of control systems. There, the combination of feedback loops across multiple realities (e.g., simulations, physical setups) enables various forms of experimentation with
robots and autonomous systems. However, the handling of multiple realities requires a more specific support by the modelling techniques used in control engineering. To streamline the latter in the system life cycle, we propose a framework for mixing (i.e., fusing, defusing) control input and feedback signals (i.e., stimuli, responses) in simultaneously operated plants for multiple realities.
Read the full abstract ▾
Robotics
Coordi-pi: Coordination-Aware Dual-Hand Action Representation for Bimanual Dexterous Manipulation
Yiming Meng, Yanming Shao, Weijie Shen et al. (+4 more)
Vision-
language-action (VLA) models have shown strong potential for general-purpose robotic manipulation, but extending them to bimanual dexterous manipulation introduces a high-dimensional and structured action space. Existing bimanual VLA policies commonly concatenate right- and left-side arm–hand actions at each control timestep, leaving cross-hand interactions implicit within a single action vector. We introduce Coordi-$\pi$, a structured action representation for bimanual dexterous manipulation. Coordi-$\pi$ represents the right- and left-side arm–hand actions as individual action
tokens and organizes them into an interleaved sequence, explicitly exposing the two sides within the action sequence. To explicitly encode hand identity and temporal synchronization in this interleaved representation, a learnable hand-identity
embedding distinguishes the two sides, while sequence–control-timestep factorized rotary positional encoding (RoPE) separates sequence position from control timestep, allowing actions at the same control timestep to share a temporal coordinate while retaining distinct sequence positions. Across all five bimanual dexterous manipulation tasks in DexJoCo, Coordi-$\pi$ achieves state-of-the-art performance under both rand-obj and rand-full, improving the average success rate over $\pi_{0.5}$ from 31.9% to 51.3% and from 18.3% to 32.3%, respectively. Coordi-$\pi$ also consistently outperforms $\pi_{0.5}$ across four real-world bimanual dexterous manipulation tasks.
Read the full abstract ▾
All authors Yiming Meng, Yanming Shao, Weijie Shen, Kunyang Lin, Jiayu Sun, Ping Luo, Yao Mu
Robotics
One-shot Human Video-Guided Adaptation for Vision-Language-Action Models
Qi Chen, Yuwei Wu, Quan Chen et al. (+5 more)
Vision-
language-action (VLA) models have recently achieved remarkable progress by leveraging large-scale
robot datasets and foundation models for general-purpose
robot manipulation. However, their ability to rapidly adapt to novel manipulation tasks beyond the training distribution remains limited. Existing approaches often require task-specific
robot demonstrations and additional training, which are labor- and compute-intensive, limiting the scalability of VLA adaptation to open-world scenarios. To reduce this dependence on target-task
robot data, we propose CapPro, a framework that enables a VLA policy to adapt to previously unseen manipulation tasks using only a single human demonstration
video. CapPro consists of two components. First, a phase-aware human
video representation module constructs a compact
video representation through learned visual
token selection and learns task-relevant query representations under phase-caption supervision. Second, a
video-conditioned adaptation module jointly incorporates the compact
video and query-derived representations into the action-conditioning context of the pretrained VLA policy. % This design introduces task-specific guidance while largely preserving the original VLA action-generation backbone. In this way, the human demonstration provides task-specific execution guidance rather than an embodiment-aligned trajectory to imitate, without requiring target-task
robot demonstrations or target-task fine-tuning. Experiments across source and unseen target tasks show that CapPro improves average performance across unseen target groups while maintaining comparable performance on the source tasks. These results demonstrate the effectiveness of human-
video guidance for cross-task VLA adaptation.
Read the full abstract ▾
All authors Qi Chen, Yuwei Wu, Quan Chen, Kunpeng Gao, Jin Chen, Che Sun, Zhi Gao, Yunde Jia
Robotics · Cross-listed from cs.CV
HyperLoop: Hyperspectral-Assisted Loop Closure Detection in 3D-SLAM for Planetary Exploration
Annie Bhalla, Dennis Rotondi, Riccardo Giubilato et al. (+2 more)
Autonomous navigation of planetary rovers is challenged by visually repetitive, featureless terrain. SLAM systems based on 3D Gaussian Splatting (3DGS) offer dense mapping but rely on RGB appearance descriptors that degrade under perceptual aliasing - a common failure mode in extraterrestrial environments where physically distinct locations appear visually identical. This paper presents HyperLoop, the first framework integrating hyperspectral imaging (HSI) into the loop closure module of a 3DGS-based SLAM system. Hyperspectral descriptors encode material-level spectral signatures complementing RGB features, providing robustness to illumination changes and visual repetitiveness. Two HSI descriptor families are introduced: Bag-of-Spectral-Words (BoSW), encoding spectral material distributions via TF-IDF weighted histograms, and PCA-K VLAD (PCAK-VLAD), capturing first-order residual statistics of PCA-compressed spectral signatures. Both are evaluated with SIFT-VLAD and NetVLAD RGB descriptors under Strict-AND and Weighted fusion strategies on a custom
dataset from the Moon-Mars Outdoor Test Site (MMOTS) at DLR, comprising synchronized RGB-D and hyperspectral sequences across four planetary analog traverses (65.5 m total). A key finding is that the sensor's Red-NIR range (600-860 nm) provides limited discriminability for rock terrain: the best HSI fusion (SIFT-VLAD + BoSW, Strict-AND) achieves 5.04 cm ATE RMSE, within 1.8% of SIFT-VLAD alone (4.95 cm). The gain over NetVLAD (8.81 cm) is attributable to SIFT-VLAD, not HSI fusion. This motivates future adoption of SWIR sensors capturing diagnostic mineral absorption features absent from the 600-860 nm range for planetary SLAM missions.
Read the full abstract ▾
All authors Annie Bhalla, Dennis Rotondi, Riccardo Giubilato, Kai O. Arras, Wolfgang Stuerzl
Robotics · Cross-listed from cs.LG
What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization
Julia Barth, Armin Sommer, Kunvar Thaman et al. (+1 more)
Goal misgeneralization is a common failure mode in which an
agent behaves competently out of distribution but does not pursue the intended goal. However, we still have a weak understanding of what happens to the intended goal in misgeneralizing
agents. Across three Procgen environments and five
robot tasks, we find three distinct forms of policy-goal misalignment: (1) functional target substitution, where behavior follows an alternative goal, (2) a policy shortcut, where behavior follows a learned action strategy, and (3) cue competition between intended and proxy goal cues. We distinguish these regimes by tracing where goal information remains available internally, how strongly it influences the policy, and what role it plays in action selection, and find that the goal is lost at different depths. Where the goal reaches the decision, as in cue competition, steering redirects the policy to the goal's location, raising intended-goal reach from 16.0% to 95.3%. Where it reaches the policy but not the decision, only new downstream wiring recovers goal reach, from 31.6% to 84.3% in the environment with the action shortcut. Where it is inert, neither works. Our results show that goal misgeneralization is not a single failure of goal representation, but distinct failures of policy
alignment at different depths, tightly connected to which recovery works.
Read the full abstract ▾
All authors Julia Barth, Armin Sommer, Kunvar Thaman, Jannik Schilling
Robotics
Uncertainty-Driven Self-Play for Robust Multi-Agent Driving Policies
Sajad Marvi, Julian Schmidt, Alessandro Canevaro et al. (+2 more)
Learning robust driving policies that generalize beyond their training domain remains a central challenge in motion planning. We propose an uncertainty-driven self-play scheme, in which half of the
agents are rewarded for increasing their neighbors' epistemic uncertainty, while the other half learn to drive safely as the exploring
agents continuously seek out their weaknesses. Epistemic uncertainty indicates where the policy is least confident and serves as an automatic proxy for scenario difficulty. Exploring
agents use this signal to seek out hard cases, generating a challenging curriculum without manual scenario design. The resulting policy outperforms standard self-play, achieving best-in-class zero-shot transfer to nuplan Val14 (99.12% vs. 97.97%) and Argoverse 2 (82.39% vs. 78.03%) without fine-tuning, while also improving performance over standard self-play on Waymo. Our diversity-driven approach achieves 85% higher LSTM hidden-state coverage than the baseline, indicating a much richer set of behaviors. Further analysis shows that epistemic uncertainty serves as a calibrated difficulty detector, with highest-uncertain
agents exhibiting 14.8 times the combined failure rate across collision, off-road, and DNF, of lowest-uncertain
agents, enabling automatic prioritization of challenging scenarios without ground-truth labels.
Read the full abstract ▾
All authors Sajad Marvi, Julian Schmidt, Alessandro Canevaro, Julian Jordan, Abhinav Valada
Robotics
ACRO: Actor-Critic Rollout Orchestration
Junhoo Lee, Seungyeon Kim, Baekseung Kim et al. (+4 more)
Executing a
vision-
language-action (VLA) policy involves deciding when to continue an attempt and when to intervene to improve its chances of success. We introduce ACRO, an Actor–Critic rollout orchestration layer that makes these decisions for a fixed VLA. ACRO seeks interventions that offer a more promising continuation, are physically feasible, and leave sufficient time to complete the task. A Critic learned from policy rollouts estimates success prospects to decide when to intervene and select a re-entry target along the executed trajectory. Trajectory Retraction shortcuts the visited path to return the
robot to that target, where the same VLA resumes execution from a fresh observation. We evaluate ACRO with GR00T N1.7 and $\pi_{0.5}$ across SimplerEnv, RoboCasa, and real-
robot manipulation. Under the same physical execution budget, including all return motion, ACRO improves success by 10.4, 7.3, and 21.7 percentage points, respectively. It also completes additional tasks as the execution budget grows. These results show that controlling when and where a policy resumes can turn its existing capabilities into more successful task executions. Project page: https://junhoo.me/acro
Read the full abstract ▾
All authors Junhoo Lee, Seungyeon Kim, Baekseung Kim, Minkyu Kim, Suhyun Jeon, Jimyeong Kim, Nojun Kwak
Robotics
Accelerating VLA Inference with Perturbation Informed Execution
Jehun Lee, Hajun Lee, Jimin Ryu et al. (+4 more)
Action-chunking
vision-
language-action (VLA) models predict sequences of future
robot actions. Under fixed-horizon execution, the
robot executes a fixedlength prefix of each predicted chunk before invoking the policy again, leaving the remaining predictions unused. For a given number of executed actions, short execution horizons require more policy calls. Extending the execution horizon amortizes policy inference cost over more actions. However, uniformly applying long horizons across episodes can limit opportunities for precise corrections. To address this limitation, we propose PICES (Perturbation-Informed Chunk and Execution Horizon Selection), a training-free method that jointly selects an action chunk and the number of actions executed before the next policy call. Within a single policy call, the action expert generates a batch of action chunks from multiple initial noise samples and their locally perturbed counterparts, using the same conditioning features across all samples. PICES exploits differences between corresponding actions in each nominal chunk and its counterpart to determine candidate-specific execution horizons and select a nominal chunk for execution. Experiments on simulation
benchmarks and real-world manipulation tasks demonstrate that PICES achieves speedup by up to 2.3x in wall-clock policy inference time on Jetson Orin and Thor relative to fixed horizon baselines while improving the success rate.
Read the full abstract ▾
All authors Jehun Lee, Hajun Lee, Jimin Ryu, Youhan Lee, Beomseok Kang, Jonghyun Choi, Jae-Joon Kim
Robotics
Eye, Robot: Learning Expressive Gaze Dynamics via a VLM-Guided Transformer-VAE
Yuhang Hu, Junlin Lyu, Yuan Fang et al. (+6 more)
Gaze coordinates
attention in human interaction through coupled eye, head, and eyelid motion. A humanoid
robot must choose where to look while generating a suitable movement toward that target; fixed tracking rules alone do not model the temporal structure of human gaze. We separate these tasks with an asynchronous
vision-
language model that selects a scene referent, a reactive perception module that maintains fixation, and a Realistic Gaze
Transformer (RGT) that generates target-conditioned motion. RGT is a
Transformer-VAE trained on markerless human recordings retargeted to the
robot's 11-DoF joint space: self-supervised pretraining encodes recent full-head history, and autoregressive fine-tuning exposes the model to its own predictions during rollout. At inference, a four-dimensional binocular-eye trajectory conditions coordinated neck and eye motion; an auxiliary blink predictor triggers eyelid templates during playback. On a two-second
benchmark in which all controllers receive the same ground-truth target, RGT-Base obtains the lowest reported overall seven-joint MSE, and physical demonstrations show the integrated targeting and control pipeline. The evidence establishes
robot-space kinematic fidelity and real-time RGT inference; perceived naturalness and open-vocabulary target-selection accuracy remain questions for controlled evaluation.
Read the full abstract ▾
All authors Yuhang Hu, Junlin Lyu, Yuan Fang, Xiaoying Shen, Lechen Zhang, Jiong Lin, Judah Goldfeder, Steven Tian, Hod Lipson
Robotics
E2DRL: End-to-End Differentiable Reinforcement Learning for 3D Obstacle Avoidance in Quadruped Robots
Cheng Fan, Bingyang Zhou, Gang Yang et al. (+7 more)
Differentiable reinforcement learning shows strong promise for
robot policy learning, but its application to legged locomotion has remained limited to blind walking or simulated 2D tasks. Extending it to end-to-end visual obstacle avoidance is challenging because hard collision constraints and early termination upon contact corrupt first-order gradients, trapping optimizers in local minima. We present E2DRL, which addresses this with a soft differentiable obstacle-interaction formulation: penetration penalties across multiple body keypoints with velocity-conditioned stop-gradient shaping to provide the first-order training signals for 3D obstacle avoidance. To realize a deployable perception-control system, a privileged teacher trained via Soft Analytic Policy Optimization (SAPO) in a differentiable simulator is distilled into a recurrent depth-based student via DAgger, then blended with a base locomotion policy using a terrain-aware gating module for deployment. In simulation, E2DRL achieves higher sample efficiency than PPO and demonstrates superior velocity-tracking performance and 3D obstacle-avoidance capability compared with both PPO and Duality-based DCBF NMPC. We present the first sim-to-real validation of end-to-end differentiable RL for 3D obstacle avoidance on a Unitree Go2 in static and dynamic environments.
Read the full abstract ▾
All authors Cheng Fan, Bingyang Zhou, Gang Yang, Zixuan Zhuang, Aoqian Zhang, Pengzhi Yang, Xin Liu, Chunzheng Wang, Yan Rui Tan, Fan Shi
Robotics
F2A-ObjectNav: An Adaptive Agentic Architecture for Object-Goal Navigation
Seyed Ali Mirghasemi and Alireza Taheri
Object-goal navigation requires an embodied
agent to explore an unfamiliar environment, identify a target object, navigate toward candidate instances, and reliably determine whether the observed object satisfies the mission goal. This work investigates this problem through a two-stage development process. In Stage I, a fixed modular framework is used to examine combinations of exploration, object detection, and target validation methods. The results reveal clear trade-offs among navigation performance, validation reliability, execution time, and model-related cost, with no single fixed configuration remaining preferable across all evaluated conditions. These observations motivate F2A-ObjectNav (Fixed-to-Adaptive Object-Goal Navigation), an adaptive agentic architecture in which a rule-based Supervisor coordinates specialized components for navigation and exploration, object detection, target validation, semantic guidance, and Mission
Memory. The architecture uses event-driven recovery, strategy revision, candidate management, and conditional semantic guidance while retaining lightweight execution during normal operation. F2A-ObjectNav is evaluated over 25 episodes in each of AI2-THOR, RoboTHOR, and ProcTHOR, achieving success rate (SR) / success-weighted path length (SPL) values of 0.84/0.52, 0.40/0.22, and 0.44/0.37, respectively. In addition to task-level metrics, architecture-specific behavioral metrics are used to characterize semantic-guidance usage, strategy switching, recovery activation,
memory-supported behavior, and decision stability. The results show different patterns of adaptive intervention across the three environments, with higher intervention frequencies observed in RoboTHOR and ProcTHOR than in AI2-THOR.
Read the full abstract ▾
Robotics
Instruction-Conditioned Motion Control for Robotic Arms: Language-to-Code Decomposition and Reinforcement Learning
Fan Huang, Dezhong Tong, Changyu Chen et al. (+2 more)
Instruction-Conditioned Motion Control (ICMC) separates parameter specification, instruction generation, and code synthesis for
language-driven
robot control, making execution outcomes traceable to instructions and programs. We compare one-shot generation with ICMC under o1 and o4-mini on cube pick-and-place and letter-reshaping tasks for a UR5e arm in MuJoCo. We then train an open-weight instruction policy by GRPO through a frozen code generator on cube placement onto two- and five-cube letter layouts, averaging the rewards of five programs per instruction. The logged positive-reward rate rose from 31.6% to 35.3% (253/800 to 282/800) between the first and last five training steps. A log audit identifies positive binary labels associated with
geometric-failure messages: at least 19 in that run, 18 in the dense-reward run, and 5 in the run with one program per instruction. In one case, all five rewards were 1.0 while the log reports a 0.9515 m miss against a 0.05 m tolerance; the verdict came from the program's own output stream. We distinguish logged reward rates from independently verified task success and report descriptive results for each configuration's single training trajectory. We introduce an independent evaluation protocol that checks task completion against fixed specifications, separating
robot control from success assessment to support reproducible evaluation.
Read the full abstract ▾
All authors Fan Huang, Dezhong Tong, Changyu Chen, Haewoon Kwak, Jisun An
Robotics
PolyMorph: Learning Locomotion Policy for Cross-Embodiment Adaptation
Hyunsik Oh, Seunghun Jeon, Donghoon Youm et al. (+5 more)
Reinforcement learning has enabled robust locomotion for legged
robots, but learned policies are typically tied to a specific
robot morphology and require additional engineering when the number of legs, actuators, or mechanical properties change. We present PolyMorph, a morphology-conditioned framework that learns a single locomotion policy for heterogeneous legged
robots. PolyMorph converts each
robot's URDF into joint-link modules and combines a recurrent encoder, graph-based
Transformer, and per-joint decoder to generate actions across variable leg counts and joint configurations. To improve training stability, PolyMorph uses morphology-normalized rewards, adaptive
robot sampling, and URDF-derived PD-gain initialization. We evaluate PolyMorph in simulation across bipedal, tripedal, and quadrupedal
robots, including unseen morphology variants and platforms, and deploy the policy on real quadruped platforms operated in multiple configurations. The results show that a single PolyMorph policy generalizes across diverse
robot morphologies without per-
robot policy retraining.
Read the full abstract ▾
All authors Hyunsik Oh, Seunghun Jeon, Donghoon Youm, Jungwoo Hur, Hyeongjun Kim, Jeong Hyun Lee, Moonkyu Jung, Jemin Hwangbo
Robotics
ForceBench: Benchmarking Force Integration for Robotic Manipulation
Xuan Han, Zinuo Jin, Tianlin Li et al. (+8 more)
Force feedback provides
vision-
language-action (VLA) policies with physical observations that can guide action selection and execution. However, differences in tasks, demonstrations, and control settings across existing systems hinder reliable comparisons of force integration designs. We introduce ForceBench, a simulation
benchmark for evaluating six-axis wrist-force integration under a unified protocol. ForceBench combines eight tasks spanning physical-property inference, resistance-dependent interaction, and contact-rich assembly with 1,600 synchronized demonstrations and task-specific success criteria. We compare one no-force baseline and six force integration designs on a shared $\pi_{0.5}$ backbone.
Prompt Fusion achieves the highest observed mean simulation success of 35.31%, compared with 23.25% for the baseline, and improves success rates across all three task categories. Predictive and temporal gains in simulation concentrate on physical-property inference, while both temporal designs underperform the baseline on assembly. Real-world evaluation of eight systems across five tasks supports the main qualitative trends:
Prompt Fusion and Dual-Path Fusion achieve 63.00% mean success, compared with 40.00% for the baseline, but only
Prompt Fusion improves every task. These findings support ForceBench's practical value for identifying the benefits and limitations of force integration across manipulation demands. The project page is available at https://forcebench-platform.github.io/.
Read the full abstract ▾
All authors Xuan Han, Zinuo Jin, Tianlin Li, Yunfan Lou, Xin Liu, Yuchen Xue, Yuanpeng He, Boxian Yao, Shuhan Zhan, Ruihai Wu, Zhibo Pang
Robotics
Cracking Mobile Manipulation: From Failure Diagnosis to Agentic Robot System
Zixuan Wang, Jia-Ning Li, Yinzhou Tang et al. (+5 more)
Recent
vision-
language-action (VLA) models have shown strong performance on local
robot manipulation, yet their success does not readily translate to long-horizon mobile manipulation. In our experiments, a pretrained subtask VLA achieves 80% success rate on atomic manipulation tasks but only 19% when these skills are composed into mobile manipulation. We argue that this gap stems from failures across the execution process rather than from local manipulation capability alone. To uncover these bottlenecks, we systematically diagnose failures in long-horizon mobile manipulation and identify three dominant regimes: task planning, navigation and base positioning, and local manipulation. Based on this diagnosis, we develop failure-guided agentic intervention that augments a fixed subtask VLA with demonstration-based planning guidance, adaptive visual
memory, accessibility-aware base adjustment, and local end-effector adjustment. On RoboCasa365, these interventions progressively improve task success from 19% to 47%, with the finalized system reaching 49% on held-out episode seeds, compared with 18% for the baseline. The same design improves performance from 14% to 37% on BEHAVIOR-1K and from 5/20 to 13/20 on real-
robot tasks. These results suggest that reliable mobile manipulation can be advanced by diagnosing and intervening on the current bottleneck, rather than uniformly scaling the underlying policy. Our code will be released at https://github.com/PKU-VaLuE-Lab/Cracking-Mobile-Manipulation.git.
Read the full abstract ▾
All authors Zixuan Wang, Jia-Ning Li, Yinzhou Tang, Yanzhao Yu, Biao Wu, Meng Fang, Xin Li, Yiwu Zhong
Robotics
RoboSpan: Learning Modular Coordination Policies for High-DoF Whole-Body Mobile Manipulation with Teleoperation
Keqiang Ren, Xuecheng Ye, Hanyu Liu et al. (+6 more)
Whole-body mobile manipulation suffers from two fundamental challenges: inherent motion heterogeneity across locomotion, arm, and hand modules, and progressive error accumulation over long-horizon multi-stage tasks. Existing teleoperation platforms also lack integrated dexterous tactile sensing to support coordinated whole-body decision-making. This paper presents RoboSpan, a high-DoF whole-body teleoperation and demonstration platform, and the Hierarchical Modular Framework (HMF) for learning modular coordination policies. HMF functionally decouples the 29-DoF action space into fine hand manipulation, arm pose adjustment, and mobile-base spatial configuration branches, and introduces the Coordinated Modular Embodied
Transformer (COMET) with inter-modular cross-
attention to enable dynamic information exchange and action coordination.
Vision-
language model driven hierarchical task decomposition structurally suppresses temporal error propagation, while a full-palm 2D tactile representation encodes distributed contact signals for cross-module tactile empowerment. Experiments on a physical 29-DoF
robot demonstrate that the proposed method achieves over 20% higher task success rate than mainstream baselines across a range of long-horizon whole-body tasks, markedly improving stability and robustness in contact-rich manipulation.
Read the full abstract ▾
All authors Keqiang Ren, Xuecheng Ye, Hanyu Liu, Zhihua Wang, Tao Jian, Yunsheng Ma, Jiayi Wen, Xinran Li, Zhigong Song
Robotics
Back on Track: Deployment-time Policy Improvement via World Model Planning
Colton Crosby, Hao Wang, Mac Schwager et al. (+1 more)
State-of-the-art visuomotor policies are increasingly capable of executing complex tasks, yet can fail during deployment in ways they cannot predict or recover from on their own. Improving their performance and robustness typically requires collecting more data, retraining, or modifying the policy architecture, all of which are time- and resource-intensive in practice. We instead propose an offline-trained monitoring-and-intervention framework that improves a policy during deployment using only black-box query access. From a limited set of labeled on-policy rollouts, we learn a policy-conditioned liveness value that measures the nominal policy's progress toward task completion; we additionally train a compact latent world model using these rollouts together with play data. At deployment, the liveness value continuously monitors execution. When it detects a loss of progress, a short-horizon planner searches the world model for corrective actions using the same value as its planning objective, after which control returns to the nominal policy. We evaluate our method on the LIBERO-Goal task suite and on a Franka Panda manipulator, using the
vision-
language-action model $\pi_{0.5}$ and a
diffusion policy as nominal policies, and demonstrate significant improvements in task success rates, while supporting real-time control. Additional results and
videos are available on the project website (https://policy-improvement-wm.pages.dev/).
Read the full abstract ▾
All authors Colton Crosby, Hao Wang, Mac Schwager, Somil Bansal
Robotics
GS-PRM: Gaussian Visual Roadmap for Image-Goal Navigation
Ashraf Siddique, Kyung Min Han and Young J. Kim
Image-goal navigation requires a global planner that connects visually specified endpoints while providing observations that a local policy can follow. Three-dimensional Gaussian Splatting (3DGS) provides map-registered RGB, opacity, and depth renderings, but does not explicitly encode navigable
topology. We introduce \method, a Gaussian visual roadmap whose milestones, local connections, and visual references are constructed directly through 3DGS rendering without deriving an occupancy grid or signed distance field. We define render-based collision-free (RC-free) configurations and paths. Panoramic depth supplies conservative clearance for milestone validation and adaptive coverage, while edge-aligned orthographic rendering tests the
robot's swept volume. We establish conditional soundness and sufficient conditions for deterministic query completeness and probabilistic construction completeness of GS-PRM. To validate the performance of GS-PRM, we conduct experiments in both simulated and real-world environments, demonstrating long-horizon navigation using goal-conditioned policies guided by GS-PRM. Compared with a state-of-the-art 3DGS-based navigation system, our GS-PRM-based navigation system achieves up to a twice higher average success rate and reduces travel time by up to 57.2% when both systems complete the navigation task.
Read the full abstract ▾