Robotics
Strategizing at Speed: A Learned Model Predictive Game for Multi-Agent Drone Racing
Andrei-Carlo Papuc, Lasse Peters, Sihao Sun et al. (+2 more)
Autonomous drone racing pushes the boundaries of high-speed motion planning and multi-
agent strategic decision-making. Success in this domain requires drones not only to navigate at their limits but also to anticipate and counteract competitors' actions. In this paper, we study a fundamental question that arises in this domain: how deeply should an
agent strategize before taking an action? To this end, we compare two planning paradigms: the Model Predictive Game (MPG), which finds interaction-aware strategies at the expense of longer computation times, and contouring Model Predictive Control (MPC), which computes strategies rapidly but does not reason about interactions. We perform extensive experiments to study this trade-off, revealing that MPG outperforms MPC at moderate velocities but loses its advantage at higher speeds due to latency. To address this shortcoming, we propose a Learned Model Predictive Game (LMPG) approach that amortizes model predictive gameplay to reduce latency. In both simulation and hardware experiments, we
benchmark our approach against MPG and MPC in head-to-head races, finding that LMPG outperforms both baselines.
Read the full abstract ▾
All authors Andrei-Carlo Papuc, Lasse Peters, Sihao Sun, Laura Ferranti, Javier Alonso-Mora
Robotics
Consensus-based optimization (CBO): Towards Global Optimality in Robotics
Xudong Sun, Armand Jordana, Massimo Fornasier et al. (+2 more)
Zero-order optimization has recently received significant
attention for designing optimal trajectories and policies for robotic systems. However, most existing methods (e.g., MPPI, CEM, and CMA-ES) are local in nature, as they rely on gradient estimation. In this paper, we introduce consensus-based optimization (CBO) to robotics, which is guaranteed to converge to a global optimum under mild assumptions. We provide theoretical analysis and illustrative examples that give intuition into the fundamental differences between CBO and existing methods. To demonstrate the scalability of CBO for robotics problems, we consider three challenging trajectory optimization scenarios: (1) a long-horizon problem for a simple system, (2) a dynamic balance problem for a highly underactuated system, and (3) a high-dimensional problem with only a terminal cost. Our results show that CBO is able to achieve lower costs with respect to existing methods on all three challenging settings. This opens a new framework to study global trajectory optimization in robotics.
Read the full abstract ▾
All authors Xudong Sun, Armand Jordana, Massimo Fornasier, Jalal Etesami, Majid Khadiv
Robotics
SURE: Safe Uncertainty-Aware Robot-Environment Interaction using Trajectory Optimization
Zhuocheng Zhang, Haizhou Zhao, Xudong Sun et al. (+2 more)
Robotic tasks involving contact interactions pose significant challenges for trajectory optimization due to discontinuous dynamics. Conventional formulations typically assume deterministic contact events, which limit robustness and adaptability in real-world settings. In this work, we propose SURE, a robust trajectory optimization framework that explicitly accounts for contact timing uncertainty. By allowing multiple trajectories to branch from possible pre-impact states and later rejoin a shared trajectory, SURE achieves both robustness and computational efficiency within a unified optimization framework. We evaluate SURE on two representative tasks with unknown impact times. In a cart-pole balancing task involving uncertain wall location, SURE achieves an average improvement of 21.6% in success rate when branch switching is enabled during control. In an egg-catching experiment using a robotic manipulator, SURE improves the success rate by 40%. These results demonstrate that SURE substantially enhances robustness compared to conventional nominal formulations.
Read the full abstract ▾
All authors Zhuocheng Zhang, Haizhou Zhao, Xudong Sun, Aaron M. Johnson, Majid Khadiv
Robotics
Perception-Control Coupled Visual Servoing for Textureless Objects Using Keypoint-Based EKF
Allen Tao, Jun Yang, Stanko Oparnica et al. (+1 more)
Visual servoing is fundamental to robotic applications, enabling precise positioning and control. However, applying it to textureless objects remains a challenge due to the absence of reliable visual features. Moreover, adverse visual conditions, such as occlusions, often corrupt visual feedback, leading to reduced accuracy and instability in visual servoing. In this work, we build upon learning-based keypoint detection for textureless objects and propose a method that enhances robustness by tightly integrating perception and control in a closed loop. Specifically, we employ an Extended Kalman Filter (EKF) that integrates per-frame keypoint measurements to estimate 6D object pose, which drives pose-based visual servoing (PBVS) for control. The resulting camera motion, in turn, enhances the tracking of subsequent keypoints, effectively closing the perception-control loop. Additionally, unlike standard PBVS, we propose a probabilistic control law that computes both camera velocity and its associated uncertainty, enabling uncertainty-aware control for safe and reliable operation. We validate our approach on real-world robotic platforms using quantitative metrics and grasping experiments, demonstrating that our method outperforms traditional visual servoing techniques in both accuracy and practical application.
Read the full abstract ▾
All authors Allen Tao, Jun Yang, Stanko Oparnica, Wenjie Xue
Robotics
DynaRetarget: Dynamically-Feasible Retargeting using Sampling-Based Trajectory Optimization
Victor Dhedin, Ilyass Taouil, Shafeef Omar et al. (+4 more)
In this paper, we introduce DynaRetarget, a complete pipeline for retargeting human motions to
humanoid control policies. The core component of DynaRetarget is a novel Sampling-Based Trajectory Optimization (SBTO) framework that refines imperfect kinematic trajectories into dynamically feasible motions. SBTO incrementally advances the optimization horizon, enabling optimization over the entire trajectory for long-horizon tasks. We validate DynaRetarget by successfully retargeting hundreds of
humanoid-object demonstrations and achieving higher success rates than the state of the art. The framework also generalizes across varying object properties, such as mass, size, and geometry, using the same tracking objective. This ability to robustly retarget diverse demonstrations opens the door to generating large-scale synthetic
datasets of
humanoid loco-manipulation trajectories, addressing a major bottleneck in real-world data collection.
Read the full abstract ▾
All authors Victor Dhedin, Ilyass Taouil, Shafeef Omar, Dian Yu, Kun Tao, Angela Dai, Majid Khadiv
Robotics
A 26-Gram Tailless Butterfly-Inspired Flapping-Wing Robot with Onboard Attitude Control
Weibin Gu, Chenrui Feng, Lian Liu et al. (+7 more)
Butterfly-inspired flapping-wing
robots use broad compliant wings and low-frequency actuation, but pronounced wingbeat-synchronous body dynamics and time-varying inertia complicate onboard control. Here, we introduce AirPulse, a 26 g two-winged tailless flapping-wing
robot with integrated venation-inspired wings, sensing, computation, and power, representing the lowest flight-ready mass among compared butterfly-inspired platforms. We analyze the
robot's dynamic structural coupling and map prescribed modulation parameters of flapping kinematics to experimental six-axis wrench profiles for systematic control channel allocation. To enable stable flapping motion, we formulate Stroke Timing Asymmetry Rhythm (
STAR), a phase-domain modulation strategy that ensures smooth stroke velocity changes while preserving mean flapping frequencies. Coupled with state estimation, the onboard feedback control architecture demonstrates successful untethered pitch and directional tracking during climbing and turning maneuvers. Ultimately, the AirPulse
robot offers an experimentally validated framework for stabilizing strongly oscillatory, low-mass bio-inspired platforms, providing a basis for future operation in sensitive, confined environments.
Read the full abstract ▾
All authors Weibin Gu, Chenrui Feng, Lian Liu, Chen Yang, Xingchi Jiao, Yuhe Ding, Xiaofei Shi, Chao Gao, Alessandro Rizzo, Guyue Zhou
Robotics
SuReNav: Superpixel Graph-based Constraint Relaxation for Navigation in Over-constrained Environments
Keonyoung Koh, Moonkyeong Jung, Samuel Seungsup Lee et al. (+1 more)
We address the over-constrained planning problem in semi-static environments. The planning objective is to find a best-effort solution that avoids all hard constraint regions while minimally traversing the least risky areas. Conventional methods often rely on pre-defined area costs, limiting generalizations. Further, the spatial continuity of
navigation spaces makes it difficult to identify regions that are passable without overestimation. To overcome these challenges, we propose SuReNav, a superpixel graph-based constraint relaxation and
navigation method that imitates human-like safe and efficient
navigation. Our framework consists of three components: 1) superpixel graph map generation with regional constraints, 2) regional-constraint relaxation using graph neural network trained on human demonstrations for safe and efficient
navigation, and 3) interleaving relaxation, planning, and execution for complete
navigation. We evaluate our method against state-of-the-art baselines on 2D semantic maps and 3D maps from OpenStreetMap, achieving the highest human-likeness score of complete
navigation while maintaining a balanced trade-off between efficiency and safety. We finally demonstrate its scalability and generalization performance in real-world urban
navigation with a quadruped
robot, Spot. Code and
Videos are available at https://sure-nav.github.io/.
Read the full abstract ▾
All authors Keonyoung Koh, Moonkyeong Jung, Samuel Seungsup Lee, Daehyung Park
Robotics
Constraint Manifold Exploration for Efficient Continuous Coverage Estimation
Robert Wilbrandt and Rüdiger Dillmann
Many automated manufacturing processes rely on industrial
robot arms to move process-specific tools along workpiece surfaces. In applications like grinding, sanding, spray painting, or inspection, they need to cover a workpiece fully while keeping their tools perpendicular to its surface. While there are approaches to generate trajectories for these applications, there are no sufficient methods for analyzing the feasibility of full surface coverage. This work proposes a sampling-based approach for continuous coverage estimation that explores reachable surface regions in the configuration space. We define an extended ambient configuration space that allows for the representation of tool position and orientation constraints. A continuation-based approach is used to explore it using two different sampling strategies. A thorough evaluation across different kinematics and environments analyzes their runtime and efficiency. This validates our ability to accurately and efficiently calculate surface coverage for complex surfaces in complicated environments.
Read the full abstract ▾
Robotics
Crowd-FM: Learned Optimal Selection of Conditional Flow Matching-generated Trajectories for Crowd Navigation
Antareep Singha, Laksh Nanwani, Mathai Mathew P. et al. (+4 more)
Safe and computationally efficient local planning for mobile
robots in dense, unstructured human crowds remains a fundamental challenge. Moreover, ensuring that
robot trajectories are similar to how a human moves will increase the acceptance of the
robot in human environments. In this paper, we present Crowd-FM, a learning-based approach to address both safety and human-likeness challenges. Our approach has two novel components. First, we train a Conditional Flow-Matching (CFM) policy over a
dataset of optimally controlled trajectories to learn a set of collision-free primitives that a
robot can choose at any given scenario. The chosen optimal control solver can generate multi-modal collision-free trajectories, allowing the CFM policy to learn a diverse set of maneuvers. Secondly, we learn a score function over a
dataset of human demonstration trajectories that provides a human-likeness score for the flow primitives. At inference time, computing the optimal trajectory requires selecting the one with the highest score. Our approach improves the state-of-the-art by showing that our CFM policy alone can produce collision-free
navigation with a higher success rate than existing learning-based baselines. Furthermore, when augmented with inference-time refinement, our approach can outperform even expensive optimisation-based planning approaches. Finally, we validate that our scoring network can select trajectories closer to the expert data than a manually designed cost function.
Read the full abstract ▾
All authors Antareep Singha, Laksh Nanwani, Mathai Mathew P., Samkit Jain, Phani Teja Singamaneni, Arun Kumar Singh, K. Madhava Krishna
Robotics
RAPID: Reconfigurable, Adaptive Platform for Iterative Design
Zi Yin, Fanhong Li, Shurui Zheng et al. (+1 more)
Developing robotic manipulation policies is iterative and hypothesis-driven: researchers test tactile sensing, gripper geometries, and sensor placements through real-world data collection and training. Yet even minor end-effector changes often require mechanical refitting and system re-integration, slowing iteration. We present RAPID, a full-stack reconfigurable platform designed to reduce this friction. RAPID is built around a tool-free, modular hardware architecture that unifies handheld data collection and
robot deployment, and a matching software stack that maintains real-time awareness of the underlying hardware configuration through a driver-level Physical Mask derived from USB events. This modular hardware architecture reduces reconfiguration to seconds and makes systematic multi-modal ablation studies practical, allowing researchers to sweep diverse gripper and sensing configurations without repeated system bring-up. The Physical Mask exposes modality presence as an explicit runtime signal, enabling auto-configuration and graceful degradation under sensor hot-plug events, so policies can continue executing when sensors are physically added or removed. System-centric experiments show that RAPID reduces the setup time for multi-modal configurations by two orders of magnitude compared to traditional workflows and preserves policy execution under runtime sensor hot-unplug events. The hardware designs, drivers, and software stack are open-sourced at https://rapid-kit.github.io/ .
Read the full abstract ▾
All authors Zi Yin, Fanhong Li, Shurui Zheng, Jia Liu
Robotics
Humanoid Manipulation Interface: Humanoid Whole-Body Manipulation from Robot-Free Demonstrations
Ruiqian Nai, Boyuan Zheng, Junming Zhao et al. (+8 more)
Current approaches for
humanoid whole-body manipulation, primarily relying on teleoperation or visual sim-to-real reinforcement learning, are hindered by hardware logistics and complex reward engineering. Consequently, demonstrated autonomous skills remain limited and are typically restricted to controlled environments. In this paper, we present the
Humanoid Manipulation Interface (HuMI), a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments. HuMI enables
robot-free data collection by capturing rich whole-body motion using portable hardware. This data drives a hierarchical learning pipeline that translates human motions into dexterous and feasible
humanoid skills. Extensive experiments across five whole-body tasks–including kneeling, squatting, tossing, walking, and bimanual manipulation–demonstrate that HuMI achieves a 3x increase in data collection efficiency compared to teleoperation and attains a 70% success rate in unseen environments.
Read the full abstract ▾
All authors Ruiqian Nai, Boyuan Zheng, Junming Zhao, Haodong Zhu, Sicong Dai, Zunhao Chen, Yihang Hu, Yingdong Hu, Tong Zhang, Chuan Wen, Yang Gao
Robotics · Cross-listed from eess.SY
Efficient and Robust Modeling of Nonlinear Mechanical Systems
Davide Tebaldi and Roberto Zanasi
The development of efficient and robust dynamic models is fundamental in the field of systems and control engineering. In this paper, a new formulation for the dynamic model of nonlinear mechanical systems, that can be applied to different automotive and robotic case studies, is proposed, together with a modeling procedure allowing to automatically obtain the model formulation. Compared with the Euler-Lagrange formulation, the proposed model is shown to give superior performances in terms of robustness against measurement noise for systems exhibiting dependence on some external variables, as well as in terms of execution time when computing the inverse dynamics of the system.
Read the full abstract ▾
Robotics
Force Generative Imitation Learning: Bridging Position Trajectory and Force Commands through Control Technique
Hiroshi Sato, Sho Sakaino and Toshiaki Tsuji
In contact-rich tasks, while position trajectories are often easy to obtain, appropriate force commands are typically unknown. Although it is conceivable to generate force commands using a pretrained foundation model such as
Vision-
Language-Action (VLA) models, force control is highly dependent on the specific hardware of the
robot, which makes the application of such models challenging. To bridge this gap, we propose a force generative model that estimates force commands from given position trajectories. However, when dealing with unseen position trajectories, the model struggles to generate accurate force commands. To address this, we introduce a feedback control mechanism. Our experiments reveal that feedback control does not converge when the force generative model has
memory. We therefore adopt a model without
memory, enabling stable feedback control. This approach allows the system to generate force commands effectively, even for unseen position trajectories, improving generalization for real-world
robot writing tasks.
Read the full abstract ▾
Robotics
Think Proprioceptively: State-Grounded Visual Token Selection for VLA Policies
Fangyuan Wang, Peng Zhou, Jiaming Qi et al. (+4 more)
Vision-
language-action (VLA) models typically inject proprioception only as a late conditioning signal, preventing
robot state from grounding instruction understanding or directing visual
attention. We introduce ThinkProprio, which discretizes proprioception into VLM-vocabulary
tokens and uses them jointly with the instruction to gate visual patches before VLM computation, steering the model toward action-relevant evidence while discarding redundant
tokens early. We find that proprioception added as a passive conditioning signal leaves performance essentially unchanged; its value emerges when
token-form state acts as an active query that, with the instruction, selects which visual patches the VLM processes. Systematic ablations show that VLM-vocabulary
tokens outperform learned projectors as the state encoding, and that retaining only about \SI{12}{\percent} of the visual
tokens surpasses on CALVIN ABC$\to$D. Across CALVIN, LIBERO, and real-world manipulation, ThinkProprio reduces end-to-end inference latency while improving the matched full-
token baseline.
Read the full abstract ▾
All authors Fangyuan Wang, Peng Zhou, Jiaming Qi, Shipeng Lyu, Chengyang He, David Navarro-Alarcon, Guodong Guo
Robotics
The Law of Task-Achieving Body Motion: Axiomatizing Success of Robot Manipulation Actions
Malte Huerkamp, Jonas Dech and Michael Beetz
Autonomous
agents that perform everyday manipulation actions need to ensure that their body motions are semantically correct with respect to a task request, causally effective within their environment, and feasible for their embodiment. In order to enable
robots to verify these properties, we introduce the Law of Task-Achieving Body Motion as an axiomatic correctness specification for body motions. To that end we introduce scoped Task-Environment-Embodiment (TEE) classes that represent world states as Semantic Digital Twins (SDTs) and define applicable physics models to decompose task achievement into three predicates: SatisfiesRequest for semantic request satisfaction over SDT state
evolution; Causes for causal sufficiency under the scoped physics model; and CanPerform for safety and feasibility verification at the embodiment level. This decomposition yields a reusable, implementation-independent interface that supports motion synthesis and the verification of given body motions. It also supports typed failure diagnosis (semantic, causal, embodiment and out-of-scope), feasibility across
robots and environments, and counterfactual
reasoning about
robot body motions. We demonstrate the usability of the law in practice by instantiating it for articulated container manipulation in kitchen environments on three contrasting mobile manipulation platforms
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
LIBERO-X: Robustness Litmus for Vision-Language-Action Models
Guodong Wang, Chenkai Zhang, Qingjie Liu et al. (+4 more)
Reliable benchmarking is critical for advancing
Vision-
Language-Action (VLA) models, as it reveals their generalization, robustness, and
alignment of perception with
language-driven manipulation tasks. However, existing
benchmarks often provide limited or misleading assessments due to insufficient evaluation protocols that inadequately capture real-world distribution shifts. This work systematically rethinks VLA benchmarking from both evaluation and data perspectives, introducing LIBERO-X, a
benchmark featuring: 1) A hierarchical evaluation protocol with progressive difficulty levels targeting three core capabilities: spatial generalization, object recognition, and task instruction understanding. This design enables fine-grained analysis of performance degradation under increasing environmental and task complexity; 2) A high-diversity training
dataset collected via human teleoperation, where each scene supports multiple fine-grained manipulation objectives to bridge the train-evaluation distribution gap. Experiments with representative VLA models reveal significant performance drops under cumulative perturbations, exposing persistent limitations in scene comprehension and instruction grounding. By integrating hierarchical evaluation with diverse training data, LIBERO-X offers a more reliable foundation for assessing and advancing VLA development.
Read the full abstract ▾
All authors Guodong Wang, Chenkai Zhang, Qingjie Liu, Jinjin Zhang, Jiancheng Cai, Junjie Liu, Xinmin Liu
Robotics
Primary Experimental Feedback on a Co-manipulated Robotic System for Assisted Cervical Surgery
Seifeddine Sellemi, Abdelbadia Chaker, Tanguy Vendeuvre et al. (+2 more)
Robotic-assisted surgery has emerged as a promising approach to improve
surgical ergonomics, precision, and workflow efficiency, particularly in complex procedures such as cervical spine surgery. In this study, we evaluate the performance of a collaborative robotic system designed to assist surgeons in drilling tasks by assessing its accuracy in executing predefined trajectories. A total of 14 drillings were performed by eight experienced cervical surgeons, utilizing a robotic-assisted setup aimed at ensuring stability and
alignment. The primary objective of this study is to quantify the deviations in the position and orientation of the drilling tool relative to the planned trajectory, providing insights into the system's reliability and potential impact on clinical outcomes. While the primary function of robotic assistance in surgery is to enhance surgeon comfort and procedural guidance rather than solely optimizing precision, understanding the system's accuracy remains crucial for its effective integration into
surgical practices part of this primary experimental feedback, the study offers an in-depth analysis of the co-manipulated robotic system's performance, focusing on the experimental setup and error evaluation methods. The findings of this study will contribute to the ongoing development of robotic-assisted cervical surgery, highlighting both its advantages and areas for improvement in achieving safer and more efficient
surgical workflows
Read the full abstract ▾
All authors Seifeddine Sellemi, Abdelbadia Chaker, Tanguy Vendeuvre, Terence Essomba, Med Amine Laribi
Robotics · Cross-listed from cs.CV
DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Driving
Feiyang jia, Lin Liu, Ziying Song et al. (+4 more)
End-to-end (E2E) autonomous driving has recently attracted increasing interest in unifying
Vision-
Language-Action (VLA) with World Models to enhance decision-making and forward-looking imagination. However, existing methods fail to effectively unify future scene
evolution and action planning within a single architecture due to inadequate sharing of latent states, limiting the impact of visual imagination on action decisions. To address this limitation, we propose DriveWorld-VLA, a novel framework that unifies world modeling and planning within a latent space by tightly integrating VLA and world models at the representation level, which enables the VLA planner to benefit directly from holistic scene-
evolution modeling and reducing reliance on dense annotated supervision. Additionally, DriveWorld-VLA incorporates the latent states of the world model as core decision-making states for the VLA planner, facilitating the planner to assess how candidate actions impact future scene
evolution. By conducting world modeling entirely in the latent space, DriveWorld-VLA supports controllable, action-conditioned imagination at the feature level, avoiding expensive pixel-level rollouts. Extensive open-loop and closed-loop evaluations demonstrate the effectiveness of DriveWorld-VLA, which achieves state-of-the-art performance with 91.3 PDMS on NAVSIMv1, 86.8 EPDMS on NAVSIMv2, and 0.16 3-second average collision rate on nuScenes. Code and models will be released in https://github.com/liulin815/DriveWorld-VLA.git.
Read the full abstract ▾
All authors Feiyang jia, Lin Liu, Ziying Song, Caiyan Jia, Hangjun Ye, Xiaoshuai Hao, Long Chen
Robotics
Beyond the Majority: Long-tail Imitation Learning for Robotic Manipulation
Junhong Zhu, Ji Zhang, Jingkuan Song et al. (+2 more)
While generalist
robot policies hold significant promise for learning diverse manipulation skills through imitation, their performance is often hindered by the long-tail distribution of training demonstrations. Policies learned on such data, which is heavily skewed towards a few data-rich head tasks, frequently exhibit poor generalization when confronted with the vast number of data-scarce tail tasks. In this work, we conduct a comprehensive analysis of the pervasive long-tail challenge inherent in policy learning. Our analysis begins by demonstrating the inefficacy of conventional long-tail learning strategies (e.g., re-sampling) for improving the policy's performance on tail tasks. We then uncover the underlying mechanism for this failure, revealing that data scarcity on tail tasks directly impairs the policy's spatial
reasoning capability. To overcome this, we introduce Approaching-Phase Augmentation (APA), a simple yet effective scheme that transfers knowledge from data-rich head tasks to data-scarce tail tasks without requiring external demonstrations. Extensive experiments in both simulation and real-world manipulation tasks demonstrate the effectiveness of APA. Our code and demos are publicly available at: https://mldxy.github.io/Project-VLA-long-tail/.
Read the full abstract ▾
All authors Junhong Zhu, Ji Zhang, Jingkuan Song, Lianli Gao, Heng Tao Shen
Robotics
World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policy
Xiaokang Liu, Zechen Bai, Hai Ci et al. (+2 more)
Reinforcement learning (RL) can refine
Vision-
Language-Action (VLA) policies beyond behavior cloning, but real-world RL remains expensive due to extensive rollouts, resets, supervision, and safety risks. Action-conditioned
video world models offer an option to train in virtual environments, yet they exhibit imprecise action following, particularly on subtle near-success failures. Besides, they lack native reward signals for RL. Computing rewards based on inaccurate visual predictions remain unreliable. We introduce World-VLA-Loop, structured around two foundational designs and a higher-level co-evolving paradigm. We first curate SANS, dedicatedly mixing successful and near-success trajectories to improve action-outcome
alignment. Then, we train a state-aware
video world model that jointly predicts future frames and binary rewards from
diffusion latents. It couples reward estimation to the generator rather than a separate module, and in turn, benefits visual prediction. Since VLA behavior shifts during RL, a fixed simulator can misalign with the updated policy, World-VLA-Loop therefore closes the loop by using the refined world model for iterative VLA post-training while feeding rollouts from each improved policy back to augment and fine-tune the world model. Across simulation and real-
robot experiments, World-VLA-Loop substantially improves VLA performance while reducing reliance on costly physical interaction.
Read the full abstract ▾
All authors Xiaokang Liu, Zechen Bai, Hai Ci, Kevin Yuchen Ma, Mike Zheng Shou
Robotics
MultiGraspNet: A Multitask 3D Vision Model for Multi-gripper Robotic Grasping
Stephany Ortuno-Chanelo, Paolo Rabino, Enrico Civitelli et al. (+2 more)
Vision-based models for robotic grasping automate critical, repetitive, and draining industrial tasks. Existing approaches are typically limited in two ways: they either target a single gripper and are potentially applied on costly dual-arm setups, or rely on custom hybrid grippers that require ad-hoc learning procedures with logic that cannot be transferred across tasks, restricting their general applicability. In this work, we present MultiGraspNet, a novel multitask 3D deep learning method that predicts feasible poses simultaneously for parallel and
vacuum grippers within a unified framework, enabling a single
robot to handle multiple end effectors. The model is trained on the richly annotated GraspNet-1Billion and SuctionNet-1Billion
datasets, which have been aligned for the purpose, and generates graspability masks quantifying the suitability of each scene point for successful grasps. By sharing early-stage features while maintaining gripper-specific refiners, MultiGraspNet effectively leverages complementary information across grasping modalities. This design preserves a compact architectural footprint of only 15.75M parameters and enables fast inference on a single GPU, enhancing adaptability and efficiency in cluttered scenes. We characterize MultiGraspnet's performance with an extensive experimental analysis, demonstrating its competitiveness with single-task models on relevant
benchmarks while reducing computational cost. Moreover, real-world experiments on a single-arm multi-gripper robotic setup show that our approach outperforms normalization-based multi-gripper approaches. Project page: https://vandal-lab.github.io/multigraspnet-project
Read the full abstract ▾
All authors Stephany Ortuno-Chanelo, Paolo Rabino, Enrico Civitelli, Tatiana Tommasi, Raffaello Camoriano
Robotics
User-Centric Object Navigation: A Benchmark with Integrated User Habits for Personalized Embodied Object Search
Hongcheng Wang, Jinyu Zhu and Hao Dong
In the evolving field of robotics, the challenge of Object
Navigation (ON) in household environments has attracted significant interest. Existing ON
benchmarks typically place objects in locations guided by general scene priors, without accounting for the specific placement habits of individual users. This omission limits the adaptability of
navigation agents in personalized household environments. To address this, we introduce User-centric Object
Navigation (UcON), a new
benchmark that incorporates user-specific object placement habits, referred to as user habits. This
benchmark requires
agents to leverage these user habits for more informed decision-making during
navigation. UcON encompasses approximately 22,600 user habits across 489 object categories. UcON is, to our knowledge, the first
benchmark that explicitly formalizes and evaluates habit-conditioned object
navigation at scale and covers the widest range of target object categories. Additionally, we propose a habit
retrieval module to extract and utilize habits related to target objects, enabling
agents to infer their likely locations more effectively. Experimental results demonstrate that current SOTA methods exhibit substantial performance degradation under habit-driven object placement, while integrating user habits consistently improves success rates. Code is available at https://github.com/whcpumpkin/User-Centric-Object-
Navigation.
Read the full abstract ▾
Robotics
ECO: Energy-Constrained Optimization with Reinforcement Learning for Humanoid Walking
Weidong Huang, Jingwen Zhang, Jiongye Li et al. (+6 more)
Achieving stable and energy-efficient locomotion is essential for
humanoid robots to operate continuously in real-world applications. Existing MPC and RL approaches often rely on energy-related metrics embedded within a multi-objective optimization framework, which require extensive hyperparameter tuning and often result in suboptimal policies. To address these challenges, we propose ECO (Energy-Constrained Optimization), a constrained RL framework that separates energy-related metrics from rewards, reformulating them as explicit inequality constraints. This method provides a clear and interpretable physical representation of energy costs, enabling more efficient and intuitive hyperparameter tuning for improved energy efficiency. ECO introduces dedicated constraints for energy consumption and reference motion, enforced by the Lagrangian method, to achieve stable, symmetric, and energy-efficient walking for
humanoid robots. We evaluated ECO against MPC, standard RL with reward shaping, and four state-of-the-art constrained RL methods. Experiments, including sim-to-sim and sim-to-real transfers on the kid-sized
humanoid robot BRUCE, demonstrate that ECO significantly reduces energy consumption compared to baselines while maintaining robust walking performance. These results highlight a substantial advancement in energy-efficient
humanoid locomotion. All experimental demonstrations can be found on the project website: https://sites.google.com/view/eco-
humanoid.
Read the full abstract ▾
All authors Weidong Huang, Jingwen Zhang, Jiongye Li, Shibowen Zhang, Jiayang Wu, Jiayi Wang, Hangxin Liu, Yaodong Yang, Yao Su
Robotics · Cross-listed from cs.CV
Bridging the Indoor-Outdoor Gap: Vision-Centric Instruction-Guided Embodied Navigation for the Last Meters
Yuxiang Zhao, Yirong Yang, Yanqing Zhu et al. (+6 more)
Embodied navigation holds significant promise for real-world applications such as last-mile delivery. However, most existing approaches are confined to either indoor or outdoor environments and rely heavily on strong assumptions, such as access to precise coordinate systems. While current outdoor methods can guide
agents to the vicinity of a target using coarse-grained localization, they fail to enable fine-grained entry through specific building entrances, critically limiting their utility in practical deployment scenarios that require seamless outdoor-to-indoor transitions. To bridge this gap, we introduce a novel task: out-to-in prior-free instruction-driven
embodied navigation. This formulation explicitly eliminates reliance on accurate external priors, requiring
agents to navigate solely based on
egocentric visual observations guided by instructions. To tackle this task, we propose a
vision-centric
embodied navigation framework that leverages image-based
prompts to drive decision-making. Additionally, we present the first open-source
dataset for this task, featuring a pipeline that integrates trajectory-conditioned
video synthesis into the data generation process. Through extensive experiments, we demonstrate that our proposed method consistently outperforms state-of-the-art baselines across key metrics including success rate and path efficiency.
Read the full abstract ▾
All authors Yuxiang Zhao, Yirong Yang, Yanqing Zhu, Yanfen Shen, Chiyu Wang, Zhining Gu, Pei Shi, Wei Guo, Mu Xu
Robotics · Cross-listed from cs.CV
TFusionOcc: T-Primitive Based Object-Centric Multi-Sensor Fusion Framework for 3D Occupancy Prediction
Zhenxing Ming, Yaoqi Huang, Julie Stephany Berrio et al. (+2 more)
The prediction of 3D semantic occupancy enables autonomous vehicles (AVs) to perceive the fine-grained
geometric and semantic scene structure for safe
navigation and decision-making. Existing methods mainly rely on either voxel-based representations, which incur redundant computation over empty regions, or on object-centric Gaussian primitives, which are limited in modeling complex, non-convex, and asymmetric structures. In this paper, we present TFusionOcc, a T-primitive-based object-centric multi-sensor fusion framework for 3D semantic occupancy prediction. Specifically, we introduce a family of Students t-distribution-based T-primitives, including the plain T-primitive, T-Superquadric, and deformable T-Superquadric with inverse warping, where the deformable T-Superquadric serves as the key geometry-enhancing primitive. We further develop a unified probabilistic formulation based on the Students t-distribution and the T-mixture model (TMM) to jointly model occupancy and semantics, and design a tightly coupled multi-stage fusion architecture to effectively integrate camera and
LiDAR cues. Extensive experiments on nuScenes show state-of-the-art performance, while additional evaluations on nuScenes-C demonstrate strong robustness under most corruption scenarios. The code will be available at: https://github.com/DanielMing123/TFusionOcc
Read the full abstract ▾
All authors Zhenxing Ming, Yaoqi Huang, Julie Stephany Berrio, Mao Shan, Stewart Worrall
Robotics
Now You See That: Learning End-to-End Humanoid Locomotion from Raw Pixels
Wandong Sun, Yongbo Su, Leoric Huang et al. (+10 more)
Achieving robust
vision-based
humanoid locomotion remains challenging due to two fundamental issues: the sim-to-real gap introduces significant perception noise that degrades performance on fine-grained tasks, and training a unified policy across diverse terrains is hindered by conflicting learning objectives. To address these challenges, we present an end-to-end framework for
vision-driven
humanoid locomotion. For robust sim-to-real transfer, we develop a high-fidelity depth sensor simulation that captures stereo matching artifacts and calibration uncertainties inherent in real-world sensing. We further propose a
vision-aware behavior
distillation approach that combines latent space
alignment with noise-invariant auxiliary tasks, enabling effective knowledge transfer from privileged height maps to noisy depth observations. For versatile terrain adaptation, we introduce terrain-specific reward shaping integrated with multi-critic and multi-discriminator learning, where dedicated networks capture the distinct dynamics and motion priors of each terrain type. We validate our approach on two
humanoid platforms equipped with different stereo depth cameras. The resulting policy demonstrates robust performance across diverse environments, seamlessly handling extreme challenges such as high platforms and wide gaps, as well as fine-grained tasks including bidirectional long-term staircase traversal.
Read the full abstract ▾
All authors Wandong Sun, Yongbo Su, Leoric Huang, Alex Zhang, Dwyane Wei, Mu San, Daniel Tian, Ellie Cao, Baoshi Cao, Yang Liu, Finn Yan, Ethan Xie, Zongwu Xie
Robotics
A Consistency-Improved LiDAR-Inertial Bundle Adjustment
Xinran Li, Shuaikang Zheng, Pengcheng Zheng et al. (+4 more)
Simultaneous Localization and Mapping (SLAM) using 3D
LiDAR has emerged as a cornerstone for autonomous
navigation in robotics. While feature-based SLAM systems have achieved impressive results by leveraging edge and planar structures, they often suffer from the inconsistent
estimator associated with feature parameterization and estimated covariance. In this work, we present a consistency-improved
LiDAR-inertial bundle adjustment (BA) with tailored parameterization and
estimator. First, we propose a stereographic-projection representation parameterizing the planar and edge features, and conduct a comprehensive observability analysis to support its integrability with consistent
estimator. Second, we implement a
LiDAR-inertial BA with Maximum a Posteriori (MAP) formulation and First-Estimate Jacobians (FEJ) to preserve the accurate estimated covariance and observability properties of the system. Last, we apply our proposed BA method to a
LiDAR-inertial odometry.
Read the full abstract ▾
All authors Xinran Li, Shuaikang Zheng, Pengcheng Zheng, Xinyang Wang, Jiacheng Li, Zhitian Li, Xudong Zou
Robotics
Towards Adaptive Environment Generation for Training Embodied Agents
Teresa Yeo, Dulaj Weerakoon, Dulanga Weerakoon et al. (+1 more)
Embodied agents struggle to generalize to new environments, even when those environments share similar underlying structures to their training settings. Most current approaches to generating these training environments follow an open-loop paradigm, without considering the
agent's current performance. While procedural generation methods can produce diverse scenes, diversity without feedback from the
agent is inefficient. The generated environments may be trivially easy, providing limited learning signal. To address this, we present a proof-of-concept for closed-loop environment generation that adapts difficulty to the
agent's current capabilities. Our system employs a controllable environment representation, extracts fine-grained performance feedback beyond binary success or failure, and implements a closed-loop adaptation mechanism that translates this feedback into environment modifications. This feedback-driven approach generates training environments that more challenging in the ways the
agent needs to improve, enabling more efficient learning and better generalization to novel settings.
Read the full abstract ▾
All authors Teresa Yeo, Dulaj Weerakoon, Dulanga Weerakoon, Archan Misra
Robotics
Nipping the Drift in the Bud: Retrospective Rectification for Robust Vision-Language Navigation
Gang He, Zhenyang Liu, Kepeng Xu et al. (+5 more)
Vision-
Language Navigation (VLN) requires
embodied agents to interpret natural
language instructions and navigate through complex continuous 3D environments. However, the dominant imitation learning paradigm suffers from exposure bias, where minor deviations during inference lead to compounding errors. While DAgger-style approaches attempt to mitigate this by correcting error states, we identify a critical limitation: Instruction-State Misalignment. Forcing an
agent to learn recovery actions from off-track states often creates supervision signals that semantically conflict with the original instruction. In response to these challenges, we introduce BudVLN, an online framework that learns from on-policy rollouts by constructing supervision to match the current state distribution. BudVLN performs retrospective rectification via counterfactual re-anchoring and decision-conditioned supervision synthesis, using a geodesic oracle to synthesize corrective trajectories that originate from valid historical states, ensuring semantic consistency. Experiments on the standard R2R-CE and RxR-CE
benchmarks demonstrate that BudVLN consistently mitigates distribution shift and achieves state-of-the-art performance in both Success Rate and SPL.
Read the full abstract ▾
All authors Gang He, Zhenyang Liu, Kepeng Xu, Li Xu, Tong Qiao, Wenxin Yu, Chang Wu, Weiying Xie
Robotics
HiWET: Hierarchical World-Frame End-Effector Tracking for Long-Horizon Humanoid Loco-Manipulation
Zhanxiang Cao, Liyun Yan, Yang Zhang et al. (+7 more)
Humanoid loco-manipulation requires executing precise manipulation tasks while maintaining dynamic stability amid base motion and impacts. Existing approaches typically formulate commands in body-centric frames, fail to inherently correct cumulative world-frame drift induced by legged locomotion. We reformulate the problem as world-frame end-effector tracking and propose HiWET, a hierarchical reinforcement learning framework that decouples global
reasoning from dynamic execution. The high-level policy generates subgoals that jointly optimize end-effector accuracy and base positioning in the world frame, while the low-level policy executes these commands under stability constraints. We introduce a Kinematic
Manifold Prior (KMP) that embeds the manipulation
manifold into the action space via residual learning, reducing exploration dimensionality and mitigating kinematically invalid behaviors. Extensive simulation and ablation studies demonstrate that HiWET achieves precise and stable end-effector tracking in long-horizon world-frame tasks. We validate zero-shot sim-to-real transfer of the low-level policy on a physical
humanoid, demonstrating stable locomotion under diverse manipulation commands. These results indicate that explicit world-frame
reasoning combined with hierarchical control provides an effective and scalable solution for long-horizon
humanoid loco-manipulation.
Read the full abstract ▾
All authors Zhanxiang Cao, Liyun Yan, Yang Zhang, Sirui Chen, Jianming Ma, Tianyue Zhan, Shengcheng Fu, Yufei Jia, Cewu Lu, Yue Gao