Robotics
TACT-ful: Multi-Channel Terrain Affordance and Compliance Training for Payload-Robust Perceptive Humanoid Locomotion
Thanh Ly, Truong-Duy Dang, Chien Le et al. (+5 more)
Foothold selection on structured terrain requires explicit
reasoning about contact planarity, surface steepness, and kinematic reachability, properties not captured by a single height-based terrain signal. We propose a multi-channel terrain cost combining flatness, steepness, and velocity-aware height feasibility, plus a forward climb reward, that simultaneously drives a GPU-parallel divergent component of motion (DCM) foothold planner and shapes a dense per-step affordance reward for an asymmetric actor-critic policy trained with proximal policy optimization (PPO) from depth images. A Bézier swing trajectory with adaptive apex bias extends foothold tracking to joint position-and-orientation, using the arc tangent to guide sole orientation through riser crossings and tread landings. To support payload tasks, we introduce a lower-body compliance training procedure in which a virtual wrench is injected at a sampled load attachment point, generating physically consistent force and moment; wrench-aware compliance targets replace rigid pose penalties, and the policy learns to yield to load-induced perturbations without force sensing. The full system trains end-to-end with standard PPO, no
distillation, and no teacher-student staging, and is deployed on a
humanoid directly from simulation with configuration changes only. In simulation, the policy reaches $1.0~\mathrm{m/s}$ on stairs with risers up to $0.20~\mathrm{m}$ and improves payload robustness up to ${\sim}15~\mathrm{kg}$ centered load and for moment-dominated wrist loads without fine-tuning. We also provide a qualitative hardware demonstration on structured terrain. Project website: https://fai-rl-tech.github.io/tact-locomotion.github.io/
Read the full abstract ▾
All authors Thanh Ly, Truong-Duy Dang, Chien Le, Tan-Dzung Do, Phuong Tuan Dat, Cuc T. Trinh, Vien Anh Ngo, An T. Le
Robotics
Uncertainty-Aware Action Segmentation for Human-to-Robot Assembly Teleoperation
Fnu Heman, Yixuan Wang, Kolin Xu et al. (+7 more)
Action
segmentation supports intention estimation and execution monitoring in assisted teleoperation, but
robot demonstrations are costly to collect. We propose an uncertainty-aware framework that transfers temporal knowledge from human to
robot demonstrations through MS-TCN++~\cite{li2020mstcnpp} pretraining and fine-tuning, reliability-aware sequence correction, and class-conditional conformal prediction. Across 22 toy-car assembly action classes, fine-tuning with 9
robot demonstrations improves test Edit score from 55.23 to 72.51 and segmental F1@50 from 10.97 to 16.86. The reliability analyzer identifies incorrect segments with 0.755 AUROC. Unrestricted correction reduces Edit to 67.86, while selective acceptance maintains 73.45, primarily safeguarding against harmful relabeling. Conformal prediction achieves 95.38% empirical coverage at a 93% target, but prediction sets average 17.41 of 22 classes. These findings support human temporal pretraining while revealing persistent challenges in reliable correction and informative uncertainty quantification with limited
robot supervision.
Read the full abstract ▾
All authors Fnu Heman, Yixuan Wang, Kolin Xu, Conner Wallace, John Dang, Akhil Joshi, Vignesh Thondikulam, Jun Sheng, Pinhas Ben-Tzvi, Mingyu Cai
Robotics
MotionVLA: Injecting Geometric Motion into Vision-Language-Action Model
Shanglin Yuan, Weiheng Zhao, Xianda Guo et al. (+4 more)
Vision-
language-action (VLA) models increasingly condition
robot policies on history, depth, or 4D features to resolve ambiguity in long-horizon manipulation. However, more spatiotemporal evidence is not necessarily better: when the injected evidence is not motion-consistent, it can introduce
geometric drift, fragmented temporal cues, and unstable action generation. This raises a simple question: should a VLA remember past frames, or remember the motion that connects them? We introduce MotionVLA, a motion-history interface that converts a short past-only
video window into compact, time-continuous trajectory-field
tokens. Instead of treating history as a sparse set of ndependently lifted frames, MotionVLA represents recent observations as physically coherent motion evidence. Current visual
tokens query this history to retrieve task-relevant motion information, which is then recoupled into the VLA stream under trajectory-grounded supervision. Experiments across simulation
benchmarks and preliminary real-
robot rollouts show that MotionVLA improves long-horizon manipulation while producing smoother and more direct executions. These results suggest that effective VLA
memory is not just about providing more 4D context, but about exposing motion-consistent evidence that is usable for control.
Read the full abstract ▾
All authors Shanglin Yuan, Weiheng Zhao, Xianda Guo, Wei Sui, Li Yu, Wenyu Liu, Xinggang Wang
Robotics · Cross-listed from cs.CV
G2G: Exploiting Intra-Group Geometry for Inter-Group Pose Estimation
Yufei Wei, Shuhao Ye, Chenxiao Hu et al. (+5 more)
Recovering the relative 6-DoF pose between two image groups underlies cross-sequence relocalization and multi-camera rig odometry. Each group carries known intra-group geometry from visual odometry or rig calibration, and pretrained multi-view backbones already fuse such geometry into visual features. Yet current models treat all views as an unstructured set, leaving cross-group
reasoning as the missing piece. We introduce G2G, which keeps the foundation model entirely frozen and adds three lightweight trainable modules to bridge the two groups: a perceiver resampler, a cross-group bridge with merged self-
attention, and a multi-frame pose head. The trainable footprint totals about 32M parameters, under 6% of the full model, and is supervised only by relative poses. Across four
datasets that span indoor and outdoor simulation, real-world cross-season capture, and zero-shot sim-to-real transfer, G2G attains state-of-the-art accuracy on both tasks, while trainable baselines are retrained with their original supervision. Code and visualizations: https://github.com/WeiYuFei0217/G2G.
Read the full abstract ▾
All authors Yufei Wei, Shuhao Ye, Chenxiao Hu, Yiyuan Pan, Dongyu Feng, Rong Xiong, Yue Wang, Yanmei Jiao
Robotics
Toward Interaction Dynamics: A Predictive Framework for Safe Physical Human Robot Interaction
Yongyan Cao and Jinshan Tang
Physical human-
robot interaction requires yielding transiently to contact yet recovering the commanded reference under sustained load. Finite-stiffness impedance control retains a static deflection there, while predictive alternatives typically optimize a nonlinear
robot or impedance model online. Operational-space cancellation instead exposes a translational error double integrator with a fixed transition
matrix and a configuration-scheduled input map, making interaction a predictive quantity rather than a property re-derived per configuration. We build on it a compact offset-free interaction-error MPC for torque-controlled manipulators: a force-domain random-walk state estimates persistent interaction and model error, and a 30-variable convex QP maps the correction through the current task inertia while constraining the applied joint torque. Conditional results establish impedance equivalence of the unconstrained passive feedback, offset-free regulation at feasible frozen configurations, and quadratic stabilizability of the scheduled backbone. In a 1kHz MuJoCo simulation of a 7-DOF Franka FR3, the
estimator cuts steady-state error under a repeated 15N step from 2.77mm to 0.042mm when added to the otherwise identical 100Hz MPC. A stiffness-and-damping-calibrated impedance baseline attains 2.59mm but briefly saturates and needs 3.3x the peak positive joint power. Adding ideal measured-force cancellation to that baseline gives 1.39mm, so constant-load rejection is not unique to MPC; the sensorless controller still reaches 0.042mm in the moving task, a 65x reduction without force sensing and without the baseline's saturation or power cost. Demonstrated in simulation under a shared actuator budget, the contribution is an efficient operational-space realization complementing rather than replacing broader interaction-control architectures.
Read the full abstract ▾
Robotics
SIMPLE: Simulation-Based Policy Learning and Evaluation for Humanoid Loco-manipulation
Songlin Wei, Zhenhao Ni, Jie Liu et al. (+9 more)
Humanoid foundation models are advancing faster than we can evaluate them. While real-world testing is expensive and difficult to reproduce, existing simulation
benchmarks focus primarily on table-top or wheeled
robots. A scalable and reproducible
benchmark for whole-body
humanoid loco-manipulation remains an open problem. To this end, we present SIMPLE, a unified simulation testbed for
humanoid policy learning and evaluation. SIMPLE couples the accurate contact-rich dynamics of MuJoCo with the photorealistic rendering of IsaacSim. It provides a large-scale environment comprising 16 diverse whole-body tasks, 50 indoor scenes, and over 1,000 object assets. To facilitate scalable data collection, the framework integrates two data generation pipelines: automated trajectory generation via motion planning and a low-latency VR teleoperation interface. We further integrate and
benchmark mainstream
humanoid policies at scale in SIMPLE, including lightweight imitation networks, large
vision-
language-action (VLA) models, and recent world action models (WAMs). Our experiments reveal a strong correlation between policy performance in simulation and the real world. Furthermore, we demonstrate that policies trained on data collected in SIMPLE can be transferred zero-shot to physical
humanoid robots under similar settings, providing a robust and reproducible foundation for
humanoid robotics research.
Read the full abstract ▾
All authors Songlin Wei, Zhenhao Ni, Jie Liu, Zhenyu Zhao, Junjie Ye, Hongyi Jing, Junkai Xia, Xiawei Liu, Michael Leong, Liang Heng, Di Huang, Yue Wang
Robotics
Mind Your Steps: A General Learning Framework for Accurate Humanoid Foothold Tracking
Alessandro Montenegro, Shihao Li, Puze Liu et al. (+2 more)
Enabling
humanoid robots to operate in complex, dynamic environments remains a critical challenge, fundamentally limited by the ability to navigate robustly, safely, and accurately. While reinforcement learning with velocity-commanded policies has achieved remarkable robustness in
humanoid locomotion, this approach lacks explicit control of the foothold placement, leading to unsafe behavior, such as stepping onto human feet, or imprecise
navigation, hindering the following manipulation task. Conversely, explicit foothold-tracking policies offer a promising alternative by directly being commanded with target foot poses. However, existing approaches are often limited by unrealistic state assumptions, compromising real-world deployment, or they are part of staged pipelines, making them tied to specific downstream tasks. In this work, we introduce a novel, lightweight framework for training general-purpose 3D foothold-tracking policies. By dynamically providing footstep support through a goal sampler, this method enables the learned policy to be agnostic to specific terrains. Our new target representation effectively mitigates challenges arising in the real world, such as noisy and inaccurate pose estimation and foot contact estimation. Designed for direct real-world transfer, our policy acts as a standalone low-level controller that can be seamlessly paired with various high-level foothold generators. We demonstrate the effectiveness of our framework through extensive experiments in simulation and in the real world. By coupling our policy with different upstream planners, we achieve natural and accurate locomotion in challenging settings, paving the way for loco-manipulation tasks in complex environments.
Read the full abstract ▾
All authors Alessandro Montenegro, Shihao Li, Puze Liu, Alberto Maria Metelli, Jan Peters
Robotics
Disturbance-Aware Aerial Robotics for Ethical Wildlife Monitoring
Mahmut Osmanovic, Isac Paulsson and Teddy Lazebnik
Reliable wildlife monitoring is essential for ecology and conservation, yet many existing methods, such as tagging, capture, and close-range observation, can alter the very behaviors they aim to measure. Aerial
robots offer a scalable alternative, which has shown promising performance in multiple studies. Nonetheless, existing approaches typically lack behavioral awareness, rely on fixed heuristics, or require real-world training data that are costly, impractical, and ethically difficult to obtain. As a result, there remains no general framework for adaptive drone-based monitoring that can both preserve ecological validity and scale across species, behaviors, and robotic platforms. In this study, we introduce a disturbance-aware reinforcement-learning-based framework for heterogeneous aerial robotic fleets that enables autonomous wildlife tracking while explicitly minimizing behavioral disruption. We couple a zoologically grounded simulation environment with fitted animal movement models derived from real trajectory statistics, and train control policies using a reward formulation that captures the trade-off between observation quality and disturbance risk. Across three species (pigeon, jackal, and spur-winged lapwing) with distinct ecologies and motion patterns and four increasingly strategic behavior models common in nature, the learned policies consistently surpassed currently used rule-based baselines and generalized across monitoring tasks, animal dynamics, and drone types. These results establish disturbance-aware learning as a viable foundation for non-invasive autonomous wildlife observation, opening a path towards scalable, ethically responsible, and scientifically reliable robotic monitoring in ecology and conservation.
Read the full abstract ▾
Robotics
Agentic Neuro-Symbolic Planning and Commissioning for Human-in-the-Loop Industrial Robotics with Digital Twins
Zhihao Liu, Victor Nan Fernandez-Ayala, Tianyu Wang et al. (+4 more)
Flexible robotic automation requires systems that interpret operator intent, verify physical feasibility, and recover from execution failures across both the planning and execution stages. This paper proposes an agentic neuro-symbolic framework for human-in-the-loop industrial robotics, in which LLMs are used for tasks that require
language understanding or contextual
reasoning, while all verification, sequencing, and execution remain deterministic. The framework adapts the Planner-Generator-Evaluator (PGE) harness pattern from software engineering into a Specifier-Designer-Inspector (SDI) architecture for industrial robotics, combined with LangGraph-based dynamic routing for failure recovery. A two-tier recovery mechanism addresses structure-level replanning through context-aware orchestration and execution-level
geometric failures through deterministic recovery skills. A Unity3D digital twin supports human inspection, modification, and re-verification prior to physical execution. Evaluated on natural-
language commands across multiple difficulty levels against ten baselines, the proposed method achieves the highest task success. Ablation results confirm that structured command expansion, symbolic verification, selective LLM routing, and recovery skills are each individually necessary.
Read the full abstract ▾
All authors Zhihao Liu, Victor Nan Fernandez-Ayala, Tianyu Wang, Qiang Qin, Xi Vincent Wang, Dimos V. Dimarogonas, Lihui Wang
Robotics
Propeller-Assisted Robust 3D Hopping Robot with Hierarchical Force Allocation
Chuhan Zhang, Hongbo Zhang, Yanlin Chen et al. (+4 more)
Monopedal hopping
robots are conceptually simple but highly dynamic and inherently unstable. Achieving robust 3D hopping is still difficult because ground reaction forces are available only during the short stance phase, while the
robot is underactuated in flight. A key unresolved issue is how to improve flight-phase control authority. Propeller assistance provides a promising solution, but it requires careful coordination of leg-generated contact forces and propeller thrusts across stance and flight. This paper presents Pro-OMEGA2, a propeller-assisted 3D monopedal hopping
robot with an active 3-RSR parallel leg and a trunk-mounted tri-rotor for auxiliary attitude regulation. To address the force coordination challenge, we propose a Hierarchical Force Allocation (HFA) framework based on a single rigid body (SRB) model. The leg generates the main stance contact wrench, while the tri-rotor provides auxiliary attitude regulation, compensating the residual attitude moment in stance and maintaining attitude during flight. Real-
robot experiments in indoor and outdoor scenarios demonstrate sustained 3D hopping, including terrain transitions and impulsive push recovery, validating robustness under unmodeled contact and external disturbances.
Read the full abstract ▾
All authors Chuhan Zhang, Hongbo Zhang, Yanlin Chen, Yunxi Tang, Yun-Hui Liu, Mingyi Liu, Xiangyu Chu
Robotics
Learning from Human Driving: A Human-in-the-Loop Online Behavior Cloning Framework for Autonomous Driving
Yuhong Shi, Jianyi Liu, Lihang Sun et al. (+2 more)
With the
evolution of large foundation models (LFMs), data-driven autonomous driving has made significant strides. However, existing paradigms still face severe challenges in complex interaction and long-tail scenarios due to distribution shift and causal confusion. These limitations often result in a lack of human-level decision-making flexibility and safety in extreme conditions. To overcome this limitation, this paper proposes a Human-in-the-Loop Online Behavior Cloning frame work (HiL-OBC) for autonomous driving, which aims to deeply integrate the cross-modal perceptual capabilities of LFMs with the high-level driving intelligence of human experts. Specifically, HiL-OBC deployment is executed through three critical phases: policy initialization with human intervention, latent behavioral modeling with
Bayesian policy adaptation, and online deploy ment and updates. Furthermore, we design a Multi-modal Online Behavior Cloning (MOBC) model, which optimizes the base driving policy online through a lightweight network architecture, a takeover trigger mechanism, and a multi-variant loss function, thereby enhancing the system's decision-making robustness in complex environments. We evaluated the HiL-OBC on the LangAuto-Human CARLA
benchmark. Experimental results demonstrate that the driving policies optimized via the human-in-the-loop mechanism achieve substantial performance gains: the DS of StructNav, LFG, and LMDrive increased by 47.25%, 31.59%, and 32.12%, respectively, with a simultaneous of various experimental settings and key components highlights the advantages of human-in-the-loop learning in improving decision-making robustness and overall driving performance.
Read the full abstract ▾
All authors Yuhong Shi, Jianyi Liu, Lihang Sun, Li Li, Xudong Dong
Robotics
CLASP: Language-Driven Robot Skill Selection and Composition using Task-Parameterized Learning
Markus Knauer, Valentin Gieraths, Tai Mai et al. (+4 more)
Enabling
robots to understand and execute tasks from natural
language commands while maintaining data efficiency remains challenging. Foundation models such as
vision-
language-action (VLA) and
vision-
language models (VLMs) provide intuitive interaction channels but require extensive data; task-parameterized imitation learning achieves data efficiency but lacks natural
language grounding. This work bridges this gap through a modular architecture combining task-parameterized kernelized movement primitives (TP-KMPs) with pretrained VLMs. During learning, skills are acquired from 2 to 5 kinesthetic demonstrations, and the VLM generates skill schemas describing each skill's parameters and preconditions. During execution, the VLM interprets commands to select skills, reason about parameter bindings, and create novel behaviors through covariance-weighted composition. When no skill or composition suffices, the system identifies capability gaps and requests targeted demonstrations, all without fine-tuning. Validation on a 7-DoF manipulator shows success rates of 73.3%-100% in scenarios requiring skill selection, composition, and active learning.
Read the full abstract ▾
All authors Markus Knauer, Valentin Gieraths, Tai Mai, Samuel Bustamante, Alin Albu-Schäffer, Freek Stulp, João Silvério
Robotics
SynthICL: Scalable In-context Imitation Learning with Synthetic Data
Cheng Qian, Ruomeng Fan, Yifei Ren et al. (+2 more)
In-context imitation learning (ICIL) enables
robots to learn new tasks from a small number of demonstrations by conditioning a pre-trained policy on task-specific examples, without retraining at test time. Despite this promise, training generalizable and scalable in-context imitation policies remains an open challenge. We present SynthICL, a scalable framework that trains ICIL policies entirely from RGB-only synthetic data. Specifically, we build a data generation pipeline to produce high-fidelity ICIL data and train a flow-matching
transformer policy on the resulting
dataset. SynthICL avoids the need for depth sensing, precise camera calibration, and real-world training data in prior approaches, offering a simpler and more scalable alternative. We further incorporate subgoal prediction by training the model to predict the next subgoal images, enabling more precise and visually grounded control. Evaluated on 16 unseen real-world manipulation tasks, SynthICL achieves an average success rate of 79% with only one demonstration provided at test time and outperforms prior methods. Project page: https://synth-icl.github.io
Read the full abstract ▾
All authors Cheng Qian, Ruomeng Fan, Yifei Ren, Yilong Wang, Edward Johns
Robotics
Vision-Guided Dual-Arm Humanoid Robotic Disassembly of End-of-Life 18650 Lithium-ion Battery Packs
Yile Chen, Zhihao Liu, Xi Vincent Wang et al. (+1 more)
The growing volume of retired lithium-ion battery packs from electric vehicles and portable electronics calls for automated disassembly that is safe, flexible, and selective down to the individual
cell. Existing robotic systems, however, mostly assume known pack poses, external fixtures, or specialised tooling, leaving fixture-free
cell-level disassembly under pose uncertainty largely unsolved. This paper presents a
vision-guided dual-arm pipeline that disassembles a 21-
cell 18650 pack from an arbitrary initial pose using only general-purpose parallel-jaw grippers, RGB-D sensing, and a pre-trained grasp detector. Pose uncertainty is absorbed by a learn-and-filter perception stack with discrete look-and-move wrist-camera corrections, while a mid-task support transfer between the two arms extends the effective workspace without any external clamp. The pipeline achieves an 8/10 end-to-end success rate, a
cell-localisation root-mean-square error of $2.4$\,mm, and a mean cycle time of 6.0\,minutes per pack, providing a practical, fixture-free building block for industrial battery recycling.
Read the full abstract ▾
All authors Yile Chen, Zhihao Liu, Xi Vincent Wang, Lihui Wang
Robotics
Learning Predictive Control with Deep Koopman Operators for Autonomous Vehicle Motion Planning
Xinglong Zhang, Yongqian Xiao, Haotian Cao et al. (+3 more)
Model Predictive Control (MPC) is widely used for autonomous-vehicle (AV) motion planning, but its real-time applicability is often limited by the need for accurate models and online solution of nonlinear, nonconvex optimization problems in dynamic road environments. Actor-critic reinforcement learning offers a promising alternative for online policy generation, yet its policy-learning process often lacks explicit control-theoretic structure. This article proposes a learning predictive control (LPC) framework with deep Koopman operators for efficient real-time motion planning under nonconvex constraints. To address nonlinear and uncertain vehicle dynamics, a deep-Koopman-based predictor is used to lift the system into an interpretable linear observable space in a data-driven manner. Unlike traditional MPC, which computes open-loop control sequences, the proposed LPC framework yields a closed-loop state-feedback policy within each prediction interval through receding-horizon actor-critic learning. To ensure safety under nonconvex environmental constraints, LPC constructs convex local surrogate representations of obstacles and defines corresponding potential-field functions. These functions and their gradients are directly embedded into the actor-critic structure, enabling efficient, safety-aware policy learning. Extensive simulations and real-world experiments on the HongQi-EHS3 platform demonstrate favorable performance in diverse obstacle-avoidance scenarios in terms of safety, computational efficiency, and driving comfort, compared with
benchmark methods such as CBF-MPC and LMPCC.
Read the full abstract ▾
All authors Xinglong Zhang, Yongqian Xiao, Haotian Cao, Xing Zhou, Xin Yin, Xin Xu
Robotics
Ego-Pi: VLA Fine-Tuning for Ego-Centric Human and Robot Data
Ji Woong Kim, Ke Wang, Zipeng Fu et al. (+4 more)
Robotics faces a fundamental challenge of data scarcity. Unlike
language or
vision research, there is no internet-scale
dataset for robotic manipulation. A promising path forward is to leverage
egocentric human data, which can be collected more easily, with greater breadth, and at a larger scale. Towards this end, we investigate key design choices for learning across human and
humanoid embodiments equipped with dexterous five-finger hands, using the $\pi_{0.5}$ model as a foundation. Our results show that human data enables
robots to learn new task semantics and compose existing skills into novel behaviors without corresponding
robot data. The paper website is here: https://egopipaper.github.io/
Read the full abstract ▾
All authors Ji Woong Kim, Ke Wang, Zipeng Fu, Sirui Chen, Cong Zhao, Jeff Lai, Chelsea Finn
Robotics
Reinforcement learning in linear embedding space unlocks generalizable control across soft robot configurations
Xinglong Zhang, Cong Li, Hangjie Mo et al. (+13 more)
Soft-bodied organisms such as octopuses and elephant trunks exhibit remarkable morphological adaptability, dynamically reconfiguring body shape and stiffness, and flexibly adjusting their control strategies to enable versatile behaviors. Inspired by these biological systems, various soft
robots have emerged in recent decades, featuring diverse materials, stiffnesses, and morphologies tailored to specific tasks. Despite substantial advances in the materials and structural designs of soft
robots, developing a generalizable control framework capable of rapid adaptation across diverse configurations remains a long-standing challenge. Existing controllers are limited to fixed configurations, demanding laborious configuration-specific remodelling and policy redesign for new configurations. Here, we introduce a generalizable control system that enables rapid adaptation across diverse soft
robot configurations via reinforcement learning in a shared linear Koopman
embedding space. By encoding
robot dynamics into this
embedding space, our method decouples control policies from specific morphologies, allowing real-time, model-free policy adaptation across diverse configurations without retraining from scratch. We validate our system across 33 distinct
robot configurations. Our system achieves a 75 times reduction in transfer samples across configurations, while sustaining robust performance under high-speed motion, heavy payloads, and multiactuator faults, and achieving real-world skills previously unattainable in soft robotics. This work establishes a unified and adaptable control paradigm for diverse soft
robot configurations, bridging mechanical reconfigurability with control flexibility, and may offer broader insights for generalizable control in complex physical systems.
Read the full abstract ▾
All authors Xinglong Zhang, Cong Li, Hangjie Mo, Yue Jiang, Xin Xu, Wei Jiang, Zhenshan Bing, Yihe Yang, Xiaojian Li, Yueneng Yang, Huimin Lu, Ling-li Zeng, Alois Knoll, Dewen Hu, Li Wen, Wei Pan
Robotics
Revisiting Articulated Parts Perception in Robot Manipulation
Xiaoqian Wu, Yejie Guo, Xiaoyang Chen et al. (+3 more)
We are surrounded by various objects with movable, articulated parts, e.g., box, handle, door. An accurate and generalizable perception of articulated parts is essential to enhance robotic manipulation capabilities. Building on this need, recent efforts in articulated parts perception have followed two main directions: One line of work uses pose-based representation, which requires high manual cost; in parallel, affordance-based methods extract future object motion from point tracking without additional manual efforts, but suffer from low-quality data. In this paper, we propose a new representation of articulated parts,
Geometric Primary Structure (GPS), an abstraction of the part geometry structure to balance scalability and quality. For efficient and scalable data collection, GPS is integrated with a portable Virtual Reality (VR) device and requires only one minute to annotate one object sequence. This direct human annotation provides higher quality than the estimated affordance. With this efficient VR-GPS system, we collect 41K frames for 234 objects across six part classes, and train a generalizable GPS model with a single RGB-D object image as input. For object manipulation, we deploy a heuristic policy based on GPS prediction. Without any in-domain fine-tuning, our method achieves an 73% success rate, covering 270 initial states for 9 objects. Our code, data and reusable tool are available at https://enlighten0707.github.io/gps.
Read the full abstract ▾
All authors Xiaoqian Wu, Yejie Guo, Xiaoyang Chen, Lixin Yang, Cewu Lu, Yong-Lu Li
Robotics
Continual Quadruped Robots Coordination via Semantic Skill Discovery
Daoqing Wang, Yuchen Xiao, Weixuan Huang et al. (+5 more)
Multi-quadruped coordination has attracted increasing
attention due to its enhanced payload capacity, broader contact coverage, and improved adaptability to challenging tasks. Existing methods for multi-quadruped manipulation typically focus on predefined or closed task families, often relying on multi-
agent reinforcement learning (MARL) to train task-specific coordination policies. However, such methods struggle in open-ended continual learning settings, where tasks arrive sequentially and
robots are expected to acquire new coordination skills while reusing previously learned ones without catastrophic forgetting. To address this challenge, we propose Conquer, a semantic skill-library framework that formulates continual multi-quadruped coordination as a retrieve-adapt-update process. First, to accommodate varying team sizes across tasks, we design a team-structured Self-Allies-Goal (SAG) backbone that supports variable-cardinality
robot teams by explicitly modeling each
robot's own state, teammate context, and task goal. For each incoming task, Conquer constructs a task-level semantic descriptor from pre-execution information and retrieves a relevant skill from the library for adaptation. After successful execution, Conquer updates the skill library by extracting trajectory-level semantic descriptors and organizing them according to semantic distance, thereby enabling continual skill accumulation and cross-task knowledge transfer. Simulation experiments show that Conquer achieves a final average success rate of 95.6%, demonstrating strong forward transfer and negligible catastrophic forgetting. Real-world rollouts on Unitree Go2 teams further validate the deployment feasibility of Conquer for practical multi-quadruped coordination. Simulation and real-
robot demonstration
videos are available at: https://conquer-project.pages.dev/.
Read the full abstract ▾
All authors Daoqing Wang, Yuchen Xiao, Weixuan Huang, Zhilong Zhang, Shenghua Wan, Meng Li, Lei Yuan, Yang Yu
Robotics
Cybernetic Android Avatar "Yui": System Integration, Field Deployment, and Evaluation
Kaoruko Shinkawa, Mizuki Nakajima, Taisei Mogi et al. (+1 more)
Remote communication technologies have become widely used; however, supporting a sense of shared physical space and conveying rich non-verbal cues remain challenging in many social interaction scenarios. This study presents "Yui," a full-body cybernetic android avatar designed to integrate operator-side immersive teleoperation with interlocutor-side human-like social signaling. Yui combines a 55-degrees of freedom full-body mechanism with a previously developed android head, facial expression and gaze control, upper-body and arm motion, hand actuation, and a mobile platform. It can be operated through either the immersive mode using a head mounted display-based interface or desktop mode using a webcam-based interface. We evaluated the system through three real-world deployments: a long-term public exhibition at Expo 2025 in Osaka, Kansai, Japan; a remote educational exchange between elementary school students; and a public interaction study with general participants. During the Expo deployment, two units accumulated approximately 1131 h of operation, demonstrating both operational feasibility and maintenance challenges. In the public study, both operators and interlocutors reported positive impressions of co-presence and willingness to use the system. Interlocutors also rated the avatar positively in terms of human likeness and the transmission of
emotions and intentions. The results indicate usability for general operators while suggesting room for improvement in precise controllability. These findings provide field-derived evidence and design implications for socially deployable full-body android avatars.
Read the full abstract ▾
All authors Kaoruko Shinkawa, Mizuki Nakajima, Taisei Mogi, Yoshihiro Nakata
Robotics
vla.cpp: A Unified Inference Runtime for Vision-Language-Action Models
Khanh D. Nguyen, Hung T. Ho, Chinh T. Nguyen et al. (+5 more)
Deploying
vision–
language–action (VLA) models on
robots requires adapting model-specific inference pipelines to heterogeneous processors and limited onboard
memory. We present vla.cpp, a unified C++ inference runtime for eleven VLA models, with no PyTorch dependency for model execution. The runtime shares model loading, tensor execution, and serving while retaining architecture-specific
attention, conditioning, and action heads. Iterative policies reuse observation-dependent computation across solver steps, while regression policies predict actions directly. We evaluate task success on LIBERO-Object and profile supported configurations on NVIDIA, Apple, and Intel hardware. BitVLA completes 200/200 LIBERO-Object episodes on an 8,GB Jetson Orin Nano. A ternary tensor-core kernel accelerates its client inference by $4.0$–$4.6\times$ over the CUDA-core baseline on RTX 3060 and AGX Orin. A SmolVLA case study links positional-index precision to gripper commands and task success, showing why fixed-input numerical checks should accompany rollout evaluation. Deployments on UR10e and ALOHA demonstrate physical
robot integration; delay and execution-horizon studies characterize synchronous chunked control. The results demonstrate a common deployment path across VLA architectures and hardware, with numerical validation and control settings guiding deployment alongside inference efficiency.
Read the full abstract ▾
All authors Khanh D. Nguyen, Hung T. Ho, Chinh T. Nguyen, Thanh Q. Duong, Linh D. Le, Duy M. H. Nguyen, Vien A. Ngo, An T. Le
Robotics
Cooperative Long Rope Skipping via Multi-Agent Reinforcement Learning
Zihao Wang, Shijie Peng, Kerui Wu et al. (+6 more)
Humans exhibit remarkable motor agility, enabling a wide range of dynamic skills such as running and jumping, which highlights the great potential of
humanoid robots for athletic locomotion. Among athletic sports, long rope skipping requires two rope turners to cooperatively swing the rope while adapting to a player under different jumping rhythms, making it a meaningful yet challenging task for
humanoid robots. Although existing methods for
humanoid sports have achieved success in single-
agent and interaction-free settings, such as running, dancing, and parkour, task scenarios that require precise coordination among multiple participants remain largely unexplored. To this end, we propose Marope, a multi-
agent reinforcement learning (MARL) framework for cooperative long rope skipping with multiple
humanoid robots. Specifically, Marope adopts a hierarchical reinforcement learning framework for policy training. At the lower level, it learns decentralized rope manipulation policies through MARL, while at the upper level, a centralized scheduling policy is trained to coordinate the execution of the lower-level policies. To improve generalization across different player behavioral styles, Marope further incorporates diverse jumping policies into cooperative game training. We evaluate our approach on Unitree G1
humanoid robots in both simulation and real-world settings. Experimental results demonstrate that Marope outperforms various baselines, achieving more efficient and stable rope manipulation as well as more robust and adaptable cooperation with varied players.
Read the full abstract ▾
All authors Zihao Wang, Shijie Peng, Kerui Wu, Yu Huang, Ruiqi Xue, Dong Liu, Tian Xu, Lei Yuan, Yang Yu
Robotics
Perceptive Behavior Foundation Model: Adapting Human Motion Priors to Robot-Centric Terrain
Zifan Wang, Yizhao Li, Teli Ma et al. (+5 more)
Humanoid behavior foundation models aim to acquire reusable whole-body control policies from broad human motion priors, enabling a single controller to produce diverse and expressive behaviors. However, existing motion-centric foundation policies largely assume that the reference motion is already physically compatible with the
robot's surroundings. This assumption breaks when the demonstrator, operator, and
robot inhabit different environments: a human motion may specify the intended behavior, but not the footholds, clearance, body height, or contact timing required by the
robot's local terrain. We introduce Perceptive Behavior Foundation Model (Perceptive BFM), a terrain-aware
humanoid control framework that grounds human motion priors in
robot-centric perception. The model preserves raw kinematic motion references as the behavioral interface, while using local terrain observations to adapt contacts, posture, and timing. To provide scalable terrain supervision, we develop terrain-conformal reference synthesis (TCRS), which converts locomotion-oriented human motion clips into terrain-consistent references through contact-aware foothold construction, foot-geometry-aware swing optimization, support-aware root reconstruction, collision repair, and multi-point inverse kinematics. We then train a blind adapted-reference teacher and transfer its terrain-conformal behavior to a deployed raw-reference student through target-frame action
alignment. The student is an identity-gated
Transformer tracker whose terrain features enter through residual pathways initialized to preserve the motion-tracking prior and trained to produce local corrections only when needed.
Read the full abstract ▾
All authors Zifan Wang, Yizhao Li, Teli Ma, Qiang Zhang, Yudong Fan, Hao Xu, Shuo Yang, Junwei Liang
Robotics
EgoAERO: Learning Dexterous Manipulation from a Single Egocentric Video without Object Assets
Yichen Niu, Haoran Lv, Xinrui Zhang et al. (+12 more)
Egocentric RGB-D
videos offer a natural source of human dexterous manipulation demonstrations, but existing data is difficult to use for
robot learning because object pose, geometry, and contact information are often missing or require pre-scanned object assets. We present EgoAERO, the first framework that learns dexterous manipulation from a single
egocentric RGB-D human demonstration without object assets. EgoAERO reconstructs contact-consistent hand-object trajectories through asset-free object tracking and reconstruction, ego motion compensation, and adaptive contact optimization, then converts them into
robot policies using two-stage residual learning. We further introduce an online quality assessment mechanism and construct EgoDex-R, a large-scale
egocentric dataset with 4.3M RGB-D frames for dexterous policy learning. Simulation and real-world experiments show that EgoAERO enables single-demonstration dexterous manipulation and achieves downstream performance close to CAD-based reconstructions on HOI4D.
Read the full abstract ▾
All authors Yichen Niu, Haoran Lv, Xinrui Zhang, Xueyao Wan, Shiyu Gao, Ying Ai, Hui Xu, Yongqi Hu, Hengyi Zhang, Yang Xie, Zhaxizhuoma, Yue Zhao, Zhenshan Bing, Yan Ding, Jianxing Liu
Robotics
MuJoCo-Drones-Gym: A GPU-Accelerated Multi-Drone Simulator for Control and Reinforcement Learning
Manan Tayal
Robotic simulators are a cornerstone of modern research in aerial robotics, serving both as a vehicle for the development of new control
algorithms and as the data source for training reinforcement learning (RL) policies. Yet, existing quadcopter learning environments often face a trade-off between physical fidelity, multi-
agent support, and the throughput required by modern deep RL pipelines. In this paper, we present MuJoCo-Drones-Gym, an open-source Gymnasium-compatible multi-drone environment built on top of the MuJoCo physics engine. MuJoCo-Drones-Gym supports an arbitrary number of Bitcraze Crazyflie 2.x nano-quadcopters and exposes a modular API for selecting (i)~the physics model (rigid-body MuJoCo, explicit Python dynamics, or any subset of ground effect, blade drag, and inter-drone downwash), (ii)~the action interface (per-motor RPMs, collective normalized thrust, velocity setpoints, or PID waypoint commands), and (iii)~the observation space (kinematic state vectors, RGB / depth /
segmentation cameras, or neighbourhood adjacency information). A PettingZoo ParallelEnv wrapper enables drop-in multi-
agent reinforcement learning, while a suite of seven task environments, hover, velocity tracking, multi-drone hover, waypoint
navigation, formation flight, gate racing, and a generic multi-
agent template, demonstrates the breadth of the interface. We describe the environment design, the underlying physics and quadcopter dynamics, and illustrate its use through control and learning examples that mirror those of the closely related gym-pybullet-drones project, while taking advantage of MuJoCo's improved contact handling, rendering, and parallelizability.
Read the full abstract ▾
Robotics
IntentNav: Learning Spatial-Visual Object Navigation from Human Demonstrations
Yuxin Cai, Zongtai Li, Maonan Wang et al. (+9 more)
Object
navigation requires a
robot to search for an unobserved target in an unknown environment by deciding where to explore next under partial observability. Effective search resembles human-like exploration: selectively probing visually promising frontiers while relying on spatial
memory to avoid redundant revisits. We propose IntentNav, a spatial-visual imitation framework that learns human-like ObjectNav policies from human demonstrations. To infer high-level search intent from low-level human actions, we introduce Frontier-based Human-Intent Labeling, which looks ahead in human demonstrations and labels the frontier that best explains the demonstrator's future search direction. We construct a spatial-visual candidate space, where BEV
memory tracks explored regions, unexplored frontiers, and trajectory history, while
egocentric visual
memory provides semantic cues for each candidate. A VLM policy is trained to select among these grounded candidates, using Intent-Aligned Objective to encourage consistent and human-like exploration. IntentNav achieves state-of-the-art performance on the MP3D, HM3D-v1 and HM3D-v2 ObjectNav
benchmarks. The proposed candidate-level
navigation interface transfers zero-shot to wheeled, quadruped, and
humanoid robots without further VLM fine-tuning. \href{https://anonymous.4open.science/w/IntentNav/}{Project page}.
Read the full abstract ▾
All authors Yuxin Cai, Zongtai Li, Maonan Wang, Muyi Bao, Haokun Zhu, Ruofei Bai, Ding Zhao, Zirui Li, Wenshan Wang, Wei-Yun Yau, Ji Zhang, Chen Lv
Robotics
Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policies
Ziqian Wang, Rui Zhang, Yitian Liu et al. (+3 more)
We propose Q-Guided Value-Gradient Matching (Q-VGM), an offline-to-online RL method for fine-tuning flow-matching
vision-
language-action policies with a learned critic. Directly applying critic gradients to flow policies requires backpropagation through the multi-step denoising process (BPTT), which is costly and unstable at VLA scale. Motivated by an optimal-control view of denoising, we derive a local approximation connecting clean-action value gradients to local velocity corrections at late denoising steps. Q-VGM uses this connection to construct critic-guided targets for fine-tuning the pretrained action expert through velocity matching. Gradients pass only through local velocity predictions, avoiding backpropagation through the denoising trajectory. An action-sensitive chunk critic is trained with IQL offline and TD learning online. On LIBERO, Q-VGM improves the few-shot-SFT four-suite average from 84.6% to 90.7% offline using 150 policy rollouts per suite, and reaches 98.4% after online training. Compared with on-policy RL, Q-VGM achieves 5.3x higher sample efficiency on average across four LIBERO suites, measured by the episode budget for 95% rollout success rate. On three real-world bimanual tasks, offline Q-VGM improves average success from 66.7% to 98.3%.
Read the full abstract ▾
All authors Ziqian Wang, Rui Zhang, Yitian Liu, Xingjian Mao, Minqian Wang, Yao Mu
Robotics
PRISM: PRior-guided Imagination Sampling in world Models
Yuhai Wang, Jiawei Xia, Rongxuan Zhou et al. (+4 more)
A learned world model provides a powerful physical intuition for evaluating future states. But its effectiveness in continuous control also depends critically on how candidate actions are generated for model-based planning. Rather than solely asking how accurately a model can simulate the future, we ask: which candidate actions are worth evaluating in the first place? Existing planners typically search arbitrarily or use expert demonstrations only to initialize a sampling mean, discarding the expert's state-conditioned confidence. Properly guiding this search requires a robust action prior, yet current approaches often rely on independent visual encoders or large-scale VLMs to obtain one. We argue that this architectural bloat is unnecessary: the exact same data - and the learned representations of the world model itself - inherently encode the
agent's action intuition. We introduce PRISM, a task-agnostic framework that extracts both from a single
dataset while maintaining strict architectural simplicity. Building on a standard JEPA-style latent world model, PRISM attaches a lightweight MLP directly to its frozen encoder to predict a state-conditioned Gaussian prior. At plan time, PRISM fuses this prior into the planner's sampling distribution via a precision-weighted Product-of-Gaussians update. This parameter-free, closed-form integration steers the sampling process, making the prior confident where it is and ceding control where it is not. PRISM improves success rates by 35 percentage points over vanilla world-model-based MPC on Cube and 32 percentage points on PushT, without introducing significant inference overhead.
Read the full abstract ▾
All authors Yuhai Wang, Jiawei Xia, Rongxuan Zhou, Xiao Hu, Yongliang Shi, Jing Du, Yang Ye
Robotics
X-OP: Cross-Morphology Whole-Body Teleoperation via MPC Retargeting
Jen-Wei Wang, Sarthak Kaingade, Andrea Tagliabue et al. (+1 more)
Whole-body teleoperation is essential for scalable
robot data collection in loco-manipulation tasks, yet existing approaches relying on exoskeleton suits or multi-camera setups impose prohibitive cost, complexity, and environmental constraints. Recent methods using a single extended reality (XR) device with end-to-end reinforcement learning policies partially address these limitations but require
robot-specific retraining, suffer from out-of-distribution failures, and rely on motion retargeting that neglects dynamic feasibility. We propose a hierarchical whole-body teleoperation framework driven by a single XR device that generalizes across diverse
robot morphologies without retraining
robot-specific policies. A Model Predictive Control (MPC)-based motion retargeter jointly optimizes
alignment with the operator's intent and the
robot's dynamic feasibility, generating optimal commands for existing low-level controllers. To ensure robust online execution, we introduce a state synchronization method that resets the simulator state at each MPC step to handle noisy real-world measurements and contact sensitivity, and integrate SLAM-based global pose feedback to mitigate long-term drift. Simulation results show higher success rates on whole-body control tasks for both a
humanoid (over 30% lower completion time and 20% lower power consumption) and a mobile manipulator (zero collisions) compared to baselines. Real-world experiments further validate the effectiveness and flexibility of our method, demonstrating the successful deployment of the proposed retargeter on both platforms for whole-body control tasks and the ease of allowing users to adjust teleoperation behavior based on their preferences. This plug-and-play framework offers a scalable, morphology-agnostic solution for whole-body
robot teleoperation, enabling real-time behavioral customization and broad applicability across platforms.
Read the full abstract ▾
All authors Jen-Wei Wang, Sarthak Kaingade, Andrea Tagliabue, Nicholas Morozovsky