Robotics
How to Peel with a Knife: Aligning Fine-Grained Manipulation with Human Preference
Toru Lin, Shuying Deng, Zhao-Heng Yin et al. (+2 more)
Many essential manipulation tasks - such as food preparation, surgery, and craftsmanship - remain intractable for autonomous
robots. These tasks are characterized not only by contact-rich, force-sensitive dynamics, but also by their "implicit" success criteria: unlike pick-and-place, task quality in these domains is continuous and subjective (e.g. how well a potato is peeled), making quantitative evaluation and reward engineering difficult. We present a learning framework for such tasks, using peeling with a knife as a representative example. Our approach follows a two-stage pipeline: first, we learn a robust initial policy via force-aware data collection and imitation learning, enabling generalization across object variations; second, we refine the policy through preference-based finetuning using a learned reward model that combines quantitative task metrics with qualitative human feedback, aligning policy behavior with human notions of task quality. Using only 50-200 peeling trajectories, our system achieves over 90% average success rates on challenging produce including cucumbers, apples, and potatoes, with performance improving by up to 40% through preference-based finetuning. Remarkably, policies trained on a single produce category exhibit strong zero-shot generalization to unseen in-category instances and to out-of-distribution produce from different categories while maintaining over 90% success rates.
Read the full abstract ▾
All authors Toru Lin, Shuying Deng, Zhao-Heng Yin, Pieter Abbeel, Jitendra Malik
Robotics
ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation
Xialin He, Sirui Xu, Xinyao Li et al. (+4 more)
Achieving autonomous and versatile whole-body loco-manipulation remains a central barrier to making
humanoids practically useful. Yet existing approaches are fundamentally constrained: retargeted data are often scarce or low-quality; methods struggle to scale to large skill repertoires; and, most importantly, they rely on tracking predefined motion references rather than generating behavior from perception and high-level task specifications. To address these limitations, we propose ULTRA, a unified framework with two key components. First, we introduce a physics-driven neural retargeting
algorithm that translates large-scale motion capture to
humanoid embodiments while preserving physical plausibility for contact-rich interactions. Second, we learn a unified
multimodal controller that supports both dense references and sparse task specifications, under sensing ranging from accurate motion-capture state to noisy
egocentric visual inputs. We distill a universal tracking policy into this controller, compress motor skills into a compact latent space, and apply reinforcement learning finetuning to expand coverage and improve robustness under out-of-distribution scenarios. This enables coordinated whole-body behavior from sparse intent without test-time reference motions. We evaluate ULTRA in simulation and on a real Unitree G1
humanoid. Results show that ULTRA generalizes to autonomous, goal-conditioned whole-body loco-manipulation from
egocentric perception, consistently outperforming tracking-only baselines with limited skills.
Read the full abstract ▾
All authors Xialin He, Sirui Xu, Xinyao Li, Runpei Dong, Liuyu Bian, Yu-Xiong Wang, Liang-Yan Gui
Robotics
Tether: Autonomous Functional Play with Correspondence-Driven Trajectory Warping
William Liang, Sam Wang, Hung-Ju Wang et al. (+3 more)
The ability to conduct and learn from interaction and experience is a central challenge in robotics, offering a scalable alternative to labor-intensive human demonstrations. However, realizing such "play" requires (1) a policy robust to diverse, potentially out-of-distribution environment states, and (2) a procedure that continuously produces useful
robot experience. To address these challenges, we introduce Tether, a method for autonomous functional play involving structured, task-directed interactions. First, we design a novel open-loop policy that warps actions from a small set of source demonstrations (<=10) by anchoring them to semantic keypoint correspondences in the target scene. We show that this design is extremely data-efficient and robust even under significant spatial and semantic variations. Second, we deploy this policy for autonomous functional play in the real world via a continuous cycle of task selection, execution, evaluation, and improvement, guided by the visual understanding capabilities of
vision-
language models. This procedure generates diverse, high-quality
datasets with minimal human intervention. In a household-like multi-object setup, our method is the first to perform many hours of autonomous multi-task play in the real world starting from only a handful of demonstrations. This produces a stream of data that consistently improves the performance of closed-loop imitation policies over time, ultimately yielding over 1000 expert-level trajectories and training policies competitive with those learned from human-collected demonstrations.
Read the full abstract ▾
All authors William Liang, Sam Wang, Hung-Ju Wang, Osbert Bastani, Yecheng Jason Ma, Dinesh Jayaraman
Robotics
HoMMI: Learning Whole-Body Mobile Manipulation from Human Demonstrations
Xiaomeng Xu, Jisang Park, Han Zhang et al. (+6 more)
We present Whole-Body Mobile Manipulation Interface (HoMMI), a data collection and policy learning framework that learns whole-body mobile manipulation directly from
robot-free human demonstrations. We augment UMI interfaces with
egocentric sensing to capture the global context required for mobile manipulation, enabling portable,
robot-free, and scalable data collection. However, naively incorporating
egocentric sensing introduces a larger human-to-
robot embodiment gap in both observation and action spaces, making policy transfer difficult. We explicitly bridge this gap with a cross-embodiment hand-eye policy design, including an embodiment agnostic visual representation; a relaxed head action representation; and a whole-body controller that realizes hand-eye trajectories through coordinated whole-body motion under
robot-specific physical constraints. Together, these enable long-horizon mobile manipulation tasks requiring bimanual and whole-body coordination,
navigation, and active perception. Results are best viewed on: https://hommi-
robot.github.io
Read the full abstract ▾
All authors Xiaomeng Xu, Jisang Park, Han Zhang, Eric Cousineau, Aditya Bhat, Jose Barreiros, Dian Wang, Jeannette Bohg, Shuran Song
Robotics
ACE-Brain-0: Spatial Intelligence as a Shared Scaffold for Universal Embodiments
Ziyang Gong, Zehang Luo, Anke Tang et al. (+21 more)
Universal
embodied intelligence demands robust generalization across heterogeneous embodiments, such as autonomous driving, robotics, and unmanned aerial vehicles (
UAVs). However, existing
embodied brain in training a unified model over diverse embodiments frequently triggers long-tail data, gradient interference, and catastrophic forgetting, making it notoriously difficult to balance universal generalization with domain-specific proficiency. In this report, we introduce ACE-
Brain-0, a generalist foundation
brain that unifies spatial
reasoning, autonomous driving, and
embodied manipulation within a single
multimodal large
language model~(MLLM). Our key insight is that spatial intelligence serves as a universal scaffold across diverse physical embodiments: although vehicles,
robots, and
UAVs differ drastically in morphology, they share a common need for modeling 3D mental space, making spatial cognition a natural, domain-agnostic foundation for cross-embodiment transfer. Building on this insight, we propose the Scaffold-Specialize-Reconcile~(SSR) paradigm, which first establishes a shared spatial foundation, then cultivates domain-specialized experts, and finally harmonizes them through data-free model merging. Furthermore, we adopt Group Relative Policy Optimization~(GRPO) to strengthen the model's comprehensive capability. Extensive experiments demonstrate that ACE-
Brain-0 achieves competitive and even state-of-the-art performance across 24 spatial and embodiment-related
benchmarks.
Read the full abstract ▾
All authors Ziyang Gong, Zehang Luo, Anke Tang, Zhe Liu, Shi Fu, Zhi Hou, Ganlin Yang, Weiyun Wang, Xiaofeng Wang, Jianbo Liu, Gen Luo, Haolan Kang, Shuang Luo, Yue Zhou, Yong Luo, Li Shen, Xiaosong Jia, Yao Mu, Xue Yang, Chunxiao Liu, Junchi Yan, Hengshuang Zhao, Dacheng Tao, Xiaogang Wang
Robotics · Cross-listed from cs.CV
Chain of World: World Model Thinking in Latent Motion
Fuxiang Yang, Donglin Di, Lulu Tang et al. (+6 more)
Vision-
Language-Action (VLA) models are a promising path toward
embodied intelligence, yet they often overlook the predictive and temporal-causal structure underlying visual dynamics. World-model VLAs address this by predicting future frames, but waste capacity reconstructing redundant backgrounds. Latent-action VLAs encode frame-to-frame transitions compactly, but lack temporally continuous dynamic modeling and world knowledge. To overcome these limitations, we introduce CoWVLA (Chain-of-World VLA), a new "Chain of World" paradigm that unifies world-model temporal
reasoning with a disentangled latent motion representation. First, a pretrained
video VAE serves as a latent motion extractor, explicitly factorizing
video segments into structure and motion latents. Then, during pre-training, the VLA learns from an instruction and an initial frame to infer a continuous latent motion chain and predict the segment's terminal frame. Finally, during co-fine-tuning, this latent dynamic is aligned with discrete action prediction by jointly modeling sparse keyframes and action sequences in a unified autoregressive decoder. This design preserves the world-model benefits of temporal
reasoning and world knowledge while retaining the compactness and interpretability of latent actions, enabling efficient visuomotor learning. Extensive experiments on robotic simulation
benchmarks show that CoWVLA outperforms existing world-model and latent-action approaches and achieves moderate computational efficiency, highlighting its potential as a more effective VLA pretraining paradigm. The project website can be found at https://fx-hit.github.io/cowvla-io.
Read the full abstract ▾
All authors Fuxiang Yang, Donglin Di, Lulu Tang, Xuancheng Zhang, Lei Fan, Hao Li, Chen Wei, Tonghua Su, Baorui Ma
Robotics
Robotic Grasping and Placement Controlled by EEG-Based Hybrid Visual and Motor Imagery
Yichang Liu, Tianyu Wang, Ziyi Ye et al. (+4 more)
We present a framework that integrates
EEG-based visual and motor imagery (VI/MI) with robotic control to enable real-time, intention-driven grasping and placement. Motivated by the promise of BCI-driven robotics to enhance human-
robot interaction, this system bridges neural signals with physical control by deploying offline-pretrained decoders in a zero-shot manner within an online streaming pipeline. This establishes a dual-channel intent interface that translates visual intent into robotic actions, with VI identifying objects for grasping and MI determining placement poses, enabling intuitive control over both what to grasp and where to place. The system operates solely on
EEG via a cue-free imagery protocol, achieving integration and online validation. Implemented on a Base robotic platform and evaluated across diverse scenarios, including occluded targets or varying participant postures, the system achieves online decoding accuracies of 40.23% (VI) and 62.59% (MI), with an end-to-end task success rate of 20.88%. These results demonstrate that high-level visual cognition can be decoded in real time and translated into executable
robot commands, bridging the gap between neural signals and physical interaction, and validating the flexibility of a purely imagery-based BCI paradigm for practical human-
robot collaboration.
Read the full abstract ▾
All authors Yichang Liu, Tianyu Wang, Ziyi Ye, Yawei Li, Yu-Gang Jiang, Shouyan Wang, Yanwei Fu
Robotics
From Language to Action: Can LLM-Based Agents Be Used for Embodied Robot Cognition?
Shinas Shaji, Fabian Huppertz, Alex Mitrevski et al. (+1 more)
In order to flexibly act in an everyday environment, a robotic
agent needs a variety of cognitive capabilities that enable it to reason about plans and perform execution recovery. Large
language models (LLMs) have been shown to demonstrate emergent cognitive aspects, such as
reasoning and
language understanding; however, the ability to control
embodied robotic
agents requires reliably bridging high-level
language to low-level functionalities for perception and control. In this paper, we investigate the extent to which an LLM can serve as a core component for planning and execution
reasoning in a cognitive
robot architecture. For this purpose, we propose a cognitive architecture in which an agentic LLM serves as the core component for planning and
reasoning, while components for working and episodic
memories support learning from experience and adaptation. An instance of the architecture is then used to control a mobile manipulator in a simulated household environment, where environment interaction is done through a set of high-level tools for perception,
reasoning,
navigation, grasping, and placement, all of which are made available to the LLM-based
agent. We evaluate our proposed system on two household tasks (object placement and object swapping), which evaluate the
agent's
reasoning, planning, and
memory utilisation. The results demonstrate that the LLM-driven
agent can complete structured tasks and exhibits emergent adaptation and
memory-guided planning, but also reveal significant limitations, such as
hallucinations about the task success and poor instruction following by refusing to acknowledge and complete sequential tasks. These findings highlight both the potential and challenges of employing LLMs as
embodied cognitive controllers for autonomous
robots.
Read the full abstract ▾
All authors Shinas Shaji, Fabian Huppertz, Alex Mitrevski, Sebastian Houben
Robotics
Look Forward to Walk Backward: Efficient Terrain Memory for Backward Locomotion with Forward Vision
Shixin Luo, Songbo Li, Yuan Hao et al. (+4 more)
Legged
robots with
egocentric forward-facing depth cameras can couple exteroception and proprioception to achieve robust forward agility on complex terrain. When these
robots walk backward, the forward-only field of view provides no preview. Purely proprioceptive controllers can remain stable on moderate ground when moving backward but cannot fully exploit the
robot's capabilities on complex terrain and must collide with obstacles. We present Look Forward to Walk Backward (LF2WB), an efficient terrain-
memory locomotion framework that uses forward
egocentric depth and proprioception to write a compact associative
memory during forward motion and to retrieve it for collision-free backward locomotion without rearward
vision. The
memory backbone employs a delta-rule selective update that softly removes then writes the
memory state along the active subspace. Training uses hardware-efficient parallel computation, and deployment runs recurrent, constant-time per-step inference with a constant-size state, making the approach suitable for onboard processors on low-cost
robots. Experiments in both simulations and real-world scenarios demonstrate the effectiveness of our method, improving backward agility across complex terrains under limited sensing.
Read the full abstract ▾
All authors Shixin Luo, Songbo Li, Yuan Hao, Yaqi Wang, Jun Zheng, Jun Wu, Qiuguo Zhu
Robotics
RL-Based Coverage Path Planning for Deformable Objects on 3D Surfaces
Yuhang Zhang, Jinming Ma and Feng Wu
Currently, manipulation tasks for deformable objects often focus on activities like folding clothes, handling ropes, and manipulating bags. However, research on contact-rich tasks involving deformable objects remains relatively underdeveloped. When humans use cloth or sponges to wipe surfaces, they rely on both
vision and tactile feedback. Yet, current
algorithms still face challenges with issues like occlusion, while research on tactile perception for manipulation is still evolving. Tasks such as covering surfaces with deformable objects demand not only perception but also precise robotic manipulation. To address this, we propose a method that leverages efficient and accessible simulators for task execution. Specifically, we train a reinforcement learning
agent in a simulator to manipulate deformable objects for surface wiping tasks. We simplify the state representation of object surfaces using harmonic UV mapping, process contact feedback from the simulator on 2D feature maps, and use scaled grouped convolutions (SGCNN) to extract features efficiently. The
agent then outputs actions in a reduced-dimensional action space to generate coverage paths. Experiments demonstrate that our method outperforms previous approaches in key metrics, including total path length and coverage area. We deploy these paths on a Kinova Gen3 manipulator to perform wiping experiments on the back of a torso model, validating the feasibility of our approach.
Read the full abstract ▾
Robotics
CMoE: Contrastive Mixture of Experts for Motion Control and Terrain Adaptation of Humanoid Robots
Shihao Ma, Hongjin Chen, Zijun Xu et al. (+6 more)
For effective deployment in real-world environments,
humanoid robots must autonomously navigate a diverse range of complex terrains with abrupt transitions. While the Vanilla mixture of experts (MoE) framework is theoretically capable of modeling diverse terrain features, in practice, the gating network exhibits nearly uniform expert activations across different terrains, weakening the expert specialization and limiting the model's expressive power. To address this limitation, we introduce CMoE, a novel single-stage reinforcement learning framework that integrates contrastive learning to refine expert activation distributions. By imposing contrastive constraints, CMoE maximizes the consistency of expert activations within the same terrain while minimizing their similarity across different terrains, thereby encouraging experts to specialize in distinct terrain types. We validated our approach on the Unitree G1
humanoid robot through a series of challenging experiments. Results demonstrate that CMoE enables the
robot to traverse continuous steps up to 20 cm high and gaps up to 80 cm wide, while achieving robust and natural gait across diverse mixed terrains, surpassing the limits of existing methods. To support further research and foster community development, we release our code publicly.
Read the full abstract ▾
All authors Shihao Ma, Hongjin Chen, Zijun Xu, Yi Zhao, Ke Wu, Ruichen Yang, Leyao Zou, Zhongxue Gan, Wenchao Ding
Robotics
Architectural HRI: Towards a Robotic Paradigm Shift in Human-Building Interaction
Alex Binh Vinh Duc Nguyen
Recent advances in sensing, communication, interfaces, control, and robotics are expanding Human-Building Interaction (HBI) beyond adaptive building services and facades toward the physical actuation of architectural space. In parallel, research in robotic furniture, swarm robotics, and shape-changing spaces shows that architectural elements can now be robotically augmented to move, reconfigure, and adapt space. We propose that these advances promise a paradigm shift in HBI, in which multiple building layers physically adapt in synchrony to support occupant needs and sustainability goals more holistically. Conversely, we argue that this emerging paradigm also provides an ideal case for transferring HRI knowledge to unconventional robotic morphologies, including the interpretation of the
robot as multiple architectural layers or even as a building. However, this research agenda remains challenged by the temporal, spatial, and social complexity of architectural HRI, and by fragmented knowledge across HCI, environmental psychology, cognitive science, and architecture. We therefore call for interdisciplinary research that unifies the why, what, and how of robotic actuation in architectural forms.
Read the full abstract ▾
Robotics
MA-CoNav: A Master-Slave Multi-Agent Framework with Hierarchical Collaboration and Dual-Level Reflection for Long-Horizon Embodied VLN
Ling Luo and Qianqian Bai
Vision-
Language Navigation (VLN) aims to empower
robots with the ability to perform long-horizon
navigation in unfamiliar environments based on complex linguistic instructions. Its success critically hinges on establishing an efficient “
language-understanding – visual-perception –
embodied-execution” closed loop. Existing methods often suffer from perceptual distortion and decision drift in complex, long-distance tasks due to the cognitive overload of a single
agent. Inspired by distributed cognition theory, this paper proposes MA-CoNav, a Multi-
Agent Collaborative
Navigation framework. This framework adopts a “Master-Slave” hierarchical
agent collaboration architecture, decoupling and distributing the perception, planning, execution, and
memory functions required for
navigation tasks to specialized
agents. Specifically, the Master
Agent is responsible for global orchestration, while the Subordinate
Agent group collaborates through a clear division of labor: an Observation
Agent generates environment descriptions, a Planning
Agent performs task decomposition and dynamic verification, an Execution
Agent handles simultaneous mapping and action, and a
Memory Agent manages structured experiences. Furthermore, the framework introduces a “Local-Global” dual-stage reflection mechanism to dynamically optimize the entire
navigation pipeline. Empirical experiments were conducted using a real-world indoor
dataset collected by a Limo Pro
robot, with no scene-specific fine-tuning performed on the models throughout the process. The results demonstrate that MA-CoNav comprehensively outperforms existing mainstream VLN methods across multiple metrics.
Read the full abstract ▾
Robotics
CASSR: Continuous A-Star Search through Reachability for real time footstep planning
Jiayi Wang and Steve Tonneau
Footstep planning involves a challenging combinatorial search. Traditional A* approaches require discretising reachability constraints, while Mixed-Integer Programming (MIP) supports continuous formulations but quickly becomes intractable, especially when rotations are included. We present CASSR, a novel framework that recursively propagates convex, continuous formulations of a
robot's kinematic constraints within an A* search. Combined with a new cost-to-go heuristic based on the EPA
algorithm, CASSR efficiently plans contact sequences of up to 30 footsteps in under 125 ms. Experiments on biped locomotion tasks demonstrate that CASSR outperforms traditional discretised A* by up to a factor of 100, while also surpassing a commercial MIP solver. These results show that CASSR enables fast, reliable, and real-time footstep planning for biped
robots.
Read the full abstract ▾
Robotics
DreamFlow: Local Navigation Beyond Observation via Conditional Flow Matching in the Latent Space
Jiwon Park, Dongkyu Lee, I Made Aswin Nahrendra et al. (+2 more)
Local
navigation in cluttered environments often suffers from dense obstacles and frequent local minima. Conventional local planners rely on heuristics and are prone to failure, while deep reinforcement learning(DRL)based approaches provide adaptability but are constrained by limited onboard sensing. These limitations lead to
navigation failures because the
robot cannot perceive structures outside its field of view. In this paper, we propose DreamFlow, a DRL-based local
navigation framework that extends the
robot's perceptual horizon through conditional flow matching(CFM). The proposed CFM based prediction module learns probabilistic mapping between local height map latent representation and broader spatial representation conditioned on
navigation context. This enables the
navigation policy to predict unobserved environmental features and proactively avoid potential local minima. Experimental results demonstrate that DreamFlow outperforms existing methods in terms of latent prediction accuracy and
navigation performance in simulation. The proposed method was further validated in cluttered real world environments with a quadrupedal
robot. The project page is available at https://dreamflow-icra.github.io.
Read the full abstract ▾
All authors Jiwon Park, Dongkyu Lee, I Made Aswin Nahrendra, Jaeyoung Lim, Hyun Myung
Robotics · Cross-listed from cs.CV
Jiaxing Liu, Zexi Zhang, Xiaoyan Li et al. (+3 more)
Vision-
Language Navigation (VLN) presents a unique challenge for Large
Vision-
Language Models (VLMs) due to their inherent architectural mismatch: VLMs are primarily pretrained on static, disembodied
vision-
language tasks, which fundamentally clash with the dynamic,
embodied, and spatially-structured nature of
navigation. Existing large-model-based methods often resort to converting rich visual and spatial information into text, forcing models to implicitly infer complex visual-topological relationships or limiting their global action capabilities. To bridge this gap, we propose TagaVLM (
Topology-Aware Global Action
reasoning), an end-to-end framework that explicitly injects topological structures into the VLM backbone. To introduce topological edge information, Spatial
Topology Aware Residual
Attention (
STAR-Att) directly integrates it into the VLM's self-
attention mechanism, enabling intrinsic spatial
reasoning while preserving pretrained knowledge. To enhance topological node information, an Interleaved
Navigation Prompt strengthens node-level visual-text
alignment. Finally, with the embedded topological graph, the model is capable of global action
reasoning, allowing for robust path correction. On the R2R
benchmark, TagaVLM achieves state-of-the-art performance among large-model-based methods, with a Success Rate (SR) of 51.09% and SPL of 47.18 in unseen environments, outperforming prior work by 3.39% in SR and 9.08 in SPL. This demonstrates that, for
embodied spatial
reasoning, targeted enhancements on smaller open-source VLMs can be more effective than brute-force model scaling. The code can be found on our project page: https://apex-bjut.github.io/Taga-VLM
Read the full abstract ▾
All authors Jiaxing Liu, Zexi Zhang, Xiaoyan Li, Boyue Wang, Yongli Hu, Baocai Yin
Robotics
Self-supervised Domain Adaptation for Visual 3D Pose Estimation of Nano-drone Racing Gates by Enforcing Geometric Consistency
Nicholas Carlotti, Michele Antonazzi, Elia Cereda et al. (+4 more)
We consider the task of visually estimating the relative pose of a drone racing gate in front of a nano-quadrotor, using a convolutional neural network pre-trained on simulated data to regress the gate's pose. Due to the sim-to-real gap, the pre-trained model underperforms in the real world and must be adapted to the target domain. We propose an unsupervised domain adaptation (UDA) approach using only real image sequences collected by the drone flying an arbitrary trajectory in front of a gate; sequences are annotated in a self-supervised fashion with the drone's odometry as measured by its onboard sensors. On this
dataset, a state consistency loss enforces that two images acquired at different times yield pose predictions that are consistent with the drone's odometry. Results indicate that our approach outperforms other SoA UDA approaches, has a low mean absolute error in position (x=26, y=28, z=10 cm) and orientation ($\psi$=13${^{\circ}}$), an improvement of 40% in position and 37% in orientation over a baseline. The approach's effectiveness is appreciable with as few as 10 minutes of real-world flight data and yields models with an inference time of 30.4ms (33 fps) when deployed aboard the Crazyflie 2.1 Brushless nano-drone.
Read the full abstract ▾
All authors Nicholas Carlotti, Michele Antonazzi, Elia Cereda, Mirko Nava, Nicola Basilico, Daniele Palossi, Alessandro Giusti
Robotics
Tracing Back Error Sources to Explain and Mitigate Pose Estimation Failures
Loris Schneider, Yitian Shi, Rosa Wolf et al. (+3 more)
Robust estimation of object poses in robotic manipulation is often addressed using foundational general
estimators, that aim to handle diverse error sources naively within a single model. Still, they struggle due to environmental uncertainties, while requiring long inference times and heavy computation. In contrast, we propose a modular, uncertainty-aware framework that attributes pose estimation errors to specific error sources and applies targeted mitigation strategies only when necessary. Instantiated with Iterative Closest Point (ICP) as a simple and lightweight pose
estimator, we leverage our framework for real-world robotic grasping tasks. By decomposing pose estimation into failure detection, error attribution, and targeted recovery, we significantly improve the robustness of ICP and achieve competitive performance compared to foundation models, while relying on a substantially simpler and faster pose
estimator.
Read the full abstract ▾
All authors Loris Schneider, Yitian Shi, Rosa Wolf, Carolin Brenner, Rudolph Triebel, Rania Rayyes
Robotics
Emerging trends in Cislunar Space for Lunar Science Exploration and Space Robotics aiding Human Spaceflight Safety
Arsalan Muhammad, Yue Wang, Hai Huang et al. (+1 more)
In recent years, the Moon has emerged as an unparalleled extraterrestrial testbed for advancing cuttingedge technological and scientific research critical to enabling sustained human presence on its surface and supporting future interplanetary exploration. This study identifies and investigates two pivotal research domains with substantial transformative potential for accelerating humanity interplanetary aspirations. First is Lunar Science Exploration with Artificial Intelligence and Space Robotics which focusses on AI and Space Robotics redefining the frontiers of space exploration. Second being Space Robotics aid in manned spaceflight to the Moon serving as critical assets for pre-deployment infrastructure development, In-Situ Resource Utilization, surface operations support, and astronaut safety assurance. By integrating autonomy, machine learning, and realtime sensor fusion, space robotics not only augment human capabilities but also serve as force multipliers in achieving sustainable lunar exploration, paving the way for future crewed missions to Mars and beyond.
Read the full abstract ▾
All authors Arsalan Muhammad, Yue Wang, Hai Huang, Hao Wang
Robotics
Rhythm: Learning Interactive Whole-Body Control for Dual Humanoids
Hongjin Chen, Wei Zhang, Pengfei Li et al. (+10 more)
Realizing interactive whole-body control for multi-
humanoid systems is critical for unlocking complex collaborative capabilities in shared environments. Although recent advancements have significantly enhanced the agility of individual
robots, bridging the gap to physically coupled multi-
humanoid interaction remains challenging, primarily due to severe kinematic mismatches and complex contact dynamics. To address this, we introduce Rhythm, the first unified framework enabling real-world deployment of dual-
humanoid systems for complex, physically plausible interactions. Our framework integrates three core components: (1) an Interaction-Aware Motion Retargeting (IAMR) module that generates feasible
humanoid interaction references from human data; (2) an Interaction-Guided Reinforcement Learning (IGRL) policy that masters coupled dynamics via graph-based rewards; and (3) a real-world deployment system that enables robust transfer of dual-
humanoid interaction. Extensive experiments on physical Unitree G1
robots demonstrate that our framework achieves robust interactive whole-body control, successfully transferring diverse behaviors such as hugging and dancing from simulation to reality.
Read the full abstract ▾
All authors Hongjin Chen, Wei Zhang, Pengfei Li, Shihao Ma, Ke Ma, Yujie Jin, Zijun Xu, Xiaohui Wang, Yupeng Zheng, Zining Wang, Jieru Zhao, Yilun Chen, Wenchao Ding
Robotics
CoFL: Continuous Flow Fields for Language-Conditioned Navigation
Haokun Liu, Zhaoqi Ma, Yicheng Chen et al. (+5 more)
Existing
language-conditioned
navigation systems typically rely on modular pipelines or trajectory generators, but the latter use each scene–instruction annotation mainly to supervise one start-conditioned rollout. To address these limitations, we present CoFL, an end-to-end policy that maps a bird's-eye view (BEV) observation and a
language instruction to a continuous flow field for
navigation. CoFL reformulates
navigation as workspace-conditioned field learning rather than start-conditioned trajectory prediction: it learns local motion vectors at arbitrary BEV locations, turning each scene–instruction annotation into dense spatial control supervision. Trajectories are generated from any start by numerical integration of the predicted field, enabling simple real-time rollout and closed-loop recovery. To enable large-scale training and evaluation, we build a
dataset of over 500k BEV image–instruction pairs, each procedurally annotated with a flow field and a trajectory derived from semantic maps built on Matterport3D and ScanNet. Evaluating on strictly unseen scenes, CoFL significantly outperforms modular
Vision-
Language Model (VLM)-based planners and trajectory generation policies in both
navigation precision and safety, while maintaining real-time inference. Finally, we deploy CoFL zero-shot in real-world experiments with BEV observations across multiple layouts, maintaining feasible closed-loop control and a high success rate.
Read the full abstract ▾
All authors Haokun Liu, Zhaoqi Ma, Yicheng Chen, Masaki Kitagawa, Wentao Zhang, Zicen Xiong, Jinjie Li, Moju Zhao
Robotics
Design, Modeling and Direction Control of a Wire-Driven Robotic Fish Based on a 2-DoF Crank-Slider Mechanism
Yita Wang, Chen Chen, Yicheng Chen et al. (+5 more)
Robotic fish have attracted growing
attention in recent years owing to their biomimetic design and potential applications in environmental monitoring and biological surveys. Among robotic fish employing the Body-Caudal Fin (BCF) locomotion pattern, motor-driven actuation is widely adopted. Some approaches utilize multiple servo motors to achieve precise body
curvature control, while others employ a brushless motor to drive the tail via wire or rod, enabling higher oscillation and swimming speeds. However, the former approaches typically result in limited swimming speed, whereas the latter suffer from poor maneuverability, with few capable of smooth turning. To address this trade-off, we develop a wire-driven robotic fish equipped with a 2-degree-of-freedom (DoF) crank-slider mechanism that decouples propulsion from steering, enabling both high swimming speed and agile maneuvering. In this paper, we first present the design of the robotic fish, including the elastic skeleton, waterproof structure, and the actuation mechanism that realizes the decoupling. We then establish the actuation modeling and body dynamics to analyze the locomotion behavior. Furthermore, we propose a combined feedforward-feedback control strategy to achieve independent regulation of propulsion and steering. Finally, we validate the feasibility of the design, modeling, and control through a series of prototype experiments, demonstrating swimming, turning, and directional control.
Read the full abstract ▾
All authors Yita Wang, Chen Chen, Yicheng Chen, Jinjie Li, Yuichi Motegi, Kenji Ohkuma, Toshihiro Maki, Moju Zhao
Robotics
SPARC: Spatial-Aware Path Planning via Attentive Agent Communication
Sayang Mu, Xiangyu Wu and Bo An
Efficient communication is critical for decentralized Multi-
Robot Path Planning (MRPP), yet existing learned communication methods treat all neighboring
robots equally regardless of their spatial proximity, leading to diluted
attention in congested regions where coordination matters most. We propose Relation enhanced Multi Head
Attention (RMHA), a communication mechanism that explicitly embeds pairwise Manhattan distances into the
attention weight computation, enabling each
robot to dynamically prioritize messages from spatially relevant neighbors. Combined with a distance-constrained
attention mask and GRU gated message fusion, RMHA integrates seamlessly with MAPPO for stable end-to-end training. In zero-shot generalization from 8 training
robots to 128 test
robots on 40x40 grids, RMHA achieves approximately 75 percent success rate at 30 percent obstacle density outperforming the best baseline by over 25 percentage points. Ablation studies confirm that distance-relation encoding is the key contributor to success rate improvement in high-density environments. Index Terms-Multi-
robot path planning, graph
attention mechanism, multi-head
attention, communication optimization, cooperative decision-making
Read the full abstract ▾
Robotics
Generative adversarial imitation learning for robot swarms: Learning from human demonstrations and trained policies
Mattes Kraus and Jonas Kuckling
In imitation learning,
robots are supposed to learn from demonstrations of the desired behavior. Most of the work in imitation learning for swarm robotics provides the demonstrations as rollouts of an existing policy. In this work, we provide a framework based on generative
adversarial imitation learning that aims to learn collective behaviors from human demonstrations. Our framework is evaluated across six different missions, learning both from manual demonstrations and demonstrations derived from a PPO-trained policy. Results show that the imitation learning process is able to learn qualitatively meaningful behaviors that perform similarly well as the provided demonstrations. Additionally, we deploy the learned policies on a swarm of TurtleBot 4
robots in real-
robot experiments. The exhibited behaviors preserved their visually recognizable character and their performance is comparable to the one achieved in simulation.
Read the full abstract ▾
Robotics
Self-Evolutionary Replanning for Failure-Aware Motion Planning
Guoliang Li, Ruihua Han, Chengyang Li et al. (+5 more)
Motion planners tuned under nominal conditions can fail when execution conditions change. Existing differentiable adaptation methods provide efficient online updates but remain constrained by static loss functions and gradient-only parameter updates. We propose Self-Evolutionary RePlanning (SERP), a failure-aware framework that jointly adapts motion-planner parameters and loss function weights from closed-loop execution feedback. SERP provides two adaptation mechanisms. First, in-context learning with auto-differentiation (ILAD) enables rapid online adaptation by coupling local numerical refinement with history-conditioned semantic intervention. Auto-differentiation performs numerical updates, while a
vision-
language model (VLM) interprets recent adaptation history to jointly revise planner parameters and loss weights when local adaptation becomes ineffective. Second, the
memory-augmented rollout system (MARS) extends ILAD into a deliberative and evolutionary recovery process with
memory-guided candidate generation and parallel low-fidelity rollout evaluation. The resulting recovery trace is then consolidated into recovery
memory to guide future adaptation. Extensive simulation and real-world experiments demonstrate improved failure recovery performance over various baselines across diverse environments.
Read the full abstract ▾
All authors Guoliang Li, Ruihua Han, Chengyang Li, He Li, Shuai Wang, Wenchao Ding, Hong Zhang, Chengzhong Xu
Robotics
Robust Tightly-Coupled Filter-Based Monocular Visual-Inertial State Estimation and Graph-Based Evaluation for Autonomous Drone Racing
Maulana Bisyir Azhari, Donghun Han, Sung Jun Park et al. (+1 more)
Autonomous drone racing (ADR) demands state estimation that is simultaneously computationally efficient and resilient to the perceptual degradation experienced during extreme velocity and maneuvers. Traditional frameworks typically rely on conventional visual-inertial pipelines with loosely-coupled gate-based Perspective-n-Points (PnP) corrections that suffer from a rigid requirement for four visible features and information loss in intermediate steps. Furthermore, the absence of GNSS and Motion Capture systems in uninstrumented, competitive racing environments makes the objective evaluation of such systems remarkably difficult. To address these limitations, we propose ADR-VINS, a robust,
monocular visual-inertial state estimation framework based on an Error-State Kalman Filter (ESKF) tailored for autonomous drone racing. Our approach integrates direct pixel reprojection errors from gate corners features as innovation terms within the filter. By bypassing intermediate PnP solvers, ADR-VINS maintains valid state updates with as few as two visible corners and utilizes robust reweighting instead of RANSAC-based schemes to handle outliers, enhancing computational efficiency. Furthermore, we introduce ADR-FGO, an offline Factor-Graph Optimization framework to generate high-fidelity reference trajectories that facilitate post-flight performance evaluation and analysis on uninstrumented, GNSS-denied environments. The proposed system is validated using TII-RATM
dataset, where ADR-VINS achieves an average RMS translation error of 0.143 m, while ADR-FGO yields 0.060 m as a smoothing-based reference. Finally, ADR-VINS was successfully deployed in the A2RL Drone Championship Season 2, maintaining stable and robust estimation despite noisy detections during high-agility flight at top speeds of 20.9 m/s. We further utilize ADR-FGO for post-flight evaluation in uninstrumented racing environments.
Read the full abstract ▾
All authors Maulana Bisyir Azhari, Donghun Han, Sung Jun Park, David Hyunchul Shim
Robotics · Cross-listed from cs.AI
Retrieval-Augmented Robots via Retrieve-Reason-Act
Izat Temiraliev, Diji Yang and Yi Zhang
To achieve general-purpose utility, we argue that
robots must evolve from passive executors into active Information
Retrieval users. In strictly zero-shot settings where no prior demonstrations exist,
robots face a critical information gap, such as the exact sequence required to assemble a complex furniture kit, that cannot be satisfied by internal parametric knowledge (common sense) or past internal
memory. While recent robotic works attempt to use search before action, they primarily focus on retrieving past kinematic trajectories (analogous to searching internal
memory) or text-based safety rules (searching for constraints). These approaches fail to address the core information need of active task construction: acquiring unseen procedural knowledge from external, unstructured documentation. In this paper, we define the paradigm as
Retrieval-Augmented Robotics (RAR), empowering the
robot with the information-seeking capability that bridges the gap between visual documentation and physical actuation. We formulate the task execution as an iterative Retrieve-Reason-Act loop: the
robot or
embodied agent actively retrieves relevant visual procedural manuals from an unstructured corpus, grounds the abstract 2D diagrams to 3D physical parts via cross-modal
alignment, and synthesizes executable plans. We validate this paradigm on a challenging long-horizon assembly
benchmark. Our experiments demonstrate that grounding robotic planning in retrieved visual documents significantly outperforms baselines relying on zero-shot
reasoning or few-shot example
retrieval. This work establishes the basis of RAR, extending the scope of Information
Retrieval from answering user queries to driving
embodied physical actions.
Read the full abstract ▾
Robotics
MMH-Planner: Multi-Mode Hybrid Trajectory Planning Method for UAV Efficient Flight Based on Real-Time Spatial Awareness
Yinghao Zhao, Chenguang Dai, Liang Lyu et al. (+3 more)
Motion planning is a critical component of intelligent unmanned systems, enabling their complex autonomous operations. However, current planning
algorithms still face limitations in planning efficiency due to inflexible strategies and weak adaptability. To address this, this paper proposes a multi-mode hybrid trajectory planning method for
UAVs based on real-time environmental awareness, which dynamically selects the optimal planning model for high-quality trajectory generation in response to environmental changes. First, we introduce a goal-oriented spatial awareness method that rapidly assesses flight safety in the upcoming environments. Second, a multi-mode hybrid trajectory planning mechanism is proposed, which can enhance the planning efficiency by selecting the optimal planning model for trajectory generation based on prior spatial awareness. Finally, we design a lazy replanning strategy that triggers replanning only when necessary to reduce computational resource consumption while maintaining flight quality. To validate the performance of the proposed method, we conducted comprehensive comparative experiments in simulation environments. Results demonstrate that our approach outperforms existing state-of-the-art (SOTA)
algorithms across multiple metrics, achieving the best performance particularly in terms of the average number of planning iterations and computational cost per iteration. Furthermore, the effectiveness of our approach is further verified through real-world flight experiments integrated with a self-developed intelligent
UAV platform.
Read the full abstract ▾
All authors Yinghao Zhao, Chenguang Dai, Liang Lyu, Zhenchao Zhang, Chaozhen Lan, Hong Xie
Robotics
IMR-LLM: Industrial Multi-Robot Task Planning and Program Generation using Large Language Models
Xiangyu Su, Juzhan Xu, Oliver van Kaick et al. (+2 more)
In modern industrial production, multiple
robots often collaborate to complete complex manufacturing tasks. Large
language models (LLMs), with their strong
reasoning capabilities, have shown potential in coordinating
robots for simple household and manipulation tasks. However, in industrial scenarios, stricter sequential constraints and more complex dependencies within tasks present new challenges for LLMs. To address this, we propose IMR-LLM, a novel LLM-driven Industrial Multi-
Robot task planning and program generation framework. Specifically, we utilize LLMs to assist in constructing disjunctive graphs and employ deterministic solving methods to obtain a feasible and efficient high-level task plan. Based on this, we use a process tree to guide LLMs to generate executable low-level programs. Additionally, we create IMR-Bench, a challenging
benchmark that encompasses multi-
robot industrial tasks across three levels of complexity. Experimental results indicate that our method significantly surpasses existing methods across all evaluation metrics.
Read the full abstract ▾
All authors Xiangyu Su, Juzhan Xu, Oliver van Kaick, Kai Xu, Ruizhen Hu
Robotics
Watch Your Step: Learning Semantically-Guided Locomotion in Cluttered Environment
Denan Liang, Yuan Zhu, Ruimeng Liu et al. (+3 more)
Although legged
robots demonstrate impressive mobility on rough terrain, using them safely in cluttered environments remains a challenge. A key issue is their inability to avoid stepping on low-lying objects, such as high-cost small devices or cables on flat ground. This limitation arises from a disconnection between high-level semantic understanding and low-level control, combined with errors in elevation maps during real-world operation. To address this, we introduce SemLoco, a Reinforcement Learning (RL) framework designed to avoid obstacles precisely in densely cluttered environments. SemLoco uses a two-stage RL approach that combines both soft and hard constraints. It performs pixel-wise foothold safety inference, which enables more accurate foot placement. Additionally, SemLoco integrates semantic map, allowing it to assign traversability costs instead of relying only on
geometric data. SemLoco greatly reduces collisions and improves safety around sensitive objects, enabling reliable
navigation in situations where traditional controllers would likely cause damage. Experimental results further show that SemLoco can be effectively applied to more complex, unstructured real-world environments. A demo
video can be view at https://youtu.be/FSq-RSmIxOM.
Read the full abstract ▾
All authors Denan Liang, Yuan Zhu, Ruimeng Liu, Thien-Minh Nguyen, Shenghai Yuan, Lihua Xie