Robotics
Safe Human-to-Humanoid Motion Imitation Using Control Barrier Functions
Wenqi Cai, John Abanes, Nikolaos Evangeliou et al. (+1 more)
Ensuring operational safety is critical for human-to-
humanoid motion imitation. This paper presents a
vision-based framework that enables a
humanoid robot to imitate human movements while avoiding collisions. Human skeletal keypoints are captured by a single camera and converted into joint angles for motion retargeting. Safety is enforced through a Control Barrier Function (CBF) layer formulated as a Quadratic Program (QP), which filters imitation commands to prevent both self-collisions and human-
robot collisions. Simulation results validate the effectiveness of the proposed framework for real-time collision-aware motion imitation.
Read the full abstract ▾
All authors Wenqi Cai, John Abanes, Nikolaos Evangeliou, Anthony Tzes
Robotics
Dyadic Partnership(DP): A Missing Link Towards Full Autonomy in Medical Robotics
Nassir Navab and Zhongliang Jiang
For the past decades medical robotic solutions were mostly based on the concept of tele-manipulation. While their design was extremely intelligent, allowing for better access, improved dexterity, reduced tremor, and improved imaging, their intelligence was limited. They therefore left cognition and decision making to the surgeon. As medical robotics advances towards high-level autonomy, the scientific community needs to explore the required pathway towards partial and full autonomy. Here, we introduce the concept of Dyadic Partnership(DP), a new paradigm in which
robots and clinicians engage in intelligent, expert interaction and collaboration. The Dyadic Partners would discuss and agree on decisions and actions during their dynamic and interactive collaboration relying also on intuitive advanced media using generative AI, such as a world model, and advanced multi-modal visualization. This article outlines the foundational components needed to enable such systems, including foundation models for clinical intelligence, multi-modal intent recognition, co-learning frameworks, advanced visualization, and explainable, trust-aware interaction. We further discuss key challenges such as data scarcity, lack of standardization, and ethical acceptance. Dyadic partnership is introduced and is positioned as a powerful yet achievable, acceptable milestone offering a promising pathway toward safer, more intuitive collaboration and a gradual transition to full autonomy across diverse clinical settings.
Read the full abstract ▾
Robotics
Efficient Emotion-Aware Iconic Gesture Prediction for Robot Co-Speech
Edwin C. Montiel-Vazquez, Christian Arzate Cruz, Stefanos Gkikas et al. (+3 more)
Co-
speech gestures increase engagement and improve
speech understanding. Most data-driven
robot systems generate rhythmic beat-like motion, yet few integrate semantic emphasis. To address this, we propose a lightweight
transformer that derives iconic gesture placement and intensity from text and
emotion alone, requiring no audio input at inference time. The model outperforms GPT-4o in both semantic gesture placement classification and intensity regression on the BEAT2
dataset, while remaining computationally compact and suitable for real-time deployment on
embodied agents.
Read the full abstract ▾
All authors Edwin C. Montiel-Vazquez, Christian Arzate Cruz, Stefanos Gkikas, Thomas Kassiotis, Giorgos Giannakakis, Randy Gomez
Robotics
Using Unwrapped Full Color Space Recording to Measure the Exposedness of Vehicle Exterior Parts for External Human Machine Interfaces
Jose Gonzalez-Belmonte and Jaerock Kwon
One of the concerns with autonomous vehicles is their ability to communicate their intent to other road users, specially pedestrians, in order to prevent accidents. External Human-Machine Interfaces (eHMIs) are the proposed solution to this issue, through the introduction of electronic devices on the exterior of a vehicle that communicate when the vehicle is planning on slowing down or yielding. This paper uses the technique of unwrapping the faces of a mesh onto a texture where every pixel is a unique color, as well as a series of animated simulations made and ran in the Unity game engine, to measure how many times is each point on a 2015 Ford F-150 King Ranch is unobstructed to a pedestrian attempting to cross the road at a four-way intersection. By cross-referencing the results with a color-coded map of the labeled parts on the exterior of the vehicle, it was concluded that while the bumper, grill, and hood were the parts of the vehicle visible to the crossing pedestrian most often, the existence of other vehicles on the same lane that might obstruct the view of these makes them insufficient. The study recommends instead a distributive approach to eHMIs by using both the windshield and frontal fenders as simultaneous placements for these devices.
Read the full abstract ▾
Robotics
EagleVision: A Multi-Task Benchmark for Cross-Domain Perception in High-Speed Autonomous Racing
Zakhar Yagudin, Murad Mebrahtu, Ren Jin et al. (+5 more)
High-speed autonomous racing presents extreme perception challenges, including large relative velocities and substantial domain shifts from conventional urban-driving
datasets. Existing
benchmarks do not adequately capture these high-dynamic conditions. We introduce EagleVision, a unified
LiDAR-based multi-task
benchmark for 3D detection and trajectory prediction in high-speed racing, providing newly annotated 3D bounding boxes for the Indy Autonomous Challenge
dataset (14,893 frames) and the A2RL Real competition
dataset (1,163 frames), together with 12,000 simulator-generated annotated frames, all standardized under a common evaluation protocol. Using a
dataset-centric transfer framework, we quantify cross-domain generalization across urban, simulator, and real racing domains. Urban pretraining improves detection over scratch training (NDS 0.72 vs. 0.69), while intermediate pretraining on real racing data achieves the best transfer to A2RL (NDS 0.726), outperforming simulator-only adaptation. For trajectory prediction, Indy-trained models surpass in-domain A2RL training on A2RL test sequences (FDE 0.947 vs. 1.250), highlighting the role of motion-distribution coverage in cross-domain
forecasting. EagleVision enables systematic study of perception generalization under extreme high-speed dynamics. The
dataset and
benchmark are publicly available at https://avlab.io/EagleVision
Read the full abstract ▾
All authors Zakhar Yagudin, Murad Mebrahtu, Ren Jin, Jiaqi Huang, Yujia Yue, Dzmitry Tsetserukou, Jorge Dias, Majid Khonji
Robotics
ComSim: Building Scalable Real-World Robot Data Generation via Compositional Simulation
Yiran Qin, Jiahua Ma, Li Kang et al. (+11 more)
Recent advancements in foundational models, such as large
language models and world models, have greatly enhanced the capabilities of robotics, enabling
robots to autonomously perform complex tasks. However, acquiring large-scale, high-quality training data for robotics remains a challenge, as it often requires substantial manual effort and is limited in its coverage of diverse real-world environments. To address this, we propose a novel hybrid approach called Compositional Simulation, which combines classical simulation and neural simulation to generate accurate action-
video pairs while maintaining real-world consistency. Our approach utilizes a closed-loop real-sim-real data augmentation pipeline, leveraging a small amount of real-world data to generate diverse, large-scale training
datasets that cover a broader spectrum of real-world scenarios. We train a neural simulator to transform classical simulation
videos into real-world representations, improving the accuracy of policy models trained in real-world environments. Through extensive experiments, we demonstrate that our method significantly reduces the sim2real domain gap, resulting in higher success rates in real-world policy model training. Our approach offers a scalable solution for generating robust training data and bridging the gap between simulated and real-world robotics.
Read the full abstract ▾
All authors Yiran Qin, Jiahua Ma, Li Kang, Wenzhan Li, Yihang Jiao, Xin Wen, Xiufeng Song, Heng Zhou, Jiwen Yu, Zhenfei Yin, Xihui Liu, Philip Torr, Yilun Du, Ruimao Zhang
Robotics
Minimal Embodiment Enables Efficient Learning of Number Concepts in Robot
Zhegong Shangguan, Alessandro Di Nuovo and Angelo Cangelosi
Robots are increasingly entering human-interactive scenarios that require understanding of quantity. How intelligent systems acquire abstract numerical concepts from sensorimotor experience remains a fundamental challenge in cognitive science and artificial intelligence. Here we investigate
embodied numerical learning using a neural network model trained to perform sequential counting through naturalistic robotic interaction with a Franka Panda manipulator. We demonstrate that
embodied models achieve 96.8% counting accuracy with only 10% of training data, compared to 60.6% for
vision-only baselines. This advantage persists when visual-motor correspondences are randomized, indicating that embodiment functions as a structural prior that regularizes learning rather than as an information source. The model spontaneously develops biologically plausible representations: number-selective units with logarithmic tuning, mental number line organization, Weber-law scaling, and rotational dynamics encoding numerical magnitude ($r = 0.97$, slope $= 30.6{\deg}$/count). The learning trajectory parallels children's developmental progression from subset-knowers to cardinal-principle knowers. These findings demonstrate that minimal embodiment can ground abstract concepts, improve data efficiency, and yield interpretable representations aligned with biological cognition, which may contribute to
embodied mathematics tutoring and safety-critical industrial applications.
Read the full abstract ▾
Robotics
MR.ScaleMaster: Scale-Consistent Collaborative Mapping from Crowd-Sourced Monocular Videos
Hyoseok Ju and Giseop Kim
Crowd-sourced cooperative mapping combines
monocular sessions from different front-ends, each an independently reconstructed keyframe sequence. Each session has its own local coordinate frame and may follow an incompatible scale convention. We present MR.ScaleMaster, a backend that accepts image, Sim(3) pose, and point-map packets without requiring a common reconstruction model, camera intrinsics, or front-end-provided metric scale. Our Cross-Front-End Loop Factor (CFL) uses a shared matcher for image correspondences but retrieves matched 3D points from the input point maps, so its scale estimates the inter-session ratio. Scale Preconditioning (SPC) initializes session scales for a Sim(3) anchor-node graph, which then corrects residual scale and drift. For front-ends exposing an incremental scale trajectory, an
agent-side Scale Collapse Alarm (SCA) rejects or rolls back false intra-session loops that would otherwise collapse the session scale. We evaluate seven front-ends on KITTI and five on CODa. Based on ground-truth path lengths, session scales in our heterogeneous KITTI setting differ by 57-103x. CFL reduces mean ATE by 37% and inter-session scale error by 60% relative to loop factors from independent pairwise reconstructions. Adding SPC raises the reductions to 74% and 77%, respectively. On CODa, five-session fusion improves mean ATE over single-session runs for all five front-ends, while joint fusion registers 15 sessions from three different front-ends in a single map. Code will be released.
Read the full abstract ▾
Robotics
WM-DAgger: Enabling Efficient Data Aggregation for Imitation Learning with World Models
Anlan Yu, Zaishu Chen, Peili Song et al. (+6 more)
Imitation learning is a powerful paradigm for training robotic policies, yet its performance is limited by compounding errors: minor policy inaccuracies could drive
robots into unseen out-of-distribution (OOD) states in the training set, where the policy could generate even bigger errors, leading to eventual failures. While the Data Aggregation (DAgger) framework tries to address this issue, its reliance on continuous human involvement severely limits scalability. In this paper, we propose WM-DAgger, an efficient data aggregation framework that leverages World Models to synthesize OOD recovery data without requiring human involvement. Specifically, we focus on manipulation tasks with an eye-in-hand robotic arm and only few-shot demonstrations. To avoid synthesizing misleading data and overcome the
hallucination issues inherent to World Models, our framework introduces two key mechanisms: (1) a Corrective Action Synthesis Module that generates task-oriented recovery actions to prevent misleading supervision, and (2) a Consistency-Guided Filtering Module that discards physically implausible trajectories by anchoring terminal synthesized frames to corresponding real frames in expert demonstrations. We extensively validate WM-DAgger on multiple real-world robotic tasks. Results that our method significantly improves success rates, achieving a 93.3% success rate in soft bag pushing with only five demonstrations. The source code is publicly available at https://github.com/czs12354-xxdbd/WM-Dagger.
Read the full abstract ▾
All authors Anlan Yu, Zaishu Chen, Peili Song, Zhiqing Hong, Haotian Wang, Desheng Zhang, Tian He, Yi Ding, Daqing Zhang
Robotics
Learning Racket-Ball Bounce Dynamics Across Diverse Rubbers for Robotic Table Tennis
Thomas Gossard
Accurate dynamic models for racket-ball bounces are essential for reliable control in robotic table tennis. Existing models typically assume simple linear models and are restricted to inverted rubbers, limiting their ability to generalize across the wide variety of rackets encountered in practice. In this work, we present a unified framework for modeling ball-racket interactions across 10 racket configurations featuring different rubber types, including inverted, anti-
spin, and pimpled surfaces. Using a high-speed multi-camera setup with
spin estimation, we collect a
dataset of racket-ball bounces spanning a broad range of incident velocities and
spins. We show that key physical parameters governing rebound, such as the Coefficient of Restitution and tangential impulse response, vary systematically with the impact state and differ significantly across rubbers. To capture these effects while preserving physical interpretability, we estimate the parameters of an impulse-based contact model using Gaussian Processes conditioned on the ball's incoming velocity and
spin. The resulting model provides both accurate predictions and uncertainty estimations. Compared to the constant parameter baselines, our approach reduces post-impact velocity and
spin prediction errors across all racket types, with the largest improvements observed for nonstandard rubbers. Furthermore, the GP-based model enables online identification of racket dynamics with few observations during gameplay.
Read the full abstract ▾
Robotics
CLASP: Closed-loop Asynchronous Spatial Perception for Open-vocabulary Desktop Object Grasping
Yiran Ling, Wenxuan Li, Siying Dong et al. (+5 more)
Robot grasping of desktop object is widely used in intelligent manufacturing, logistics, and agriculture.Although
vision-
language models (VLMs) show strong potential for robotic manipulation, their deployment in low-level grasping faces key challenges: scarce high-quality
multimodal demonstrations, spatial
hallucination caused by weak
geometric grounding, and the fragility of open-loop execution in dynamic environments. To address these challenges, we propose Closed-Loop Asynchronous Spatial Perception(CLASP), a novel asynchronous closed-loop framework that integrates
multimodal perception, logical
reasoning, and state-reflective feedback. First, we design a Dual-Pathway Hierarchical Perception module that decouples high-level semantic intent from
geometric grounding. The design guides the output of the inference model and the definite action tuples, reducing spatial illusions. Second, an Asynchronous Closed-Loop Evaluator is implemented to compare pre- and post-execution states, providing text-based diagnostic feedback to establish a robust error-correction loop and improving the vulnerability of traditional open-loop execution in dynamic environments. Finally, we design a scalable multi-modal data engine that automatically synthesizes high-quality spatial annotations and
reasoning templates from real and synthetic scenes without human teleoperation. Extensive experiments demonstrate that our approach significantly outperforms existing baselines, achieving an 87.0% overall success rate. Notably, the proposed framework exhibits remarkable generalization across diverse objects, bridging the sim-to-real gap and providing exceptional robustness in geometrically challenging categories and cluttered scenarios.
Read the full abstract ▾
All authors Yiran Ling, Wenxuan Li, Siying Dong, Yize Zhang, Xiaoyao Huang, Jing Jiang, Ruonan Li, Jie Liu
Robotics
Learning to Forget – Hierarchical Episodic Memory for Lifelong Robot Deployment
Leonard Bärmann, Joana Plewnia, Alex Waibel et al. (+1 more)
Robots must verbalize their past experiences when users ask "Where did you put my keys?" or "Why did the task fail?" Yet maintaining life-long episodic
memory (EM) from continuous
multimodal perception quickly exceeds storage limits and makes real-time query impractical, calling for selective forgetting that adapts to users' notions of relevance. We present H$^2$-EMV, a framework enabling
humanoids to learn what to remember through user interaction. Our approach incrementally constructs hierarchical EM, selectively forgets using
language-model-based relevance estimation conditioned on learned natural-
language rules, and updates these rules given user feedback about forgotten details. Evaluations on simulated household tasks and 20.5-hour-long real-world recordings from ARMAR-7 demonstrate that H$^2$-EMV maintains question-answering accuracy while reducing
memory size by 45% and query-time compute by 35%. Critically, performance improves over time - accuracy increases 70% in second-round queries by adapting to user-specific priorities - demonstrating that learned forgetting enables scalable, personalized EM for long-term human-
robot collaboration.
Read the full abstract ▾
All authors Leonard Bärmann, Joana Plewnia, Alex Waibel, Tamim Asfour
Robotics
3D-Anchored Lookahead Planning for Persistent Robotic Scene Memory via World-Model-Based MCTS
Bronislav Sidik and Dror Mizrahi
We present 3D-Anchored Lookahead Planning (3D-ALP), a System 2
reasoning engine for robotic manipulation that combines Monte Carlo Tree Search (MCTS) with a 3D-consistent world model as the rollout oracle. Unlike reactive policies that evaluate actions from the current camera frame only, 3D-ALP maintains a persistent camera-to-world (c2w) anchor that survives occlusion, enabling accurate replanning to object positions that are no longer directly observable. On a 5-step sequential reach task requiring spatial
memory (Experiment E3), 3D-ALP achieves 0.650 0.109 success rate on
memory-required steps versus 0.006 0.008 for a greedy reactive baseline ({\Delta}=+0.645), while step 5 success reaches 0.822 against 0.000 for greedy. An ablation study (30 episodes, 3 seeds) isolates tree search spatial
memory as the primary driver (+0.533, 82% of gain) with additional benefit from deeper lookahead (+0.111, 17%). We also identify and resolve four structural failure modes in applying UCT-MCTS (Upper Confidence Bounds applied to Trees [10]) to continuous robotic manipulation.
Read the full abstract ▾
Robotics
Modeling, Analysis and Activation of Planar Viscoelastically-combined Rimless Wheels
Fumihiko Asano, Yuxuan Xiang, Yanqiu Zheng et al. (+1 more)
This paper proposes novel passive-dynamic walkers formed by two cross-shaped frames and eight viscoelastic elements. Since it is a combination of two four-legged rimless wheels via viscoelastic elements, we call it viscoelastically-combined rimless wheel (VCRW). Two types of VCRWs consisting of different cross-shaped frames are introduced; one is formed by combining two Greek-cross-shaped frames (VCRW1), and the other is formed by combining two-link cross-shaped frames that can rotate freely around the central axis (VCRW2). First, we describe the model assumptions and equations of motion and collision. Second, we numerically analyze the basic gait properties of passive dynamic walking. Furthermore, we consider an activation of VCRW2 for generating a stable level gait, and discuss the significance of the study as a novel walking support device.
Read the full abstract ▾
All authors Fumihiko Asano, Yuxuan Xiang, Yanqiu Zheng, Cong Yan
Robotics
CLAW: Composable Language-Annotated Whole-body Motion Generation
Jianuo Cao, Yuxin Chen and Masayoshi Tomizuka
Training
language-conditioned whole-body controllers for
humanoid robots demands large-scale motion-
language datasets. Existing approaches based on motion capture are costly and limited in diversity, while text-to-motion generative models produce purely kinematic outputs that are not guaranteed to be physically feasible. We present CLAW, a pipeline for scalable generation of
language-annotated whole-body motion data for the Unitree G1
humanoid robot. CLAW composes motion primitives from a kinematic planner, parameterized by movement, heading, speed, pelvis height, and duration, and provides two browser-based interfaces–a real-time keyboard mode and a timeline-based sequence editor–for exploratory and batch data collection. A low-level controller tracks these references in MuJoCo simulation, yielding physically grounded trajectories. In parallel, a template-based engine generates diverse natural-
language annotations at both segment and trajectory levels. To support scalable generation of motion-
language paired data for
humanoid robot learning, we make our system publicly available at: https://github.com/JianuoCao/CLAW
Read the full abstract ▾
Robotics
EmbodiedGovBench: A Benchmark for Governance, Recovery, and Upgrade Safety in Embodied Agent Systems
Xue Qin, Simin Luan, John See et al. (+2 more)
Recent progress in
embodied AI has produced a growing ecosystem of
robot policies, foundation models, and modular runtimes. However, current evaluation remains dominated by task success metrics such as completion rate or manipulation accuracy. These metrics leave a critical gap: they do not measure whether
embodied systems are governable – whether they respect capability boundaries, enforce policies, recover safely, maintain audit trails, and respond to human oversight. We present EmbodiedGovBench, a
benchmark for governance-oriented evaluation of
embodied agent systems. Rather than asking only whether a
robot can complete a task, EmbodiedGovBench evaluates whether the system remains controllable, policy-bounded, recoverable, auditable, and
evolution-safe under realistic perturbations. The
benchmark covers seven governance dimensions: unauthorized capability invocation, runtime drift robustness, recovery success, policy portability, version upgrade safety, human override responsiveness, and audit completeness. We define a
benchmark structure spanning single-
robot and fleet settings, with scenario templates, perturbation operators, governance metrics, and baseline evaluation protocols. We describe how the
benchmark can be instantiated over
embodied capability runtimes with modular interfaces and contract-aware upgrade workflows. Our analysis suggests that
embodied governance should become a first-class evaluation target. EmbodiedGovBench provides the initial measurement framework for that shift.
Read the full abstract ▾
All authors Xue Qin, Simin Luan, John See, Cong Yang, Zhijun Li
Robotics
ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation
Arjun Bhardwaj, Maximum Wilder-Smith, Mayank Mittal et al. (+2 more)
In-hand object reorientation requires precise estimation of the object pose to handle complex task dynamics. While RGB sensing offers rich semantic cues for pose tracking, existing solutions rely on multi-camera setups or costly ray tracing. We present a sim-to-real framework for
monocular RGB in-hand reorientation that integrates 3D Gaussian Splatting (3DGS) to bridge the visual sim-to-real gap. Our key insight is performing domain randomization in the Gaussian representation space: by applying physically consistent, pre-rendering augmentations to 3D Gaussians, we generate photorealistic, randomized visual data for object pose estimation. The manipulation policy is trained using curriculum-based reinforcement learning with teacher-student
distillation, enabling efficient learning of complex behaviors. Importantly, both perception and control models can be trained independently on consumer-grade hardware, eliminating the need for large compute clusters. Experiments show that the pose
estimator trained with 3DGS data outperforms those trained using conventional rendering data in challenging visual environments. We validate the system on a physical multi-fingered hand equipped with an RGB camera, demonstrating robust reorientation of five diverse objects even under challenging lighting conditions. Our results highlight Gaussian splatting as a practical path for RGB-only dexterous manipulation. For
videos of the hardware deployments and additional supplementary materials, please refer to the project website: https://rffr.leggedrobotics.com/works/viserdex/
Read the full abstract ▾
All authors Arjun Bhardwaj, Maximum Wilder-Smith, Mayank Mittal, Vaishakh Patil, Marco Hutter
Robotics
AIM: Intent-Aware Unified world action Modeling with Spatial Value Maps
Liaoyuan Fan, Zetian Xu, Chen Cao et al. (+3 more)
Pretrained
video generation models provide strong priors for
robot control, but existing unified world action models still struggle to decode reliable actions without substantial
robot-specific training. We attribute this limitation to a structural mismatch: while
video models capture how scenes evolve, action generation requires explicit
reasoning about where to interact and the underlying manipulation intent. We introduce AIM, an intent-aware unified world action model that bridges this gap via an explicit spatial interface. Instead of decoding actions directly from future visual representations, AIM predicts an aligned spatial value map that encodes task-relevant interaction structure, enabling a control-oriented abstraction of future dynamics. Built on a pretrained
video generation model, AIM jointly models future observations and value maps within a shared mixture-of-
transformers architecture. It employs intent-causal
attention to route future information to the action branch exclusively through the value representation. We further propose a self-
distillation reinforcement learning stage that freezes the
video and value branches and optimizes only the action head using dense rewards derived from projected value-map responses together with sparse task-level signals. To support training and evaluation, we construct a simulation
dataset of 30K manipulation trajectories with synchronized multi-view observations, actions, and value-map annotations. Experiments on RoboTwin 2.0
benchmark show that AIM achieves a 94.0% average success rate, significantly outperforming prior unified world action baselines. Notably, the improvement is more pronounced in long-horizon and contact-sensitive manipulation tasks, demonstrating the effectiveness of explicit spatial-intent modeling as a bridge between visual world modeling and
robot control.
Read the full abstract ▾
All authors Liaoyuan Fan, Zetian Xu, Chen Cao, Wenyao Zhang, Mingqi Yuan, Jiayu Chen
Robotics
Simulator Adaptation for Sim-to-Real Learning of Legged Locomotion via Proprioceptive Distribution Matching
Jeremy Dao and Alan Fern
Simulation trained legged locomotion policies often exhibit performance loss on hardware due to dynamics discrepancies between the simulator and the real world, highlighting the need for approaches that adapt the simulator itself to better match hardware behavior. Prior work typically quantify these discrepancies through precise, time-aligned matching of joint and base trajectories. This process requires motion capture, privileged sensing, and carefully controlled initial conditions. We introduce a practical alternative based on proprioceptive distribution matching, which compares hardware and simulation rollouts as distributions of joint observations and actions, eliminating the need for time
alignment or external sensing. Using this metric as a black-box objective, we explore adapting simulator dynamics through parameter identification, action-delta models, and residual actuator models. Our approach matches the parameter recovery and policy-performance gains of privileged state-matching baselines across extensive sim-to-sim ablations on the Go2 quadruped. Real-world experiments demonstrate substantial drift reduction using less than five minutes of hardware data, even for a challenging two-legged walking behavior. These results demonstrate that proprioceptive distribution matching provides a practical and effective route to simulator adaptation for sim-to-real transfer of learned legged locomotion.
Read the full abstract ▾
Robotics
Federated Single-Agent Robotics: Multi-Robot Coordination Without Intra-Robot Multi-Agent Fragmentation
Xue Qin, Simin Luan, John See et al. (+2 more)
As
embodied robots move toward fleet-scale operation, multi-
robot coordination is becoming a central systems challenge. Existing approaches often treat this as motivation for increasing internal multi-
agent decomposition within each
robot. We argue for a different principle: multi-
robot coordination does not require intra-
robot multi-
agent fragmentation. Each
robot should remain a single
embodied agent with its own persistent runtime, local policy scope, capability state, and recovery authority, while coordination emerges through federation across
robots at the fleet level. We present
Federated Single-
Agent Robotics (FSAR), a runtime architecture for multi-
robot coordination built on single-
agent robot runtimes. Each
robot exposes a governed capability surface rather than an internally fragmented
agent society. Fleet coordination is achieved through shared capability registries, cross-
robot task delegation, policy-aware authority assignment, trust-scoped interaction, and layered recovery protocols. We formalize key coordination relations including authority delegation, inter-
robot capability requests, local-versus-fleet recovery boundaries, and hierarchical human supervision, and describe a fleet runtime architecture supporting shared
Embodied Capability Module (ECM) discovery, contract-aware cross-
robot coordination, and fleet-level governance. We evaluate FSAR on representative multi-
robot coordination scenarios against decomposition-heavy baselines. Results show statistically significant gains in governance locality (d=2.91, p<.001 vs. centralized control) and recovery containment (d=4.88, p<.001 vs. decomposition-heavy), while reducing authority conflicts and policy violations across all scenarios. Our results support the view that the path from
embodied agents to
embodied fleets is better served by federation across coherent
robot runtimes than by fragmentation within them.
Read the full abstract ▾
All authors Xue Qin, Simin Luan, John See, Cong Yang, Zhijun Li
Robotics
Inferring World Belief States in Dynamic Real-World Environments
Jack Kolb, Aditya Garg, Nikolai Warner et al. (+1 more)
We investigate estimating a human's world belief state using a
robot's observations in a dynamic, 3D, and partially observable environment. The methods are grounded in mental model theory, which posits that human decision making, contextual
reasoning, situation awareness, and behavior planning draw from an internal simulation or world belief state. When in teams, the mental model also includes a team model of each teammate's beliefs and capabilities, enabling fluent teamwork without the need for constant and explicit communication. In this work we replicate a core component of the team model by inferring a teammate's belief state, or level one situation awareness, as a human-
robot team navigates a household environment. We evaluate our methods in a realistic simulation, extend to a real-world
robot platform, and demonstrate a downstream application of the belief state through an active assistance semantic
reasoning task.
Read the full abstract ▾
All authors Jack Kolb, Aditya Garg, Nikolai Warner, Karen M. Feigh
Robotics
{\Psi}-Map: Panoptic Surface Integrated Mapping Enables Real2Sim Transfer
Xuan Yu, Yuxuan Xie, Changjian Jiang et al. (+4 more)
Open-vocabulary panoptic reconstruction is essential for advanced robotics perception and simulation. However, existing methods based on 3D Gaussian Splatting (3DGS) often struggle to simultaneously achieve
geometric accuracy, coherent panoptic understanding, and real-time inference frequency in large-scale scenes. In this paper, we propose a comprehensive framework that integrates
geometric reinforcement, end-to-end panoptic learning, and efficient rendering. First, to ensure physical realism in large-scale environments, we leverage
LiDAR data to construct plane-constrained
multimodal Gaussian Mixture Models (GMMs) and employ 2D Gaussian surfels as the map representation, enabling high-precision surface
alignment and continuous
geometric supervision. Building upon this, to overcome the error accumulation and cumbersome cross-frame association inherent in traditional multi-stage panoptic
segmentation pipelines, we design a query-guided end-to-end learning architecture. By utilizing a local cross-
attention mechanism within the view frustum, the system lifts 2D mask features directly into 3D space, achieving globally consistent panoptic understanding. Finally, addressing the computational bottlenecks caused by high-dimensional semantic features, we introduce Precise Tile Intersection and a Top-K Hard Selection strategy to optimize the rendering pipeline. Experimental results demonstrate that our system achieves superior
geometric and panoptic reconstruction quality in large-scale scenes while maintaining an inference rate exceeding 40 FPS, meeting the real-time requirements of robotic control loops.
Read the full abstract ▾
All authors Xuan Yu, Yuxuan Xie, Changjian Jiang, Shichao Zhai, Rong Xiong, Yu Zhang, Yue Wang
Robotics · Cross-listed from cs.LG
Robust Adversarial Policy Optimization Under Dynamics Uncertainty
Mintae Kim and Koushil Sreenath
Reinforcement learning (RL) policies often fail under dynamics that differ from training, a gap not fully addressed by domain randomization or existing
adversarial RL methods. Distributionally robust RL provides a formal remedy but still relies on surrogate adversaries to approximate intractable primal problems, leaving blind spots that potentially cause instability and over-conservatism. We propose a dual formulation that directly exposes the robustness-performance trade-off. At the trajectory level, a temperature parameter from the dual problem is approximated with an
adversarial network, yielding efficient and stable worst-case rollouts within a divergence bound. At the model level, we employ Boltzmann reweighting over dynamics ensembles, focusing on more adverse environments to the current policy rather than uniform sampling. The two components act independently and complement each other: trajectory-level steering ensures robust rollouts, while model-level sampling provides policy-sensitive coverage of adverse dynamics. The resulting framework, robust
adversarial policy optimization (RAPO) outperforms robust RL baselines, improving resilience to uncertainty and generalization to out-of-distribution dynamics while maintaining dual tractability.
Read the full abstract ▾
Robotics
ScoRe-Flow: Complete Distributional Control via Score-Based Reinforcement Learning for Flow Matching
Xiaotian Qiu, Lukai Chen, Jinhao Li et al. (+3 more)
Flow Matching (FM) policies have emerged as an efficient backbone for robotic control, offering fast and expressive action generation that underpins recent large-scale
embodied AI systems. However, FM policies trained via imitation learning inherit the limitations of demonstration data; surpassing suboptimal behaviors requires reinforcement learning (RL) fine-tuning. Recent methods convert deterministic flows into
stochastic differential equations (SDEs) with learnable noise injection, enabling exploration and tractable likelihoods, but such noise-only control can compromise training efficiency when demonstrations already provide strong priors. We observe that modulating the drift via the score function, i.e., the gradient of log-density, steers exploration toward high-probability regions, improving stability. The score admits a closed-form expression from the velocity field, requiring no auxiliary networks. Based on this, we propose ScoRe-Flow, a score-based RL fine-tuning method that combines drift modulation with learned variance prediction to achieve decoupled control over the mean and variance of
stochastic transitions. Experiments demonstrate that ScoRe-Flow achieves 2.4x faster convergence than flow-based SOTA on D4RL locomotion tasks and up to 5.4% higher success rates on Robomimic and Franka Kitchen manipulation tasks.
Read the full abstract ▾
All authors Xiaotian Qiu, Lukai Chen, Jinhao Li, Qi Sun, Cheng Zhuo, Guohao Dai
Robotics
Diffusion Reinforcement Learning Based Online 3D Bin Packing Spatial Strategy Optimization
Jie Han, Tong Li, Qingyang Xu et al. (+3 more)
The online 3D bin packing problem is important in logistics, warehousing and intelligent manufacturing, with solutions shifting to deep reinforcement learning (DRL) which faces challenges like low sample efficiency. This paper proposes a
diffusion reinforcement learning-based
algorithm, using a Markov decision chain for packing modeling, height map-based state representation and a
diffusion model-based actor network. Experiments show it significantly improves the average number of packed items compared to state-of-the-art DRL methods, with excellent application potential in complex online scenarios.
Read the full abstract ▾
All authors Jie Han, Tong Li, Qingyang Xu, Yong Song, Bao Pang, Xianfeng Yuan
Robotics
Fast-SegSim: Real-Time Open-Vocabulary Segmentation for Robotics in Simulation
Xuan Yu, Yuxuan Xie, Shichao Zhai et al. (+3 more)
Open-vocabulary panoptic reconstruction is crucial for advanced robotics and simulation. However, existing 3D reconstruction methods, such as NeRF or Gaussian Splatting variants, often struggle to achieve the real-time inference frequency required by robotic control loops. Existing methods incur prohibitive latency when processing the high-dimensional features required for robust open-vocabulary
segmentation. We propose Fast-SegSim, a novel, simple, and end-to-end framework built upon 2D Gaussian Splatting, designed to realize real-time, high-fidelity, and 3D-consistent open-vocabulary
segmentation reconstruction. Our core contribution is a highly optimized rendering pipeline that specifically addresses the computational bottleneck of high-channel
segmentation feature accumulation. We introduce two key optimizations: Precise Tile Intersection to reduce rasterization redundancy, and a novel Top-K Hard Selection strategy. This strategy leverages the
geometric sparsity inherent in the 2D Gaussian representation to greatly simplify feature accumulation and alleviate bandwidth limitations, achieving render rates exceeding 40 FPS. Fast-SegSim provides critical value in robotic applications: it serves both as a high-frequency sensor input for simulation platforms like Gazebo, and its 3D-consistent outputs provide essential multi-view 'ground truth' labels for fine-tuning downstream perception tasks. We demonstrate this utility by using the generated labels to fine-tune the perception module in object goal
navigation, successfully doubling the
navigation success rate. Our superior rendering speed and practical utility underscore Fast-SegSim's potential to bridge the sim-to-real gap.
Read the full abstract ▾
All authors Xuan Yu, Yuxuan Xie, Shichao Zhai, Shuhao Ye, Rong Xiong, Yue Wang
Robotics
Ro-SLM: Onboard Small Language Models for Robot Task Planning and Operation Code Generation
Wenhao Wang, Yanyan Li, Long Jiao et al. (+1 more)
Recent advances in large
language models (LLMs) provide
robots with contextual
reasoning abilities to comprehend human instructions. Yet, current LLM-enabled
robots typically depend on cloud-based models or high-performance computing infrastructure, which limit their deployment on
robots under unreliable internet environments or with constrained computational resources, such as
UAVs and small ground vehicles. Thus, deploying fine-tuned small
language models (SLMs) that support onboard deployment offers a promising alternative. This paper introduces Ro-SLM, a framework that enables reliable SLM-driven
robot operation by distilling LLMs' knowledge and
reasoning. Ro-SLM starts from
dataset synthesis by leveraging LLMs to generate diverse task instructions, produce corresponding ground truth code with minimal human assistance, and augment instructions into real-world application scenarios. Ro-SLM is then fine-tuned with the
dataset, in which LLM serves as a reward function to guide the training. Extensive experiments on
UAV operation tasks demonstrate that Ro-SLM improves the performance of SLM from being incapable of supporting robotic task planning and code generation to achieving performance that approaches LLM.
Read the full abstract ▾
All authors Wenhao Wang, Yanyan Li, Long Jiao, Jiawei Yuan
Robotics · Cross-listed from cs.HC
Teaching Robots to Interpret Social Interactions through Lexically-guided Dynamic Graph Learning
Tongfei Bian, Mathieu Chollet and Tanaya Guha
For a
robot to be called socially intelligent, it must be able to infer users internal states from their current behaviour, predict the users future behaviour, and if required, respond appropriately. In this work, we investigate how
robots can be endowed with such social intelligence by modelling the dynamic relationship between user's internal states (latent) and actions (observable state). Our premise is that these states arise from the same underlying socio-cognitive process and influence each other dynamically. Drawing inspiration from theories in Cognitive Science, we propose a novel multi-task learning framework, termed as SocialLDG that explicitly models the dynamic relationship among the states represent as six distinct tasks. Our framework uses a
language model to introduce lexical priors for each task and employs dynamic graph learning to model task affinity evolving with time. SocialLDG has three advantages: First, it achieves state-of-the-art performance on two challenging human-
robot social interaction
datasets available publicly. Second, it supports strong task scalability by learning new tasks seamlessly without catastrophic forgetting. Finally, benefiting from explicit modelling task affinity, it offers insights on how different interactions unfolds in time and how the internal states and observable actions influence each other in human decision making.
Read the full abstract ▾
Robotics
HECTOR: Human-centric Hierarchical Coordination and Supervision of Robotic Fleets under Continual Temporal Tasks
Shen Wang, Yinhang Luo, Jie Li et al. (+1 more)
Robotic fleets can be extremely efficient when working concurrently and collaboratively, e.g., for delivery, surveillance, search and rescue. However, it can be demanding or even impractical for an operator to directly control each
robot. Thus, autonomy of the fleet and its online interaction with the operator are both essential, particularly in dynamic and partially unknown environments. The operator might need to add new tasks, cancel some tasks, change priorities and modify planning results. How to design the procedure for these interactions and efficient
algorithms to fulfill these needs have been mostly neglected in the related literature. Thus, this work proposes a human-centric coordination and supervision scheme (HECTOR) for large-scale robotic fleets under continual and uncertain temporal tasks. It consists of three hierarchical layers: (I) the bidirectional and
multimodal protocol of online human-fleet interaction, where the operator interacts with and supervises the whole fleet; (II) the rolling assignment of currently-known tasks to teams within a certain horizon, and (III) the dynamic coordination within a team given the detected subtasks during online execution. The overall mission can be as general as temporal logic formulas over collaborative actions. Such hierarchical structure allows human interaction and supervision at different granularities and triggering conditions, to both improve computational efficiency and reduce human effort. Extensive human-in-the-loop simulations are performed over heterogeneous fleets under various temporal tasks and environmental uncertainties.
Read the full abstract ▾
All authors Shen Wang, Yinhang Luo, Jie Li, Meng Guo