Robotics · Cross-listed from cs.NI
Network-in-the-Loop at Scale: GPU-Batched 5G Simulation for Massively Parallel Robot Learning
Zifan Zhang, Mingzhe Han, Kannan Athreya et al. (+1 more)
Massively parallel GPU simulators train multi-
robot policies in thousands of environments, and many fleets use private Fifth-Generation (5G) networks, where each
robot's delay depends on its teammates' traffic. Network-in-the-loop training places a simulated 5G network inside this loop. However, GPU
robot simulators reduce the network to an independent delay per message, while packet-level simulators run one scenario per CPU process and cannot keep pace with thousands of parallel environments. To bridge this gap, we present Isaac-Net, a GPU-batched 5G New
Radio (NR) module that advances the uplink of thousands of environments in lockstep with Isaac Lab physics. Isaac-Net simulates every slot, the 0.5~ms interval in which the base station decides which
robots transmit, for all environments at once. Extensive experiments confirm that its NR engine reproduces the median delay of ns-3 5G-LENA across loads, with a median delay 5–10% low on an unseen carrier and 9% high at 32
robots per environment in closed loop. The engine also reproduces the Age of Information (AoI), the age of each
robot's newest delivered report, while an independent delay per message leaves the AoI tail about three times too light. In a configuration validated against 5G-LENA, Isaac-Net keeps the network in the loop for about one million
robots on one GPU at 83% of the Isaac Lab rate without the network, measured under a random policy. Isaac-Net is open source at https://github.com/ZzZTripleZzZ/isaac-net
Read the full abstract ▾
All authors Zifan Zhang, Mingzhe Han, Kannan Athreya, Yuchen Liu
Robotics
Rethinking World-Action Model for Compositional and In-Context Robotic Manipulation
Shukai Gong, Xuanran Zhai, Yintianrun Zhang et al. (+16 more)
Long-horizon compositional manipulation has become increasingly important for real-world
robot deployment, where a single task involves multiple coordinated subtasks. Existing world-action models (WAMs) jointly predict short-horizon visual futures and actions, but typically lack explicit subtask-level
reasoning. We propose Visual Goal-conditioned Action
Reasoning (ViGAR), a hierarchical framework that factorizes manipulation into a visual subgoal planner and a subgoal executor. Given the current observation and global instruction, the subgoal planner predicts a visual subgoal for the next subtask. The subgoal executor then jointly generates future visual trajectories and actions conditioned on the predicted subgoal. Both components share a pretrained world-model representation, enabling task-level planning and action generation to benefit from common physical knowledge. Moreover, our framework naturally supports in-context learning: using a global goal image as context can induce different subtask decompositions and behaviors without parameter updates. On the RoboTwin Clean2Random
benchmark, ViGAR achieves 82.00% and 67.02% success rates under the Clean and Random settings, respectively, surpassing the strongest baseline by 12.86 percentage points in average success rate. Real-world
robot experiments on five compositional and two in-context learning tasks further confirm the effectiveness of ViGAR.
Read the full abstract ▾
All authors Shukai Gong, Xuanran Zhai, Yintianrun Zhang, Ruopeng Cui, Ye Huang, Yiyang Fu, Dexuan Lyu, Chaojie Li, Xinyi Song, Peiwen Lin, Chuang Wang, Mingyuan Jia, Yufan Deng, Jiaxin Fang, Bo Liang, Jiaxin Li, Yuxiang Gao, Hao Liu, Daquan Zhou
Robotics · Cross-listed from cs.LG
Co-design Gym: A Unified Benchmark for Embodiment-Policy Co-optimization
Aviraj Newatia, Yordan Tsvetkov, Leonard Pleiss et al. (+2 more)
Finding an optimal behaviour policy within a given environment is a widely studied problem in domains as diverse as games, robotics, energy infrastructure, communication networks, and multi-
agent systems. Numerous
benchmarks have been developed to support such research, but the vast majority assume that the
agent's embodiment (design) is fixed, focusing instead on policy learning alone. Lifting this assumption gives rise to a broader class of problems in which optimizing embodiment and policy separately is highly suboptimal. An
agent's embodiment strongly shapes which control policies can be discovered, while the optimal embodiment is in turn defined by the policies it admits. To help the research community study this class of problems explicitly and systematically, we introduce Co-Design Gym - a suite of
benchmark environments for jointly optimizing embodiment and policy. Our environments span domains such as robotic manipulation and locomotion, multi-
robot cooperation, deformable and soft dynamics,
video games, electricity grids, wireless networks, F1 racing, multi-
agent warehouses, and optimal control, offering 20 environment families (domains), with over 85 distinct co-design presets in total. We further contribute a systematic evaluation of representative co-design
algorithms, characterizing the current state of the art. Together, these contributions lay the groundwork for cumulative, comparable progress in co-design.
Read the full abstract ▾
All authors Aviraj Newatia, Yordan Tsvetkov, Leonard Pleiss, Andrew Spielberg, Rika Antonova
Robotics
SocialVLA: A Social Perception Gateway for Human-Reaction-Based Failure Detection and Recovery in VLA Manipulation
Sofya Konstantinova, Miguel Altamirano Cabrera, Artem Lykov et al. (+1 more)
Vision-
language-action (VLA) policies enable diverse robotic manipulation but can fail during execution without recognizing their own errors. Human observers provide complementary signals, as unexpected
robot behavior can trigger rapid vocal, facial, or verbal reactions before failure is completed. We introduce SocialVLA, a local, policy-agnostic social perception gateway that converts spontaneous human reactions into runtime intervention signals for VLA manipulation. SocialVLA combines causal paralinguistic audio detection, visual reaction recognition, explicit stop phrases, and
robot-relevance estimation. An asynchronous first-event fusion mechanism triggers a VLA hold from the earliest sufficiently confident signal, while a separate
speech channel captures verbal corrections for participant-directed continuation, restart, or instruction revision. We evaluate SocialVLA on physical Unitree G1 manipulation using 15 participants, with 238 annotated intervention-worthy episodes and 1.038 h of non-intervention behavior. Frozen offline replay achieves 54.6% recall and 69.5% precision, while unfiltered audio-
video fusion reaches 64.3% recall. Relevance estimation reduces false-stop episodes from 100 to 57 and increases precision from 60.5% to 69.8%. In prospective deployment on an unseen 16th participant, the frozen system achieves 59.5% recall and 91.7% precision. Median detector-to-fusion latency is 47.9 ms, VLA-gate-to-physical-hold latency is 336 ms, and reaction-onset-to-hold latency is 1.021 s. These results demonstrate a complete local pathway from spontaneous social reaction to physical VLA interruption and participant-directed recovery.
Read the full abstract ▾
All authors Sofya Konstantinova, Miguel Altamirano Cabrera, Artem Lykov, Dzmitry Tsetserukou
Robotics
Filter-Aware Fine-Tuning for Safe Humanoid Whole-Body Tracking
Pranit Mohnot, Christian Helten, Daniele Gammelli et al. (+1 more)
Safe whole-body motion is essential for deploying humanoid
robots in unstructured environments. Modern humanoid control commonly separates reference specification from execution, with a planner, teleoperator, or motion generator providing a reference that a reinforcement-learning policy tracks through dynamically feasible whole-body control. Runtime safety filters, such as control barrier functions (CBFs), offer a promising approach for enforcing newly introduced constraints via interventions on the tracker's outputs. We show, however, that treating the tracking policy and safety filter independently induces fundamental mismatches, as filtering alters both the executed actions and the induced state distribution. We study this policy-filter interface through case studies that isolate dynamics, objective, and information mismatches, highlight their root causes, and use these insights to develop CoFiT (Constrained Filter-aware Tuning), a filter-aware fine-tuning method for pretrained trackers. Across diverse constraint scenes, CoFiT reduces violation time relative to filter-only training by 91% on TWIST2 and 21% on SONIC, while requiring smaller safety filter corrections. On Unitree G1 hardware, CoFiT reduces violation time by 83% for TWIST2 and completes every trial without operator intervention, whereas 50% of baseline trials require an operator stop. Together, these results provide actionable insights into policy-filter interactions and establish design principles for integrating learned trackers with runtime safety filters.
Read the full abstract ▾
All authors Pranit Mohnot, Christian Helten, Daniele Gammelli, Marco Pavone
Robotics
NEEDLEWORK: Offline Rewriting of Robot Data with Verified Local Stitches
Juntao Ren, Yifan Hou and Shuran Song
Robot demonstrations may contain useful behavior even when individual episodes are inefficient or unsuccessful. Trajectory stitching offers a way to compose these behaviors into improved training data, but identifying useful connections and verifying their feasibility is difficult in high-dimensional
robot data, where many prior methods rely on low-dimensional state representations. We introduce NEEDLE, an offline
dataset-augmentation
algorithm that addresses these challenges by adding short, verified action bridges between recorded observations in high-dimensional
robot demonstrations. First, NEEDLE identifies and creates connections that bypass suboptimal detours, broaden action coverage, and augment the original
dataset with failed trajectories, using only RGB images, proprioception, and episode-level outcomes, without new environment interaction or privileged object state. Next, we present a sampling technique that incorporates accepted bridges into policy training without synthesizing intermediate images or discarding the original demonstrations, allowing policies to learn alternative actions while retaining the original
dataset's coverage. On real-
robot tasks, NEEDLE improves success rate over the strongest baseline on each task by an average of 21 percentage points.
Videos and supplementary materials are on https://needle-work.github.io/.
Read the full abstract ▾
Robotics
SoTa: Soft Tactile Skins for Dexterous Manipulation
Jingyun Yang, Baiyu Shi, Timothy Yu et al. (+6 more)
A growing body of work suggests that tactile sensing gives
robot policies contact information that complements
vision in dexterous manipulation. However, visuo-tactile
robot data remains scarce: dexterous demonstrations require teleoperating
robots, which limits
dataset scale. Human demonstrations are far cheaper to collect and offer a path to scale this data, but only if human and
robot hands carry tactile sensors with corresponding signals. This requires sensors that conform to different hand geometries, cover the full hand, and share a common layout across embodiments. We present SoTa, a low-cost capacitive tactile skin that provides full-hand coverage on humans and
robots while preserving a shared layout of 202 taxels across corresponding finger and palm regions. Our multilayer design with fabric electrodes enables in-house fabrication of thin, soft skins with customizable geometry for under $10 in materials per skin. The sensor retains over 97% of its initial response span after 10,000 loading-unloading cycles with traces retaining continuity through 1,280 tight-fist folding cycles. The shared taxel layout supports human-
robot co-training with a common tactile encoder and no learned cross-sensor mapping. Across three contact-rich manipulation tasks, tactile observations improve in-distribution success over
vision-only policies. With a fixed
robot demonstration budget, adding human demonstrations more than doubles mean success across eight evaluation conditions, from 22.8% to 45.9%, improving success in all five out-of-distribution conditions. We plan to open-source the resources needed to fabricate and operate these skins.
Read the full abstract ▾
All authors Jingyun Yang, Baiyu Shi, Timothy Yu, Haitian Liu, Alberta Longhini, Weichen Wang, Rika Antonova, Zhenan Bao, Jeannette Bohg
Robotics
World-Calibrated Proposal-to-Action Flow for Vision-Language-Action Models
Jie He, Wei Li, Junwen Tong et al. (+3 more)
Flow-based
Vision-
Language-Action (VLA) policies generate action chunks by transporting samples from a task-agnostic isotropic Gaussian source. As this source is conditioned on neither recent execution nor predicted future
evolution, (i) it discards the local continuity established by recently executed motion. (ii) Even when predictive world representations are introduced, they often only condition the transport dynamics rather than determine where generation starts, how far it may deviate, or along which action directions it may expand. Building on this observation, we introduce ProAct, a world-calibrated proposal-to-action framework that makes the generative source itself predictable. (i) To preserve motion continuity, a lightweight Proposal Expert converts recent actions into a scene-aware hypothesis via one motion-anchored endpoint flow-matching step, initializing generation near the demonstrated action
manifold. (ii) To jointly capture intended scene
evolution and proposal-future compatibility, a prospective World Expert treats the hypothesis as a soft motion prior while predicting the task-consistent latent future. (iii) From this compatibility, the model calibrates a proposal-centered anisotropic source, where a bounded per-step extent controls the allowed deviation and a trace-normalized low-rank geometry under a condition-number budget allocates refinement over coupled translation, rotation, and gripper directions. Compared with $\pi_{0.5}$, ProAct improves performance across simulation and real-world tasks while reducing denoising steps by 50%, inference latency by up to 25.8%, and increasing throughput by up to 34.8%.
Read the full abstract ▾
All authors Jie He, Wei Li, Junwen Tong, Rui Shao, Wei-Shi Zheng, Liqiang Nie
Robotics
Awomo-SimDataEngine: Agentic Simulation-ReadyWorld Generation
Awomo-PhysicalRSI Team, Danjiao Ma, Enhui Ma et al. (+21 more)
Generating useful
robot-training data requires more than visually plausiblescenes: objects must support interaction, placements must remain physicallyvalid, and tasks must admit repeatable execution. We presentAwomo-SimDataEngine, an agentic system that connects asset and scenegeneration to
robot demonstration synthesis. Shared asset services providerigid and articulated objects, including structure-grounded part and jointgeneration with ISArt. Scene generation supports two complementary routes:Unravel reconstructs editable scenes from images, while SimForge buildssingle-room and multi-room environments from text. A graph-native harnesscoordinates construction, validation, andbounded repair, routing failures to the responsible module while retainingunaffected scene state. PolicyForge binds validated worlds to tasks and robotembodiments to produce replayable demonstrations. Evaluations cover assetgeometry, scene quality, and downstream policy learning. On MuJoCo-basedLIBERO-Plus, co-training with Isaac Sim demonstrations improves the overallsuccess rate of a World-Action Model (WAM) from $77.17\%$ to $89.43\%$. Goal and spatialsuccess improve by $31.66$ and $6.25$ percentage points, respectively.These results support the utility of the generated data for cross-simulatorpolicy training, with more limited gains on long-horizon tasks.
Read the full abstract ▾
All authors Awomo-PhysicalRSI Team, Danjiao Ma, Enhui Ma, Haohan Liu, Heng Jia, Hui Shan, Jianhua Xu, Jiahuan Zhang, Jiangdi Xu, Kaiwen Guo, Kaicheng Yu, Linwei Zhang, Liyang Jin, Maochun Luo, Pengyao Niu, Shiwen Li, Shuangyu Feng, Tong Zhang, Tianheng Wang, Xin Wang, Xiangru Huang, Yongqiang Huang, Zhaozhi Wang, Zijian Ma
Robotics
Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents
Yen-Jen Wang, Haozhe Jiang, Shuying Deng et al. (+7 more)
Building reliable
robot capabilities across diverse tasks requires substantial human effort to develop and maintain skills, design rewards, and integrate perception with control. We present Reconstruct, Practice, Go Real (RPG), a framework for autonomous improvement of
robot execution systems without updating model weights. RPG identifies manipulation capabilities in an offline
dataset and constructs related practice tasks in simulation. During practice, RPG uses execution feedback, privileged simulator state, and available
dataset videos to diagnose failures. It develops new reusable symbolic skills, refines existing skills, and revises the system
prompt based on these diagnoses. Cross-task evaluation tests individual candidate changes and merged revisions before they are retained for reuse. At test time, a
multimodal LLM uses the resulting system
prompt and skill library to coordinate perception and
robot control. On held-out initializations of 22 manipulation tasks, RPG improves task success from 28.6% after the first practice round to 95.0% after 15 rounds, outperforming all evaluated baselines, including ASPIRE (75.5%) and CaP-
Agent0 powered by GPT-6 Astra Pro (60.0%). After a common calibration and hardware-adaptation procedure, the frozen system succeeds in all 30 physical trials, with ten trials on each of three tasks. Project Website: https://rpg-
robot.github.io/
Read the full abstract ▾
All authors Yen-Jen Wang, Haozhe Jiang, Shuying Deng, Haoru Xue, Weirui Ye, Rocky Duan, Nika Haghtalab, S. Shankar Sastry, Pieter Abbeel, Haozhi Qi
Robotics · Cross-listed from cs.LG
FERPO: Forward Entropy-Regularized Policy Optimization
Sebastian Sanokowski, Alireza Sarmadi and Majid Khadiv
Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates. We propose Forward
Entropy-Regularized Policy Optimization (FERPO), an on-policy maximum
entropy reinforcement learning
algorithm that performs policy improvement using critic values without differentiating the critic with respect to actions. FERPO derives an optimal target action distribution from a policy-improvement objective regularized by
entropy and Kullback-Leibler (KL) divergence. We then fit the actor to this target by minimizing a forward-KL objective, estimated using self-normalized importance sampling (SNIS) with actions drawn from the rollout policy. By limiting the target distribution's deviation from the rollout policy, the KL regularization helps keep these importance weights well behaved. In contrast to reverse-KL objectives, which can favor a subset of the target distribution's modes, the forward-KL objective encourages coverage of multiple high-value modes and thereby promotes exploration. Experiments and ablations on MuJoCo Playground and ManiSkill show competitive performance and sample-efficiency gains. Computational
benchmarks also demonstrate faster actor updates than Relative
Entropy Pathwise Policy Optimization (REPPO).
Read the full abstract ▾
Robotics
InterEvolve: Test-Time Evolution of Reward Programs for Humanoid Loco-Manipulation
Zhuo Lin, Sirui Xu, Liuyu Bian et al. (+2 more)
We study test-time
evolution for humanoid loco-manipulation: solving tasks that a controller was never trained for by repurposing its existing skills, improving from its own attempts, and retaining what it learns, without retraining. Our key insight is that a broad controller already holds much of the competence a new task needs, and that this competence becomes accessible through an interface between planning and control that is expressive enough to specify contact-rich, multi-stage interactions, yet executable and measurable enough that execution feedback can guide planning from experience. InterEvolve realizes this interface with two components. First, we develop an object-aware forward-backward (FB) behavioral foundation model, whose object residuals on a frozen body prior turn a new reward about the body or objects into loco-manipulation behavior at test time. Second, we specify tasks as reward programs: staged rewards with completion conditions and tunable constants. A large
language model (LLM)
agent revises the program structure in context, drawing on execution feedback and a skill library of verified programs, while a numerical optimizer tunes its constants. With every candidate verified across parallel simulation scenarios, the program explores new ways to induce, repurpose, and compose the controller's existing motor competence for the task at hand, and thus improves over iterations. Experiments show that human-designed rewards leave much of the FB model's loco-manipulation competence untapped, whereas the programs InterEvolve evolves release it, sometimes through novel strategies. It further produces behaviors for diverse tasks, complex scenes, and long-horizon compositions in simulation, and evolved skills run autonomously on a physical Unitree G1 from egocentric onboard perception.
Read the full abstract ▾
All authors Zhuo Lin, Sirui Xu, Liuyu Bian, Yu-Xiong Wang, Liang-Yan Gui
Robotics
Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordination
Suyu Ye, Zheyuan Zhang, Vaishnav Tadiparthi et al. (+5 more)
Robots operating in the physical world will increasingly need to coordinate with other
robots, particularly in manipulation tasks where an object may be too large or heavy for a single
robot to carry alone. Physical limitations caused by hardware degradation or actuator faults can restrict the actions a
robot can reliably execute, yet these limitations may be unknown to its partner. We study whether a helper can infer a
robot partner's physical constraints from observing it coordinate with another
robot, then use the inferred capability to coordinate with the same partner on a new task. This is difficult because a demonstration shows what the constrained
robot did, but not what it could have done. In physically coupled tasks, the other
robot may also compensate for its limitations, making those limitations difficult to identify from the constrained
robot's behavior alone. Our key insight is that these constraints shape the joint behavior of the team, making the actions of both
robots informative about the constrained partner's capability. We introduce Watch, Infer, Coordinate, a
benchmark spanning three physically coupled manipulation settings, together with an inference approach that scores candidate constraints using observed joint behavior. Across all three settings, our method substantially improves constraint inference and zero-shot coordination, approaching an oracle with access to the true constraints.
Read the full abstract ▾
All authors Suyu Ye, Zheyuan Zhang, Vaishnav Tadiparthi, Hossein Nourkhiz Mahjoub, Ehsan Moradi Pari, Tianmin Shu, Homanga Bharadhwaj, Nakul Agarwal
Robotics
DuoMind: Enabling Distributed Multi-Robot Coordination with Semantic Communication
Hanchu Zhou, Dechen Gao, Hang Wang et al. (+5 more)
Vision-
language models (VLMs) and
vision-
language-action models (VLAs) have recently driven rapid progress in general-purpose
robots, yet most progress has focused on single-
robot settings. Extending these capabilities to multi-
robot systems remains challenging because
robots must coordinate long-horizon behaviors while maintaining reliable, fine-grained execution. We introduce DuoMind, a distributed hierarchical framework for multi-
robot coordination through semantic communication. Each
robot uses a VLA-based action model for low-level execution and a VLM-based orchestrator for high-level
reasoning and inter-
agent coordination. At each planning step, the orchestrator at each
robot reasons over the task instruction, local observations, and messages received from other
robots. It then generates low-level instructions for the action model and semantic messages for peer
robots. This architecture exploits the complementary strengths of pretrained models by combining the semantic
reasoning capabilities of VLMs with the precise action-generation capabilities of VLAs. To address the scarcity of
benchmarks for multi-
robot coordination, we further develop RoboPoly, a
benchmark comprising long-horizon manipulation tasks that require coordinated, closed-loop execution under distributed control. Experiments on RoboPoly and RoboTwin demonstrate that DuoMind improves multi-
robot task performance, while ablation studies confirm the contributions of hierarchical orchestration and semantic communication. More details are available on our project page.
Read the full abstract ▾
All authors Hanchu Zhou, Dechen Gao, Hang Wang, Brendan Lynch, Boqi Zhao, Qiyao Ma, Raman Goyal, Junshan Zhang
Robotics
SkeleWAM: Skeleton World-Action Modeling for Efficient Robotic Manipulation
Juyi Sheng, Hua Wang and Mengyuan Liu
World action models (WAMs) combine
robot action generation with future state prediction. Existing WAMs typically predict
videos or learned visual latents, which represent interaction geometry only implicitly and may retain appearance information unrelated to control. We introduce SkeleWAM, a compact WAM that represents a manipulation scene as a sparse 3D skeleton composed of
robot joints, object centers, and interaction points. Constructed online from current RGB-D observations and
robot proprioception, the skeleton provides a unified
geometric state for action generation and future skeleton prediction. Future skeleton prediction provides additional
geometric supervision for action learning without requiring visual reconstruction. At inference, SkeleWAM generates actions directly from the current skeleton and
language instruction, while Medoid Action Consensus (MAC) serves as an auxiliary consensus strategy for
stochastic action samples. On LIBERO-Plus, SkeleWAM achieves an overall success rate of 85.9% with 57.1M parameters, outperforming Cosmos-Policy by 3.7 percentage points. These results demonstrate that sparse 3D
robot–object structure provides an effective state space for robust and parameter-efficient world action learning. The project is available at https://skelewam-project.github.io/.
Read the full abstract ▾
Robotics
GlassGuard: Verified Glass Plane Mapping for Robot Navigation
Hanwen Guo, Zhengzhi Lin, Yusen Xie et al. (+1 more)
Transparent and specular surfaces pose a serious challenge to
LiDAR-based SLAM and navigation because
laser returns may pass through glass, leaving collision boundaries absent from the map. Prior work attempts to reconstruct the missing surfaces, but inaccurate obstacle placement can create the opposite failure: contamination of traversable free space. Recognizing this dual requirement, we present GlassGuard, a navigation-oriented framework for reconstructing planar architectural glass from complementary visual and
LiDAR evidence. We formulate success in terms of both glass coverage and free-space contamination and apply this principle throughout proposal verification and global map construction. A foundation
vision model provides glass-instance masks, structural 3D cues generate metric plane hypotheses, and depth-free 2D projective geometry checks their orientations before they enter a consolidated global map. We evaluate GlassGuard in nine building-scale scenes spanning diverse glass structures, spatial scales, and lighting conditions, with more than one hour and 2.1 km of real-world
robot traversal. GlassGuard achieves 85% of total glass coverage for its panoramic version. Under identical pinhole inputs, GlassGuard achieves 82% total coverage, compared with at most 61% for the evaluated baselines, while producing 5-17x fewer false voxels per frame. Qualitative examples with a navigation planner illustrate the reconstructed planes blocking paths through glass while leaving traversable routes open. The project page is available at https://glassguardproject.github.io/.
Read the full abstract ▾
All authors Hanwen Guo, Zhengzhi Lin, Yusen Xie, Ji Zhang
Robotics
HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution
Kyochul Jang, Seohyeon Park, Ohchul Kwon et al. (+9 more)
As robotic hardware and learning methods advance, humanoids need tools to perform tasks beyond their inherent physical limits. Successful tool use requires selecting a suitable tool and coordinating manipulation and, when needed, locomotion to complete the task. Existing
benchmarks do not jointly evaluate these capabilities on a humanoid. We introduce HumanoidToolBench, an 18-task
benchmark spanning three scenarios, three execution levels, and two tool-set modes, together with ToolBook, a
dataset of 3.1k demonstrations collected in simulation and on a real Unitree G1. Evaluation of seven policies in simulation and three on the real
robot reveals substantial gaps between selecting a suitable tool and completing the task. Focused GR00T N1.7 probes show reduced selection accuracy on unseen tools and continued task execution under unrelated instructions. Code and data are available at https://snu-pi.github.io/HumanoidToolBench/.
Read the full abstract ▾
All authors Kyochul Jang, Seohyeon Park, Ohchul Kwon, Sangjun Park, Junhyeok Choi, Seungyeop Yi, Chaeyun Kim, Sangkyu Lee, Idan Szpektor, Avi Caciularu, Jongmin Park, Youngjae Yu
Robotics
UniWAM: Unified World-Action Model
Wenxuan Song, Jiayi Chen, Jingbo Wang et al. (+13 more)
Vision-
language-action models benefit from the understanding and
reasoning capabilities of pretrained
vision-
language models, but action-only supervision provides limited grounding in world dynamics. Conversely, world-action models inherit spatiotemporal priors from
video generation models, yet remain limited in semantic understanding and
reasoning under distribution shifts. We introduce UniWAM, a unified architecture that integrates a physical reasoner, a world generator, and an action predictor to jointly learn semantic understanding of the physical world, visual generation, and action prediction. To ensure the quality of the training data, we developed a rigorous data cleaning and annotation pipeline for both human egocentric data and
robot data. To adapt the
vision-
language component to embodied tasks while preserving its inherited
language capabilities, we represent low-level actions in natural
language and introduce a pre-training recipe that assigns complementary supervision from visual question answering (VQA) data, human egocentric data, and
robot demonstrations to the appropriate model components. During post-training, future visual noise augmentation reduces reliance on precise future predictions, while history-conditioned flow matching uses encoded action history to initialize action generation. Together, these designs significantly reduce denoising steps while maintaining performance. UniWAM achieves state-of-the-art (SOTA) performance across multiple evaluations, including in-distribution performance, robustness, generalization, instruction following, and long-horizon task execution. Furthermore, we uncover a log-linear scaling law of unified human-
robot co-training, demonstrating the effectiveness of large-scale pre-training on a mixture of human and
robot data.
Read the full abstract ▾
All authors Wenxuan Song, Jiayi Chen, Jingbo Wang, Shuai Zhou, Xicheng Gong, Zehua Fan, Ziyang Zhou, Junwu E, Haodong Yan, Fuhao Li, Qize Yu, Xu Huang, Pengwei Wang, Wen Chen, Shunbo Zhou, Haoang Li
Robotics
H-SPAR: Hydrodynamic-aware Simulation for Particle Transport and Autonomous Robots
Navid Zarrabi, Nariman Yousefi and Sajad Saeedi
Environmental robotic sampling requires considering the dual influence of water currents on robotic motion and particle transport. Existing marine robotics simulators generally model flow, autonomy, and sampling targets separately, limiting joint evaluation of mission cost and sampling performance. H-SPAR integrates spatially and temporally varying velocity fields, Lagrangian particle transport, probabilistic sampling, and ROS 2/Gazebo-based uncrewed surface vehicle (USV) autonomy. In this work, shared precomputed flow fields drive particle advection and current-induced forces during closed-loop vehicle execution. Path-planning experiments show that the existing current-aware planner SVF-RRT* achieves 69.4% lower upstream cost than conventional RRT* at the planning level, but this reduction falls to 41.7% during execution under time-varying currents, reflecting temporal flow variation, vehicle motion constraints, and path deviation omitted during planning. Coverage experiments show that sweep orientation changes the particle-sampling rate by up to 22.2% under the complete H-SPAR configuration. These findings highlight the importance of evaluating planning, vehicle execution, particle transport, and sampling together under consistent hydrodynamic conditions. The project webpage is available at https://sites.google.com/view/h-spar, and the open-source code is available on GitHub at https://github.com/naviiidz/h-spar-sim.
Read the full abstract ▾
Robotics
BLT*: Informed Belief Localization Trees for Uncertainty-Aware Planning on Digital Twins
Elliot Preston-Krebs, Abhishek Goudar and Timothy D. Barfoot
We present Informed Belief Localization Trees* (Informed BLT*), a sampling-based belief space planning (BSP)
algorithm that scales to large outdoor digital twins with point-cloud observations. We adapt RRT* and Informed RRT* to belief space using the $2$-Wasserstein ($W_2$) metric. Assuming isotropic Gaussian beliefs, sampled belief states can be connected efficiently while accounting for available information and probabilistic collision constraints. This enables steering and rewiring without repeatedly propagating observations, and allows previously computed measurement information to be reused. We present a framework to generate semantically labelled digital twins for planning in real-world environments with point-cloud-based localization. Experiments in simulated environments and digital twins show faster initial solution discovery in most maps with competitive cost convergence.
Read the full abstract ▾
Robotics
Training-Free Diffusion Planning with Analytical Local Scores
Michael Y. Fatemi, Jinhao Liang and Ferdinando Fioretto
Path finding and multi-
robot motion planning require trajectories that are smooth, goal-directed, and collision-free in environments with complex
geometric constraints. Recent
diffusion-based planners have shown that trajectory generation can be cast as iterative denoising which has opened the doors to learning-based approaches that can handle multi-modal trajectory distributions and refine entire trajectories. However, a key limitation is that
diffusion planners require training on large collections of feasible trajectories, rendering them map-specific, and difficult to deploy when high-quality demonstrations are unavailable. This paper introduces a training-free
diffusion-based motion planner that replaces learned global trajectory scores with analytical local scores derived from obstacle, smoothness, velocity, and inter-
agent feasibility terms. The proposed idea relies on a key observation: the score of a trajectory can be reconstructed by considering only local interactions between neighboring waypoints and nearby constraints. This structure exploitation yields a decomposed denoising procedure that retains the optimization structure of classical trajectory methods while inheriting the iterative refinement behavior of
diffusion models. Experiments on a large collection of complex environments and large multi-
agent planning tasks show that the proposed analytical score produces smooth and feasible trajectories within limited computational costs, for example in generating feasible paths for 300+
agents in environments containing 100+ obstacles in under 6 seconds on a GPU, outperforming strong learning-based and optimization baselines, while avoiding the data requirements of learned
diffusion planners.
Read the full abstract ▾
Robotics
TouchTherm: Building Multimodal Digital Twins of Objects for Tactile and Thermal Rendering
Yitao Zhang, Hong Ying, Haoran Guo et al. (+3 more)
Robotic simulation and virtual reality increasingly require object assets that capture not only visual geometry but also the physical cues underlying tactile and thermal interaction. Existing 3D
datasets and reconstruction methods primarily represent object-scale geometry and visual appearance, overlooking microscale surface structure for high-fidelity haptic rendering and transient temperature dynamics for temperature-aware interaction. We present TouchTherm, a framework for constructing simulation-ready visuo-tactile-thermal object assets from real-world objects. For visual and tactile reconstruction, we combine structured-light scanning with multiview normal maps obtained from photometric stereo. The normal maps are registered to the scanned geometry and transformed into tangent space to recover local micro-height fields for optical tactile rendering, while the coarse mesh handles collision detection. For thermal reconstruction, we capture synchronized multiview infrared
videos of natural cooling following controlled heating and reconstruct a physics-regularized dynamic thermal field. Experiments on 20 objects show that the reconstructed micro-height fields preserve dominant surface structures and recover higher-frequency details beyond the coarse geometry, while the thermal fields achieve held-out surface-temperature MAEs of 0.465 degrees C and 0.592 degrees C at 30 s and 45 s, respectively. The resulting tactile assets support synthetic-to-real object recognition from tactile observations, while a glove-based VR system demonstrates spatially and temporally varying thermal feedback. These results highlight the potential of TouchTherm for
multimodal sensory simulation and temperature-aware virtual interaction.
Read the full abstract ▾
All authors Yitao Zhang, Hong Ying, Haoran Guo, Xiaoying Zhou, Guanyu Chen, Chenxi Xiao
Robotics · Cross-listed from cs.CV
Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens
Ruiyang Si, Jianxin Bi, Shunyu Yang et al. (+12 more)
Vision-
language model (VLM)
agents can control
robots through visual feedback and action primitives, but repeated LLM calls and redundant observations incur substantial
token overhead. We introduce PyRUA-Lean, an interactive code-execution framework in which the
agent composes
robot primitives, including
vision-
language-action (VLA) policies, into Python
cells that check and retry locally and return only the images and state they request. Across 700 simulated task instances from LIBERO-PRO, RoboTwin 2.0, and RoboCasa365, it raises success from 63.1% to 71.7% over tool calling with the same GPT-6 Astra planner and primitives, using 49% fewer LLM calls and 65% fewer input
tokens on jointly solved instances. On RoboDojo, without any VLA policy or privileged information, it reaches 51.6% average success rate (Score 58.2), 19.4 points above the top entry of the public leaderboard. Based on the PyRUA-Lean framework, we train PyRUA-Lean Coder v0.1-4B, an open 4B model, with GRPO on 45 LIBERO-PRO instances; it improves from 18.8% to 44.1% on unseen task instances, surpassing the Qwen3.5-27B base model.
Read the full abstract ▾
All authors Ruiyang Si, Jianxin Bi, Shunyu Yang, Rui Ni, Wenbo Huang, Qiang Wang, Shulong Jiang, Yintianrun Zhang, Feixiang Ruan, Kewei Zhang, Duomin Wang, Xiuyu Li, Haiwen Feng, Zhen Dong, Daquan Zhou
Robotics · Cross-listed from cs.CV
CLoSeR: Closing the Loop for Long-Context Streaming Reconstruction
Moyang Li, Zihan Zhu, Wei Zhang et al. (+2 more)
Feedforward foundation models have recently shown remarkable 3D reconstruction capabilities. However, existing models exhibit large tracking drift in long-context streaming reconstruction due to error accumulation. In this paper, we revisit loop closure with streaming reconstruction foundation models to enable accurate, drift-free, kilometer-scale reconstruction. Specifically, our method detects loop candidates through global descriptor
retrieval, and constructs loop-conditioned windows to estimate the relative poses between looped frames. Given the observation that our adopted streaming reconstruction backbone produces a globally consistent scale, we optimize all frame poses on the SE(3)
manifold with sequential and loop closure constraints, avoiding the pose graph optimization on the Sim(3) or higher-dimensional SL(4)
manifolds employed in prior works. Extensive experiments show that our method reduces drift and produces consistent geometry on kilometer-scale sequences, significantly outperforming the state of the art. Code is available at https://github.com/MoyangLi00/CLoSeR.git.
Read the full abstract ▾
All authors Moyang Li, Zihan Zhu, Wei Zhang, Marc Pollefeys, Daniel Barath
Robotics
Robot Learning on Discrete Surfaces: Theory and Applications
Matteo Dalle Vedove, Fares J. Abu-Dakka, Luigi Palopoli et al. (+2 more)
All the objects composing our world are enclosed within surfaces. Yet, most
robot learning and motion generation frameworks treat surfaces as constraints ignoring their intrinsic geometry. This gap is acute for polyhedral meshes–the standard output of CAD and 3D reconstruction–whose discrete
geometric structure remains unexploited. In this paper, we propose a unified discrete Riemannian framework that enables
robot learning directly on polyhedral surface meshes. Using discrete differential geometry, we define logarithmic and exponential maps, parallel transport, and ambient-space projections that remain well-defined across faces, edges, and vertices. We instantiate the framework in three learning paradigms: (i) Dynamic Movement Primitives (DMPs), an improved exponential-map computation and a fixed-tangent-cone forcing-term encoding with parallel transport yield better cross-surface generalisation and stability over prior mesh-based approaches. (ii) Gaussian Process (GP), a geodesic-based kernel with practical admissibility control, enables regression at arbitrary mesh locations without smoothness assumptions. (iii) Riemannian Flow Matching (RFM), mesh-native operators improve generative quality over spectral baselines while reducing training time. The framework is validated in simulation against state-of-the-art methods and demonstrated on two real-
robot scenarios: generalising user-drawn trajectories across different surfaces and planning polishing motions on RGB-D-reconstructed surfaces.
Read the full abstract ▾
All authors Matteo Dalle Vedove, Fares J. Abu-Dakka, Luigi Palopoli, Daniele Fontanelli, Matteo Saveriano
Robotics
Towards Physical Underwater Robotic Assistance for Scuba Diver Movement in Confined Spaces
Demetrious T. Kutzke and Junaed Sattar
Scuba divers are taught to control their depth to avoid rapid ascents and descents, which could result in serious injuries such as gas embolisms and barotrauma. However, many underwater tasks necessitate lateral control, maintaining distance between subsea structures such as coral reefs, submerged drilling instrumentation, or unexploded ordnance. In this work, we discuss a first-of-its-kind wearable robotic solution providing thruster-actuated directional guidance to a diver, as distinct from prior propulsive-assistance exoskeletons. We introduce “Robotic Assisted Diver Movement in Confined Spaces” (RADMCS), a wearable
robot that assists divers in maintaining a fixed distance from subsea structures by leveraging perception techniques in monocular depth estimation and force-feedback from submersible thrusters to provide haptic feedback. Its small and compact form factor creates a foundational platform that could be expanded to include more sophisticated control and navigation behaviors. We present results from Institutional Review Board (IRB) in-water studies with eight human scuba diver participants on threshold sensitivity tests in both a closed-water swimming facility and ocean environments; distance-maintaining experiments in a closed-water facility; and form, fit, and function testing in the ocean. We demonstrate that relatively low thrust values (10 percent of maximum) allow robotic direction of a human's movement using the physical sensation of the
robot's guidance.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
LiteReality-Agent: An Agentic System for Interactable 3D Indoor Scene Reconstruction
Zhening Huang, Yueyan Li, Johnathan Chiu et al. (+5 more)
We present LiteReality-
Agent, an agentic system for reconstructing real indoor environments as realistic, articulated, and simulation-ready 3D scenes from RGB-D scans. At its core, LiteReality-
Agent formulates 3D reconstruction as a coding problem, in which a coding
agent gathers evidence using specialised tools and iteratively edits a Python script, Room.py, which can be executed to produce a 3D digital twin of the room. With this formulation, we develop a robust observe-edit-verify harness that supports evidence gathering, measurement, verification, layout optimisation, simulation readiness, and quality control throughout the reconstruction process. LiteReality-
Agent produces high-quality reconstructions suitable for simulation and downstream embodied AI tasks. Furthermore, as
agent capabilities continue to improve rapidly, the system introduced by LiteReality-
Agent remains a strong orchestration framework for future
agents: it equips them with specialised tools, structured workflows, and robust verification mechanisms that substantially improve reconstruction quality and reliability. We demonstrate that LiteReality-
Agent produces reconstructions that are more geometrically accurate, visually realistic, and simulation-compatible than those generated by recent frontier models, such as Astra and Fable. We therefore view LiteReality-
Agent as a practical and important building block for robust real-to-sim systems. Both the source code and the data-capture application are publicly available. Code:https://github.com/LiteReality/LiteReality-
Agent/
Read the full abstract ▾
All authors Zhening Huang, Yueyan Li, Johnathan Chiu, Xiaoyang Lyu, Matt Zhou, Yuxin Yao, Joan Lasenby, Shangzhe Wu
Robotics
ChunkVLA-AM: Parallel Action Chunking for Vision-Language-Action Robot Control in Additive Manufacturing
Zhugang Liu, Kaichuang Zhang, Jinman Zhang et al. (+7 more)
Vision-
language-action (VLA) models unify visual perception,
language understanding, and action generation, offering new opportunities for automation in additive manufacturing (AM). However, deployment in AM remains challenging because adapting these models to unseen
robot embodiments is costly, and performance can degrade under environment changes. In this work, we present a framework for deploying OpenVLA-OFT on a FAIRINO FR3
robot in a fixed AM workcell. A data pipeline converts monocular real-world demonstrations into OpenVLA-compatible TFDS/RLDS
datasets to support adaptation to the FR3 embodiment. At runtime, each inference request predicts an eight-step chunk of 7-D actions. The FR3 executes each chunk open loop before capturing a new observation, providing closed-loop feedback between chunks. The system uses a cloud-edge architecture in which the FR3 client streams observations to a remote inference server through a FastAPI interface. In 42 physical A-to-B object-transfer trials, evenly split between red and blue targets, the system succeeded in 39 (92.9%). All three failures occurred during final placement, when insufficient release-height control caused the object to topple. An illumination sweep identified a low-error luminance range of 85-125 on a 0-255 scale, with the lowest mean spatial error at 95.
Read the full abstract ▾
All authors Zhugang Liu, Kaichuang Zhang, Jinman Zhang, Pu Sun, Martha Asare, Jose Hernandez, Maxim Ermolinsky, Efren Saenz, Qi Lu, Jinghao Yang
Robotics
FlashDexRetarget: Accelerating Dexterous Manipulation Data Generation through Multi-Motion Retargeting
Kyungmin Lee, Sibeen Kim, Dongyoon Hwang et al. (+6 more)
Human hand-object demonstrations provide a scalable source of data for dexterous
robot learning, but transferring them across embodiments requires physically feasible retargeting. Existing physics-based methods typically optimize each demonstration independently, leading to either limited success under finite simulation budgets or training costs that grow with
dataset size. We introduce FlashDexRetarget, an RL framework for multi-reference dexterous retargeting. We formulate retargeting as multi-reference tracking, jointly learning a single policy across many demonstrations with off-policy RL and
geometric supervision of the demonstrated interactions. This shared training formulation amortizes optimization across references while enabling the policy to track diverse hand-object interactions. On a 50-motion
benchmark from TACO, OakInk2, and HOT3D using XHand and Sharpa Wave Hand as target embodiments, FlashDexRetarget retargets 90% of demonstrations using about 30 GPU-hours, compared with about 46% at about 3,000 GPU-hours for CHORD. This corresponds to about 100 times lower training compute and a 44-percentage-point improvement in retargeting success. Ablations examine the key design choices, while experiments with up to 1,000 motions and real-world replay further demonstrate the scalability and practical applicability of our method.
Read the full abstract ▾
All authors Kyungmin Lee, Sibeen Kim, Dongyoon Hwang, Yoonsang Oh, Donghu Kim, Youngdo Lee, I Made Aswin Nahrendra, Jaegul Choo, Hojoon Lee
Robotics · Cross-listed from eess.SY
Finite-Data Safety Informativity Under Dynamic Asymmetric Actuation
Abhinav Sinha, Praveen Kumar Ranjan and Yongcan Cao
When the system model is not fully known, measurement error and limited excitation can leave several models consistent with the same finite data. A command judged safe for one model may fail for another, while limited control authority can prevent the corrective action needed to preserve safety. To ensure safety under model uncertainty and asymmetric input limits, we develop a finite-data certificate that determines whether a command can enforce a prescribed safety inequality. For a linearly parameterized safety channel with exactly known regressors and bounded aggregate residual error, we derive a support formula for the worst-case safety contribution of all data-consistent models. The formula identifies the regressor directions that admit a finite bound, allowing rank-deficient records to contribute to safety certification. Using certified componentwise bounds on actuator tracking error yields an affine inequality with a necessary and sufficient test for pointwise command feasibility. The affine inequality reduces computation of the closest certified command to a scalar root-finding problem. It also yields a closed-form gate that selects the largest certified fraction of a prescribed command segment. The proposed certificate guarantees output safety within its operating domain, provided the feedback is locally Lipschitz and the uncertainty bounds remain valid. Domain retention and full-state continuation extend this guarantee to all time. A vehicle study demonstrates that output safety can be certified from finite measurements in a safety-critical setting with model and actuator uncertainty.
Read the full abstract ▾