Robotics
MonoEgo: Monocular Metric Egocentric Demonstration Capture with Passive Wrist Constellations and Sparse Workstation Anchors
Jie Xu, Kangjin Yu, Ziyi Jin et al. (+2 more)
Image-aligned metric demonstrations often require dedicated tracking hardware and synchronization across devices. We present MonoEgo, a capture system that replaces active wrist instrumentation with offline monocular reconstruction. One 90-FPS global-shutter camera observes calibrated passive wrist constellations, sparse workstation anchors, and the scene on a shared image clock. MonoTag SLAM combines marker corners with ORB geometry and uses visual evidence to reject ambiguous planar-marker poses. Its metric Atlas supports interval scale re-anchoring, verified map merging, and retrospective localization of earlier frames supported by the final map. Camera and wrist-constellation outputs retain validity and map provenance, and unsupported motion is left missing. Experiments show metric tracking beyond continuous anchor visibility, reconnection of supported map components, and recovery of some missing camera poses. Comparisons against a multisensor camera reference and separate stationary-constellation tests characterize trajectory agreement and precision while revealing incomplete coverage and residual
geometric uncertainty. The results indicate that passive fixtures and offline reconstruction can reduce capture-side requirements. Dynamic accuracy, deployment, and downstream policy benefits require further study.
Read the full abstract ▾
All authors Jie Xu, Kangjin Yu, Ziyi Jin, Beichen Wang, Zhongpu Xia
Robotics
Robot-Assisted Deployment and Maintenance of Inflatable Modules for Lunar Habitation: A Field Demonstration
Kentaro Uno, Shamistan Karimov, Ashutosh Mishra et al. (+5 more)
Long-term human habitation and in-situ development on the Moon open a new era of space utilization. In this context,
robots are a key technology for facilitating the construction of future human outposts. Toward the deployment and establishment of human habitation modules on the lunar surface, we propose a combined system consisting of inflatable modules and a modular, reconfigurable robotic system. This paper presents a report demonstrating various
robot-assisted task executions using real hardware, namely the modular and reconfigurable
robot MoonBot and the inflatable module HIDAS, to enhance the reliability of their deployment and maintenance. The demonstrated tasks include robotic inspection during inflation, module position
alignment, final safety locking, and three-dimensional mapping for post-deployment maintenance. All demonstrations were conducted either in a laboratory environment or at a lunar analogue test site. Finally, lessons learned are discussed to provide essential insights for this robotic application to future lunar habitation.
Read the full abstract ▾
All authors Kentaro Uno, Shamistan Karimov, Ashutosh Mishra, Elian Neppel, Hazal Gozbasi, Shreya Santra, Shinichi Kimura, Kazuya Yoshida
Robotics
Model-Informed Safe Reinforcement Learning for Bipedal Locomotion via Step-to-Step Prediction
Victor Paredes and Ayonga Hereid
Humanoid
robots promise versatile mobility in cluttered, human-centric environments, but real deployment demands principled safety. Classical model-based gait generators yield interpretable motions but often lack the robustness and adaptability of modern reinforcement learning (RL) based approaches. We propose a model-informed reinforcement learning framework anchored to the analytical Angular Momentum Linear Inverted Pendulum (ALIP) template. We provide a step-to-step safety certificate for ALIP stepping via a discrete exponential control barrier function (DECBF) and use it as (i) a training-time shaping signal and (ii) a runtime action filter that minimally adjusts swing-foot placement to satisfy template-level constraints. Full-order safety is evaluated empirically on the Digit humanoid in MuJoCo with a whole-body controller stack. Compared to an unconstrained baseline, our approach reduces safety-violation events in the reported external-disturbance trial, while larger lateral-velocity transients reveal a safety-tracking tradeoff.
Read the full abstract ▾
Robotics
Trajectory-Safe Orienteering for Human-Robot Shared Environments
Songqun Gao, Elena Basei, Marco Roveri et al. (+2 more)
Orienteering problem (OP) has wide real-world applications and also great potential in human-
robot collaboration. However, existing approaches struggle to simultaneously ensure safe and feasible trajectories while achieving high-quality task execution in shared workspaces. To this end, this work studies the OP with time windows and variable profits (OPTWVP). A two-stage DEcoupled discrete-Continuous Optimization with Service-time-guided Trajectory (DeCoST) approach is proposed to effectively solve OPTWVP in shared spaces. Meanwhile, the safety-aware time windows of nodes and the discretized workspace are introduced to ensure collision-free trajectories between the end effector and the human. Preliminary results validate the effectiveness of DeCoST in generating collision-free trajectory plans while preserving the quality of orienteering tasks.
Read the full abstract ▾
All authors Songqun Gao, Elena Basei, Marco Roveri, Luigi Palopoli, Daniele Fontanelli
Robotics
ARS: Agentic Reward System for Robot Learning
Sheng Hu, Weiyi Lu, Lingbing Zeng et al. (+5 more)
Progress reward modeling is the problem of estimating how a
robot's behavior changes task progress over time. Reliable estimation requires distinguishing meaningful state changes from failed attempts and task-irrelevant actions. We introduce the Agentic Reward System (ARS), an inference framework for progress reward modeling with general-purpose
vision-
language models (VLMs), without additional reward-model training. Given an offline trajectory and a task instruction, ARS uses adaptive visual inspection for both event proposal and verification. A subagent proposes a task-relevant event timeline, which a primary
agent verifies and revises before estimating per-frame progress. ARS can incorporate optional terminal outcome labels and visual references to inform its judgments. It can also audit progress estimates from external reward models. We evaluate ARS with a 27B VLM on a controlled semantic-mismatch
benchmark and downstream policy learning in simulation and on a real
robot. The
benchmark reveals that several evaluated reward baselines assign spurious progress to wrong-object manipulation even in simple pick-and-place scenes. ARS better suppresses these errors and outperforms these baselines in simulation policy learning. We further demonstrate that ARS supports long-horizon policy learning from mixed-quality offline experience on real-
robot multi-screw fastening in a full-scale laboratory replica of an industrial washing-machine assembly line. These results suggest that structured inference and verification can improve the usefulness of general-purpose VLMs for
robot reward modeling. Code is at https://github.com/midea-ai/ars
Read the full abstract ▾
All authors Sheng Hu, Weiyi Lu, Lingbing Zeng, Gan Weng, Weiwei Zhang, Kai Xie, Xiaofeng Mou, Yi Xu
Robotics · Cross-listed from cs.CV
When the Score Becomes the Target: Rethinking Metric Validity in Autonomous Driving
Morui Zhu, Deyuan Qu, Qi Chen et al. (+2 more)
Driving
benchmark scores are increasingly used not only for evaluation but also as optimization targets. This raises a fundamental question: do score gains remain reliable evidence of driving improvement once the score itself is optimized? We address this question by examining how the scoring process responds to changes in driving behavior and whether the resulting gains persist under repeated execution and replanning. We decompose the process into execution, measurement, subscore mapping, and aggregation. Controlled interventions reveal substantial behavioral changes that receive little score response because distinctions are omitted, thresholded, or attenuated between requested and executed motion. Closed-loop comparisons further show that optimization gains can reverse when the execution interface changes, demonstrating their dependence on how requests are executed and returned as feedback. Together, these findings connect the behavioral distinctions preserved by a metric to the conditions under which its gains transfer. Metric validity under optimization therefore requires examining both what the scoring process measures and how the optimized behavior is executed.
Read the full abstract ▾
All authors Morui Zhu, Deyuan Qu, Qi Chen, Kentaro Oguchi, Qing Yang
Robotics
From World Models to World Action Models: Rethinking Next-State Prediction
Tingyu Yuan, Ziming Ji, Biaoliang Guan et al. (+9 more)
Predicting the next state is a core paradigm of World Models for modeling physical dynamics, emphasizing prediction fidelity. As World Models evolve into World-Action Models (WAMs), existing methods still fix the next state before training as RGB, a single latent feature, or a static combination of predefined targets, thereby constraining action learning to the inductive biases preserved by a particular representation. To address this limitation, we propose CF-WAM, a dynamic next-state prediction framework that samples visual, semantic,
geometric, and interaction projections of the same future, standardizes them into a common
video form, and supervises a unified WAM across these projections. The action-relevant constraints exposed by these projections accumulate across training steps, forcing WAM to capture the underlying state-transition structure that supports multiple projections of the same action-conditioned future. This dynamic mechanism also provides a natural cross-embodiment dynamics reference frame for Human and
Robot learning. By jointly learning across different next-state parameterizations, heterogeneous Human and
Robot experience can bypass appearance differences and directly contribute to shared state-transition learning, improving cross-embodiment generalization. Experiments show that CF-WAM improves both training efficiency and final control performance, while translating Human experience effectively into policy gains. CF-WAM achieves state-of-the-art performance on RoboCasa-GR1 with an average success rate of 82.50%, while reaching 82.65% on LIBERO-Plus and up to 84.00% in real-world evaluations.
Read the full abstract ▾
All authors Tingyu Yuan, Ziming Ji, Biaoliang Guan, Wen Ye, Wenrui Tian, Zhaopeng Gu, Feihong Zhang, Xu Yang, Yan Huang, Zhaowen Li, Chaoyang Zhao, Jinqiao Wang
Robotics
From Language to Task Maps: Compiling Semantic Relations While Preserving Task-Relevant Freedom
Jaegyun Park, Jingwang Lee, Jungsoo Lee et al. (+2 more)
Natural-
language manipulation instructions specify qualitative relations, whereas continuous controllers require state-evaluable task quantities, differentials, and completion conditions. Because a qualitative relation generally leaves part of the relative configuration unspecified, expanding it into a complete pose can introduce unintended constraints. We present a typed semantic-to-
geometric interface in which
language specifies entities, relations, and phases, while each relation indexes a registered specification of its task-relevant distinctions and preserved freedoms. A
robot-side compiler grounds these specifications, constructs relation-specific task maps and consistent differentials using conformal
geometric algebra, and composes the resulting policies through RMPflow. To evaluate the division of responsibility between the
language model and the compiler, we compared a Semantic
Topology interface with one that additionally requires relation-specific
geometric specifications over 60 instructions. Both produced correct shared semantic content in 41/60 cases, but critical errors under their respective interface requirements occurred in 19/60 and 58/60 cases. Across 64 grounded evaluations spanning eight
geometric relation forms, the task maps preserved registered null directions and responded to relation-relevant perturbations; analytic directional derivatives agreed with finite differences, and Jacobian ranks matched the registered dimensions. In three closed-loop ablations using a simulated Franka Emika Panda in MuJoCo, fixing a relation-preserved coordinate increased median terminal progress error by 20.24–71.00~mm while the retained relation errors remained within their evaluation bounds. These results support compiling relation-visible geometry and preserved freedom together into composable continuous objectives.
Read the full abstract ▾
All authors Jaegyun Park, Jingwang Lee, Jungsoo Lee, Soonwoong Hwang, Wansoo Kim
Robotics
Contact-Aware Impedance Controller for Robot-Assisted Ultrasound Imaging
MD Miraj Arefin and M Efe Tiryaki
Safe
robot-assisted ultrasound imaging requires a reliable controller able to detect and localize probe–tissue interaction. In this paper, we present a B-mode ultrasound image-based contact perception method and a contact-aware impedance controller for robotic ultrasound imaging. The proposed method detects acoustic contact independently of force measurements, enabling contact-conditioned force/torque taring to reduce residual wrench bias. During contact, the method continuously estimates the effective contact location along the curved probe surface and uses it to update the controller interaction frame, enabling visual servoing of the physical probe–tissue contact point during imaging. Experiments on an agar phantom demonstrated a contact-localization RMSE of $\mathbf{1.46 \pm 0.14}$~mm over probe roll angles from $\mathbf{-15^\circ}$ to $\mathbf{15^\circ}$. During static rolling, the proposed controller maintained task-space tracking accuracy comparable to a conventional fixed-frame impedance controller while reducing the maximum compressive interaction force from $\mathbf{31.56}$~N to $\mathbf{20.09}$~N, corresponding to a $\mathbf{36.3\%}$ reduction. These results demonstrate the potential of ultrasound images as direct contact feedback for safe and accurate
robot-assisted ultrasound imaging.
Read the full abstract ▾
Robotics
RoboIRGBench: Benchmarking Implicit Referential Grounding in Vision-Language-Action Models
Aernaer Akelijiang, Jiannan Li, Zhineng Chen et al. (+2 more)
Vision-
Language-Action (VLA) models have shown strong capabilities in robotic manipulation, yet existing
benchmarks typically assume that task-relevant information is explicitly specified in the instruction. In practice, however, humans frequently refer to objects, quantities, and relations implicitly, requiring
robots to recover the intended target from linguistic and perceptual context. We study this capability as Implicit Referential Grounding (IRG) and introduce RoboIRG-Bench, a manipulation
benchmark designed to systematically evaluate it. Built upon RoboMME, RoboIRG-Bench contains 40 variants derived from 11 tasks and covers four challenges, including direct,
reasoning-mediated, spatial, and contextual referential grounding. As IRG often requires retaining and retrieving previously established context, we evaluate representative VLAs spanning different
memory mechanisms. Our evaluation reveals a noticeable referential robustness gap. Models that perform well under explicit instructions can degrade sharply when the same task-relevant information must be recovered from context.
Reasoning-mediated and spatial references are particularly challenging, while models using external VLMs show greater robustness but still exhibit significant failures. Moreover, replacing the external VLM with a stronger model does not eliminate these gaps. We further validate these findings on a Franka Research 3
robot arm, where the gap persists under real-world manipulation and manifests as both incorrect referent grounding and downstream execution failures. These results establish IRG as a distinct and underexplored capability for reliable robotic instruction following and highlight the need for VLAs that can robustly integrate
language, perception,
reasoning, and action.
Read the full abstract ▾
All authors Aernaer Akelijiang, Jiannan Li, Zhineng Chen, Jingjing Chen, Bin Zhu
Robotics · Cross-listed from cs.LG
LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models
Luzhe Huang, Lei Chu, Jingyi Liang et al. (+1 more)
Compact JEPA world models enable efficient latent-space planning, but low-dimensional representation trained under reward-free self-supervision must encode both action-conditioned dynamics and predictable visual context. This competition can entangle controllable state with high-rank nuisance appearance and degrade planning as scenes become more complex. We introduce LRC-JEPA, a lightweight end-to-end world model that routes information into a compact predictive latent $\mathbf{z}$ and learned-query residual-context
embeddings $\mathbf{u}$. Only $\mathbf{z}$ is propagated by the dynamics model and used for planning, while $\mathbf{u}$ captures temporally persistent information for cross-
attention reconstruction; a differentiable residual connection encourages the latent to retain complementary dynamic content. Under explicit assumptions, we show that the resulting representation is sufficient, minimal, nuisance-invariant, and disentangled. Across four simulated control environments, LRC-JEPA improves average planning success over a parameter-matched JEPA baseline by 9 percentage points and matches or exceeds substantially larger pretrained models. On the real-world Bridge-v2 set, its 5.5M-parameter active encoder outperforms DINO-WM (22.1M) and V-JEPA2 (303.9M) encoders while also enabling faster planning. Physical-state probes, reconstruction interventions, and ablations confirm the effectiveness of LRC-JEPA's representation disentanglement.
Read the full abstract ▾
All authors Luzhe Huang, Lei Chu, Jingyi Liang, Yuhuan Zhao
Robotics · Cross-listed from cs.LG
Emergence, Not Bandwidth: Physical Coupling and the Limits of Learned Multi-Agent Communication
Mihir Chauhan and Aniket Bera
Rate-limited multi-
agent teams raise three questions the emergent-communication literature has answered only empirically: what an optimal message should encode, what compression costs over a horizon, and when a learned protocol is unique enough for a teammate to read. We answer them for rate-limited Dec-POMDPs, then measure how far reinforcement learning falls short of the optimum. Our
theorems fix what is achievable independently of any learner, so a gap between an engineered and a learned sender at the same bit budget is an optimization fact, not an information-theoretic one. We instantiate this on three MuJoCo arenas spanning zero, partial and rigid physical coupling, charging every condition exactly 2 bits per decision, and create the discriminating regime by closing a physical side channel within one arena, holding bodies, task and reward fixed. Communication value is governed by coupling: under rigid coupling through a shared object, no channel beats silence (+0.001 +/- 0.001, p = 0.982, n = 25), since proprioception already carries that information; without coupling, every condition solves the task; under partial coupling, the engineered 2-bit sender reaches an interquartile mean of 1.000 but the learned one reaches 0.482, indistinguishable from silence (p = 0.400, n = 25). With a shared alphabet, bandwidth cannot explain the gap. Warm-starting from an engineered receiver localizes the failure: the same channel reaches 0.857 versus 0.562 cold-started (p < 0.001), so it is neither representational nor one of maintenance; reinforcement learning fails to discover the protocol. Cross-play shows learned protocols are individually meaningful but mutually unintelligible: self-play 0.980 collapses to 0.144 across seeds, and our best constructed
alignment leaves at least 77% of that gap. All headline results use 25 seeds per arena and seven published baselines at matched rate.
Read the full abstract ▾
Robotics
FutureDuet: Decoupling Observation Access from Future Supervision in World Action Models
Jie Wu, Yuzhi Huang, Junqi Liu et al. (+5 more)
World Action Models (WAMs) augment
robot action generation with future visual supervision. Existing WAMs commonly fuse main and wrist observations into one visual stream and train both with the same future-
video objective, despite their different visual dynamics. A stable main camera reveals scene-level task
evolution, whereas wrist cameras move with the end effector, mixing local interaction changes with viewpoint shifts and self-occlusion. These contrasting predictive demands suggest that the two views may benefit from different future objectives. We introduce FutureDuet, which retains both views for control, while allowing each visual stream to receive a different future objective. For the main view, future RGB models task
evolution, while interaction masks and
robot skeletons focus supervision on task objects and
robot motion. For the wrist stream, future latent prediction models short-horizon interaction changes without requiring pixel-level reconstruction. ActionDiT jointly reads the resulting Task State and Interaction State, combining scene-level progress with close-range interaction evidence. All auxiliary prediction modules are training-only, adding no inference overhead. FutureDuet achieves 94.2% clean and 94.1% randomized success on RoboTwin50 and 99.2% average success on LIBERO. The improvements are most pronounced on six RoboTwin50 tasks that require precise interaction, averaging gains of 9.2% and 12.8% over Fast-WAM in clean and randomized settings. Controlled studies further show complementary gains from separating the wrist pathway and designing future supervision separately for the two views.
Read the full abstract ▾
All authors Jie Wu, Yuzhi Huang, Junqi Liu, Weichen Zhang, Haibin Huang, Yin Chen, Jingyan Jiang, Chi Zhang
Robotics
Predictive Semantic Safety: From Visual Physical Reasoning to Safety-Critical Control
Taekyung Kim, Salem Fradi, Yanning Dai et al. (+2 more)
Physical interactions can create future hazards that are not apparent from the
robot's current
geometric surroundings. We present a framework termed Predictive Semantic Safety (PSS), which connects visual physical
reasoning to backup-based safety filtering. A
vision-
language model (VLM) predicts physical events and their timing or directly predicts object displacements. An explicit motion model converts event hypotheses into object trajectories. Split conformal prediction calibrates position errors jointly across specified objects, observation times, and future times;
geometric shape bounds convert the resulting position regions into predicted object occupancy. PSS evaluates a prescribed backup maneuver against this occupancy and derives input-affine constraints for minimally modifying the nominal input while preserving backup feasibility under the
robot dynamics and input limits. MuJoCo experiments with a Unitree Go1 consider falling fixtures, impact-driven support loss, and contact propagation. PSS achieves a safe episode rate of 99.3%, compared with 43.3% for a Backup Control Barrier Function baseline that only uses current obstacle geometry.
Read the full abstract ▾
All authors Taekyung Kim, Salem Fradi, Yanning Dai, Mateusz Ostaszewski, Jürgen Schmidhuber
Robotics · Cross-listed from cs.CV
Text-Vision Synergistic Token Caching: A Training-Free Framework for Efficient Vision-Language-Action Inference
Qianer Li, Chengjie Zhang, Jingwen Chen et al. (+3 more)
Vision-
Language-Action (VLA) models enable generalizable robotic control but remain computationally expensive.
Token caching provides a training-free, plug-and-play acceleration alternative. However, existing VLA caching does not fully exploit a key inductive bias of VLA models: text-
vision synergy, wherein textual semantics guide the precise visual grounding of task-relevant regions. In particular, existing designs insufficiently account for head-wise reliability in
attention aggregation and layer-wise stability in cache reuse. To address this, we propose Text-
Vision Synergistic
Token Caching (TVCache), a training-free framework for efficient VLA inference. TVCache filters
attention heads based on text-
vision information focus to improve task-relevant and physically consistent visual grounding. Concurrently, we introduce a reuse-layer selection mechanism guided by text-
vision entropy differences to avoid caching unstable representations and improve cache resource allocation. Extensive experiments across four representative VLA models, two simulation
benchmarks, and real-world robotic tasks demonstrate the effectiveness and generality of TVCache. At matched
token-retention ratios, TVCache consistently improves task success over existing VLA caching with comparable computational cost. On OpenVLA-OFT, it improves average success by up to 14.5 percentage points over VLA-Cache at 12.5% retention while reducing FLOPs by 2.45x relative to full-
token inference.
Read the full abstract ▾
All authors Qianer Li, Chengjie Zhang, Jingwen Chen, Zanjia Tong, Jiyuan Zhang, Hong Zhang
Robotics
When World Models Lie: Adaptive Safety Analysis Under Wrong Imaginations
John Cao and Somil Bansal
World models offer a powerful substrate for safety
reasoning in high-dimensional robotic systems, but they are also fallible: their predictions can be biased, miscalibrated, or confidently wrong. This creates a central challenge for latent-space safety filters, which often learn Hamilton-Jacobi safety value functions on the dynamics of a world model. If the world model is incorrect, the resulting value function can inherit its errors and produce overconfident safety estimates. Existing latent safety filters often rely on auxiliary signals such as ensemble disagreement or value-target consistency residuals for adaptation, but these signals can remain small even when the world model's predictions deviate from observations. We propose an adaptive latent safety filter that calibrates safety
reasoning using directly observed world-model error. Our method uses Adaptive Conformal Inference to construct online uncertainty sets from discrepancies between predicted and observation-inferred latent states, then evaluates safety pessimistically by minimizing the learned value function over these sets. This allows the filter to remain minimally conservative when the world model is accurate, while becoming more cautious when observations reveal model mismatch. We provide a finite-time coverage guarantee for the adaptive uncertainty radius. Through simulation and hardware experiments, we show that our method significantly reduces failures relative to state-of-the-art latent safety filters while preserving task completion.
Read the full abstract ▾
Robotics
TLC-DiT: Task-Aligned Local Visual Conditioning for Robust Multitask Robot Manipulation
Xianbo Cai, Hideyuki Ichiwara, Zihang Wang et al. (+2 more)
Language-conditioned
robot policies have made clear progress in multitask manipulation, but task-relevant local visual evidence usually stays hidden inside a visual backbone or
attention layers. This leaves the policy difficult to inspect and fragile under visual change, two symptoms of a missing explicit, task-aligned local visual channel. We present TLC-DiT, a plug-in extension of the Multitask
Diffusion Transformer (DiT) policy that adds explicit task-guided local visual feature maps without changing the
diffusion objective or the action-generation process. For each camera view, frozen DINOv2 patch features are modulated by the CLIP task
embedding through FiLM and refined by a lightweight CoordConv CNN adapter into smooth spatial maps, which are concatenated with the original global image,
language, joint-state, and timestep conditions. On LIBERO, TLC-DiT reaches a 93.5% average success rate, compared with 86.5% for Multitask DiT and 79.25% for SmolVLA. On LIBERO-plus, the total success rate improves from 54.07% to 57.24%, with larger gains under camera, background, and sensor-noise changes. In real-world bimanual tasks, TLC-DiT raises Teabag Putting completion from 44% to 89% while maintaining comparable Match Box Opening performance. Feature-map visualizations confirm that the model attends to task-relevant regions across views and perturbations, providing a direct way to inspect the visual evidence.
Read the full abstract ▾
All authors Xianbo Cai, Hideyuki Ichiwara, Zihang Wang, Yijun Lu, Tetsuya Ogata
Robotics · Cross-listed from cs.CV
Dexterous Tactile World Model
Ziyao Zeng, Xiatao Sun, Hao Wang et al. (+6 more)
World models for manipulation are typically trained from
video, yet the events that determine how manipulation unfolds, such as making and releasing contact, are difficult to observe visually and are often easier to sense through touch. We present the Dexterous Tactile World Model (DTWM), a
video world model for future-frame prediction of egocentric manipulation from both observed
video and tactile signals from a glove worn on each hand. We condition a pretrained
video diffusion transformer on each hand's tactile signal through a zero-initialized residual at the corresponding hand location in the
video tokens, while a causal mask prevents predicted frames from accessing future information. Compared with a
vision-only model matched in architecture, parameters, and training, DTWM reduces the underestimation of hand motion from 23% to 9%, while reducing the perceptual error in the hand region by 7.4% across three training runs per model. The benefit also increases over the prediction horizon, with the improvement in the later predicted chunks being about 4.1x larger than in the first. DTWM also outperforms other visual-tactile world models under the same setting, and training with touch improves future-frame prediction even when no touch is available at inference. Ablations show that the model benefits from both the magnitude and spatial location of force: replacing the tactile signal with binary contact states, either per hand or per location, increases prediction error. The observed course of the force indicates whether the interaction will persist or change.
Read the full abstract ▾
All authors Ziyao Zeng, Xiatao Sun, Hao Wang, Yueyang Pan, Zhengxiang Yu, Fengyu Yang, Tianyu Liu, Zhiwen Fan, Daniel Rakita
Robotics
NavHarness: Towards Lifelong Embodied Navigation
Xunyi Zhao, Jian Zhou, Sihao Lin et al. (+6 more)
Frontier models can now perform well on individual embodied navigation tasks through multi-round
multimodal reasoning with simple tools. Across successive tasks, however, an
agent must also rely on an evolving map and earlier search records, both of which may be incomplete or conflict with new observations. We present NavHarness, a training-free embodied harness towards lifelong navigation that makes
memory processing part of the navigation loop. During navigation, its multi-round agentic session draws on maps, task records, and house knowledge, checking them against observations and recording corrections to guide its actions. NavHarness preserves this experience across fresh conversations for new tasks or recovery attempts, while outcome verification and run-end summaries support its later reuse. On GOAT-Bench, NavHarness improves s-SR over context-only independent sessions by 18.6 points with Astra and 22.6 with Opus 5. Using SLAM-estimated poses, NavHarness with GPT-6 Astra achieves state-of-the-art task success of 83.7 s-SR with 36.9 e-SR on GOAT-Bench and 85.9 s-SR on IR2R-CE. To understand these gains, we examine how experience is carried between sessions and find that structured recovery handovers outperform length-matched summaries. In extended deployments across houses, consolidation improves navigation beyond retaining maps and task records, with case studies showing how
agents use earlier experience to interpret new goals, investigate unresolved questions, and resume failed searches. We suggest that progress towards lifelong navigation depends on how successive
reasoning sessions build on prior experience, alongside improvements in single-task capability.
Read the full abstract ▾
All authors Xunyi Zhao, Jian Zhou, Sihao Lin, Gengze Zhou, Zerui Li, Xinyu Yan, Jiajun Liu, Anton van den Hengel, Qi Wu
Robotics
Bayesian Active Learning for Intent Disambiguation in Interactive Robot Planning
Huao Li, Carson Sobolewski, Augustinos Saravanos et al. (+3 more)
Interactive
robot planning requires
robots to infer and execute human intentions from natural
language instructions that are often ambiguous, incomplete, or underspecified. Although large
language models (LLMs) provide a powerful interface for clarification, relying on the generative model to drive an multi-turn conversation can introduce systematic failures. We propose a
Bayesian framework that treats clarification as an active learning problem over grounded Signal Temporal Logic (STL) task specifications. Our method uses LLMs to initialize candidate formal specifications and translate informative contrasts into natural-
language clarification questions, while
Bayesian optimization maintains uncertainty estimation over user intent and selects queries that maximize information gain. After convergence, the inferred STL specification is passed to a formal planner to synthesize a verifiable
robot trajectory. Across four simulated and real-world task domains, our approach generally achieves higher task satisfaction and requires fewer clarification rounds than LLM baselines, while helping smaller models close the performance gap against larger
reasoning models.
Read the full abstract ▾
All authors Huao Li, Carson Sobolewski, Augustinos Saravanos, William Tan, John Karigiannis, Chuchu Fan
Robotics
SAGE: Symbolic Action-Gating and Editing for LLM Task Planners
Trung Minh Bui, JongSul Moon, YoungOuk Kim et al. (+3 more)
Large
language models (LLMs) are now the default cognitive core of embodied household
agents, yet the plans they emit are rarely checked against a grounded model of the environment before execution, and the task-success they report is often measured on
benchmarks so saturated that no method can be separated from another. We present SAGE (Symbolic Action-Gating and Editing), a single-LLM planner built from two lightweight mechanisms: a domain-agnostic symbolic gate (~250 lines of Python, zero
tokens, $O(|\pi|)$) that blocks precondition-violating actions with typed reasons as a runtime safety monitor, and a local edit that regenerates only the failed sub-goal's suffix, keeping completed and untouched work intact; a hybrid seed+live
memory store supports cold-start coverage. We evaluate under a leak-free protocol (leave-one-out
retrieval) over five open-weight models and a 75-task AI2-THOR
benchmark. On the standard
benchmark goal-completeness saturates (52% of instances trivially solved) and SAGE ties strong hierarchical baselines. On a harder, method-agnostic multi-goal composition, SAGE's completeness lead re-emerges large (+0.06 to +0.23 across four models). Under injected mid-execution failures, SAGE recovers as reliably as whole-plan replanners at 2.4-3.3x fewer LLM calls. As a verify-before-execute gate, the symbolic monitor blocks unsafe actions before actuation and raises simulator-reported step-success for every planner tested (up to +0.11), a signal the verifier never sees (non-circular). Because the gate calls no model (0.008 ms/plan), it is a safety layer that runs essentially free on the edge: SAGE planning reproduces its quality on a Jetson AGX Orin, where small-model verification helps most. We release the
benchmark, the leak-free protocol, the recovery and safety-gate harnesses, and a verifier-portability study (auto-induced on ALFWorld, 0.89 held-out).
Read the full abstract ▾
All authors Trung Minh Bui, JongSul Moon, YoungOuk Kim, Quang-Ngoc Phung, Se-Woong Jun, Dongin Shin
Robotics
RoboICL: Embodied In-Context Learning with GPT-6 Astra
Fangcheng Liu, Yeqing Shen, Anda Cheng et al. (+7 more)
General-purpose
vision-
language models offer a promising way to zero-shot
robot control: \gptastra{} excels at open-ended and
language- or image-conditioned manipulation but remains substantially weaker on high-precision and long-horizon tasks. We introduce RoboICL, an in-context
robot-control framework that narrows these gaps without
robot-specific parameter updates or a learned VLA. RoboICL separates demonstration context, which provides recorded examples when available, from interaction
memory, which accumulates the model's own actions and observed outcomes. Both use a shared observation–action–receipt–observation grammar. To preserve experience across task stages, RoboICL combines sampled demonstration blocks with bounded anchored
memory. Fixed anchors keep earlier rollout interactions available for in-context learning, while the latest interaction supports immediate error correction. Across 30 RoboDojo tasks, using zero shot for Open and one demonstration elsewhere, RoboICL improves on official zero-shot \gptastra{} by 20–27 progress-score points in every category. It leads the leaderboard baselines on
Memory and Open, achieves comparable performance to the strongest Precision baseline, and remains competitive on Long-Horizon. Its 30-task Overall score is 50.64, versus 33.68 for the strongest baseline. On a separate ten-task subset, RoboICL scores 60.60, within 2.00 points of the $\pi_{0.5}$ + \gptastra{} hybrid approach. On three real-
robot tasks, mean progress rises from 14.45 at zero shot to 63.33 at one shot and 78.89 at three shots. On two development tasks, optional Jev-gated action reuse reduces \gptastra{} calls by 33–48%. Code is available at \href{https://github.com/Mosi-AI/RoboICL}{https://github.com/Mosi-AI/RoboICL}.
Read the full abstract ▾
All authors Fangcheng Liu, Yeqing Shen, Anda Cheng, Weishi Mi, Chao Tang, Chenyuan Liu, Yushun Xiang, Tingguang Li, Yong-Lu Li, Yehui Tang
Robotics
UMR: Universal Manipulation Representation
Song Liu, Linyi Li, Yanshun Zhao et al. (+11 more)
General-purpose embodied manipulation hinges on a unified action representation that generalizes across embodiments and scales readily. Yet existing policies rely on embodiment-specific action spaces, making cross-embodiment demonstrations difficult to leverage at scale and limiting transfer to new embodiments and spatial variations. To this end, we introduce Universal Manipulation Representation (UMR), a unified action representation that enables zero-shot skill transfer from human demonstrations to heterogeneous
robots. UMR decomposes manipulation into two functionally distinct yet geometrically linked components: embodiment-agnostic World Flow, which describes task-relevant object motion in the world frame, and Ego Trajectory, which represents end-effector motion relative to the current pose. We instantiate UMR as World–Ego Point VLA (WEPVLA), a compact 0.5B-parameter policy that learns in the unified
geometric action space through a dual-stream Point Action Adapter and a unified Point Action Expert, with an $SE(3)$ conjugation coupling the two components. To improve data efficiency, we complement UMR with a Data-Efficient Strategy (DES) that diversifies object configurations through stage-aware point-cloud editing while preserving demonstrated contact geometry. In simulation, WEPVLA achieves average success rates of 97.5% on LIBERO and 85.7% on the 10-task RLBench
benchmark. In real-world experiments, a single policy trained on human demonstrations augmented by DES transfers zero-shot to diverse deployment conditions. With about 10 minutes of collected human demonstrations per task and no
robot demonstrations, it achieves 91.7% average success across six evaluation settings, compared with 60.8% for HumanEgo. Code and additional materials are available at https://umr-wepvla.github.io/.
Read the full abstract ▾
All authors Song Liu, Linyi Li, Yanshun Zhao, Rxuan Li, Xinrui Xu, Yi Ju, Yahui Deng, Senge Zhang, Guoyu Liu, Yixuan Li, Wuyang Zhang, Yao Li, Congcong Zhu, Jingrun Chen
Robotics
WAM-OPD: Sharpening World Action Models via On-Policy Distillation
Panjun Liu, Xiaohan Lei, Shiqi Zhang et al. (+8 more)
Pretrained world action models (WAMs) provide generalist capabilities across diverse robotic manipulation tasks, yet improving target-task performance to an expert level without degrading pretrained skills remains challenging. We explore on-policy
distillation (OPD) for WAMs and introduce WAM-OPD. WAM-OPD inherits the advantage of OPD methods that transfer task-specific teacher knowledge under the student's own induced distribution, rather than directly fitting the student to a narrow task-specific data distribution. However, in closed-loop manipulation, the observation histories change as the student policy evolves, requiring fresh environment rollouts to remain on-policy. Applying OPD to WAMs entails repeated data collection, which is costly even in simulation and often impractical on real
robots. To avoid repeated environment rollouts during
distillation, we introduce prefix-weighted trajectory replay (PWTR). PWTR uses a fixed trajectory pool composed primarily of initial-student rollouts, supplemented with task-specific teacher rollouts to broaden trajectory coverage. For each trajectory replayed from this pool, PWTR conditions the current policy on successive stored histories to generate fresh denoising paths, along which the task-specific teacher provides supervision. Although these denoising paths are refreshed as the policy evolves, the replayed environment trajectories remain fixed. PWTR therefore reweights per-decision
distillation losses using proxy importance weights derived from path scores accumulated over the trajectory prefix preceding each decision to mitigate the resulting shift in the history distribution. Simulated and real-world experiments demonstrate task adaptation without additional environment interaction during
distillation. In both settings, WAM-OPD improves target-task performance while retaining near-initial performance on tasks excluded from adaptation.
Read the full abstract ▾
All authors Panjun Liu, Xiaohan Lei, Shiqi Zhang, Yikun Wang, Yongxin Zhang, Mingyi Hu, Shida Sun, Jiateng Shou, Wengang Zhou, Jiajun Deng, Zhiwei Xiong
Robotics
GAE: General Action Expert for Real-Time Humanoid Teleoperation
Yuefan Wang, Huaicheng Zhou, Xiao He et al. (+6 more)
Humanoid avatars extend human physical presence beyond the body, enabling people to participate in social, service, and labor activities through remotely operated
robots. This requires teleoperation systems capable of realizing diverse and dynamic whole-body behaviors while maintaining responsive human-
robot synchronization. We present General Action Expert(GAE), a unified learning framework for general-purpose, low-latency humanoid whole-body teleoperation. To cover diverse human behaviors, GAE builds a large-scale human motion
dataset from heterogeneous sources, including
videos, animations, and motion capture, followed by standardization and augmentation. GAE then addresses the noise and embodiment mismatch in human motions with a two-stage training paradigm: a privileged generator policy first tracks human motion references in simulation and rolls out feasible humanoid trajectories; a deployable executor policy then learns to track these generated trajectories under curriculum domain randomization. For responsive human-
robot synchronization, GAE introduces a latency-conditioned anticipation mechanism that adaptively compensates for end-to-end delay during real-time teleoperation. Simulation and real-world experiments on Unitree G1 and Westlake O1
robots demonstrate that GAE enables humanoids to smoothly mirror diverse, agile, and expressive human behaviors. Project website: https://wangyf0928.github.io/gae-wlrobotics/
Read the full abstract ▾
All authors Yuefan Wang, Huaicheng Zhou, Xiao He, Zhijie He, Mingchuan Yang, Huayi Zhang, Li Chai, Jinxin Liu, Donglin Wang
Robotics
Proprioceptive Force Estimation for Quadruped Locomotion and Human-Robot Interaction
Run Wang, Xu Yang, Alapati Tuerxun et al. (+1 more)
Payload forces must be accommodated during locomotion, while leash forces can specify desired motion. We investigate whether a shared three-dimensional force estimate in newtons, inferred from proprioceptive history under sustained loading, can support both tasks. An
estimator and locomotion policy are jointly trained with supervised force and velocity outputs and learned latent context. The estimated force conditions locomotion and additionally generates planar-velocity and yaw-rate commands for leash guidance through an analytical map. In sustained-force simulation sweeps, temporal means of componentwise force root mean square error range from 1.44 to 2.83\,N. Compared with a domain-randomized baseline, the framework reduces velocity-tracking and base-orientation error scores by 21.6% and 46.5%, respectively, and increases mean survival from 68.29% to 94.60% in separate sustained-force tests. Unitree Go1 experiments demonstrate stationary vertical and horizontal force estimation, locomotion with an 8.5\,kg payload whose weight exceeds the 70\,N training force limit, and leash guidance using the same force-estimation interface.
Read the full abstract ▾
All authors Run Wang, Xu Yang, Alapati Tuerxun, Yilin Mo
Robotics
mmHRI: Towards Privacy-Preserving Human-Robot Interaction with Millimeter-Wave Radar
Junqiao Fan, Yuxuan Hu, Bofan Lyu et al. (+7 more)
Assistive
robots increasingly operate in many human-centered environments and perform various human-
robot interaction (HRI) tasks, such as object delivery. However, most existing HRI systems rely on RGB cameras that continuously observe humans to respond to non-verbal commands, such as hand gestures. This raises
privacy concerns in
privacy- critical environments, such as hospital wards or restaurants, where direct camera observation of humans is restricted. To develop
privacy-preserving HRI, we leverage millimeter-wave (mmWave) radar, which can sense human motion through
privacy barriers without identifiable imagery. We propose mmHRI, the first multi-modal
robot manipulation framework that achieves mmWave radar-guided
privacy-preserving HRI. mmHRI introduces two key designs to mitigate the sparsity and temporal inconsistency of radar data in cluttered
robot manipulation environments. First, we propose a dual-stream architecture that jointly learns from unfiltered raw radar tensors and radar point clouds to estimate both human actions and 3D poses. To mitigate signal inconsistency, mmHRI further incorporates a
memory-based state-space model (MSSM) that retains historical radar features to reduce abrupt changes in pose/action. These estimated human states are then converted into structured textual
robot instructions, which control a
vision-
language-action (VLA) policy for closed-loop
robot manipulation and human-aware reactions. Our evaluation covers human action recognition and closed-loop delivery and
retrieval. In the
privacy-preserving curtain setting, mmHRI achieves 85.09% action-recognition accuracy, outperforming existing radar-based alternatives.
Robot trials further demonstrate successful delivery and
retrieval under visual occlusion, with stable task performance across unseen subjects, clutter configurations, and environments.
Read the full abstract ▾
All authors Junqiao Fan, Yuxuan Hu, Bofan Lyu, Yanshuo Lu, Pengfei Liu, Jiarui Zhang, Fangqiang Ding, Lihua Xie, Gen Li, Jianfei Yang
Robotics
RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers
Haitong Ma, Chenxiao Gao, Rushi Qiang et al. (+2 more)
Coding
agents are beginning to move beyond purely digital tasks to tackle physical-world challenges, particularly in robotics. Existing robotics
benchmarks, however, primarily focus on the performance of individual artifacts, such as policies or controllers, offering limited coverage of coding
agents' broader engineering capabilities. Real-world robotics extends beyond control:
agents must build, integrate, diagnose, and improve heterogeneous artifacts under resource constraints and reason from
multimodal feedback. To evaluate these broader capabilities, we introduce RLE-Bench, a
benchmark of
robot-learning tasks spanning four representative robotics development workflows: interactive control, policy learning, perception and estimation, and mechanical design. We use diverse task-specific metrics to evaluate the artifacts submitted by the coding
agents, from the success rate the
agents achieved to the policy
agents trained, the harness
agent built, and the mechanical structures the
agent designed. We aggregate these metrics into an overall RLE Index and report workflow-specific capability profiles, enabling systematic comparison of coding
agents' capabilities across multiple capability dimensions. Beyond performance ranks, we also conduct in-depth case studies examining
agent behavior on representative tasks, highlighting both current capabilities and limitations, and pointing to the opportunities robotics tasks have to offer for future
agent training.
Read the full abstract ▾
All authors Haitong Ma, Chenxiao Gao, Rushi Qiang, Bo Dai, Na Li
Robotics
WB-WAM: Heterogeneous Body-Hand Pre-training for Humanoid Loco-Manipulation
Chuan Qin, Shaoting Zhu, Siyuan Luo et al. (+4 more)
Humanoid loco-manipulation demands coordinated body and hand behavior, while conventional
robot pre-training data provide limited coverage of such whole-body motion. We present WB-WAM, a World Action Model that incorporates explicit whole-body action supervision into generative
video pre-training. A shared physical action space integrates body, root, and dexterous hand annotations from heterogeneous sources, enabling joint
video and action learning from 1880.2 hours of partially annotated
video and motion data. The resulting priors are refined through PICO mid-training and adapted to
robot tasks with auxiliary forward kinematics supervision. We construct WB-
Datasets to support these stages with retargeted egocentric human demonstrations and
robot trajectories, allowing task-aligned human motion to supplement limited
robot data. Evaluations in simulation demonstrate strong whole-body task performance with 81.9% in HumanoidArena, while real-world experiments further validate WB-WAM with 84.0% mean success across five tasks. Moreover, task-aligned PICO mid-training improves downstream task performance while reducing the need for real-
robot demonstrations. These results support heterogeneous whole-body pre-training and human motion transfer as a practical route to data-efficient humanoid loco-manipulation.
Read the full abstract ▾
All authors Chuan Qin, Shaoting Zhu, Siyuan Luo, Siqiao Huang, Hongyu Zhao, Shanaka Baduge, Hang Zhao
Robotics
Unified Visual-Tactile-Action Modeling from Human Demonstrations for Dexterous Manipulation
Wenqiao Li, Qianyou Zhao, Jiawen Hao et al. (+5 more)
Dexterous manipulation requires tactile feedback. However,
robot tactile demonstrations are difficult to scale,because dexterous-hand teleoperation provides limited tactile feedback to the operator. In contrast, human demonstrations offer a substantially more scalable source of diverse tactile interactions. Motivated by a simple premise: hands can change, but the underlying physics of interaction does not. We leverage human tactile data to improve dexterous manipulation policies. Specifically, we first build a tactile motion-capture system that synchronously records images, tactile signals, and hand motions. Using this system, we construct the UVTA
dataset spanning five contact-rich tasks, with 1,000 human demonstrations covering diverse interaction patterns and 150
robot demonstrations per task. To transfer the underlying physics of human interaction to
robot control, we propose a Unified Visual-Tactile-Action Model that maps both embodiments into aligned tactile and action representations and jointly predicts future action and tactile trajectories. The joint objective enables human demonstrations to supervise contact-aware representation learning, while only
robot actions are executed during deployment. In real-
robot evaluations across five tasks, our method achieves an average success rate of 70%, outperforming the strongest visual-tactile baseline, which achieves 29%, and an architecture ablation, which achieves 42%. Performance improves consistently with additional human demonstrations and exhibits no saturation at 1,000 demonstrations per task, validating the effectiveness of scalable human tactile data for dexterous manipulation. Project page is available at https://uni-vta.github.io/.
Read the full abstract ▾
All authors Wenqiao Li, Qianyou Zhao, Jiawen Hao, Xuezhou Zhu, Tengyu Liu, Kaifeng Zhang, Chuan Wen, Siyuan Huang