Robotics
Linear Recurrent Memory Suffices to Distil a World-Model Policy for Robot Air Hockey
F. Olivia Fan and Oliver Obst
Does
memory-dependent control need nonlinear recurrent dynamics? We study simulated air-hockey defence under temporary loss of puck tracking. A DreamerV3 teacher outperforms a memoryless policy under tracking loss, while resetting the teacher's recurrent state sharply reduces performance, which demonstrates that the task requires
memory. We distil this teacher into compact recurrent policies with a 64 dimensional state, with a combination of a diagonal linear recurrence and an optional rank-$k$ nonlinear innovation while retaining nonlinear observation encoders and action heads. Across five matched seeds, the purely linear recurrent model ($k=0$) matches both the GRU baseline and the teacher throughout the tested range of tracking loss. Increasing nonlinear innovation rank providing no measured benefits. This result is obtained on a fresh test split, which will be only opened after all models and analyses are frozen. The linear model requires fewer recurrent parameters and less computation than GRU, but performs comparably. These results suggest that, for this
memory dependent control task, nonlinear representation learning around a simple linear
memory mechanism can be sufficient, and that nonlinear recurrent dynamics are not necessarily required. These conclusions are limited to the simulated task, teacher, state dimension, and blackout horizon considered here, and to policies whose observation encoder and action head remain nonlinear.
Read the full abstract ▾
Robotics
Blackout vs. Freeze: Analyzing Physical Failure Modes of VLAs under Camera Faults
Heejae Suh, Jongwook Han, Zahra Gholami et al. (+1 more)
Unreliable visual inputs can harm task performance and cause potential physical safety risks for
vision-
language-action (VLA) models. We analyze how $\pi 0.5$ and GR00T models act under input faults such as image blackouts and freezing. We find that blackout and freezing produce distinct physical failure modes even when task-success rates are similarly low: freezing causes more extreme joint behavior, whereas blackout after gripper closure can cause more object drops, most markedly without proprioception. Selective intervention studies reveal that proprioception (current
robot state) partly compensates for the removed
robot depictions and reduces non-target contact. However, it cannot sufficiently restore task success when wrist-view object information is removed, even when aided by the remaining scene view. We then evaluate two mitigation approaches: camera-blackout training and training-free replacement of faulty visual
embeddings. Both improve task success in selected conditions, but can increase unintended contact or disturbance to surrounding objects. Real-
robot trials further show that successful execution under camera faults can still involve unintended physical interactions. These findings motivate designing VLA policies that use the
robot and object information still available under camera faults to limit hazardous motion.
Read the full abstract ▾
All authors Heejae Suh, Jongwook Han, Zahra Gholami, Yohan Jo
Robotics
Drape-Compatible Tool-Tip Localization for Hand-Held Laparoscopic Instruments via UWB Carrier-Phase Ranging and Trocar-Constrained Geometry
Jinseok Lee, Dongho Yee, Minsung Kim et al. (+8 more)
A surgical
robot policy needs to know where each instrument's working end sits relative to the camera and to the other instrument, yet a laparoscopic operation leaves only the endoscope
video, and a draped instrument hides every optical path on itself. We estimate the tool tips from distances and an IMU alone. The distances are measured by ultra-wideband carrier phase between
antenna nodes on the handle side of the instruments: one per hand-held instrument, one on the endoscope, all outside the sterile barrier. Take the endoscope
antenna as the reference. The two instrument
antennas carry six coordinates while the three pairs give three distances, so the problem is short by three, and averaging cannot close that gap because what is missing is information, not precision. Carrier phase adds one more unknown per pair, an integer that leaves distance fixed only modulo lambda/2 = 23.1 mm. The trocar settles both difficulties: each shaft passes through a port whose position is known and whose direction the on-board IMU measures, so its
antenna keeps a single degree of freedom, the insertion depth. Two unknowns now face three distances – two fix the depths and the third is left over, and that spare equation exposes a wrong integer or a wrong mounting offset instead of absorbing it. The same solve returns both tool tips in the endoscope frame, without the second carrier frequency that integer resolution usually needs, so the link never leaves a single PLL lock. In an 11-hole phantom among metal instruments the three distances hold sigma = 0.10-0.33 mm at 35-46 Hz, a sterile drape pressed onto the
antennas leaves the link unchanged where an optical path would be blocked, and the logger was carried into a live rabbit appendectomy.
Read the full abstract ▾
All authors Jinseok Lee, Dongho Yee, Minsung Kim, Younghoon Noh, Seonho Shim, Juahn Oh, Yechan Seo, Jiyul Lee, Seong Jeong, Hyuk Choi, Hyoun-Joong Kong
Robotics
LBDU-VIO: Learned Bias Dynamics and Uncertainty for Visual-Inertial Odometry with Unreliable Vision
Qizhi Guo, Junning Lyu, Defu Lin et al. (+1 more)
Visual-inertial odometry (VIO) for aerial
robots relies on high rate inertial measurement unit (IMU) propagation between visual updates. However, conventional multi state constraint Kalman filters (MSCKFs) use random walk bias assumptions and fixed noise parameters, which can limit robustness when visual information is unreliable. To address this problem, we propose LBDU-VIO, a learning-augmented MSCKF with learned continuous time bias dynamics and an IMU uncertainty model. A neural ordinary differential equation (ODE) models continuous time bias dynamics to propagate the filter's bias states, replacing their random walk model. The IMU uncertainty model predicts motion adaptive measurement noise covariances for covariance propagation. Both models are trained with pose supervision without direct labels. Experiments on real world EuRoC and TUM-VI
benchmarks show lower errors than representative visual-inertial baselines, including a 25.1% reduction in mean relative position error compared with S-MSCKF on EuRoC sequences with 10s visual outage.
Read the full abstract ▾
All authors Qizhi Guo, Junning Lyu, Defu Lin, Shaoming He
Robotics
Occlusion-Aware, Quasi-Static, Stability-Oriented Trajectory Planning on Uneven Terrain
Amith Manoharan, Chinmay Mundane, Aayush Bahukhandi et al. (+3 more)
Autonomous navigation in unstructured off-road environments requires
reasoning about both vehicle–terrain interaction and environmental unknowns. We propose a model-based framework for generating quasi-static, stability-oriented reference trajectories for rigid, non-articulated four-wheeled vehicles on highly uneven terrain. Our work makes three primary contributions. First, we model blind spots caused by terrain occlusion as coverage-induced epistemic uncertainty in a fixed-feature
Fourier terrain representation, quantified through a regularized inverse-Hessian estimate. Second, we propagate this uncertainty through the Nonlinear Least-Squares (NLS) pose/contact model using implicit differentiation and incorporate the resulting pose, contact-point, and per-wheel surface-normal uncertainty terms into trajectory optimization based on the Cross-
Entropy Method (CEM). Third, we introduce a Flow Matching model that warm-starts terrain fitting, and we evaluate its fitting-accuracy–latency trade-off while retaining model-based refinement. Across six synthetic terrains with 30 matched start–goal pairs per terrain, the complete framework produced an observed failure rate of 18.9%, compared with 46.1% and 41.7% for two representative baselines and 34.4% for an ablation that removed the propagated-uncertainty scoring. Hardware evaluations span six distinct outdoor environments, with two representative executions presented in the paper and four additional executions included in the supplementary
video. The evaluation also reports the accuracy–latency trade-off for the Flow Matching warm start.
Read the full abstract ▾
All authors Amith Manoharan, Chinmay Mundane, Aayush Bahukhandi, K. Madhava Krishna, Karel Zimmermann, Arun Kumar Singh
Robotics
DiFF: Doppler-informed Flow Matching for Human Motion Flow
Kai Wang and Mingle Zhao
Perceiving human motion via
privacy-preserving 4D millimeter-wave (mmWave) radar is critical for next-generation human-
robot interaction (HRI), where point cloud scene flow serves as a foundational motion representation. Yet the extreme sparsity and noise of 4D radar point clouds make non-rigid motion flow estimation severely ill-posed–a challenge that existing rigid-centric methods and prior works fail to adequately address, largely because they neglect the rich Doppler velocity cues inherent in 4D radar. We propose DiFF, a generative framework that marries Doppler-informed motion priors with a Kolmogorov-Arnold Network (KAN)-based conditional flow matching model. At its core, a KAN-
attention mechanism enables expressive feature extraction, while a prior-guided generative process harnesses Doppler cues to regularize the ill-posed solution space. Extensive experiments show that DiFF achieves state-of-the-art (SOTA) performance across diverse real-world
datasets, reducing 3D endpoint error to the millimeter scale on the mmBody
benchmark.
Read the full abstract ▾
Robotics
SteerQuant: Steering Quantization Error with Action-Guided Scaling in World-Action Models
Yunhan Wang, Haodong Wang, Zhiming Liu et al. (+7 more)
World-action models (WAMs) jointly generate future world states and actions through iterative denoising, using shared weights to process heterogeneous semantic streams of
video, proprioceptive, and action
tokens. Quantization reduces inference cost, but comparable numerical errors in different streams can have markedly different effects on final actions, making numerical accuracy alone insufficient for reliable control. We introduce SteerQuant, a 4-bit quantization framework for WAMs that steers errors toward computations with less influence on final actions. It maps how each stream's quantization errors affect final actions and uses this map to guide shared channel scaling. Activation scaling is further calibrated for each stream and denoising step to accommodate changes in activation ranges and action impact. This adapts quantization to different stream requirements without duplicating weights or increasing bit-widths for selected streams. To reduce the extra kernel launches and
memory traffic introduced by scaling, we develop Rudder, a 4-bit inference engine for WAMs that fuses scaling and output compensation into low-bit kernels. Under W4A8 and W4A4, SteerQuant maintains mean LIBERO success within 0.8 percentage points of full precision, while delivering up to $2.23\times$ denoising speedup over BF16 across three WAMs with reduced peak GPU
memory usage. On a real dual-arm
robot, W4A8 deployment achieves a $1.35\times$ end-to-end inference speedup while maintaining average task success relative to BF16.
Read the full abstract ▾
All authors Yunhan Wang, Haodong Wang, Zhiming Liu, Zicong Hong, Qianli Liu, Xiaoyi Pang, Yangjia Hu, Quanxin Shou, Yikun Miao, Song Guo
Robotics
CLIPPER Beyond Shortlisting: Auditable Decision Support for Changing Municipal Micromobility Policies
Julian Teusch, Jörg Philipp Müller and Monika Sester
In municipal planning workshops, planners and other stakeholders compare shared-micromobility parking policies by varying no-parking zones, retained sites, spacing, or area allocations. Each edit changes feasible sites and how much demand they cover, so the alternative must be reoptimized on the same spatial data. Full-set greedy, the transparent reference for this task, takes tens of seconds per alternative at city scale. We present Constraint-exact Low-latency Iterative Planning with Pooled Evaluation and Replay (CLIPPER), an optimizer with audit functions developed for requirements elicited with the City of Braunschweig. In each greedy round, it forms a deterministic candidate pool of bounded size, computes how much still-uncovered demand each candidate would add, and rejects candidates that violate an active constraint. An optional offline audit scans every remaining feasible candidate and records what the restricted pool omitted. We evaluate these functions on complete eleven-state edit chains ($E_0,\ldots,E_{10}$) in Braunschweig, Munich, and Berlin. With $K=1024$ candidates per group, the fixed-width mode CLIPPER-F has mean coverage gaps to full-set greedy under the same policy of 0.245, 0.003, and 0.001 percentage points in Braunschweig, Munich, and Berlin, respectively, while mean rollout time falls by factors of 13.6–28.9; no audited run terminates while a candidate outside the pool could still increase coverage. Plans computed from two checksummed versions of Braunschweig's official no-parking-zone data differ in 30 of about 540 selected sites although coverage moves by only about 0.1 percentage points. These changes still require municipal assessment and implementation. The findings inform a proposed municipal process that versions policy inputs, reports site changes beside coverage, and scans the full candidate set before a final decision.
Read the full abstract ▾
Robotics
Looking Back to Move Forward: Temporal Verification for Generative Robot Policies
Haoxuan Wang, Wayne Wu, Yan Yan et al. (+1 more)
Generative policies have emerged as a promising paradigm for
robot learning, combining expressive generative action modeling with scalable imitation learning from large demonstration corpora. However, heterogeneous demonstrations can induce suboptimal action chunks whose errors compound over time, eventually driving the
robot into out-of-distribution states from which recovery is difficult. Action verification offers a test-time scaling strategy for mitigating this failure mode by sampling multiple candidate actions and using a verifier to select one for execution. Existing approaches, however, remain temporally myopic and costly to train, evaluating candidates from the current observation alone without accounting for trajectory continuity and often relying on large verifiers and additional expert demonstrations. In this paper, we introduce Temporal Verification (TeV), an efficient temporally aware action verification framework for flow-matching VLAs. TeV first learns a temporal
token that summarizes recent observation–action history, enabling candidate chunks to be evaluated as continuations of the execution trajectory rather than as isolated predictions. Conditioned on this
token, TeV constructs positive–negative pairs without additional expert demonstrations or preference annotations and trains an energy-based verifier contrastively to assign lower energy to higher-quality, trajectory-consistent action chunks. Beyond post-hoc ranking, TeV further uses the learned energy landscape to guide intermediate flow samples toward lower-energy regions, improving candidates before final selection. Extensive experiments in simulation and real-world settings demonstrate that TeV provides reliably ranks action candidates, improves task success rates, and produces smoother execution trajectories.
Read the full abstract ▾
All authors Haoxuan Wang, Wayne Wu, Yan Yan, Bolei Zhou
Robotics
Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Tools
Shijia Ge, Alex Zhou, Jianshu Zeng et al. (+12 more)
Frontier models can control
robots, but
reasoning through every reach, grasp, and retreat makes manipulation slow and
token-intensive. We revisit code as policy with a different division of labor: models build executable tools, code handles multi-phase motions, and models decide what to do next. We introduce URAI (Universal
Robot-
Agent Interface), which couples a programming
agent that constructs
robot tools with an execution
agent that uses them in a feedback loop. The programming
agent writes reusable and task-specific tools from task intent and refines them through execution feedback and human guidance. The execution
agent selects and parameterizes these tools from current observations; each call runs a complete motion locally before returning control to the
agent. Unlike delegating subsequent decisions to a generated program, this design retains model-level decision-making between tool executions. Validated tool revisions persist across episodes without updating foundation-model weights, and a shared GUI and API make the same tools available to humans and
agents. Across five RoboDojo tasks and four frozen execution
agents, URAI raises aggregate success from 18.0% to 53.0% relative to direct fingertip control, with the largest gain on Swap Blocks; with the same tools, a program written in advance reaches only 24% against 56% for two
agents deciding after each call. Three of the four
agents also finish episodes 1.3-1.5 times faster with 1.5-1.7 times fewer execution-
agent output
tokens; DeepSeek-V4-Flash's cost barely changes. We further evaluate URAI on seven real-world AgileX dual-arm tasks, spanning object manipulation, cloth folding, and human-interactive tic-tac-toe. URAI connects the coding and decision-making capabilities of frontier
agents, organizing
robot control around reusable tools that
agents can both invoke and revise.
Read the full abstract ▾
All authors Shijia Ge, Alex Zhou, Jianshu Zeng, Yexing Wan, Di Wu, Zelin Zheng, Yazhe Wang, Zhiqi Jia, Xuan Shangguan, Jay Zhu, Yijun Liu, Lingyu He, Sihang Wu, Xiao He, Hongcheng Gao
Robotics
OccluDex: Hierarchical 3D Visuo-Tactile Representation Learning for Egocentric Dexterous Manipulation under Self-Occlusion
Ziheng Xu, Yueyuan Chen, Xinyuan He et al. (+6 more)
Reliable dexterous manipulation requires continuous estimation of object geometry and hand-object contact throughout interaction. With egocentric sensing, however, the manipulating hand frequently occludes task-relevant object surfaces and contact regions, reducing the visual evidence available for state estimation and thereby making robust closed-loop control and generalization to unseen object geometries particularly challenging. To address this, we present OccluDex, a hierarchical 3D visuo-tactile representation learning framework that integrates global
geometric structure with local contact information for robust manipulation under dynamic self-occlusion during hand-object interaction. OccluDex adopts multi-scale masked autoencoding to progressively encode partial 3D geometry and fuses tactile contact
tokens with high-level
geometric features through cross-modal
attention. The encoder is pretrained from synchronized human visuo-tactile demonstrations and transferred as a frozen perceptual backbone for downstream reinforcement learning. We evaluate OccluDex on a faucet rotation task, requiring one full clockwise handle revolution, and a tabletop object reorientation task, requiring a 180-degree tabletop object reorientation without toppling. In simulation experiments, OccluDex demonstrated 12.6% higher accuracy for unseen objects and 8.3% higher accuracy for previously seen objects than the strongest state-of-the-art baseline models. Physical experiments were further performed with a Shadow Hand to demonstrate successful zero-shot sim-to-real generalization on unseen physical objects. This results could enable humanoid egocentric object manipulation for seen and unseen objects even when the manipulating robotic hand occludes
vision.
Read the full abstract ▾
All authors Ziheng Xu, Yueyuan Chen, Xinyuan He, Guoxing Liu, Yuanshuo Tan, Huiming Pan, Bin He, Shuo Jiang, Peter B. Shull
Robotics
Function beyond Form: Functional Correspondence for Cross-Embodiment Dexterous Grasp Generation
Bolin Zou, Wenlong Dong, Mu Ai et al. (+4 more)
Cross-embodiment dexterous grasp generation remains challenging because robotic hands differ substantially in geometry,
topology, and kinematics. Existing approaches often lack explicit correspondences between structurally different hand regions that play similar functional roles in a grasp, a concept we refer to as functional correspondence. Consequently, their models tend to learn hand-specific interaction patterns rather than transferable grasp knowledge, limiting generalization to unseen hands. To address this limitation, we introduce FunCo-Grasp, which establishes functional correspondences across heterogeneous hand embodiments. Specifically, Functional Part
Alignment aligns each hand to a canonical functional schema by mapping physical links to shared functional parts according to their grasping roles, while Canonical Frame
Alignment expresses these parts in canonical local frames. These two
alignments provide a consistent representation for inter-part and hand-object interactions, allowing the model to learn transferable grasp knowledge across hands. Conditioned on the aligned hand representation and object geometry, a
diffusion model generates the target spatial arrangement of the functional parts, which are then converted into an executable joint configuration. Adapting FunCo-Grasp to an unseen hand requires only its
geometric and kinematic models and a one-time lightweight functional annotation, without target-hand grasp data, fine-tuning, or learned retargeting. In simulation on held-out objects from the filtered CMapDataset, we achieves average success rates of 92.40% on three seen hands and 74.02% on four unseen hands. In real-world experiments, the same model achieves an overall success rate of 76.00% on two unseen hands without additional training or fine-tuning. These results demonstrate the effectiveness of FunCo-Grasp in transferring grasp knowledge to unseen hands.
Read the full abstract ▾
All authors Bolin Zou, Wenlong Dong, Mu Ai, Chao Tang, Aoxiang Gu, Lipeng Chen, Hong Zhang
Robotics
NEXUS: Perceptive Whole-Body Control for Terrain-Adaptive Teleoperation
Xiangyu Miao, Junsong Wu, Jiyuan Shi et al. (+6 more)
Whole-body teleoperation requires a humanoid
robot to reproduce a human operator's behavior even when their terrains differ. This demands that the
robot perceive local terrain and adapt its posture and contacts accordingly, rather than copy the operator's motion frame by frame. However, paired motion data linking the same behaviors across flat ground and different terrains remain scarce, limiting supervision for learning terrain-adaptive control. To enable whole-body teleoperation across mismatched terrains, we introduce NEXUS, a perceptive whole-body control framework that combines human motion commands with onboard sensory feedback. We first develop a scalable terrain-aware adaptation
algorithm that efficiently generates high-quality motion pairs across motions and terrains without per-motion or per-terrain tuning. Using a paired motion corpus totaling nearly 1,000 hours, we train a perceptive whole-body controller through teacher-student learning to reproduce commanded behaviors across terrains. Experiments demonstrate efficient, scalable generation of high-quality motion data and show that NEXUS combines broad behavioral coverage with terrain adaptability and tracking fidelity, outperforming existing whole-body controllers on the evaluated
benchmarks. Zero-shot real-world deployment enables real-time whole-body teleoperation on diverse unseen terrains, further validating the generalization of our method. Project website: https://nexus-humanoid.github.io/
Read the full abstract ▾
All authors Xiangyu Miao, Junsong Wu, Jiyuan Shi, Weiji Xie, Jinrui Han, Xingyi Wang, Weinan Zhang, Chenjia Bai, Xuelong Li
Robotics
Cue the Flow: Steering Flow-Matching Policies for Open-World Delivery Manipulation
Haoxuan Wang, Griffin Galimi, Junhua Huang et al. (+4 more)
Open-world goods delivery requires mobile manipulators to follow free-form user instructions and manipulate potentially novel objects. Existing dual-system approaches use high-level grounding models to convert
language into grounded visual
prompts, but their low-level controllers can remain brittle under noisy perception, dynamic scenes, and contact-rich interactions. We instead use a pretrained flow-matching
vision-
language-action model as the low-level control interface, leveraging its reactivity and robustness to environmental changes while treating the grounding output as a spatial cue for policy steering. Our key insight is that the pretrained VLA already provides a strong manipulation prior, while the spatial cue supplies the missing target information needed to guide actions under novel
language–object mappings. Concretely, we introduce a lightweight cue-conditioned adapter. The adapter is first trained with contrastive objectives to produce salient and spatially discriminative cue representations, and is then supervised to predict a diagonal affine transformation over the generated action chunk, aligning policy steering with the cued target. Across tabletop and mobile-base settings, our method improves instruction following and manipulation success on both in-domain and out-of-domain objects, achieving up to near $2\times$ improvement in average task success rate with negligible inference overhead.
Read the full abstract ▾
All authors Haoxuan Wang, Griffin Galimi, Junhua Huang, Selina Song, Wayne Wu, Yan Yan, Bolei Zhou
Robotics
Sparse-WAM: Accelerating World Action Models via Action-Guided Sparse Imagination
Xinling Xie, Haodong Wang, Jiazhi Mi et al. (+8 more)
World-action models (WAMs) leverage pretrained
video models to improve generalization in
robot control by jointly predicting future visual states and actions. This capability comes at a substantial inference cost, as dense future-frame
tokens are repeatedly processed during denoising. Prior methods address this by
token pruning that prioritizes visual fidelity to reduce denoising costs in
video diffusion models. However, these methods do not use action relevance to determine which future-frame
tokens to retain during joint denoising in WAMs. In this paper, we propose Sparse-WAM, a training-free framework for action-guided sparse imagination that selectively processes future-frame
tokens to accelerate WAM inference. We observe substantial overlap in the spatial distribution of
attention from action
tokens to future-frame
tokens (action-to-future
attention) between consecutive denoising steps, despite continued updates to the future representations. Motivated by this, we develop Action-Guided
Token Selection to retain frame-specific action-relevant regions together with cross-frame context. However, a naive implementation can incur
attention-scoring and
token-packing overhead that offsets the computational savings from
pruning. We therefore introduce Pilot, an efficient engine that reduces sparse inference overhead through lightweight scoring and cross-step reuse of
token selections. On LIBERO with FastWAM-Joint and RoboLab-120 with Cosmos 3 Edge, Sparse-WAM achieves inference speedups of approximately $2.0\times$ and $1.8\times$, respectively, over dense eager inference on an NVIDIA RTX 4090, while largely preserving task performance.
Read the full abstract ▾
All authors Xinling Xie, Haodong Wang, Jiazhi Mi, Zhiming Liu, Zicong Hong, Xiaoyi Pang, Qianli Liu, Yangjia Hu, Ying Chen, Zhengyang Yan, Song Guo
Robotics
SimEX: Simulation-Integrated Robotics AutoResearch
Jiaheng Hu, Roberto Martin-Martin, Peter Stone et al. (+3 more)
Coding
agents powered by large
language models (LLMs) have shown remarkable abilities to autonomously reason about and achieve goals in the digital world. However, bringing this success to the physical world remains challenging. On the one hand, direct generation methods (e.g., Code as Policies) often suffer from the LLMs' insufficient understanding of
robots and physical environments. On the other hand, iterative trial-and-error tuning in the physical world (e.g., physical autoresearch) induces significant experimental cost and safety concerns. We introduce SimEX: Simulation-Integrated Robotics AutoResearch, an autoresearch framework that tightly integrates simulated experimentation, enabling coding
agents to efficiently acquire physical capabilities for controlling real
robots. SimEX operates in two stages. First, the
agent conducts open-ended probe-and-optimize iterations in simulation, developing a
robot toolbox with robust and generalizable capabilities. Second, the
agent adapts the toolbox and the simulator together through only a few physical trials: each trial corrects the simulator, and the corrected simulator is used to diagnose failures and screen candidate repairs. We evaluate SimEX extensively in sim-to-sim settings and on physical
robots. On challenging real-world manipulation tasks including towel folding, barcode scanning, and plate manipulation, SimEX enables coding
agents to efficiently acquire
robot skills without any demonstration and with only 10 minutes of real-
robot interaction. These results suggest that simulation can be a critical component in achieving physical intelligence, not only as a source of training data that must closely replicate the real world, but also as a roughly correct laboratory where a coding
agent develops the knowledge and procedures needed to act on the
robot. More details and
robot videos at https://robo-simex.github.io/
Read the full abstract ▾
All authors Jiaheng Hu, Roberto Martin-Martin, Peter Stone, Rocky Duan, Zhenyu Jiang, Guanya Shi
Robotics · Cross-listed from cs.CR
Refusals That Bend: Measuring and Predicting Task Malleability in Embodied VLM Planners
Leo Y. Lin, Mikhail Kuznetsov, Muslum Ozgur Ozmen et al. (+1 more)
Embodied
vision-
language models (VLMs) are increasingly deployed as high-level planners for
robots because they generalize across diverse environments. However, this requires their safety
alignment to also hold in unseen environments. Existing red-teaming assumes an adversary who optimizes the
prompt, the pixels, or text in the environment, and existing
benchmarks ask whether a planner recognizes or mitigates a hazard in a fixed scene. Neither asks whether a refusal the planner has already given survives an ordinary change to the environment. We ask that question by placing a single everyday object into the environment, with no pixel, gradient, or
prompt under
adversarial control. On $846$ tasks that a constitution-guarded planner initially refuses, we find $20.2\%$ of tasks can be flipped to compliance by one or more objects, and the number of objects differs from one task to another. In addition, the object need not be chosen for the task, i.e., items drawn from a fixed list, with no knowledge of the environment or the instruction, bypass safety about as often as items proposed for the specific task. We qualitatively contrast the tasks bypassed most and least often and find that the distinction lies in how conspicuous the hazard is in the instruction and environment. Susceptibility to safety bypass is therefore a property of the task, which we call its malleability, and we show that it can be predicted before the target is ever queried. A composite of signals read from a small open-source VLM identifies malleable tasks $2.4\times$ as often as picking at random. Everyday objects, whether placed by an adversary or introduced by ordinary rearrangement of the environment, are thus sufficient to overturn a refusal. Because susceptibility is determined by how a task is specified, we recommend assessing malleability per task prior to deployment.
Read the full abstract ▾
All authors Leo Y. Lin, Mikhail Kuznetsov, Muslum Ozgur Ozmen, Z. Berkay Celik
Robotics
Video2SwimFish: An Automated Pipeline for Reconstructing Controllable Fish Models and Biological Locomotion from Real Fish Videos
Hangong Chen, Linfeng Cheng, Tahsin Zaman Jilan et al. (+4 more)
We present
Video2SwimFish, an automated pipeline and
benchmark for building controllable fish assets from real-fish
videos for underwater embodied AI. Given synchronized multi-view
videos of an individual fish, the pipeline reconstructs a metrically scaled deformable mesh from a VLM-selected canonical frame, generates internal articulation adapted to that individual's morphology through a VLM actor-critic loop, and extracts a Biological Locomotion
Manifold (BLM) from the fish's observed midline
curvature. The BLM provides a low-dimensional action space bounded by real-fish motion, enabling an individual swimming policy to be learned for each reconstructed fish. We release two paired
datasets: synchronized top- and front-view recordings of 120 individual fish across 6 species, and the controllable assets and individual swimming policies derived from them. Because every asset is tied to the animal it came from, the
dataset supports a
benchmark that evaluates locomotion learning not only on task success but on fidelity to that individual in trajectory shape, body
curvature, and tail-beat frequency, across trajectory following, reward-free swimming behavior transfer from
video, and a downstream case study in which a simulated BlueROV underwater
robot captures one of the assets. We find that task success and locomotion fidelity do not necessarily improve together: the method achieving the highest task completion is not the method achieving the highest locomotion fidelity, and we identify faithful reproduction of individual animal locomotion as an open challenge for the community. Project website: https://hangongchen.github.io/
video2swimfish-web/.
Read the full abstract ▾
All authors Hangong Chen, Linfeng Cheng, Tahsin Zaman Jilan, Ian Fuller, Lee Caesar, Jiaye Wu, Yantian Zha
Robotics · Cross-listed from cs.MA
Fast and Scalable Multi-Agent Distribution Matching via Partitioned Optimal Transport
Kooktae Lee and Ruchika Singh
This paper presents a scalable optimal-transport-based framework for terminal distribution matching in multi-
agent systems. While optimal transport provides a natural way to measure distributional mismatch and assign
agents to a desired spatial distribution, global discrete transport can become computationally expensive for large-scale systems. We address this bottleneck by partitioning
agents and target samples into spatially corresponding blocks and solving smaller local transport problems. Under a mass-balance condition, the resulting restricted coupling remains feasible for the global problem and provides an upper bound on the Wasserstein cost. The local assignments generate target locations for finite-horizon
agent control, applicable to both linear and nonlinear dynamics. By alternating local assignment and control, we establish a cycle-to-cycle descent guarantee for the resulting transport surrogate. The proposed framework therefore enables scalable terminal distribution matching while retaining a rigorous connection to the Wasserstein objective. The technical soundness of the proposed results is validated through simulations.
Read the full abstract ▾
Robotics
DrivingBench: Can Vision-Language Models Drive a Toyota Corolla?
Aditya Ramabadran, Simon Mahns and Tobias Gessler
Frontier models excel at many digital
benchmarks, yet their ability to drive a real car, an everyday human skill, remains largely untested. We present DrivingBench, to our knowledge the first
benchmark where general-purpose
vision-
language models must drive a real car. Through three tools, the models see camera frames from a Toyota Corolla and directly command its steering and velocity around a parking lot cone course at low speeds. The car may continue moving while the model thinks and new commands replace the currently running one, so inference latency is part of the task, testing the models' abilities to observe, act, monitor, recover, and complete a long-horizon objective under such constraints. We
benchmark GPT-6 Astra, Claude Fable 5.1, GPT-5.6 Sol, and Grok 4.6 in vendor-native harnesses (Codex, Claude Code, Cursor) with up to three attempts each in one conversation; Astra is the only model to finish the course, on its second attempt, with no other attempt passing 50% of the course. Two of the four models improved materially across attempts with retained context. We also detail the design principles behind our action interface, and show how the tool output format and the framing of the task combined to determine whether models would drive at all or refuse. We release our harness,
prompts, course map, and traces with
video and telemetry for reproducibility.
Read the full abstract ▾
Robotics
Local-Minimum Escaper: Programmatic Subgoal Generation for Robust Navigation in Unknown Environments
Yin Gu, Xinming Zhang, Shanze Wang et al. (+2 more)
Mapless navigation in unknown and partially observable environments remains challenging for mobile
robots, particularly when local minima prevent the
robot from making progress toward its goal. Existing local navigation methods often lack an explicit mechanism for escaping such situations, while deep reinforcement learning (DRL) approaches typically learn recovery behaviors implicitly through reward design and policy optimization. In this work, we propose LME (Local-Minimum Escaper), a programmatic hierarchical framework that explicitly generates and reasons subgoals to guide
robots out of local-minimum regions. LME operates solely on local observations and selects candidate subgoals using interpretable heuristic criteria that account for both surrounding obstacle geometry and candidate-location safety. A local planner then generates low-level motion commands toward the selected subgoal. This design enables LME to handle environments both with and without local minima within a unified framework, while remaining independent of the underlying local planner and requiring no additional training. Extensive experiments in simulated and real-world environments demonstrate that LME provides robust navigation performance and generalizes to challenging unseen scenarios. Furthermore, the generated subgoals can be used to guide different local planners, substantially improving their ability to escape local minima. Successful deployments on both differential-drive and quadruped
robots further demonstrate the practical applicability and generality of the proposed framework.
Read the full abstract ▾
All authors Yin Gu, Xinming Zhang, Shanze Wang, Siwei Cheng, Wei Zhang
Robotics
EmbodiRSI: Recursive Self-Improvement for Data-Efficient Robot Adaptation
Haoran Lang, Haotao Lu, Shiyu Sang et al. (+6 more)
Adapting
robot manipulation policies to new tasks and environments remains highly data-intensive, while the data needed for further improvement depends on the policy's current capabilities and failure modes. We introduce EmbodiRSI, an agentic system for recursive self-improvement (RSI) in a real-to-sim-to-real setting, where task-specific simulations are constructed from target deployment scenarios and used as low-cost environments for iterative policy improvement before transfer back to the physical world. EmbodiRSI uses policy execution feedback to guide subsequent experience acquisition and policy updates. Two complementary mechanisms close this loop: Collaborative Error Correction generates
agent-assisted corrective trajectories from policy-reached states, while Adaptive Data Collection directs expert demonstration generation toward the current policy's weaknesses. The task-specific simulation serves as a reusable workspace for policy warm-up, repeatable evaluation, failure diagnosis, and targeted data generation across successive RSI rounds. Across three tabletop environments and 14 subtasks, EmbodiRSI increases scene-balanced autonomous simulation success from 50.4% to 83.5% over two RSI updates. With 400 adaptive simulated trajectories and only ten real-world refinement trajectories per subtask, EmbodiRSI achieves 83.1% scene-balanced autonomous real-world success, compared with 75.0% for adaptation using 200 real-world demonstrations per subtask. These results demonstrate that feedback-driven recursive improvement in deployment-specific simulations can enable data-efficient adaptation of embodied policies to physical environments.
Read the full abstract ▾
All authors Haoran Lang, Haotao Lu, Shiyu Sang, Haoyang Luo, Guo Chen, Qun Li, Jingyi Yu, Ye Shi, Jingya Wang
Robotics
PRICE the Action Chunks: Physical Relational Credit Assignment for Embodied Reinforcement Learning
Yangang Zou, Jiajun Lu, Weitao Zhou et al. (+6 more)
Outcome-based reinforcement learning (RL) post-trains
vision–
language–action policies using terminal success signals, but assigns the same trajectory-level advantage to every action chunk. A failed episode can thus penalize useful early actions as if they caused the failure. Existing approaches seek finer-grained feedback through learned evaluators, adding task-specific supervision or additional model training. We explore, for the first time to our knowledge, whether physical relations across trajectories can provide action-chunk credit in embodied RL from terminal outcomes alone, without an auxiliary evaluator. The key insight is that rollouts reaching corresponding physical situations can serve as references for one another: their terminal outcomes provide evidence for assessing local progress. We introduce Physical Relations for Inferring Credit from Episodes(PRICE), with two components: (i) a physical relational graph that pools current and historical outcomes at corresponding chunk boundaries to estimate success potentials; and (ii) confidence-gated credit assignment that uses changes in these potentials to refine trajectory-level supervision. Our analysis connects oracle potential changes to the terminal-success objective and provides a finite-sample directional bound for outcome-independent evidence pools. Independent continuation tests show that PRICE's retained credits align with local progress, while experiments on LIBERO, RoboTwin 2.0, and real
robots demonstrate improved task success over outcome-based baselines and faster learning.
Read the full abstract ▾
All authors Yangang Zou, Jiajun Lu, Weitao Zhou, Haibao Yu, Bozhou Zhang, Jiawei Wang, Honglong Tian, Minglei Li, Li Zhang
Robotics
Benchmarking and Enhancing Skill-Level Memory for Partially Observable Robotic Manipulation
Yansong Shi, Jiange Yang, Xijie Yang et al. (+4 more)
Recent advances in
robot learning have enabled manipulation policies to perform increasingly diverse tasks and generalize across environments. However, reliable execution often depends on hidden task states that cannot be determined from current observations alone, making interaction history essential. We introduce $HIDE$, a
benchmark for evaluating manipulation
memory under partial observability. HIDE comprises 15 tasks covering repetition counting, historical-state recall, and execution-progress tracking, with randomized initial configurations and decision points where similar observations require different actions depending on prior events. We further propose $SEEK$, a framework combining three complementary
memory mechanisms to retain historical evidence and track execution state. Evaluations reveal substantial limitations in existing policies on HIDE, while
memory augmentation improves task success in both simulation and real-world experiments. Individual mechanisms benefit some tasks but can degrade others; their combination achieves the highest average success rate on HIDE among the evaluated configurations. These findings highlight the importance of maintaining internal representations of hidden task states and matching
memory design to task-specific information requirements.
Read the full abstract ▾
All authors Yansong Shi, Jiange Yang, Xijie Yang, Shaowei Zhang, Yuhan Zhu, Tao Lu, Limin Wang
Robotics
DODGER: Safety-Guided Reinforcement Learning for Robot Navigation Among Dynamic Obstacles
Sanghyuk Park, Kwanwoo Lee, Taekyung Kim et al. (+2 more)
Robots operating in human-centered environments must safely navigate among multiple dynamic obstacles to avoid collisions with people and surrounding infrastructure. Control barrier functions (CBFs) provide an effective mechanism for safety filtering, and recent CBF-based reinforcement learning (RL) methods embed such safety information into learned policies. However, executing only safety-filtered actions during training can restrict policy exploration, a limitation that becomes particularly consequential in dynamic scenes where safety depends on relative
robot-obstacle motion. We propose DODGER, a safety-guided RL framework that directly executes policy-generated actions to drive training rollouts while using CBF-filtered references and constraint violations to shape the policy toward collision-avoidance behavior. We evaluate DODGER through a Dubins-car safety analysis and demonstrate goal-directed navigation among multiple dynamic obstacles in full-order humanoid simulation and real-world humanoid experiments using
LiDAR-based perception, without a runtime safety filter.
Read the full abstract ▾
All authors Sanghyuk Park, Kwanwoo Lee, Taekyung Kim, Seohyeon Lim, Yisoo Lee
Robotics
Efficient Multi-Modal Planning with Reward-Guided Preference Optimization for Autonomous Driving
Chenglin Chen, Lujia Wang, Xinhu Zheng et al. (+2 more)
Safe and efficient trajectory planning is essential in autonomous driving. However, existing end-to-end approaches often fall short in both computational efficiency and safety guarantees. Methods based on imitation learning suffer from causal confusion, while rule-based scoring approaches often incur heavy computational overhead and suffer from objective misalignment. Additionally, preference-based methods rely on strict pairwise annotations, limiting data utilization. To overcome these limitations, we propose EMPlan, an efficient multi-modal trajectory planning method powered by reward-guided fine-tuning. We design a hybrid architecture that combines sparse anchors with an offset refinement module for efficient multi-modal trajectory prediction. Sparse anchors provide coarse trajectory candidates with low latency, which are subsequently refined by the offset module for higher prediction accuracy. To enhance safety without incurring additional inference costs, we adopt a two-stage training paradigm consisting of pretraining and reward-guided fine-tuning. During fine-tuning, we leverage rule-based reward signals and unpaired preference supervision to refine the pretrained policy toward safer trajectory selection. We evaluate EMPlan on the non-reactive NAVSIM
benchmark, where it strikes a favorable balance between planning accuracy and efficiency, demonstrating superior performance under real-time constraints.
Read the full abstract ▾
All authors Chenglin Chen, Lujia Wang, Xinhu Zheng, Jun Ma, Haoang Li
Robotics
Plan-Conditioned Imitation for Robust Object Retrieval under Self-Occlusion in Dense Clutter
Kowndinya Boyalakuntla, Ajinkya Pawar, Abdeslam Boularias et al. (+1 more)
Retrieving objects from dense clutter requires rearrangement during which the manipulator can occlude objects while moving them. Repeated arm withdrawals to restore visibility interrupt execution. We introduce TRACE, a plan-conditioned imitation framework for
retrieval under self-occlusion. A single unoccluded observation initializes a digital twin, where a privileged teacher generates a fixed nominal rollout. A recurrent student combines local rollout context, partial object observations, and proprioception to select actions that can correct deviations from the prediction. Behavior cloning initializes the student; DAgger refines it with teacher labels on student-visited states. The rollout remains fixed throughout execution, so the deployed student needs neither online teacher queries nor additional simulator rollouts during pushing. On 511 simulation test scenes, TRACE achieves 90.7% success versus 43.4% for nominal replay and 96.7% for the privileged closed-loop teacher. At a matched 26,373-label budget, student-state supervision achieves 87.8% versus 66.7% for expert-only cloning, demonstrating gains beyond additional labels. On a UR5e, TRACE achieves 90.0% success versus 95.0% for the closed-loop teacher, while reducing total execution time from 192.7 s to 67.3 s. It avoids the teacher's 16.8 sensing-related arm retractions per trial during pushing, retaining a final withdrawal for graspability evaluation. Code and data will be released at: https://trace-
retrieval.github.io.
Read the full abstract ▾
All authors Kowndinya Boyalakuntla, Ajinkya Pawar, Abdeslam Boularias, Jingjin Yu
Robotics
Online Evolution Strategy for Flow-Matching VLA Policies via Self-Supervised Trajectory Distribution Optimization
Gongxin Yao, Yongsheng Zhao, Jiayin Deng et al. (+4 more)
Vision-
Language-Action (VLA) models based on generative frameworks, such as Flow Matching, have recently achieved impressive performance in robotic manipulation. Unlike deterministic policies, Flow Matching enables VLA models to learn conditional action trajectory distributions, where latent noise vectors induce different actions under the same task scenario. However, we observe that these distributions are often ill-formed, with successful and failed behaviors coexisting while considerable probability mass remains in unfavorable regions. To this end, we propose Online-ES, an online adaptation framework for Flow Matching VLAs based on
Evolution Strategy (ES), which refines the learned action trajectory distribution through interaction feedback. Instead of
pruning the latent noise space, our method performs evolutionary exploration directly in the action trajectory space, where diverse trajectories generated by Flow Matching provide candidate solutions for adaptation. By perturbing sampled trajectories and evaluating their execution outcomes, we derive a self-supervised MSE objective that transfers the
evolution direction from trajectory space into model parameter space. Mathematically, we prove that the proposed objective provides an unbiased
estimator of the optimal
evolution direction. Moreover, we also incorporate failure experiences as negative feedback to regularize the
evolution direction, steering the policy away from previously explored failure regions. Experiments in both simulation and real-world environments demonstrate that Online-ES achieves policy improvement comparable to reinforcement fine-tuning, without learning a value model or computing advantages.
Read the full abstract ▾
All authors Gongxin Yao, Yongsheng Zhao, Jiayin Deng, Deng Liang, Han Gao, Lei Zhao, Baoping Cheng
Robotics
Locomotion-Grounded Humanoid Soccer: Task-Gated Reinforcement Learning of a Multi-Directional Kicking Library
Abu Hanif Muhammad Syarubany, Jaehyun Jang, Hwanhee Kim et al. (+3 more)
Recent humanoid soccer systems make motion tracking the substrate and derive locomotion from it, typically by steering a motion-reference anchor toward the ball. This yields strong shooting results, but locomotion is trained only on the narrow, deterministic command distribution ball approach induces, never evaluated as a capability in its own right. We invert the stack: a general, command-conditioned locomotion policy is trained first as the substrate, and N motion-guided kicking skills are added on top as task-gated layers, so the reachable gait space is set by the locomotion curriculum rather than any reference clip. Because every skill starts from and returns to this same commandable state, locomotion also becomes a composition hub (O(N) transitions rather than O(N^2)), and post-strike stabilisation is handed back to the trained controller rather than scripted per clip. We instantiate this on a 29-DoF Unitree G1 with seven retargeted kicking skills spanning 259.5 degrees of nominal aim direction, including lateral, rearward and weak-foot strikes a single forward-facing reference cannot express, and report shooting accuracy alongside command-tracking, terrain and push-recovery results with the full skill library attached, an axis prior humanoid soccer systems do not report. The library is validated on hardware across forward, lateral, rearward and commanded approaches.
Read the full abstract ▾
All authors Abu Hanif Muhammad Syarubany, Jaehyun Jang, Hwanhee Kim, Kyuwon Kim, Seungyeon Ryu, Chang D. Yoo
Robotics
Coral Grow-out Robotic Assessment System (CGRAS): Scaling Coral Recruit Monitoring Through Robotics and Computer Vision
Dorian Tsai, Scarlett Raine, Emilio Olivastri et al. (+12 more)
Climate change is the largest threat to coral reefs, with increasing global impacts accelerating the need for scalable reef restoration technologies. Large-scale reef restoration depends on the mass production of corals, such as through coral aquaculture. Coral seeding with recruits grown in aquaculture facilities is a feasible restoration approach, but effective production requires consistent, high-frequency monitoring of tens of thousands of macroscopic (0.5-2mm diameter) recruits, making conventional manual assessment prohibitively labor-intensive. To address this monitoring bottleneck, we introduce the Coral Grow-out Robotic Assessment System (CGRAS) which combines robotic imaging and computer
vision to automate data acquisition, perform multi-species detection and counting of corals, and evaluate coral health. CGRAS automatically extracts coral growth, survival and spatial distribution metrics, with the aim of providing timely feedback to operators for optimizing production, grow-out and deployment workflow processes. We demonstrate CGRAS in a large aquaculture facility on standardized coral settlement tiles, reducing the time and labor costs by a factor of 9.6 as compared to manual monitoring, whilst achieving 96.4% agreement for Acropora kenti corals relative to expert counts.
Read the full abstract ▾
All authors Dorian Tsai, Scarlett Raine, Emilio Olivastri, Riki Lamont, Andrew Lui, Timothy Morris, Joshua Esplin, Christopher A. Brunner, F. Mikaela Nordborg, Reginald Wardleworth, Garima Samvedi, Karen Jackel, Matthew Dunbabin, Tobias Fischer, Andrea Severati