Robotics
Walk Like This: Stylized Locomotion for Physics-Based Characters through Example-Driven Diffusion
Joshua Aurand, Agon Serifi, Sammy Christen et al. (+4 more)
We present a real-time framework for expressive legged locomotion that enables independent control over velocity and style. Unlike existing data-driven controllers, which typically require explicit style labels for training, our method efficiently leverages large-scale unstructured motion data and generalizes to unseen styles. We disentangle style from functional locomotion through a tailored sampling strategy used to train a style encoder whose output conditions an autoregressive
diffusion model. During inference, the user provides a style reference clip and a joystick velocity command. The
diffusion model then generates a kinematic reference tracked by a reinforcement learning controller. To improve user controllability and stylization strength, we propose an
attention biasing mechanism and a velocity command correction strategy. We demonstrate our method on a physics-based humanoid character in simulation and on a physical
robot, showcasing robust real-time control and continuous style modulation. The results can be best seen in our supplemental
video (https://www.youtube.com/watch?v=roFnS6N7nJ4).
Read the full abstract ▾
All authors Joshua Aurand, Agon Serifi, Sammy Christen, Ruben Grandia, Espen Knoop, Marco Hutter, Moritz Bächer
Robotics
SkipWorld: Latent Skip Planning with Distilled World Models for Efficient Visual Navigation
Yifei Yang, Zehua Fan, Zike Yan et al. (+2 more)
World-model-based visual navigation enables action planning by predicting future states under candidate trajectories. However, long-horizon rollout remains challenging: models are often trained with ground-truth visual histories but deployed in closed-loop prediction, and evaluating many candidates with step-by-step rollout is computationally expensive. We propose SkipWorld, a latent skip planning framework for efficient visual navigation. SkipWorld introduces three components: (i) Latent AutoRoll, which trains a world model to recursively roll out future states in latent space, reducing the mismatch between training and closed-loop inference; (ii) SkipDistill, which distills the latent autoregressive model into a skip world model that directly predicts long-stride future latent states; and (iii) Skip-then-Refine Planning, a dual-model coarse-to-fine planner that first filters candidate trajectories with the skip model and then reranks selected candidates with the stronger autoregressive model. This design preserves accurate trajectory evaluation while reducing redundant rollout computation. Experiments on RECON, HuRoN, SCAND, and TartanDrive show that, compared with NWM, SkipWorld reduces the average ATE by 4.2% and RPE by 2.7%, while decreasing planning time by 62.4% with a 2.7-fold speedup.
Read the full abstract ▾
All authors Yifei Yang, Zehua Fan, Zike Yan, Zhongyu Li, Yan Wang
Robotics
MARTS: A Modular Agricultural Robot for Targeted Spraying of Diseased Tomato Plants in Greenhouses
Rajmeet Singh, Manveen Kaur, Yusuf Adamu et al. (+2 more)
Broadcast spraying remains the dominant method of disease control in greenhouse tomato production, yet it treats healthy and diseased tissue alike, exposes workers to hazardous aerosols, and depends on operator skill. This paper presents MARTS (Modular Agricultural
Robot for Targeted Spraying), a low-cost robotic platform that closes the loop from plant-level disease detection to leaf-localised spray delivery in commercial greenhouses. MARTS integrates a Clearpath Husky A200 mobile base, a UR5 manipulator, a novel modular spraying end-effector built around a single 12 V micro-diaphragm pump, and a wrist-mounted Intel RealSense D435 RGB-D camera. Disease detection reuses our previously published Tomato Leaf Disease
Vision Language Model (TLDVLM; Grounding DINO, SAM-2 and LoRA-tuned BLIP-2, 97.27% accuracy over 10 classes); this work focuses on actuation, control and system integration. A ROS pipeline lifts detections into 3D targets, generates collision-aware nozzle poses at a 120 mm stand-off using fitted leaf normals, and scales pump pulse duration to lesion area. Water-sensitive paper calibration showed a linear pulse-volume relationship (R^2 = 0.994). In MuJoCo simulation, an adaptive base-repositioning strategy achieved 100% coverage of 18 waypoints with a maximum orientation error of 1.1 deg. Deployed at the SILAL commercial greenhouse (Al Ain, UAE) on cherry tomatoes under natural infection, MARTS treated 312 diseased leaves in 12 autonomous trials with 92.4% targeting success, 18.7 mm mean spray-centre error, 6.1% adjacent-leaf overspray, and a 71% reduction in pesticide volume versus broadcast spraying. Mean cycle time was 4.8 s per leaf, and the spraying subsystem costs USD 412, 6-10x lower than comparable systems, showing that accurate leaf-level targeting is achievable on commodity hardware.
Read the full abstract ▾
All authors Rajmeet Singh, Manveen Kaur, Yusuf Adamu, Lochan Kshetrimayum, Irfan Hussain
Robotics
ForeTac: Predictive Contact Guidance for Generative Robot Policies
Zhiyuan Chen, Shuai Chen, Haoran Zhou et al. (+7 more)
Visually similar actions can produce contact loss, jamming, slippage, or irreversible damage, while tactile feedback may arrive after a critical transition. Contact-rich manipulation therefore requires generative policies to anticipate and evaluate whether candidate actions will produce desirable contact outcomes, then adjust action generation before execution. We present ForeTac, a predictive contact-guidance framework organized as a represent-predict-evaluate-guide loop. ForeTac encodes the current tactile state, predicts the tactile consequences of a candidate action sequence, and evaluates the predicted trajectory using a contact-quality objective learned from successful and failed physical interactions. At deployment, its gradient propagates through the tactile predictor to guide action generation while the pretrained model parameters remain frozen. We evaluate ForeTac on four real-
robot tasks (planar and curved-surface wiping, card swiping, and fragile-object grasping) with
diffusion and flow-matching policies. Across these tasks, ForeTac raises the average task success rate from 36.3% to 71.3% and the average contact success rate (CSR) from 20.0% to 66.3% compared with
vision-only
Diffusion Policy. Component analyses support the tactile representation and action-conditioned foresight, and show that predicted touch preserves contact-quality ordering. Project page: https://foretac.github.io/.
Read the full abstract ▾
All authors Zhiyuan Chen, Shuai Chen, Haoran Zhou, Yizhe Zhao, Kaiyuan Yang, Zhixiang Duan, Shaoqing Xu, Ran Cao, Diyun Xiang, Yangwei You
Robotics
LynxScenes: A Large-Scale Real-World Benchmark for Outdoor Wheel-Legged Robot Perception, Navigation, Imagination and Reconstruction
Jun Tian, Hao Wu, Ruo Chen et al. (+5 more)
Wheel-legged
robots operating in outdoor spaces shared with humans require accurate perception, socially compliant navigation, future-scene
reasoning, and scene reconstruction. However, existing
datasets typically support these capabilities in isolation and rarely provide dense spatial and behavior-level supervision on real wheel-legged platforms. We introduce LynxScenes, a large-scale real-world
dataset and
benchmark collected with a LYNX M20 wheel-legged
robot. It contains approximately 100K 30-s navigation clips spanning 833.6 h and 3,294.7 km, with 3.0M annotated frames. The synchronized recordings include four fisheye cameras, dual
LiDARs, IMU, odometry, and
robot states. Annotations comprise 20.82M oriented 3D boxes across 27 categories, 13.89M road-structure annotations, and 20 navigation instructions organized into three layers and aligned with continuous future trajectories. LynxScenes supports four complementary tasks: instruction-conditioned trajectory prediction, BEV-based perception of 3D objects and road structures, future scene prediction, and multi-view scene reconstruction. For trajectory prediction, we further introduce a log-replay-based pseudo-closed-loop evaluator that measures safety, progress, and instruction compliance beyond trajectory imitation. Extensive evaluations across representative navigation, perception, world-model, and reconstruction methods reveal substantial remaining challenges in close-range outdoor
robot navigation. The
dataset and evaluation toolkit are planned for public release.
Read the full abstract ▾
All authors Jun Tian, Hao Wu, Ruo Chen, Yili Liao, Hongyi Lin, Jincheng Yu, Chao Li, Yingjie Niu
Robotics
LADDER: A Long-Horizon Robot Manipulation Benchmark
Petko Petkov, Giuliano Albanese, Nikolay Nikolov et al. (+3 more)
General-purpose
robot manipulation policies learn short-horizon skills effectively, but degrade rapidly on long-horizon tasks. Existing
benchmarks cannot isolate failures because task horizon is either too short or entangled with scene and object variations. We propose LADDER, a long-horizon
robot manipulation
benchmark scaling from 1 to 10 subgoals within the same environment over three suites (SimplerEnv-Long, RobotArena-$\infty$-Long, and RoboLab-Long), 145 tasks, eight categories, two embodiments, and 2,024 simulated episodes per system. We evaluate two orchestration paradigms: text subgoals for learned low-level policies (VoLo) and tool selection orchestration (CaP-X, GPT-Policy, PSP-
Agent, TiPToP). Given the full instruction directly, low-level policies achieve 85.4% on single-subgoal tasks, but drop to near zero on extended tasks. While orchestration remedies this, it remains bounded by physical execution errors of the low-level policy rather than high-level planning and achieves only ~25%. Tool selection orchestration with GPT-Policy on GPT-6 Astra significantly outperforms learned-policy methods, but incurs significant time and cost budget and succeeds only ~62%. Real-world experiments across two embodiments confirm the collapse of the low-level policy and the gain from orchestration. Project website: https://ladder.insait.ai
Read the full abstract ▾
All authors Petko Petkov, Giuliano Albanese, Nikolay Nikolov, Luc Van Gool, Jonathan Tremblay, Danda Pani Paudel
Robotics
Contractor Filters
Simon Rohou
We introduce Contractor Filters (CFs), a set-membership architecture for nonlinear continuous-time state estimation with bounded uncertainties and sporadic outliers. CFs represent feasible states symbolically by compositions of contractors rather than rematerializing a
geometric enclosure after each update. This provides an abstraction layer between the set-theoretic filtering equations and their numerical realization, so prediction, correction, ambiguity, and outlier handling can be expressed within one
algebra without prescribing a state geometry. We derive an exact nominal recursion and a guaranteed robust enclosure, with validated flow inclusions preserving soundness. Optional sub-paving compression then provides finite outer representations. CFs are evaluated against a robust particle filter on range-only and terrain-based localization under outliers, probabilistic model mismatch, association ambiguity, and multimodality. CFs retain the simulated truth in all tested configurations, while particle-filter reliability degrades under persistent bias and spatial aliasing. Contractor strength and compression budget control the trade-off between paving tightness and computational cost, while preserving the enclosure guarantee.
Read the full abstract ▾
Robotics
Data-Free Continual Learning for World Action Model with Rectified Generative Replay
Kaipeng Fang, Hangxu Liu, Chaojian Shi et al. (+5 more)
World action models (WAMs) built on pretrained
video generation models adapt quickly to new robotic tasks, but they forget previously learned tasks under sequential fine-tuning. Experience replay (ER) mitigates such forgetting by rehearsing data of previous tasks, yet this data is often unavailable in robotics, where a policy is deployed as weights only and its training data is not shared along with it. In this work, we ask whether a WAM can achieve data-free continual learning by regenerating its own past. Our key observation is that a WAM can indeed autoregressively regenerate complete rollouts of previously learned tasks and replay them to mitigate forgetting. However, replaying such raw rollouts still lags far behind replaying real data, which we attribute to two issues. First, the WAM is unable to recognize task completion and keeps generating purposeless motion after the task is done. Second, under open-loop generation, the generated actions gradually drift away from the generated observations. To address both issues, we present rectified generative replay (R$^2$-WAM), which truncates generated rollouts at task completion with an agentic evaluator and filters out rollouts whose actions are physically inconsistent with the observations. On the four LIBERO suites and five real-world tasks, R$^2$-WAM reduces forgetting by over 90% relative to sequential fine-tuning and nearly matches ER without accessing any data from previous tasks. Our analysis further reveals that forgetting in WAMs concentrates in inverse dynamics modeling: a forgotten WAM still produces visually plausible observations of previous tasks while its generated actions fail. Project page: https://kaipengfang.github.io/r2-wam/
Read the full abstract ▾
All authors Kaipeng Fang, Hangxu Liu, Chaojian Shi, Yunsong Zhou, Hongzi Zhu, Minyi Guo, Jingkuan Song, Lianli Gao
Robotics
TRACE: Efficient Transfer Learning for Neural Whole-Body Control Across Humanoid Embodiments
Qingyu Lan, Zhihao Tian, Zifan Wang et al. (+1 more)
Humanoid behavior foundation models (BFMs) enable diverse whole-body control, but adapting a pretrained controller to a new embodiment remains challenging because of kinematic and dynamic mismatch. We present TRACE, a framework for efficient cross-embodiment adaptation that preserves pretrained motion representations while specializing feedback control to the target
robot.TRACE first injects kinematic priors into input and output projection layers and aligns nominal joint dynamics to reduce embodiment mismatch and support stable fine-tuning. It then combines encoder
distillation with low-rank reinforcement learning on a compact, quality-screened motion set, using a two-phase curriculum to establish stable tracking before refining difficult motions. We apply TRACE to transfer SONIC from Unitree G1 to Unitree H2 using 10 million motion frames and approximately eight hours of training on a single RTX PRO 6000 GPU, at less than 0.1% of the GPU-hour cost of training SONIC from scratch. The adapted controller achieves 96.08% success on 67,953 evaluation motions under a height-scaled tracking criterion. Real-
robot demonstrations span dynamic motion tracking and whole-body teleoperation, with collected demonstrations supporting downstream
vision-
language-action policy post-training. These results demonstrate practical reuse of a pretrained humanoid controller across
robots with different body scales and dynamics.
Read the full abstract ▾
All authors Qingyu Lan, Zhihao Tian, Zifan Wang, Junwei Liang
Robotics
Breaking the Average: Overcoming Convergent Defaults in Robotic Manipulation via Diagnostic Chain-of-Thought and Structured Guidance
Mohammed Daba and Waleed Abdulrahman Saleh AlAkwa
Recent
Vision-
Language-Action (VLA) models, such as $\pi_{0.5}$, have demonstrated impressive generalization capabilities by scaling behavior cloning across massive
datasets. However, these models frequently suffer from distributional convergence, a phenomenon where the policy regresses to "safe" but suboptimal mean behaviors due to mode averaging in the training data. This often results in generic grasp strategies, Convergent Defaults, that fail in constrained or long-tail scenarios. In this work, we propose a novel framework to identify and correct these behaviors without parameter updates. We introduce a three-stage methodology: (1) Probing for Defaults, identifying task-specific mode collapse; (2) Diagnostic Root Cause Analysis, utilizing the VLA's latent
reasoning capabilities to verbalize the logic behind suboptimal actions; and (3) Structured Guidance Injection, where concrete, distinct alternatives are encoded into reusable "Lens
Prompts" that shift the sampling distribution. We validate this approach using $\pi_{0.5}$ in ManiSkill3 simulation environments. We hypothesize that our method outperforms vanilla $\pi_{0.5}$ and standard Chain-of-Thought prompting by effectively breaking convergent defaults in complex manipulation tasks.
Read the full abstract ▾
Robotics
ReEnvision: Future-Guided Selective Action Repair for Long-Horizon Manipulation
Xiyin Zeng, Xuewei Huang, Yi Lu et al. (+2 more)
Long-horizon robotic manipulation requires robust execution over sequences of interdependent actions. Existing methods typically handle execution deviations by replanning the remaining action chunk or correcting actions step by step as new observations arrive. However, a local deviation may affect only a few actions under distribution shift, yet replanning regenerates the entire chunk from an unfamiliar state, potentially introducing errors into otherwise valid actions. Step-wise correction may react too late, as it does not explicitly anticipate how the deviation affects subsequent actions. We propose ReEnvision, a future-guided selective action repair framework that updates a joint future-action belief using execution feedback. It then learns how the resulting belief revision affects each unexecuted action. The resulting belief revision enables selective repair of only the affected actions within a cached action chunk, preserving valid actions and reserving full replanning for severe deviations. On CALVIN, L-CALVIN, and RLBench, ReEnvision achieves 86.6% 5-step success, improves 10-step success from 56% to 63%, and reaches an average success rate of 88.8%, respectively. On five real-world manipulation tasks, it achieves an average success rate of 82.8%. Controlled repair experiments further demonstrate the benefits of localizing affected actions and preserving valid ones during execution.
Read the full abstract ▾
All authors Xiyin Zeng, Xuewei Huang, Yi Lu, Weihao Yuan, Hao Wang
Robotics
Locomotion with Online Awareness of Dynamics for Precision Footholds
Kunzhao Ren, Zhishuo Zhu and Xiaobin Xiong
Unknown payloads alter
robot dynamics and make precise contact execution difficult when foothold margins are small. We present LOAD, a perceptive locomotion framework that combines explicit inertial estimates with implicit dynamics features for narrow-beam traversal. Separate recurrent modules infer torso inertial changes and task-relevant history from proprioception, while a height-map teacher is distilled into a depth-based student. We extensively evaluate LOAD and its variants in simulation and on hardware across beam widths and payload configurations. The results show that implicit dynamics features accelerate training, while explicit inertial estimation further improves traversal success; combining both enables LOAD to approach the performance of a privileged policy supplied with ground-truth inertial parameters. Hardware experiments demonstrate traversal with unilateral payloads of 6 kg on a 10 cm beam and 9 kg on a 15 cm beam, and an evenly distributed 11 kg payload on a 15 cm beam. These results support combining physical estimates and history features for precision locomotion under unknown loads.
Read the full abstract ▾
Robotics
GLST: Global-Local Spatio-Temporal Memory Decomposition Framework for Online Chained-Goal Navigation
Yousef Liang, Ryan Ardalan, Yarden Goraly et al. (+6 more)
Autonomous navigation in real-world environments often requires long-horizon operation under partial observability and evolving goals. Existing mapless navigation approaches predominantly assume fixed goals throughout an episode, limiting applicability to practical scenarios where goals change online during execution. We study online chained-goal navigation, where goals evolve through existing-goal redirection or new-goal emergence without explicit transition signals. Under this setting, we find navigation may require either continued exploration or strategic revisitation, creating conflicting
memory requirements. We propose Global-Local Spatio-Temporal Belief Learning (GLST), a
memory-decomposition framework that separates latent beliefs into global-local spatial and long-short temporal representations for long-horizon
reasoning. We further introduce goal-conditioned rewards to balance exploration and exploitation under evolving goals. Extensive experiments in simulation across diverse terrains demonstrate improved chained-goal navigation performance over baselines. Real-
robot experiments using the Spot quadruped further validate our
memory decomposition framework.
Read the full abstract ▾
All authors Yousef Liang, Ryan Ardalan, Yarden Goraly, Chenhao Zhu, Yue Yin, Jonas Frey, Claire Tomlin, Marco Pavone, Shankar Sastry
Robotics
WoVR+: World Model as Reliable Simulator for Versatile VLA Policy Optimization with RL
Shangqing Zhou, Zhennan Jiang, Ning Ma et al. (+9 more)
Action-conditioned world models can reduce costly real-
robot interaction in reinforcement learning for
Vision-
Language-Action models (VLA-RL), but
hallucinations can mislead policy optimization. Extending them to richer VLA interfaces and long-horizon tasks raises challenges in spatiotemporal and cross-modal consistency, exploration-reliability balance in RL, and compounding rollout errors. We present WoVR+, extending WoVR's
hallucination-aware WM-RL framework by redesigning the learned environment and imagined interaction protocol: A VLA-interface-aligned world model provides multi-view prediction and proprioceptive updates while mitigating
hallucinations. Chunk-wise noise preserves flow-based policy exploration while limiting prediction-degrading action jitter. For long-horizon tasks, stage-local imagined training combines stage-entry resets, stage-specific rewards, local horizons, and controllable sampling to shorten generated prefixes and balance training across stages. WoVR+ improves success over SFT by 21.09% and 16.7% on RoboTwin and 6 real-
robot tasks. Long-horizon phone-packaging success rises from 30% to 75%.
Videos are available at https://youmo445.github.io/WoVR_Plus.github.io/.
Read the full abstract ▾
All authors Shangqing Zhou, Zhennan Jiang, Ning Ma, Xin Fu, Yupeng Zheng, Shuai Tian, Huangrui Li, Jiahao Liu, Chao Yu, Zhongpu Xia, Haoran Li, Dongbin Zhao
Robotics · Cross-listed from cs.CV
DriveWLA: Unifying Future Dynamics and Scene Semantics through World-Language-Action Modeling for Autonomous Driving
Shuzheng Wang, Haoxuan Li, Shengzhen Li et al. (+5 more)
Vision-
Language Models (VLMs) and world models have demonstrated strong capabilities in semantic understanding and future scene
evolution for autonomous driving. However, effectively integrating both capabilities for trajectory generation remains challenging. In this work, we propose DriveWLA, a unified World-
Language-Action model based on Mixture-of-
Transformers (MoT) that jointly models semantic understanding, future scene dynamics, and trajectory generation. DriveWLA consists of three experts: a World Expert for future prediction, a
Language Expert for semantic understanding, and an Action Expert for trajectory generation. We further introduce Cross-Expert
Attention, which enables controlled information flow across experts while aligning semantic and dynamic representations. To improve trajectory generation beyond imitation learning, we apply Flow-GRPO reinforcement learning post-training to refine the Action Expert. Extensive experiments show that DriveWLA achieves superior performance in both open-loop and closed-loop evaluation. DriveWLA achieves 92.3 PDMS on NAVSIM v1 and 90.1 EPDMS on NAVSIM v2, demonstrating strong performance among VLM-based and world-model-based planners. Without specific fine-tuning, DriveWLA further demonstrates strong zero-shot generalization to HUGSIM, nuScenes, and Bench2Drive. These results validate the effectiveness of our unified World-
Language-Action modeling, establishing DriveWLA as a generalizable framework for autonomous driving.
Read the full abstract ▾
All authors Shuzheng Wang, Haoxuan Li, Shengzhen Li, Chenxi Wu, Yajing Sun, Lu Hou, Wenyu Liu, Xinggang Wang
Robotics
Agentic Robot Creation through Generation, Verification, and Revision
Xuebing Li, Yibo He, Sitong Liu et al. (+2 more)
Can an AI
agent turn high-level user intent into a functioning
robot and teach it how to behave? We study this question from two complementary aspects: embodiment and behavior creation. For embodiment creation, we investigate how well current
multimodal models capture mechanical requirements, and introduce computational metrics for evaluating generated designs before fabrication. For behavior creation, we present Roboforge, an agentic framework that turns task specifications into executable policies through motion generation, policy learning, and execution-grounded revision. Given a
robot embodiment and task, the
agent generates a reference motion, configures the learning objective, and trains a policy. Crucially, motion remains in the loop: rollout feedback can revise the reference motion, reward, or both. We evaluate a generated embodiment across multiple MLLMs and behavior creation across four diverse
robots against reward-refinement and one-shot motion baselines. Our computational metrics expose recurring mechanical failures in generated designs, while behavior experiments show that execution-grounded refinement improves task-level behavior and execution quality. Finally, we fabricate an
agent-generated Gorilla and directly deploy learned behaviors on the physical
robot.
Read the full abstract ▾
All authors Xuebing Li, Yibo He, Sitong Liu, Ran Ji, Jiayuan Gu
Robotics
Information-based Unified Design Framework for NMR Magnetic Tracking System
Seçkin Eren Yetim and Mehmet Efe Tiryaki
Nuclear
magnetic resonance (NMR)
magnetic tracking offers a promising alternative to
magnetic resonance imaging (MRI)-based localization for medical robotic guidance. However, scalar field measurements and field-aligned marker moments constrain position and orientation tracking, requiring careful configuration design. We present a unified information-based framework that uses local covariance analysis to evaluate receiver geometry, marker separation, and external source configuration in on/off-board NMR tracking. Numerical evaluation shows that planar-array precision depends jointly on receiver count and geometry, with triangular
lattice arrays achieving the lowest best-case position uncertainty among the evaluated layouts for 9–20 receivers. Two spatially separated, field-aligned markers enable device-axis tracking, with predicted angular uncertainty below $2^\circ$ at the evaluated workspace center for suitable marker separations. We further establish point-dipole source–receiver duality, unifying the analysis of on/off-board NMR tracking. These
Read the full abstract ▾
Robotics
LAG-IO: Multi-IMU State Estimation for Legged Robots Using Factor Graphs
Yicheng Zhou, Longwen Chen, Qinghua Yu et al. (+3 more)
Robust proprioceptive state estimation is crucial for legged
robots, particularly in challenging environments where exteroceptive sensors are unreliable. While traditional leg odometry typically relies on a single body-mounted inertial measurement unit (IMU) fused with leg kinematics via an extended Kalman filter (EKF), such methods are vulnerable to linearization errors and hard-contact assumptions, leading to severe drift during slippage or dynamic locomotion.In this paper, we propose LAG-IO, a factor graph-based multi-IMU state estimation framework that tightly couples measurements from a main body IMU, multiple leg-mounted IMUs, and joint encoders within a sliding-window optimization. To handle non-stationary foot-ground interactions, we introduce a continuous probabilistic contact detection module based on kernel density estimation (KDE), which adaptively scales the covariance of zero-velocity update (ZUPT) factors. Furthermore, kinematic relative position factors are formulated to structurally anchor the independent leg inertial states to the
robot's main body. Experimental results on a quadruped
robot across various unstructured terrains demonstrate that, compared to existing baselines, LAG-IO achieves higher planar trajectory accuracy, reduces unobservable vertical divergence, and provides smoother trajectory transitions during unstable contacts, while maintaining competitive attitude observability.
Read the full abstract ▾
All authors Yicheng Zhou, Longwen Chen, Qinghua Yu, Junhao Xiao, Huimin Lu, Xieyuanli Chen
Robotics
Physics-Aware Object-Centric SE(3) Forecasting for Real-Time Proactive Dynamic Grasping
Yirui Zhou, Hongli Xu, Ying Zheng et al. (+2 more)
Proactive dynamic grasping requires
robots to predict the future state at which physical contact will occur, rather than reacting to the currently observed object pose. But what future must the
robot predict, and at what level of abstraction? We argue that contact-relevant rigid-body state in SE(3) is a compact task-sufficient abstraction for this problem: it directly determines where and when the
robot should establish contact, without requiring visually complete future scenes. This insight leads to a lightweight object-centric forecast-then-select framework that predicts future rigid-body trajectories from short pose histories and uses a reachability-aware interception selector to choose feasible contact states under kinematic and timing constraints. Trained with randomized physics simulation over primitive shapes, the forecaster transfers to real objects once their metric geometry and SE(3) state histories are recovered. Experiments across simulation and real-world dynamic grasping tasks show that our framework improves the average success rate from \(24.5\%\) to \(62.0\%\) in simulation, and from \(15.3\%\) to \(67.3\%\) in real-world catching, conveyor, and turntable interception. The SE(3) forecaster runs in \(15\,\mathrm{ms}\), orders of magnitude faster than dense visual future prediction, enabling real-time contact decision making. Our results suggest that effective proactive grasping does not require imagining the entire future scene. Instead, predicting the compact physical future that determines contact provides an effective and practical abstraction for real-time dynamic grasping.
Read the full abstract ▾
All authors Yirui Zhou, Hongli Xu, Ying Zheng, Yi Wang, Lap-Pui Chau
Robotics
Which Trajectory Is Safer? Combining Rule and Learning for Safety Evaluation
Xuewen Luo, Linlian Jiang, Bo Yu et al. (+3 more)
Collision rate remains a dominant metric for evaluating autonomous driving trajectory safety, but its binary nature obscures substantial safety differences among collision-free trajectories. Existing surrogate-safety metrics provide finer-grained assessments, yet often trade off physical interpretability against adaptivity to interaction context. We introduce FutureRisk, a continuous metric that quantifies the latent future risk of a trajectory by how closely and persistently its realized interactions approach unsafe states. At its core, our Context-Adaptive Risk Model (CARM) combines a context-dependent safety-time distribution with an independently supervised critical-event-time distribution. Its safety criterion is bounded around a closed-form physical prior, while a bounded residual provides additional adaptation without abandoning this physical anchor. Experiments on DeepAccident and Bench2Drive show that FutureRisk achieves stronger agreement with native collision labels than existing surrogate-safety measures, discriminates risk among collision-free trajectories, and transfers zero-shot across
datasets. Comparisons across four planners further show that FutureRisk reveals safety differences hidden by binary collision outcomes. Together, these results establish FutureRisk as a fine-grained, interpretable, and context-adaptive metric for trajectory safety evaluation. Code is available at https://github.com/Xuewen2025/safety-evaluation.
Read the full abstract ▾
All authors Xuewen Luo, Linlian Jiang, Bo Yu, Fan Ding, Xinxin Zuo, Yang Wang
Robotics
Beyond the End-Effector: Learning Whole-Arm Contact Manipulation from Human Demonstrations
Bo-Ruei Huang, Daniel Seita and Erdem B{\i}y{\i}k
Robot manipulation has traditionally relied on end-effector-centric interactions. In contrast, humans routinely exploit whole-arm contacts to improve efficiency, robustness, and adaptability, especially when the hands are occupied or when precise grasping is unnecessary. Enabling similar capabilities in
robots remains challenging: the
robot must select where on its arm to make contact, generate effective non-prehensile interactions, and satisfy task-specific pose or safety constraints on the end-effector. We propose Whole-Arm Contact Learning (WACL), a hierarchical framework that describes each whole-arm contact task with three
robot-agnostic representations and hands them to a quasi-static surface-contact optimizer: semantic keypoints that summarize the scene, a reference contact trajectory that specifies where and how the arm presses on the object, and safety-aware path constraints that a
vision-
language model generates from the task description to protect the end-effector's task. The
robot's kinematic chain enters only in this optimizer, which grounds these representations onto the arm by jointly searching over which link makes contact, where on that link, and the joint configuration that realizes it. This decomposition lets WACL learn the contact trajectory from 10 bare-hand human demonstrations per task collected without teleoperation, while new end-effector constraints change only the optimizer's input rather than any trained component. Real-world experiments on a 7-DoF arm across four whole-arm tasks show that WACL sustains contact under end-effector constraints where a strong link-pose baseline fails (78% vs. 33% success) and generalizes to new constraints without retraining. Website: https://wacl-
robot.github.io
Read the full abstract ▾
Robotics
Beyond Safe Execution: Safe Online Reinforcement Learning for Autonomous Driving with Value Calibration and Adaptive Guidance
Zeqiao Li, Yijing Wang, Haoyu Wang et al. (+2 more)
Online reinforcement learning (RL) enables driving policies to adapt to changing deployment conditions, but requires safe exploration throughout training. Safety filtering alone does not ensure stable policy improvement or reduced dependence on corrections. We propose Safe Online Adaptive Refinement (SOAR-Flow), a framework for safe and stable online fine-tuning of offline-initialized Flow-based driving policies. A control barrier function (CBF)–model predictive control (MPC) filter corrects potentially unsafe actions before execution. To support stable policy updates, we introduce Gaussian-return-aware value calibration based on calibrated Q-learning (Cal-QL), using recorded behavior returns and predicted return variability to smooth conservative critic updates and mitigate policy degradation. Paired corrective learning teaches feasible action adjustments, while an intervention-to-go predictor encourages decisions requiring fewer future corrections. Its predictions also regulate guidance strength, allowing recovery supervision to weaken as predicted dependence decreases. Together with gradually reduced calibration regularization, this supports continued task improvement while preserving hard execution constraints. We analyze conditions for safe execution at control steps and diminishing auxiliary learning influences. We evaluate SOAR-Flow in simulation and on an uncrewed ground vehicle (UGV). The results demonstrate improved driving performance and reduced reliance on safety corrections during protected online fine-tuning. The demonstration
video and project overview are available at https://lzqw.github.io/SOAR-Flow/.
Read the full abstract ▾
All authors Zeqiao Li, Yijing Wang, Haoyu Wang, Zheng Li, Zhiqiang Zuo
Robotics
DexStitch: Learning and Composing Dexterous Skills from Human Videos via Contact-Aware Robotic Inpainting
Byeonggyeol Choi, Seongmin Jung, Unghui Nam et al. (+5 more)
A long-horizon manipulation task consists of primitive skills and the transitions connecting them. Learning skills that can be executed in sequence requires supervision for both, which is costly to collect on a
robot. Human
videos offer a scalable source of demonstrations, but transferring them to
robots requires extracting
robot-compatible observations and actions together with their skill structure. We present DexStitch, a framework that cuts long-horizon human
videos into skills and transitions, converts them into aligned
robot observation-action-skill data, and stitches the learned skills into a requested sequence. A skill classifier first segments the
videos into manipulation skills and transitions. DexStitch then converts the human demonstrations into
robot observations and actions while retaining their corresponding skill labels. During conversion, per-finger human-object contacts guide dexterous hand retargeting and robotic inpainting, preserving contact geometry and finger-object occlusion ordering. To stitch the learned skills into a sequence, a shared skill-conditioned policy is pretrained on the resulting synthetic data and finetuned on a mixture of synthetic data and a small set of single-skill teleoperation episodes. Conditioning on the current and next skills guides each transition toward the following skill, while predicted skill progression enables closed-loop switching. On a held-out session, the classifier agrees with manual annotations on 92.4% of frames. Across four household skills, DexStitch achieves 80.0% single-skill success and 77.5% sequence success.
Read the full abstract ▾
All authors Byeonggyeol Choi, Seongmin Jung, Unghui Nam, Youngbo Song, Sunhong Min, MinHyung Lee, Byonghyo Shim, Yongseok Lee
Robotics · Cross-listed from cs.AI
Autonomous 3D Visual-Thermal Inspection for Utility-Scale Solar Farms
Lachlan Tychsen-Smith, Mark Cox, Brett Wood et al. (+6 more)
Utility-scale photovoltaic (PV) inspection is dominated by aerial surveys, which are effective for identifying module-level hot
cells but provide limited visibility of critical components located under the array, including connectors, cabling, fasteners and structural elements. These assets are difficult to inspect consistently at scale, and their defects are often spatially small, partially occluded and difficult to localise from isolated images. We present an automated 3D visual-thermal inspection system (Tricorder) for under-panel PV asset inspection. The approach combines
LiDAR-inertial Simultaneous Localization and Mapping (SLAM) with geometrically calibrated RGB and thermal imagery and machine-learning-based object detection and segmentation. Detected assets are associated with the reconstructed 3D environment, allowing visual and thermal observations from multiple viewpoints to be spatially fused and localised. We demonstrate the approach on utility-scale PV data for MC4 connector detection and connection-state recognition, thermal anomaly identification, and detection of structural components including bolts, nuts and torque tubes. The results demonstrate a pathway from image-based inspection toward spatially grounded autonomous inspection, where defects are detected and linked to specific assets and their 3D locations within the solar farm.
Read the full abstract ▾
All authors Lachlan Tychsen-Smith, Mark Cox, Brett Wood, David Haddon, Kenrick Anderson, Benjamin Duck, Jim O'Malley, Noel W Duffy, Peyman Moghadam
Robotics
Learning Robot Policies for Whole-Body Mobility Assistance via Dynamic Human Motion Generation
Shikun Ban, Shohei Nagai, Qingyang Song et al. (+4 more)
Robotic assistance for human mobility could benefit care recipients and reduce caregiver burden during physically demanding tasks such as Sit-to-Stand (STS) and Lift-and-Transfer (L&T). Learning
robot policies for these whole-body assistive tasks is challenging: real-world data collection for human-
robot interaction is costly and safety-critical, making simulation an attractive alternative. Prior work, however, typically treats the human as static or minimally responsive and thus fails to capture the cooperative nature of mobility assistance. We present a simulation-based framework for learning whole-body assistive
robot policies. At its core is a human policy trained with an
adversarial motion prior that produces realistic care-recipient motion in response to the
robot's actions. A human-adaptive scripted controller interacts with this policy to collect diverse
robot demonstrations at scale. From these demonstrations, we train 3D
diffusion policies that map segmented human point clouds and
robot joint positions to whole-body actions for a mobile manipulator, without relying on privileged human state at deployment. The resulting
robot policies successfully perform both STS and L&T assistance in simulation. Preliminary experiments further demonstrate sim-to-real transfer for STS after fine-tuning on a small amount of real-world data.
Videos are available on our project page: https://rchi-lab.github.io/human-gen-mobility-assistance/
Read the full abstract ▾
All authors Shikun Ban, Shohei Nagai, Qingyang Song, Hanxi Wan, Eric Zhao, Thomas Wang, Zackory Erickson
Robotics
I3L: Iterative Interactive Imitation Learning for Whole-Body Mobile Manipulation
Wensi Ai, Christopher Luey, Karthik Dharmarajan et al. (+7 more)
Policies trained offline from teleoperated demonstrations can degrade outside their training distribution. This gap is particularly acute in long-horizon whole-body mobile manipulation, where long horizons amplify compounding errors and coordinated arm, torso, and base motion makes exhaustive demonstration coverage impractical. We introduce I3L, an integrated hardware and learning system that combines whole-body correction with iterative policy refinement to address this last-mile reliability gap. Its low-cost, open-source JoyLo+ interface uses policy-tracking leader arms and touch-triggered takeover to enable coordinated corrections across the body parts, while iterative deployment and retraining allow operators to target failures of the updated policy. We evaluate I3L across three
robot embodiments and eight simulated and real-world tasks, including four long-horizon whole-body mobile manipulation tasks. Controlled simulation experiments show that four-round HG-DAgger achieves the highest final success on both tasks. Under this protocol, I3L improves autonomous success over the base policies by an average of 68.3 percentage points and achieves at least 90% success on every task, outperforming demonstration-only training by 35.6 percentage points on average with the same number of training trajectories. Averaged across tasks, takeover duration per episode decreases by 72% over four rounds. Together, these results show that I3L helps close the last-mile gap between offline training and reliable deployment and provides practical insights into collecting and using corrections for long-horizon whole-body mobile manipulation.
Videos, visualizations, and appendix are at https://i-3l.github.io/.
Read the full abstract ▾
All authors Wensi Ai, Christopher Luey, Karthik Dharmarajan, Evans Han, Yunfan Jiang, Stef Ren, Jianwen Xie, Jiajun Wu, Li Fei-Fei, Ruohan Zhang
Robotics
The Human–VLA Communication Gap: What Humans Leave Implicit and What Policies Cannot Recover
Yiqing Xu and Jiajun Wu
A
robot may know how to perform a task yet fail when a human asks for it naturally. We study this human–VLA communication gap by separating an information gap, where task semantics, grounding, or event structure remain unresolved, from a residual execution gap, where even a fully specified task depends on how it is organized and invoked. To do this, we characterize how people naturally express demonstrated objectives and evaluate controlled variants of the same physical tasks, with detailed analysis on $\pi_{0.5}$ and consistent trends on $\pi_0$-FAST and Cosmos3-Edge. Our study reveals that humans increasingly compress instructions as tasks grow, relying more on contextual grounding and implicit execution structure. Current policies are misaligned with this behavior: they tolerate grounding dependence relatively well, but struggle when task semantics or event structure must be reconstructed. Even after the information gap is resolved, execution remains sensitive to command granularity and invocation state. These findings identify concrete targets for human-centered policy learning and interfaces that better bridge natural human objectives to
robot behavior. Project website: https://commgap.github.io/.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
Where Small Mistakes Matter: Quantifying State Criticality for Imitation Learning
Yusuf Umut Ciftci, Joonwon Kang, Ajay Suresha Sathya et al. (+1 more)
Imitation learning typically treats every demonstration state as equally important, optimizing a behavior-cloning objective uniformly across the
dataset. Yet not all states are created equal: in a small number of critical states, even a slight deviation from the expert action can lead to a failure, while in many other states such errors are recoverable and inconsequential. We formalize state criticality in imitation learning as the worst-case degradation in downstream value induced by a bounded perturbation of the expert action at a given state. We then propose SCRIM, a practical State CRIticality Measure that estimates this quantity by training an
adversarial disturbance policy and a value critic around a fixed imitation policy. SCRIM assigns each demonstration state a score reflecting how vulnerable the task outcome is to small, policy-relevant action errors. We show that these scores can be used to reweight behavior-cloning losses, improving imitation at consequential states and increasing overall task performance. We further show that SCRIM enables data curation by selecting compact, high-value subsets of demonstrations that preserve performance while substantially reducing the amount of training data. We evaluate SCRIM across various simulated and real-world robotic manipulation tasks, and demonstrate that SCRIM improves policy training, enables efficient data selection, and provides interpretable insights into which moments in a demonstration are most critical for task success. Project website: https://sia-lab-git.github.io/SCRIM/
Read the full abstract ▾
All authors Yusuf Umut Ciftci, Joonwon Kang, Ajay Suresha Sathya, Somil Bansal
Robotics
HelpMeOut: A Neuropsychology-Grounded Benchmark for Error-Aware Monitoring of Household Task Execution
Haoran Wang, Jingwei Huang, Anqing Duan et al. (+2 more)
We introduce HelpMeOut, an embodied AI
benchmark for error-aware monitoring of household task execution. Existing embodied household
benchmarks typically assume that humans execute tasks correctly, but this assumption does not hold for cognitively impaired persons or older adults whose everyday activities produce structured execution errors. To address this gap, HelpMeOut operationalizes research on everyday action errors into 20000 paired activity episodes, covering nine neuropsychologically motivated error categories. In each episode,
agents observe a streaming human activity trajectory and must monitor whether the execution is unfolding correctly, diagnosing any structured errors as early as possible. We evaluate learned-monitor, symbolic, change-point, and frontier LLM prompting baselines. Results show that these methods perform poorly as error monitors, often missing when execution goes wrong. These results suggest a gap left by standard goal-inference and task-completion
benchmarks, making HelpMeOut a testbed for error-aware assistive
agents.
Read the full abstract ▾
All authors Haoran Wang, Jingwei Huang, Anqing Duan, Ho Seok Ahn, Bruce Macdonald
Robotics
OccWAM: Occupancy World-Action Models with Joint Scene-to-Ego Flow Matching
Qihang Huang, Wenhui Huang, Collister Chua et al. (+6 more)
Occupancy world models provide a structured, metric representation of future driving scenes, making them a promising basis for end-to-end autonomous driving. However, existing occupancy-aware planners typically couple occupancy prediction and action generation through either hierarchical coupling or parallel multi-task learning. Hierarchical methods use occupancy either as a fully predicted future preceding action generation or as an action-conditioned rollout for trajectory evaluation, whereas parallel multi-task methods predict occupancy alongside planning through a separate branch or auxiliary objective. These interfaces retain either an explicit generation order or a role asymmetry, limiting reciprocal interaction between world prediction and action generation during trajectory formation. We propose OccWAM, an occupancy-native world-action model that jointly generates dense future occupancy and continuous ego trajectories via flow matching conditioned on scene representations. Building on this joint formulation, we introduce scene-to-ego world-action flow matching, a two-stage formulation. The first stage predicts scene-centric occupancy together with an absolute ego trajectory, providing a stable representation of future scene
evolution. The same predicted world is then re-indexed into the future ego frames induced by the initial trajectory, where ego motion is represented through adjacent pose increments and a second joint flow refines ego-centric occupancy and motion for action-aligned planning. Unlike
video-based world-action models, OccWAM preserves explicit semantic geometry for direct planning
reasoning without pixel synthesis. Extensive experiments on Bench2Drive and nuScenes demonstrate state-of-the-art performance in closed-loop driving, open-loop planning, and future occupancy prediction.
Read the full abstract ▾
All authors Qihang Huang, Wenhui Huang, Collister Chua, Shiyu Fang, Zhan Chen, Songyan Zhang, Zhidong Wang, Daiying Tian, Chen Lv