Robotics
MomWorld: Momentum-Aware Latent World Model for Long-Horizon Autonomous Driving
Ziying Song, Shengkai Zhang, Lei Yang et al. (+6 more)
Long-horizon planning enables autonomous vehicles to anticipate scene
evolution and potential risks, supporting safe and stable decisions in complex interactions. However, existing methods struggle to propagate motion trends from observed history into the future. Long rollouts based on a single latent state may further attenuate useful dynamics, retain stale motion patterns, and disrupt reliable near-term plans. We introduce MomWorld, a momentum-aware latent world model for long-horizon planning. MomWorld extracts scene motion trends from historical-to-current observations and propagates latent momentum into future horizons, jointly predicting future configuration and momentum states. A learnable momentum persistence mechanism preserves stable trends, scene-conditioned momentum updates adapt future dynamics, and a scene-adaptive reset gate suppresses stale momentum under abrupt changes. We further propose MoFlow, a momentum-conditioned flow-matching module that refines a base trajectory to align with the predicted future scene
evolution in only a few integration steps, with a horizon-aware residual fusion that preserves near-term planning stability while permitting stronger long-range corrections. Extensive experiments on NAVSIM, nuScenes and Bench2Drive demonstrate that MomWorld improves long-horizon planning consistency and reduces the average collision rate by 12.2% relative to MomAD over a 6-second planning horizon.
Read the full abstract ▾
All authors Ziying Song, Shengkai Zhang, Lei Yang, Haozhuang Chi, Yuchen Liu, Jiangtao Su, Lin Liu, Ziyang Liu, Chen Lv
Robotics · Cross-listed from cs.AI
Does Adversarial Training Improve Generalization in Multi-View VLAs? Revealing and Mitigating View Collapse
Futa Waseda, Shuhei Kurita and Isao Echizen
Vision-
language-action (VLA) models adapt pretrained
vision-
language models (VLMs) for closed-loop
robot control, transferring their perceptual and semantic capabilities to action prediction. Despite strong in-distribution performance, however, VLAs often degrade under deployment shifts.
Adversarial training (AT) offers a model-adaptive approach to robustness without explicitly anticipating individual shifts, but its effect on natural distribution-shift generalization in multi-view VLAs remains unclear. We study this question using a multi-view VLA directly adapted from a pretrained VLM and evaluate generalization across seven LIBERO-Plus shift axes. Direct AT substantially improves Camera Viewpoint and Sensor Noise, the two shifts affecting only the third-person view, yet produces mixed or negative effects on other shifts. Controlled view interventions reveal a surprising failure mode that we term view collapse: Direct AT can shift cross-view reliance so strongly that the policy becomes dominated by the wrist view. This exposes a robustness shortcut: apparent robustness to a shifted view can arise from reduced use of that view rather than more robust perception of it. This motivates a distinction between robust perception, extracting reliable information under within-view shifts, and robust fusion, adapting reliance across views according to their reliability. To reduce fixed view reliance, we use a simple View Swap intervention and then re-evaluate AT. With View Swap, AT further improves Camera Viewpoint, Sensor Noise, and
Robot Initial State, while its effects remain mixed on other shifts. Our results show that multi-view robustness requires separating improved perception from changes in cross-view reliance, and that AT provides selective rather than generic distribution-shift benefits.
Read the full abstract ▾
Robotics
Observability-Informed Optimal Sensor Placement for Soft Robots
Samuel Smocot, James Richard Forbes and Audrey Sedal
This paper presents the application and experimental evaluation of a systematic method for optimal sensor placement in soft
robots. Existing methods either lack generalizability across different soft
robot morphologies or do not account for system dynamics. The applied method uses convex optimization to find the optimal sensor configuration that maximizes an observability Gramian-based metric. The framework is experimentally evaluated using position and strain measurements on a soft continuum arm. Kalman filter state estimates using optimal sensor placements yield lower reconstruction error than a baseline across all sinusoidal input trials, with improvements on the order of millimeters. This case study shows that linear control theory tools can guide optimal sensor placement in soft
robots, suggesting an interpretable approach to sensor placement that may extend to other morphologies.
Read the full abstract ▾
Robotics
Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies
Duo Wu, Haifeng Wang, Rongwei Lu et al. (+6 more)
Reinforcement learning (RL) enables generalist
robot policies to improve through trial-and-error interaction, yet its effectiveness is fundamentally constrained by sparse task rewards. Existing general-purpose reward models typically alleviate this issue by learning task progress from expert demonstrations, but introduce a distribution mismatch with the mixed-quality rollouts encountered during policy optimization, making their estimates unreliable on suboptimal and failed behaviors from which the policy must learn. In this work, we introduce a demonstration-free reward learning paradigm where dense reward feedback can be learned directly from sparse task outcomes and policy experience. We theoretically show that terminal task outcomes implicitly define dense success-probability feedback at intermediate timesteps, which can be recursively learned through bootstrapping. Based on this insight, we introduce eVTA$_0$, which learns success probabilities from mixed-quality policy rollouts through temporal-difference-style bootstrapping, without expert demonstrations or intermediate annotations. We further introduce RL with Evolving Rewards (RLER), a closed-loop framework that adapts eVTA$_0$ using newly collected rollouts as the policy evolves. Experiments show that eVTA$_0$ provides more informative rewards than state-of-the-art reward models and achieves the best average policy performance across all LIBERO task suites under the same RL training budget, improving success rates by 5.4%-13.8% over the initial policy. In real-world manipulation, RLER further improves overall success rates by 20%-26%, with 35%-36% gains under out-of-distribution conditions. These results demonstrate the effectiveness of demonstration-free reward learning and adapting rewards as the policy evolves. Project webpage: https://duowuyms.github.io/evta0.
Read the full abstract ▾
All authors Duo Wu, Haifeng Wang, Rongwei Lu, Jinghe Wang, Tianyi Xiong, Zhimin Wang, Chao Yu, Shuai Ma, Zhi Wang
Robotics
InfraVLA: Extending Vision-Language-Action Navigation with Infrastructure Cameras
Lukas Vierling, Benjamin Ramtoula, Luke Robinson et al. (+2 more)
Many indoor environments in which
robots operate, such as warehouses, offices, and hospitals, already have cameras installed. They observe parts of the building that the
robot cannot see from where it stands, yet navigation policies, including recent
vision-
language-action (VLA) models, do not use them. We propose InfraVLA, an end-to-end method that adapts a pretrained navigation VLA to such static infrastructure views: a closed-
circuit television (CCTV) encoder turns each external view into
tokens of the input sequence. Because the views matter only at rare decision points, fine-tuning alone did not make the policy use them in our experiments; we therefore train in two stages, on demonstrations with upsampled counterfactual data and then on recovery data. We evaluate on two simulated warehouse tasks, finding an object named in the instruction and rerouting around blocked aisles, where the deciding information is often visible only to the infrastructure cameras. Tested in distribution, InfraVLA reached a success rate of 100% on both, against 34.0% and 73.6% for a baseline without CCTV input. On out-of-distribution test sets it reached 88.2% and 88.9%. On a real quadruped fine-tuned with under 10 minutes of demonstrations, the policy reached 83.3% against 29.2% for the on-board-only baseline.
Read the full abstract ▾
All authors Lukas Vierling, Benjamin Ramtoula, Luke Robinson, Ronald Clark, Daniele De Martini
Robotics · Cross-listed from eess.SY
Hierarchical Multi-agent Reinforcement Learning for Warehouse Robot Coordination under Communication Loss
Weihao Sun, Gehui Xu and Andreas A. Malikopoulos
In this paper, we propose a hierarchical multi-
agent reinforcement learning framework for coordinating
robot teams in warehouse environments under communication loss. We partition the
robot team into groups, with centralized coordination within each group and distributed coordination across groups. Each group uses a recurrent predictor to estimate unavailable interaction information due to communication loss. A higher-level policy then generates a compact coordination reference that conditions the local control policy within each group. A predictive safety filter evaluates and modifies the proposed controls when they violate safety constraints. Simulation results show improved task completion under communication loss, reduced communication growth as the team size increases, and safe operation in the tested scenarios.
Read the full abstract ▾
Robotics
Beyond One-Step Accuracy: State-Affine Latent Transition for Reliable Visual Planning
Boyuan Zhang, Yingjun Du, Xiantong Zhen et al. (+1 more)
Joint-
embedding world models enable visual planning by learning action-conditioned dynamics in latent space. Yet they are commonly trained for one-step prediction on encoded states, while planning recursively applies the learned transition to its own predictions. One-step accuracy therefore does not capture how prediction errors propagate under recursive rollout. We decompose multi-step rollout error into the errors introduced at individual steps and their propagation through subsequent transitions. We show that state-affine dynamics are precisely the differentiable transitions with state-independent Jacobians, eliminating the nonlinear propagation residual and making the error propagation operators depend only on the action sequence. Guided by this result, we introduce SALT (State-Affine Latent Transition), an action-conditioned state-affine dynamics model in which the action modulates both the state transformation and the additive update. We train SALT through recursive multi-step rollout supervision, feeding each predicted latent state back into the transition so that training matches how the model is used during planning. Across four visual planning environments, SALT exhibits $1.48$–$2.19\times$ higher one-step prediction error than the matched LeWM baseline, yet improves closed-loop success in every environment by $10.0$ percentage points on average. On OGBench-Cube, the fraction of episodes that fail with a sharp rise in model-predicted cost after execution decreases from $23.3%$ to $2.0%$.
Read the full abstract ▾
All authors Boyuan Zhang, Yingjun Du, Xiantong Zhen, Ling Shao
Robotics
SLIP-VLA: Single-Step Latent Imagination for Policy Learning in Vision-Language-Action Models
Tianfu Li, Haoxuan Xu, Wenbo Chen et al. (+7 more)
Vision-
Language-Action models are increasingly effective for robotic manipulation, yet most predict actions directly from current observations without explicitly modeling future scene
evolution. Recent methods introduce future prediction to improve action generation, but dense future modeling often requires expensive iterative denoising, while one-step alternatives can underperform their multi-step counterparts. To reconcile efficient future modeling with strong action performance, we present SLIP-VLA, a policy learning framework that equips VLA models with a Single-Step Latent Imagination for future-aware action prediction. SLIP-VLA obtains temporally dense future latent representations with a single denoising update, and we improve the perceptual sufficiency of these representations by aligning intermediate latents with future
geometric and semantic features. We further improve their control sufficiency through action-conditioned latent world modeling and inverse dynamics modeling, explicitly coupling latent transitions with
robot actions. SLIP-VLA achieves state-of-the-art performance across diverse simulation
benchmarks and real-world manipulation tasks, while its single-step latent imagination takes only 12 ms.
Read the full abstract ▾
All authors Tianfu Li, Haoxuan Xu, Wenbo Chen, Haitian Li, Changchuan Yang, Xinhu Zheng, Jun Ma, Yuan Liu, Lujia Wang, Haoang Li
Robotics
FoLD: Force-Informed Learning for Dexterous Articulated Object Manipulation
Haowei Shen, Tingai Li, Yumeng Liu et al. (+6 more)
Transferring human demonstrations to dexterous
robots remains challenging because differences in hand morphology and contact dynamics often cause retargeted motions to fail at producing the intended object behavior. We present FoLD, a framework for learning dexterous manipulation of articulated objects through explicit force guidance. FoLD compute compensatory force fields from human demonstrations together with the
robot's current interaction state, yielding a force prior that promotes the demonstrated object motion. This force prior informs a residual policy that adapts retargeted hand motions to the contact requirements of the task. We evaluate FoLD on a public
benchmark for articulated object manipulation, where it consistently outperforms state-of-the-art baselines across tasks and embodiments. We further validate FoLD on real dexterous
robot platforms, demonstrating successful transfer of human manipulation skills to
robot execution. Here is the link of our project page: https://gghgghgghgg.github.io/FoLD-project-page/.
Read the full abstract ▾
All authors Haowei Shen, Tingai Li, Yumeng Liu, Wenyuan Guang, Xuanze Yang, Qing Fang, Kai Xu, Ligang Liu, Ruizhen Hu
Robotics
Steer2Grasp: Inference-Time Embodiment-Aware Steering for Diverse Physically Feasible Grasp Diffusion
Vignesh Vembar, Ayush Kaura, A Padmaprabhan et al. (+5 more)
Current grasp
diffusion models provide rich priors for generation, yet their object-centric approach can violate the kinematic and collision constraints imposed by the embodiment and the environment. Existing embodiment-aware methods primarily perform local corrections around generated grasps through gradient guidance or optimization, making it difficult to recover from fundamentally infeasible modes. We present Steer2Grasp, a training-free, embodiment-agnostic framework for inference-time grasp steering that adapts a frozen Cartesian grasp
diffusion model using deployment-specific rewards. Through Feynman-Kac (FK) inspired particle reweighting and resampling, the method reallocates population mass from infeasible to high-reward grasp modes, enabling population-level mode transitions without modifying the pretrained
diffusion model or requiring differentiable constraints. The framework enables a unified treatment for single and dual arm grasping through reachability and collision aware rewards, followed by gradient free gripper level local refinement. Across diverse objects,
robot embodiments, and constrained environments, our method substantially improves feasible grasp generation while maintaining proximity to the underlying grasp prior.
Read the full abstract ▾
All authors Vignesh Vembar, Ayush Kaura, A Padmaprabhan, Siddharth Sinha, Kailash Nagarajan, Keshab Patra, Md Faizal Karim, K Madhava Krishna
Robotics
A Disk-Shaped Magnetoelastic Torque Sensor for Robotic Joints Using Permanent Magnetization
Bruno Brajon, Enrico Gasparin, Nicole Yazigy et al. (+5 more)
Direct torque sensing is a growing need in the robotics community to enable precise control and interactions where torque estimation from motor current is not sufficient. This paper presents a novel disk-shaped magnetoelastic torque sensor with a compact axial envelope of about 1 cm, suitable for integration in robotic joints. A four-magnetometer architecture is used to measure the field modulated by the stress affecting a narrow magnetized region while rejecting the effects of parasitic cross forces. A custom-designed
magnetic shield enhances the torque sensitivity while reducing the external stray fields by a factor 7x. The device measures the torque with an accuracy of 1.34 %FS relative to the 50 Nm full scale (FS). The paper details the development of the sensor through the mechanical design, the magnetization procedure, and the experimental validation. The results demonstrate the potential of the proposed sensor for robotic applications.
Read the full abstract ▾
All authors Bruno Brajon, Enrico Gasparin, Nicole Yazigy, Lisa Salamin, Florian Copt, Guzmán Borque Gallego, Elias Klauser, Gaël Close
Robotics
AI-Driven Collaborative Assembly Line Inspection: System Integration and Deployment Challenges
Asya Ünal, Amr Okasha, Ege Ç{\i}rakman et al. (+1 more)
Manual visual inspection on assembly lines is a persistent manufacturing bottleneck: operator fatigue over extended shifts lowers defect-detection rates. This paper presents the design, integration, and field deployment of an AI-assisted collaborative inspection
cell at the Silverline kitchen-appliance factory, developed within the AI-PRISM project. The
cell couples a Universal
Robots UR 10e cobot carrying a machine-
vision defect-detection pipeline with a Comau Racer-5 cobot for functional tests, coordinated through ROS 2 Humble on an Ubuntu 22.04 LTS server. Multi-modal data (Basler camera imagery, TIA microphone acoustics, and SPS electrical-safety measurements) are logged locally and visualised in real time with Grafana. We report the practical deployment challenges (close-proximity safety, AI robustness under glare and reflections, ROS 2 namespace collisions across two cobots, and operating-system and dependency issues) together with the engineering solutions adopted, and structure the integration through a four-level Human-
Robot Interaction analysis. The deployed
cell cuts per-unit quality-check time from 82 s to 61 s (about 25%), raises final-control resource efficiency from 0.75 to 0.88, reduces operator visual-inspection viewing time by 82%, and significantly lowers operator mental demand (p = 0.005, NASA-TLX).
Read the full abstract ▾
All authors Asya Ünal, Amr Okasha, Ege Ç{\i}rakman, Perin Ünal
Robotics
AMBIT: Anticipatory Multimodal Body Recruitment for Bimanual Tracking on a Humanoid
Hanlong Li, Sihan Tan, Takeshi Ashizawa et al. (+2 more)
A humanoid with 5-DoF arms cannot track generic bimanual end-effector trajectories with its arms alone; pelvis and waist motion must be recruited, but which motion, and when, is not uniquely determined. On a Unitree R1 in fixed double support, the set of dynamically valid recruitment strategies (pelvis pose and waist trajectories) for a task is a diverse continuous
manifold, and a deterministic regressor trained on it mode-averages into strategies valid only 35% of the time, against 52% for a conditional variational autoencoder (CVAE) and 82% for the best of 16 CVAE samples. We introduce AMBIT: the CVAE proposes strategies from a preview of the commanded trajectory, a non-learned selector filters, ranks and verifies them, and a receding-horizon loop commits to one with hysteresis. The committed strategy is the reference of the same whole-body differential-IK QP a reactive tracker runs, which keeps authority over residual error. On 160 held-out episodes that admit a valid strategy, in full MuJoCo dynamics under a torque controller, AMBIT reaches 85% success at a 3 cm/15 deg tolerance against 74% for the tracker (disjoint confidence intervals) and recruits the body before the arms saturate in 48% of episodes against 35%. Because diversity is preserved, constraints unknown at training time are enforced by selection alone: under five zero-shot shifts AMBIT beats the warm-started tracker on every shift and matches a test-time re-optimisation baseline 17x more expensive. On a Unitree G1, with hyperparameters unchanged, the protocol reproduces the structure of the valid set and widens the gap over the tracker to 0.85 against 0.53. Five selected strategies execute on the externally supported physical R1, distinct in pelvis excursion and tracking the planned end-effector motion to a median of 11 mm by encoder forward kinematics, which establishes kinematic realisability, not balance.
Read the full abstract ▾
All authors Hanlong Li, Sihan Tan, Takeshi Ashizawa, Benjamin Yen, Kazuhiro Nakadai
Robotics
VIDEAS: Distilling Explicit Action Semantics from Demonstration Videos for World Models via Prior-Guided Simulation
Jianan Wang, Haoquan Zhai, Siyang Zhang et al. (+7 more)
World models learn internal representations of environment dynamics to predict future states, enabling
agents to optimize action plans without physical interactions. However, developing world models that genuinely internalize underlying causal physical laws to explicitly reason about action preconditions and subsequent state transitions remains an open challenge. In this paper, we propose VIDEAS, a data
distillation framework that transforms continuous physical dynamics from operational
videos into explicit action semantics for foundation models. Specifically, it deconstructs visual demonstrations into discrete action trajectories and utilizes advanced
vision-
language models (VLMs) to extract structured knowledge encapsulating action preconditions and effects. To ensure physical consistency, we introduce a prior-guided trajectory simulation mechanism grounded within a text-based environment to rigorously validate the extracted knowledge. Notably, we incorporate negative trajectories to enrich knowledge completeness and enhance data diversity to mitigate cognitive bias. Furthermore, we present VIDEAS-WM, an 8B/9B-parameter suite of
language-based world models trained on 34K high-quality samples derived from AgiBot-World
dataset. Extensive experiments demonstrate that VIDEAS-WM establishes state-of-the-art performance in high-level embodied action semantic
reasoning, exhibiting profound physical understanding and robust generalization across unseen scenarios.
Read the full abstract ▾
All authors Jianan Wang, Haoquan Zhai, Siyang Zhang, Bin Li, Juan Chen, Jingtao Qi, Zhuo Zhang, Enze Wang, Haoxiang Jin, Chen Qian
Robotics · Cross-listed from cs.CV
TC-ADA: One-Shot Active Domain Adaptation for Semantic Segmentation
Weihao Yan, Yeqiang Qian, Yueyuan Li et al. (+3 more)
Manual dense annotation remains a major obstacle to deploying semantic segmentation models in new driving environments. Active domain adaptation (ADA) seeks label-efficient transfer by annotating only a selected portion of the target domain. Existing ADA methods commonly implement this process through multiple rounds of acquisition, annotation, and retraining. We study a practical one-shot image-level setting that selects and densely annotates a fixed target subset in a single round, followed by uninterrupted adaptation. Within this setting, we develop Target-Calibrated Active Domain Adaptation (TC-ADA) as a joint design of complete-image acquisition and target-calibrated adaptation. Stage~1 uses visual representations from a
vision foundation model (VFM) together with semantic predictions from a fixed unsupervised domain adaptation model to select representative and informative target images without target annotations. Stage~2 jointly uses labeled source data, labeled target data, and the remaining unlabeled target data, while calibrating source and target supervision under limited target labels. Extensive experiments across five synthetic-to-real and real-to-real driving transfers show consistent improvements over representative ADA baselines. With only 23 to 46 labeled target images on four transfers and 140 on Mapillary, TC-ADA stays within 1.9 mean intersection over union (mIoU) points of target-only full supervision. Code will be available at https://github.com/ywher/TC-ADA.
Read the full abstract ▾
All authors Weihao Yan, Yeqiang Qian, Yueyuan Li, Tao Li, Chunxiang Wang, Ming Yang
Robotics
Large Language Models for Model-Based Robot Design
Andrew Wilhelm, Angelina Zhou and Nils Napp
Large
Language Models (LLMs) can contribute useful engineering knowledge to
robot design, but directly generated designs may rely on implicit assumptions and provide no guarantees of feasibility or optimality. These assumptions are critical because different reasonable modeling choices can materially change which designs are predicted to be feasible or optimal. We therefore present a framework that uses LLMs to construct explicit engineering models containing physical relationships, compatibility constraints, and objectives, allowing these modeling choices to be inspected and revised before formal optimization. The model can then be updated with additional engineering, manufacturer, or system-specific information before formal multi-objective optimization provides feasibility and Pareto-optimality guarantees with respect to the finalized model and specified design space. We evaluate the framework on quadcopter and line-following
robot component-selection problems. Across 30 direct LLM design trials, none could be verified as feasible under the corresponding finalized model. Comparisons with an independently developed expert model and successive stages of model refinement further showed that changes in modeling assumptions substantially altered the predicted feasible and Pareto-optimal design sets. Together, these results show that using LLMs to construct explicit engineering models makes the underlying design choices available for inspection and revision before those assumptions determine the optimized designs. Explicit modeling therefore provides an interface for combining LLM-generated engineering knowledge, system-specific information, and formal design optimization.
Read the full abstract ▾
Robotics
Seungyeon Kim, Junhoo Lee, Minkyu Kim et al. (+2 more)
A central goal in robotics is to enable manipulation across changing tasks and environments.
Vision-
language-action (VLA) models provide broad manipulation capabilities but can struggle when execution requires diagnosing failures and adapting behavior. Strong
agents can discover effective interventions through interaction with these policies. We propose Recursive Harness
Distillation to accumulate this experience as reusable guidance across
agents. A strong
agent distills its experience into a playbook for a light
agent, then recursively refines the playbook using the light
agent's execution feedback. The resulting playbook enables
agents to reuse accumulated intervention knowledge in new task instances without updating model parameters. In real-world manipulation, the harness improves success from 37.3% to 64.0%. On SimplerEnv Bridge, the light
agent with the playbook achieves 66.7% success, compared with 41.7% for the GR00T-only baseline, and outperforms the strong
agent without a playbook. The same playbook also benefits the strong
agent, which reaches 79.2% success. These results demonstrate the feasibility of harness
distillation for robotics: intervention experience can be accumulated, refined through execution, and reused across
agents to improve manipulation.
Read the full abstract ▾
All authors Seungyeon Kim, Junhoo Lee, Minkyu Kim, Baekseung Kim, Nojun Kwak
Robotics
Traceable Human-to-Humanoid Sign Language Benchmarking
Ao Liu, Shengeng Tang, Lechao Cheng et al. (+3 more)
Sign data collection is costly, and teleoperation scales poorly, motivating reuse of large
video corpora. Humanoid signing requires converting
video-derived human motion into
robot trajectories while preserving linguistic motion cues. Errors from fitting, human-motion repair, retargeting,
robot geometry repair, and control are hard to separate from the final trajectory alone. We introduce HumanoidCSL-20K, a
dataset and
benchmark of 20,648 sentence-level Chinese Sign
Language sequences, each with four aligned versions: the source, the repaired human motion, the direct
robot reference, and the geometry-repaired
robot reference. Observation-supported local human-motion repair, full-
robot geometry repair, and cross-representation provenance make each transformation traceable. Paired evaluations measure human-motion continuity and content preservation,
robot-reference feasibility, and physical execution. A sign-specific kinematic-reference protocol scores handshape, location, palm orientation, and inter-hand relation over the full planned motion. Full-corpus results show fewer abnormal arm / hand steps and less inter-hand and hand-body penetration after repair. Control experiments separate reference learnability from curriculum effects, while component scores expose remaining execution errors.
Read the full abstract ▾
All authors Ao Liu, Shengeng Tang, Lechao Cheng, Yanbin Hao, Bingkun Bao, Richang Hong
Robotics · Cross-listed from cs.LG
Safe Score Matching: Diffusion Policies with Hamilton-Jacobi Reachability for Online Safe Reinforcement Learning
Boyang Li, Matthew Kim and Sylvia Lee Herbert
Online safe reinforcement learning (RL) seeks policies that maximize reward while satisfying safety constraints. A popular line of research in safe RL relaxes safety to a soft expected-cost constraint and solves the resulting Constrained Markov Decision Process via primal-dual Lagrangian updates that only enforce safety on average. To address this limitation, hard, state-wise constraints are introduced and often imposed through Hamilton-Jacobi (HJ) reachability. Yet such constraints require solving different objectives in the feasible and infeasible regions: reward maximization in the former, recovery toward the feasible regions in the latter. The resulting target action distributions are inherently
multimodal, and this structure poses a fundamental challenge for the Gaussian or deterministic actors used in existing HJ-based safe RL, which often collapse onto suboptimal modes.
Diffusion policies provide the expressiveness needed to represent such distributions, and recent work on Q-score matching offers a route to training them for online RL by score regression – but has been applied only to reward maximization. We propose Safe Score Matching (SSM), an off-policy actor-critic method that adapts Q-score matching to hard-constrained safe RL by gating a two-branch score target with HJ reachability: inside the feasible set, the denoising process degenerates to Q-score matching on actions classified as viable by the HJ critic; outside, a recovery branch biases denoising toward regions with lower worst-case violation. On quadrotor and fixed-wing trajectory-tracking and stabilize-and-avoid
benchmarks, SSM attains the best or near-best task performance with low false-safe rates, whereas the primal-dual baseline admits more unsafe behavior and reachability-based baselines tend to be more conservative; on Safety-Gymnasium velocity tasks, SSM attains the lowest cost with competitive reward.
Read the full abstract ▾
Robotics
SocialHumanoid: Towards Expressive Humanoid Behavior via One-Step Co-Speech Motion Generation
Chengqun Yang, Tengjie Zhu, Liang Xu et al. (+10 more)
Humanoid
robots are increasingly expected to serve as embodied social
agents that communicate naturally with humans through face-to-face interaction. During such communication, humanoid
robots require body behaviors that are synchronized with
speech, affectively expressive, and suitable for real-time execution. However, existing co-
speech methods are primarily developed for digital humans and lack joint support for affective control and low-latency continuous generation on physical embodiments. To bridge this gap, we present SocialHumanoid, a system for expressive humanoid behavior via one-step co-
speech motion generation. Given response
speech and a specified affective condition, SocialHumanoid generates each full-body motion window in a single forward pass and connects successive windows through motion-history conditioning. The generated human motion is further converted online into embodiment-compatible
robot references and tracked by a whole-body controller for physical execution. To provide explicit supervision for affective body expression, we further introduce AffectMoCap, a 4-hour
dataset captured from two professional actors, containing synchronized
speech, body motion, fine-grained hand motion, and emotion annotations. On BEAT2, SocialHumanoid achieves the best FGD among the compared generation methods, competitive
speech-motion synchrony, and approximately $6\times$ faster inference than GestureLSM under the same protocol. Perceptual evaluations further show that training with AffectMoCap improves affect recognition from generated body motion, while real-
robot experiments demonstrate continuous affect-conditioned behavior and stable long-horizon execution. Our project page is https://rex0191.github.io/SocialHumanoid/.
Read the full abstract ▾
All authors Chengqun Yang, Tengjie Zhu, Liang Xu, Fulong Liu, Guanzhu Ren, Yitong Xing, Xuefeng Lu, Fei Shi, Siyuan Fan, Weijie Dong, Yao Mu, Xiaokang Yang, Yichao Yan
Robotics
CompliantWBC: Whole-Body Compliance for Heavy Humanoids via Force Latent Estimation and Residual Impedance Targets
Tan-Dzung Do, Cuc T.Trinh, Tuan Dat Phuong et al. (+4 more)
Whole-body compliant control is essential for deploying heavy humanoids under high payload in human-centric environments. Most prior force-aware learning-based pipelines focus on end-effector resistance, per-link upper-body springs, or end-effector stiffness modulation, leaving arbitrary-site perturbations on heavy platforms with lower-body engagement largely unaddressed. We close this gap with CompliantWBC comprising: (1) A base policy trained with RL to maximize compliance-fidelity reward, guided by a multi-site whole-body impedance reference controller, extending classical Cartesian impedance to any controlled link; (2) A bounded residual policy that edits the per-link impedance
equilibrium over a frozen base, correcting the coarse but structured wrench estimate supplied by a force encoder co-trained behind a gradient barrier; (3) A Phong-weighted force-origin sampler with an axis-decoupled pelvis anchor induces lower-body-inclusive compliance curriculum training via two interpretable parameters. We evaluate CompliantWBC in simulation against both compliant and stiff baselines, achieving best compliant fidelity of 2.58cm deviation from analytical solutions, and demonstrate it on a real heavy humanoid across static/dynamic force reaction, board wiping, squat under payload, and cooperative payload transport. Project website: https://dotandung.github.io/compliantwbc/
Read the full abstract ▾
All authors Tan-Dzung Do, Cuc T.Trinh, Tuan Dat Phuong, Chien Le, Thanh Ly, Vien Anh Ngo, An Thai Le
Robotics
AquaWAM: A Dynamics-aware World Action Model for Underwater Embodied Agents
Cunhao Zhu, Yifeng Wang, Dongliang Xu et al. (+3 more)
World Action Models (WAMs) are becoming increasingly important and useful for embodied intelligence, as they enable
robots to anticipate the consequences of candidate actions before interacting with the physical environment. However, underwater
robots are usually subject to passive dynamics, such as inertia, buoyancy, hydrodynamic drag, and persistent drift, which can continue to affect the vehicle even after an action is completed. Existing WAMs, which primarily predict action-conditioned visual observations, are not explicitly designed to capture such passive motion dynamics. In this paper, we present AquaWAM, the first World Action Model designed for underwater embodied
agents. Instead of predicting future images, AquaWAM models both action-conditioned and passive physical dynamics, including the thruster dead band, the inertial glide that outlasts each command, and ambient currents. Specifically, it senses through the DVL, IMU, pressure sensor and joint encoders, while cameras supply only semantics for understanding goals and target pose. By modeling compact navigation states rather than high-dimensional visual observations, AquaWAM substantially reduces the model size and computational cost compared with conventional WAMs. Experimentally, AquaWAM achieves a 72.6% task success rate across 20 underwater tasks on the USIM
benchmark, outperforming existing methods while making action decisions 2.7x faster than U0 on an NVIDIA Jetson AGX Orin. Our model also remains effective when some onboard sensor measurements are unavailable. For example, without DVL velocity measurements, our method still achieves a 61.6% success rate, compared with 39.4% for U0.
Read the full abstract ▾
All authors Cunhao Zhu, Yifeng Wang, Dongliang Xu, Yunzhong Hou, Yue Yao, Chi Harold Liu
Robotics · Cross-listed from cs.CV
Informative Viewpoint Selection for Episodic-Memory Embodied Question Answering using Omnidirectional Images
Kaname Kitamura and Asako Kanezaki
Embodied Question Answering (EQA) requires
agents to answer natural
language questions about surrounding environments from visual observations. In this work, we focus on open-vocabulary episodic-
memory EQA (EM-EQA), where an
agent answers free-form questions using recorded observation histories. Omnidirectional images are promising for this task, as they provide wide field-of-view observations that can capture surrounding context without requiring explicit camera rotations. However, omnidirectional images introduce two challenges for EQA: (i) equirectangular projection causes severe
geometric distortion that degrades
vision-
language model (VLM) recognition accuracy, and (ii) feeding equirectangular images directly into VLMs introduces excessive irrelevant background information, reducing answer accuracy and increasing the visual-
token burden. To address these challenges, we propose a viewpoint selection method for EM-EQA using omnidirectional images. Our method converts equirectangular observations into perspective views via cubemap projection, estimates question-conditioned relevance with fine-tuned BLIP-2, and selects informative and diverse viewpoints through diversity-aware greedy selection. Experiments on the Habitat-Matterport 3D (HM3D) subset of OpenEQA show that our method achieves state-of-the-art model performance among the reported model results with equirectangular observations. Moreover, after removing rotation views, which reduces observation frames by 65.5%, our method largely maintains its answer accuracy.
Read the full abstract ▾
Robotics · Cross-listed from cs.AI
Feedback Makes Perfect: A Closed-Loop Framework for NL-to-STL Translation
Bowen Ye and Xiang Yin
Signal Temporal Logic (STL) enables rigorous verification and control of cyber-physical systems, but writing correct specifications requires expertise that most requirement holders lack. Large
language models can translate natural-
language (NL) requirements into STL, yet stronger translators alone approach an accuracy ceiling. We argue that this ceiling stems from how the task is posed: one-shot, open-loop translation is somewhat ill-defined. Natural
language is ambiguous, and, more fundamentally, what a person writes may not always be what they intend, so the target specification is not fully contained in the input text. We therefore reformulate NL-to-STL translation as a closed-loop feedback process. Each generated formula is translated back into natural
language for the user to check, and natural-
language corrections drive revision until the user accepts the specification. Users never read or write formal syntax. This framework rests on an asymmetry familiar from feedback control theory. The forward path, from ambiguous
language to formal logic, is hard and error-prone. The feedback path, from structured STL back to
language, can be made highly precise, and a precise feedback path lets an imprecise forward path achieve precise closed-loop behavior. Experiments on 500 expert-authored requirements and seven LLMs support this view. Back-translated explanations agree with expert judgments in 99.5% of cases. Closed-loop refinement raises strong models from about 89% open-loop accuracy to 98.0–99.2%, and yields gains of over 30 percentage points for weaker models (e.g., 17.6%$\rightarrow$48.0%). Ablations show these gains come from the semantic content of the feedback rather than from repeated attempts. An expert audit and a 280-session user study further confirm the reliability of the loop. We also identify a capability threshold above which feedback no longer helps.
Read the full abstract ▾
Robotics
Q-WAM: 4-Bit Quantization of World Action Models with Action-Subspace Protection
Arash Akbari, Arman Akbari, Jingwu Luo et al. (+7 more)
World Action Models (WAMs) jointly generate
video and
robot actions through iterative
diffusion and perform strongly in robotic manipulation. However, their prohibitive compute and
memory costs pose substantial deployment challenges. Post-training quantization (PTQ) can reduce these costs, but existing PTQ methods such as smoothing and rotation are insufficient to maintain the precision of action generation. To overcome this limitation, we propose Q-WAM, a new 4-bit weight-activation quantization for WAMs that preserves the actions the model generates. Specifically, we introduce the Action Observability Gramian (AOG), which measures how much rounding errors in each weighted combination of a layer's input channels change the final action through all denoising steps. We also develop Action-Subspace Protection (ASP), which keeps the few most action-sensitive channel combinations in a tiny 16-bit low-rank branch and quantizes the complementary weights and activations to 4 bits, both as dense
matrix multiplications that run efficiently on GPUs. Finally, to preserve action quality with minimal overhead, we identify the experts that matter most for the generated action by aggregating the AOG-derived action mass across the layers of each expert and apply ASP only to those experts. We evaluate Q-WAM on three WAMs, both in simulation and in real-world deployment. On the RoboTwin 2.0
benchmark, it reaches 89.6–93.0% average success rate, within 1.1 percentage points of the 16-bit models, while reducing the
memory of the quantized blocks by 3.1–3.4$\times$. Our method outperforms the strongest baseline, SVDQuant, by 2.5–8.7 percentage points. On a Unitree G1 humanoid and a bimanual UR3
robot, it improves success over SVDQuant by 12.8-17.6 percentage points.
Read the full abstract ▾
All authors Arash Akbari, Arman Akbari, Jingwu Luo, Yuhao Lei, Yi Gao, Weiwei Chen, Xuan Zhang, Zhenman Fang, Geng Yuan, Yanzhi Wang
Robotics · Cross-listed from cs.CV
VehDyn: A Driving World Model Benchmark for Vehicle Dynamics
Tianyi Wang, Wangsheng Du, Jiazhou Chen et al. (+12 more)
Video world models are emerging as data engines, action planners, and generative simulators for autonomous driving, but existing
benchmarks primarily assess visual fidelity and coarse physical plausibility, providing limited evidence on whether generated driving futures obey realistic vehicle kinematics and dynamics. This limitation is further compounded by the lack of
datasets in which vehicle, road, maneuver, and speed conditions are independently controlled, and ground-truth vehicle states are recorded in synchrony with
videos. We introduce VehDyn, a driving world model
benchmark for vehicle dynamics. VehDyn is built on a CARLA-CarSim co-simulation platform where photorealistic rendering is coupled with a validated multi-body dynamics model, and it contains 10,080 configurations from a full factorial design over five vehicle types, four tire-road friction coefficients, three maneuvers, four target speeds, 14 scenes, and three illuminations, each paired with synchronized position, velocity, and attitude sequences. Built on this
dataset, VehDyn introduces a hierarchical evaluation framework that measures trajectory
alignment, kinematic consistency, and dynamic consistency, and
benchmarks 12 state-of-the-art
video world models. We further assess the
video quality using two established protocols and correlate it with the VehDyn score. Trajectory-level metrics are nearly saturated, with ten of twelve models within 20% of ground truth, while no model reaches 92% of ground truth on dynamic consistency, and visual-quality metrics are only weakly correlated with vehicle-dynamics fidelity. DrivingWorld achieves the highest VehDyn score, followed by Cosmos 3 Nano and LTX-
Video 2.5, and the VehDyn score agrees closely with human judgment. VehDyn provides a systematic foundation for developing driving world models that are physically consistent and visually realistic.
Read the full abstract ▾
All authors Tianyi Wang, Wangsheng Du, Jiazhou Chen, Tianyi Zeng, Xiangyu Li, Jiseop Byeon, Yujin Wang, Yiming Xu, Yangyang Wang, Bingzhao Gao, Sikai Chen, Zhaomiao Guo, Junfeng Jiao, Christian Claudel, Alexandre Bayen
Robotics
PORTER: Edge-Cloud Residency for Persistent 3D Scene Graph Memory
Yue Chang, Yifan Tian, Jiajing Peng et al. (+5 more)
Recent task-driven and just-in-time 3D Scene Graph (3DSG) methods reduce per-task representations by constructing or activating only task-relevant information. Yet sparse per-task working sets do not bound onboard
memory usage over a
robot's lifetime: as tasks change, payloads accumulated for earlier tasks may become irrelevant to the current task but can be useful again in future tasks. Over repeated task switches and expanding environments, retaining such reusable payloads causes local
memory to grow, whereas discarding them entirely can lead to costly repeated construction of the same payloads later. We introduce PORTER, which decouples persistence from residency: lightweight anchors remain in the limited
memory of the edge
robot while heavy object payloads migrate between the edge and the cloud. Relevance alone is insufficient for deciding residency because multiple relevant payloads may provide redundant information. We therefore decompose each task into functional requirements and introduce Irreplaceable Support Erasure (ISE), which measures the loss in requirement coverage caused by offloading. ISE discounts replaceable support and penalizes losses more strongly when the remaining coverage of a requirement is weak. PORTER constructs a budget-aware local working set by repeatedly offloading the payload with the smallest marginal ISE per byte. Experiments on JITOMA-Bench evaluate PORTER across four 3DSG builders. Under progressive compression, pooled relative mR@3 remains at 100% through 91% payload-byte offloading.
Read the full abstract ▾
All authors Yue Chang, Yifan Tian, Jiajing Peng, Dazhi Huang, Rufeng Chen, Zhaofan Zhang, Li Chen, Sihong Xie
Robotics
ActionGround: Training-Free Runtime Refinement of Frozen VLA Policies
Namai Chandra, Madhur Thareja, Shriram Damodaran et al. (+1 more)
Vision-
Language-Action (VLA) models map visual observations and
language instructions directly to
robot actions, but they do not explicitly represent the phase structure of manipulation tasks or the rigid-body dynamics governing execution. We present ActionGround, a neuro-symbolic, training-free runtime layer that wraps a frozen VLA policy without retraining, fine-tuning, or weight access, adding less than 1 ms of overhead per control step. A symbolic phase-aware finite-state machine identifies the manipulation phase (approach, grasp, transport, or place) and applies a phase-specific rule-based correction. In parallel, an always-on, inertia-weighted Euler-Lagrange term incorporates the
robot's equations of motion into each control step, while its dynamics residual is logged as a consistency diagnostic rather than used as a gate. We evaluate ActionGround across OpenVLA, OpenVLA-OFT, Force-VLA, and Generalist-VLA on ten LIBERO-Spatial pick-and-place tasks using a 7-DoF Franka Panda. With fixed parameters across tasks and backbones, ActionGround improves success rate by up to 6 percentage points and stability by up to 19.3 percentage points, while improving trajectory efficiency by up to 15%. In a separate Robosuite noise sweep, ActionGround provides approximately a 10x improvement in trajectory-jerk robustness under injected action noise. In a matched-seed Robosuite simulation companion to a real Agilex Piper trial, simulated baseline success increases from 35% to 95%. The physical-hardware experiment is presented as a qualitative deployment demonstration; quantitative per-trial success on the real arm is left for future work. Our evaluation is limited to rigid-object pick-and-place manipulation.
Read the full abstract ▾
All authors Namai Chandra, Madhur Thareja, Shriram Damodaran, Addison Lin Wang
Robotics
SurgFlow: 3D Object-Centric Contact Flow for Surgical Robot Manipulation
Changwei Chen, Xiao Liang, Yinuo Yang et al. (+6 more)
Paired
video-action demonstrations enable autonomous surgical behavior, but such data is scarce:
robots perform roughly 1% of surgeries, while
video-only data is abundant. Learning 3D object flow offers an embodiment-agnostic way to utilize
video data, but flow alone specifies how an object should move, not where and when the tool should engage it, a distinction that is critical in surgery. We introduce SurgFlow, a framework that learns 3D Object-Centric Contact Flow from stereo surgical
video without action labels. For each object point, it predicts a future 3D trajectory and contact scores. We extract targets via 3D tracking and tool-object proximity, train a flow matching generator to predict them, and use predicted contact to trigger grasp and release while optimizing end effector motion from flow. On the da Vinci Research Kit (dVRK), SurgFlow succeeds in 37 of 39 stage evaluations across tissue retraction, bimanual reveal, needle pickup, and handover, outperforming baselines trained on equal data with or without action labels. Zero-shot transfer to a humanoid-based laparoscopic
robot achieves 85% and 70% average success under similar and novel camera viewpoints, respectively.
Read the full abstract ▾
All authors Changwei Chen, Xiao Liang, Yinuo Yang, Nicole Shen, Peihan Zhang, Sara Wickenhiser, Zekai Liang, Soofiyan Atar, Michael Yip
Robotics
Learning with Object-centric Representations of Tactile Interactive Perception for Robot Manipulation
Xinyi Yang, Zilin Si, Zhuowei Xu et al. (+2 more)
Implicit object properties that are difficult to directly infer from
vision, such as material, container contents, or softness, can be revealed through tactile sensing and exploratory interactions. However, because tactile signals are transient and sparse, extracting informative tactile events and effectively incorporating them into robotic manipulation remains a challenge. In this work, we present an object-centric context-aware manipulation framework that learns task-agnostic object representations through tactile exploration. A
token learner autonomously selects representative tactile segments from long-horizon exploration, while contrastive
alignment with descriptive text
embeddings enables a latent space that captures multiple physical object properties. These learned representations are then used as semantic context to guide object-centric manipulation policies and adapt strategies based on object properties. Experiments show that the learned representations achieve 93% and 84% property estimation accuracy on seen and unseen objects. Evaluated on three tasks involving visually ambiguous objects, i.e. multi-object rearrangement, pouring, and box opening, the proposed framework improves both target selection and property-dependent manipulation adaptation, raising task success, aggregated over all evaluation trials, from 41% to 92% on seen objects and from 19% to 67% on unseen objects over baseline policies without object-context conditioning.
Videos and additional results are available at https://xinyiyxyx.github.io/tactile-object-centric/.
Read the full abstract ▾
All authors Xinyi Yang, Zilin Si, Zhuowei Xu, Zeynep Temel, Oliver Kroemer