Robotics · Cross-listed from cs.CV
AevaScenes: An FMCW LiDAR Dataset and Benchmark for Long-Range Perception
Gautham Narayan Narasimhan, Heethesh Vhavle, Kumar Bhargav Viswanatha et al. (+2 more)
FMCW
LiDAR measures per-point radial Doppler velocity alongside range, providing a motion cue unavailable in conventional time-of-flight sensors. Exploiting this signal at long range remains understudied. We present an FMCW
LiDAR dataset of 575 sequences (57.5K frames) with over 8 million annotated 3D boxes across 16 detection classes and per-point labels across 24 semantic classes, captured by six commercial FMCW
LiDAR sensors and six paired 4K cameras across eight Bay Area cities, including 237 nighttime sequences, with annotations extending to 400m. We define a
benchmark with three tasks: 3D object detection, scene flow estimation, and semantic segmentation. Detection and scene flow are evaluated across three range bins to 400m, with a public evaluation server. We explore the impact of Doppler measurements on flagship recognition tasks, and find significant improvements up to 2X in detection AP of far-away vehicles and pedestrians, particularly in low-latency single-frame settings. We similarly find scene flow accuracy is significantly improved with Doppler measurements across all ranges. Our
dataset and
benchmark have been publicly released at https://scenes.aeva.com.
Read the full abstract ▾
All authors Gautham Narayan Narasimhan, Heethesh Vhavle, Kumar Bhargav Viswanatha, James Reuther, Deva Ramanan
Robotics · Cross-listed from cs.CV
Scope-WM: Scoped Computation for Efficient Visual World Models
Chunzheng Li, Zesheng Jia, Hongda Zhang et al. (+4 more)
Visual world models enable robotic planning by predicting future observations, but dense latent-state propagation and sample-intensive trajectory optimization incur high inference latency and peak
memory usage, limiting real-time deployment on resource-constrained platforms. Existing sparse world-model acceleration methods either rely on unguided
token sparsification, which may discard planning-relevant information and restrict achievable sparsity, or introduce heavy auxiliary modules and cumbersome multi-stage training pipelines. In this work, we present Scope-WM, an efficient visual world model that scopes computation to prediction-relevant latent regions and promising action sequences. Scope-WM distills prediction relevance into a lightweight action-conditioned selector and applies full dynamics prediction only to a compact subset of selected
tokens. It updates the remaining
tokens using a compact summary of foreground states and their changes, allowing the background to perceive foreground dynamics without costly
token-to-
token interactions. During planning, Scope-WM preserves and reuses high-quality action sequences discovered during the initial MPC search, focusing subsequent search under reduced rollout budgets. The resulting pipeline requires only a one-off selector
distillation followed by a single joint training stage for the sparse world model. On the challenging Push-T task, Scope-WM reduces peak GPU
memory usage and planning time to $18.1\%$ and $14.3\%$ of those of dense DINO-WM, respectively, corresponding to a $6.97\times$ planning speedup, while maintaining competitive task performance. Further evaluations across five diverse visual planning tasks demonstrate the general applicability of Scope-WM. Code is available at https://github.com/ChunZheng2022/Scope-WM.
Read the full abstract ▾
All authors Chunzheng Li, Zesheng Jia, Hongda Zhang, Jiaying Tang, Yuntian Wang, Siao Liu, Jin Wang
Robotics
TAO-DA: Towards Autonomous Operation–A Dual-Arm Vision-Language-Action Model for Coordinated Manipulation
Yongsheng Zhao, Han Gao, Baoping Cheng et al. (+7 more)
Vision-
Language-Action (VLA) models provide a unified framework for grounding high-level semantic information into low-level
robot actions, enabling scalable robotic manipulation across diverse tasks. However, existing VLA models lack explicit mechanisms to disentangle the states and intents of the two arms, leading to unintended cross-arm interference that degrades task execution success. To address this issue, we propose a symmetric Dual-Arm Expert (DAE) architecture built upon a shared
Vision-
Language Model (VLM) backbone with decoupled, arm-specific expert towers. Expert selection is carried out through a two-stage dual-arm intent routing scheme, in which experts are routed either by explicit
language instructions in the first stage or by implicit visual semantics in the second stage. Moreover, we introduce a lightweight task progress prediction module that leverages cross-
attention between the pre-chunk temporal features and semantic representations of proprioceptive and visual observations to accurately estimate frame-wise task completion progress. This module facilitates task progress synchronization to support coordinated scheduling for collaborative multi-
robot tasks. Experimental results demonstrate the effectiveness of our model in dual-arm intent routing and the disentanglement of cross-arm interference, and further provide preliminary evidence of emergent skill generalization from single- to dual-arm tasks (as well as the reverse), together with cross-arm motion-domain skill transfer.
Read the full abstract ▾
All authors Yongsheng Zhao, Han Gao, Baoping Cheng, Jingyao Tang, Dian Zhou, Deng Liang, Ji Ge, Xuanzhang Wen, Lei Zhao, Ye Wang
Robotics · Cross-listed from cs.CV
FocusDrive: Reasoning with Visual Focus for Autonomous Driving
Zhiyuan Liu, Zehong Ke, Yuanxin Tian et al. (+7 more)
Driving decisions depend on both where to focus and how to act on what is seen. Effective driving
reasoning must establish which objects matter, where they are, and how they inform the intended action. Text-based rationales can describe a driving response while leaving its correspondence to specific visual evidence implicit. Visual focus provides a concrete starting point for this connection by identifying what matters in the scene and where it is. We propose FocusDrive, a structured
multimodal reasoning framework that organizes end-to-end planning around explicit visual focus. It pairs descriptions of decision-relevant objects with image-patch references, bringing explicit visual focus into the
reasoning that generates driving plans and trajectories. We first assess this focus representation through driver gaze prediction, then investigate its role in planning
reasoning using driving-focus annotations within existing NAVSIM training scenes. Experiments on W3DA and NAVSIM demonstrate competitive gaze prediction and end-to-end planning performance, with FocusDrive improving over text-based chain-of-thought. These results support visual focus as an effective link between scene understanding and driving action.
Read the full abstract ▾
All authors Zhiyuan Liu, Zehong Ke, Yuanxin Tian, Hao Cheng, Jinhao Li, Yining Xing, Yanbo Jiang, Zhenhua Xu, Wenhao Yu, Jianqiang Wang
Robotics · Cross-listed from cs.LG
When an Evaluation Rule Writes Training Labels: Measuring Human-Reference Forgiveness in NAVSIM
Jiaxuan Guo, Jingxin Yang, Jiaqi Ye et al. (+4 more)
When the human reference scores zero on a metric, the released GTRS-Dense label generator for NAVSIM marks every candidate trajectory in the scene as passing it. NAVSIM's authors introduced this human-reference forgiveness to avoid penalizing contextually justified maneuvers when scoring one trajectory, and warned that it could overlook important failures. In label generation it sets a whole column of 16,384 candidate targets to passing. To measure the consequences for supervision, we re-run the generator with the overwrite disabled and compare the pre-overwrite targets with the released labels on all 103,288 navtrain scenes. The rule erases a candidate distinction that the training loss reads on 11,237 of them (10.8793%). Firing usually changes most of a column: lane keeping carries 9,982 of the 13,042 forgiven loss columns, and its median forgiven column had 14,391 of 16,384 candidates failing before the overwrite. On held-out navtest scenes forgiven on lane keeping, the released lane-keeping head's median AUC against the pre-overwrite outcome is 0.7095; on unforgiven scenes matched on failing-candidate count it is 0.9807. For the Hydra-MDP checkpoint released with GTRS, whose configuration takes the same label file, the two values are 0.6627 and 0.9761. Continuing the released GTRS-Dense checkpoint for 300 optimizer steps with three paired seeds, we observe the forgiven-scene AUC 0.1086-0.1251 higher with pre-overwrite than with published targets, and a narrower gap between matched groups, still above zero. Scoring with forgiveness disabled, we observe lane keeping higher by 2.478-3.524 points on navtest scenes forgiven on any of five loss metrics, with lower adjacent-frame plan consistency. Both changes are larger there than on the rest. EPDMS, scored the same way, does not separate the two target sets.
Read the full abstract ▾
All authors Jiaxuan Guo, Jingxin Yang, Jiaqi Ye, Youran Sun, Shuo Xin, Kejia Zhang, Haizhao Yang
Robotics
DeltaWAM: Change-Centric Visual Foresight via Delta Tokens for an Efficient World-Action Model
Tianyun Jiang, Wenrui Bao, Bingxin Xu et al. (+2 more)
World-Action Models (WAMs) offer visual foresight for robotic manipulation, but pixel-space models repeatedly reconstruct entire future scenes, incurring high computational cost and spatio-temporal redundancy. In physical manipulation, consecutive frames often share most of their visual context; the changes between them are what an action policy needs to anticipate. We introduce DeltaWAM, a change-centric WAM that makes a compact delta
token the unit of future prediction. Each
token is a single vector encoding changes between consecutive dense DINO feature maps. DeltaWAM builds on DeltaWorld, a latent world model pretrained on large-scale
videos, to autoregressively predict one delta
token per future frame. A flow-matching action expert then conditions on the predicted transitions and current DINO features, which serve as spatial anchors, to generate action chunks. Trained for 256 GPU hours on two H100 GPUs, DeltaWAM has 0.725B parameters and achieves 92.8% average success on LIBERO. It also shows robust generalization under procedural perturbations on LIBERO-Pro. Inference takes 142.1 ms per action chunk with 3.86 GB peak
memory.
Read the full abstract ▾
All authors Tianyun Jiang, Wenrui Bao, Bingxin Xu, Yu Tian, Yuzhang Shang
Robotics
Multi-Terrain Mastery: A Comprehensive Controller for Bipedal Locomotion
Oluwami Dosunmu-Ogunbi and Aayushi Shrivastava
Advancing bipedal
robots to navigate diverse terrains remains a significant challenge in robotics. Traditional locomotion controllers excel on specific surfaces but struggle across varied environments, limiting their practical applications. Given the unpredictable nature of real-world environments, a single controller capable of handling multiple terrains is ideal, eliminating the need for multiple specialized controllers. We propose a multi-terrain controller to enhance the versatility and robustness of bipedal locomotion. Building on previous work with a stance ankle motor for stability on inclined and rough surfaces, this paper extends capabilities to steep wet uneven grassy slopes, and compliant terrains such as sand, gravel, rocks, and constrained terrains like staircases. To address the unique demands of these terrains, we introduce a new impact map that is essential for maintaining performance and robustness against unseen terrains. We also discuss in detail the control structure for real-time deployment on the
robot. We validate our controller on the 20 degree-of-freedom Cassie bipedal
robot.
Read the full abstract ▾
Robotics
Dynamic Manipulation with World-Action Models via Counterfactual Planning
Sunwoo Park, Wonbin Lee, Seonghyun Jin et al. (+3 more)
World-Action models (WAMs) trained on static demonstrations often fail to manipulate moving targets even when they possess the required manipulation skills. We attribute this failure to target-response collapse: as execution advances, the policy becomes increasingly biased toward the learned continuation of its ongoing behavior and less responsive to target relocation. To bridge the gap between what the model has learned and what it can generate from the current context, we formulate dynamic manipulation as counterfactual planning by decoupling the context used for plan generation from the physical state used for execution. Our framework, Dynamic Predictive Planning (DPP), first uses the WAM's predictive rollout to estimate when an interaction is expected to occur, and combines this timing estimate with observed target motion to predict the target's future interaction position. DPP then constructs a counterfactual observation that places this predicted target position in a familiar
robot context, allowing the model to invoke an existing manipulation skill rather than generate a recovery behavior from an unfamiliar
robot-target configuration. The resulting plan is connected to the
robot's actual state during execution. DPP enables real-time dynamic manipulation on a single consumer GPU without additional training on dynamic data. Experiments in simulation and on a real
robot demonstrate consistent improvements across diverse target motions, with simulation performance surpassing all evaluated baselines, including methods additionally trained on dynamic data. Project page: https://methoder00.github.io/DPP/
Read the full abstract ▾
All authors Sunwoo Park, Wonbin Lee, Seonghyun Jin, Youngmin Kim, Jangho Park, Jong Chul Ye
Robotics · Cross-listed from cs.LG
When Does Backpropagating Through Policy Memory Matter? Physical Credit, Optimizer Updates, and Observability
Xingjian Li, Jianhua Z. Huang and Junli Duan
Policies with
memory can learn along two backward paths: through the physical states their actions produce and through the representations they store.
Transformer-XL and truncated backpropagation through time cut the second path at stored history while keeping its values. We ask when this cut matters. Holding the forward computation fixed and varying only derivative edges, we measure parameter gradients, the updates the optimizer applies, and continued training in a
Transformer vessel-trajectory model and a quadrotor tracking policy. In the vessel model, detaching the key-value cache shrank the gradient to about a tenth of its norm, with little rotation, when gradients flowed through all earlier physical states, but barely changed it under one-step physical credit. In this strongly clipped regime the optimizer, not the gradient, set how far updates differed: global-norm clipping removed most of the gradient difference between
memory-cut graphs, whereas AdamW turned a 2% gradient difference between two placements of the cut into update differences of up to 31% at the step where the placement was switched. In a quadrotor trained from initialization with 0.20 m/s velocity noise, removing
memory raised tracking error by 43% and cutting
memory gradients raised it by 32%; at low noise the cut's mean cost exceeded the value of
memory. Two-step truncation segments gave no measurable gain, although with hidden velocity a two-step window captured most of the value of
memory; eight-step segments removed half to three quarters of the cost. Switching the cut on only for the last fifth of training understated its cost about threefold at 0.20-0.30 m/s, but not at low noise or with hidden velocity. These results suggest measuring the cost of a
memory cut by training with it from initialization, and comparing backward graphs by the updates the optimizer applies rather than by raw gradients.
Read the full abstract ▾
Robotics
Beyond Tasks: A Vision for Reproducing an Animal-like Behavioral Substrate Using Modern Robot Learning Techniques
Samiyuru Menik and Hemadri Jayalath
Recent advances in
robot learning have produced increasingly capable embodied
agents. Yet comparatively less
attention has been given to a more basic form of competence that animals exhibit continuously: the ability to remain situated, responsive, and behaviorally coherent as physical, environmental, and social demands change over time. We propose the ethological behavioral substrate as a conceptual lens for studying this form of competence in artificial
agents. Rather than treating these behaviors that animals exhibit as a set of isolated skills, we argue that their continual coordination under competing demands constitutes an important and underexplored target for modern
robot learning. We further propose robotic animal companions as a useful research setting for studying sustained interaction and adaptation in human-centered environments. Such systems provide an opportunity to investigate how social behavior,
memory, and continual learning develop over long periods of interaction. This perspective motivates further investigation of how such persistent behavioral competence may complement higher-level capabilities in embodied
agents.
Read the full abstract ▾
Robotics
TimelyDAgger: Timing-Aware Expert Querying for VLA Policy Improvement
Zhixuan Zhao, Peiyan Li, Enhao Zhang et al. (+10 more)
DAgger improves
robot policies by aggregating expert supervision from states visited during policy execution.
Robot-gated DAgger automates expert queries, allowing the
robot to decide when to request expert takeover. While existing gates emphasize detecting the need for assistance, takeover timing also shapes the content of these demonstrations and their value for policy learning. We propose TimelyDAgger, combining Bridge-PCA monitoring of internal
vision-
language-action (VLA) features with Feedback-guided Threshold Adaptation based on expert behavior to improve takeover timing. We introduce an evaluation framework linking failure detection, takeover timing, and policy improvement, including Target-Aligned Supervision Ratio (TASR) for assessing supervision quality without retraining. Experiments show that takeover timing affects policy learning, with TimelyDAgger achieving competitive failure detection and higher post-training success in most evaluated settings under matched expert-action budgets. Project website: https://seen-e.github.io/TimelyDagger/.
Read the full abstract ▾
All authors Zhixuan Zhao, Peiyan Li, Enhao Zhang, Yueran Tao, Hao Wang, Chenghao Yue, Lei Lv, Wentao Zhao, Jiahao Chen, Xin Liu, Kangyao Huang, Yu Luo, Huaping Liu
Robotics
Beyond State-as-Action: Exploiting Command-State Discrepancy for Robot Imitation Learning
Peiyan Li, Yueran Tao, Enhao Zhang et al. (+10 more)
Constructing action targets from measured
robot motion is an established approach in imitation learning. Under interaction constraints, however, command-state discrepancy may reflect control demands that motion alone does not capture. We investigate when this information matters and how to exploit it. Across three real-
robot tasks, task and phase analyses reveal larger supervision gaps under constrained interaction, while selective command retention provides evidence of locally useful command information. Building on these findings, we propose Command-State Discrepancy Weighting (CSDW), which accounts for
robot response times and combines subsequent progress, persistent unmet demand, and demand changes into continuous weights for command supervision. The method requires no task-phase annotations or changes to policy architecture or inference. CSDW improves over uniform command supervision on constrained tasks, while methods perform similarly in the less constrained task. Project page: https://seen-e.github.io/CSDW/.
Read the full abstract ▾
All authors Peiyan Li, Yueran Tao, Enhao Zhang, Zhixuan Zhao, Chenghao Yue, Hao Wang, Lei Lv, Wentao Zhao, Jiahao Chen, Xin Liu, Kangyao Huang, Yu Luo, Huaping Liu
Robotics
Multi-Modal Non-Prehensile Estimation of Physical Parameters via Press-and-Pull Tipping
Steven M. Hyland, Jing Xiao and Cagdas D. Onal
Recovering physical properties of unknown objects through non-prehensile interaction is challenging because no single manipulation primitive reveals all relevant parameters. Planar pushing couples mass and friction, while conventional tipping cannot recover friction and may fail entirely when low-friction or curved-base objects slide or rotate instead of tipping. We introduce a multi-modal estimation framework that combines a sliding interaction with a press-and-pull tipping primitive to recover object mass, center of mass height, and surface friction. The press-and-pull interaction increases the object-table sliding threshold and stabilizes the pivot, enabling controlled tipping without any prior
geometric object model. Wrist force/torque sensing, RGB-D perception, and
robot proprioception are fused to estimate the physical parameters from the two complementary interaction modes. Experiments on an ABB IRB120 across four objects with varied geometry, mass, center of mass, and friction achieve low relative error, while successfully operating on curved-base objects that fail under conventional forward tipping. The results demonstrate that complementary non-prehensile interactions can recover a compact set of physical parameters without grasping, a prior object model, or learned interaction dynamics.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Train Together or Merge Later? Unifying VLA Experts via a Shared Action Interface
Zhizhen Zhang, Yuxia Fu, Zijian Wang et al. (+2 more)
Co-training offers a straightforward way to build a multi-task
vision-
language-action (VLA) policy, but can fall short of the performance achieved by training each task independently. The challenge is to retain these task-specific gains in a multi-task policy without joint post-training. Combining independently trained experts through model merging is a natural approach, yet strong individual experts do not necessarily yield a strong merged policy. We identify one source of this incompatibility: task-specific changes to the action interface, comprising action normalization and the action encoder and decoder. We propose PolicyWeave, combining merge-compatible post-training with context-guided sparse merging. During post-training, all experts retain the common base policy's action interface, while task adaptation is restricted to LoRA updates in the hidden layers of the action model. This makes the experts more compatible with existing model merging methods. However, merging all experts can still introduce interference from unrelated tasks at deployment. PolicyWeave scores each expert's LoRA updates using the initial visual-
language context, determines the expert set through leave-one-layer-out ranking stability, and forms a sparse weighted merge of the selected updates that remains fixed for current task. We evaluate PolicyWeave with GR00T N1.5 on 18 RoboCasa365 tasks, using only 10% of the target-task demonstrations for supervised fine-tuning (SFT). Preserving the shared action interface raises the average success rate across four static merging methods from 17.0% to 52.8%. PolicyWeave achieves 64.7% success with these SFT experts and 74.1% after task-specific reinforcement learning (RL), compared with 60.7% for joint RL. Further evaluations on LIBERO-10 and an AgileX Piper arm support the deployment of independently learned skills in long-horizon and real-world manipulation.
Read the full abstract ▾
All authors Zhizhen Zhang, Yuxia Fu, Zijian Wang, Helen Huang, Yadan Luo
Robotics
VPTwin: Real-Sim-Real Video Prediction for Robotic Manipulation Planning
Zhenghao Xiao, Minting Pan, Nantian He et al. (+2 more)
While action-conditioned
video prediction provides an intuitive world model for robotics, purely data-driven predictors often suffer from compounding errors and physically implausible
hallucinations in long-horizon rollouts, severely undermining downstream action planning. We propose VPTwin, a Real-Sim-Real
video prediction framework that anchors real-world future prediction using real-synchronized simulation twins. For a target manipulation task, a VLM reconstructs an executable digital twin from a real demonstration episode. To accommodate the ill-posed estimation of unobserved physical properties, Isaac Sim simulates multiple forward dynamic rollouts across randomized physical configurations under candidate action trajectories. Using these rollouts as in-context references, VPTwin harmonizes both domains, using simulation dynamics to enforce physical plausibility while capturing unmodeled contact interactions from real
video. Furthermore, we establish a predictive planning loop using VPTwin to visually verify VLM-proposed actions and guide reliable real-world execution. Evaluations show substantial reductions in physical
hallucinations during
video prediction and marked improvements in manipulation planning performance.
Read the full abstract ▾
All authors Zhenghao Xiao, Minting Pan, Nantian He, Dongzhan Zhou, Yunbo Wang
Robotics
Evolving Dexterous Robots from Scratch
Zihan Guo, Shuzhe Zhang, Muhan Li et al. (+2 more)
Little is known about how to manually design
agents capable of dexterous manipulation. Some design principles have been inferred from close examination of how animals manipulate objects, but these structures and behaviors have so far resisted biomimicry and may not be optimal for artificial machines. Here we evolve freeform
robots to pick up, hold, rotate, and use diverse objects. Unlike other approaches to optimizing
robot hands, we do not presuppose the presence, articulation, or geometry of any part of the body. Although familiar prehensile forms such as tails, beaks, paws and claws may emerge spontaneously under certain conditions–and while such conditions could be of interest to evolutionary biologists–de novo manipulator design can also reveal whole new solutions, overlooked or unknown structures which may be better suited for the task at hand. We use contrastive learning to create a highly searchable genetic
embedding of design space, an autoregressive developmental model to decode designs, evolutionary strategies to find good designs, and reinforcement learning to train each evolved design. Winning designs were automatically converted into a manufacturable blueprint, printed, assembled and tested in the real world in a zero-shot manner. The results represent the state-of-the-art in evolutionary robotics in terms of performance, diversity and complexity.
Read the full abstract ▾
All authors Zihan Guo, Shuzhe Zhang, Muhan Li, Peiyang Li, Sam Kriegman
Robotics · Cross-listed from cs.CV
Query, Align, and Distill: Navigation-Aware Cross-Modal Interaction for Efficient Vision-and-Language Navigation
Zhihao Chen, Yiyuan Ge, Ziyang Wang et al. (+2 more)
Recent large-scale
Vision-and-
Language Navigation (VLN) models deliver strong accuracy but remain costly to deploy due to their large parameter counts and computational requirements. We tackle efficient VLN in two steps. First, we build a high-performing teacher that makes navigation evidence selection explicit and compressible. The teacher introduces a small set of learnable query slots to extract global and local action-sufficient navigable evidence from panoramic observations via a Navigable Query Generator, then progressively grounds these evidence
tokens to the instruction with an Instruction-Query Aligner for policy prediction. Second, using this explicit query bottleneck as a
distillation interface, we train a compact student by transferring both where to attend and what to do. We distill the teacher's global and local navigable queries with a navigation-aware
token-adaptive objective, then further match action distributions during fine-tuning. Experiments on standard VLN
benchmarks demonstrate that our student nearly matches the teacher's navigation performance while reducing the number of parameters by 93.65% compared to the teacher.
Read the full abstract ▾
All authors Zhihao Chen, Yiyuan Ge, Ziyang Wang, Pu Cao, Lu Yang
Robotics
Humanoids for Robot-Assisted Surgery: Bimanual Base Placement and Tool-Mount Optimization via Capability Maps
Peihan Zhang, Zekai Liang, Florian Richter et al. (+4 more)
Rapid advances in humanoid robotics have motivated growing interest in the application of humanoids for healthcare and clinical tasks. However, it remains unclear how close contemporary humanoids are to meeting the kinematic demands of
robot-assisted laparoscopic surgery. In this work, we address the question of optimal
robot positioning through a quantitative analysis of workspace and
robot setup configurations. We present a capability-map-based
robot setup framework that optimizes humanoid base placement and tool mounting orientation to maximize bimanual humanoid reachability while accounting for tool-tip kinematics and remote-center-of-motion (RCM) constraints. We evaluate three humanoid platforms spanning different body dimensions and kinematic redundancy on workspace reachability for three representative general surgery procedures: cholecystectomy, inguinal hernia repair, and sleeve gastrectomy. The proposed joint optimization of base placement and tool mounting consistently outperforms base-only optimization and heuristic baselines. For cholecystectomy and inguinal hernia repair, which are characterized by relatively small and minimally overlapping workspaces, humanoid reachability approached 90%. For the larger, overlapping multi-port arm workspace of sleeve gastrectomy, humanoids yield substantially lower coverage. These results quantify the near-term promise of humanoids for selected laparoscopic procedures and clarify key limitations that must be addressed for broader deployment.
Read the full abstract ▾
All authors Peihan Zhang, Zekai Liang, Florian Richter, Nikita Thareja, Ryan Broderick, Shanglei Liu, Michael Yip
Robotics
REALM: A Coarse-to-Fine Generative Framework for Embodied Reactive Listening
Peizhen Li, Longbing Cao and Yang Zhang
Generating responsive listener facial motion is an important task for embodied conversational AI. Two modeling challenges are central: accounting for the timing of speaker cues while maintaining continuity with the listener's ongoing motion, and capturing locally variable facial events alongside the overall motion trajectory. Listener responses may follow preceding cues with a temporal lag, while brief expressions and blinks introduce variation that is difficult to predict deterministically. These challenges motivate a framework that combines history-aware temporal
alignment with
stochastic expression refinement. We propose REALM (Reactive Embodied Audio-driven Listening Model), a coarse-to-fine framework for audio-driven reactive listening. A Reactive Gated Speaker-Listener Fusion module combines listener motion history with speaker audio through a delay-centered
attention prior and adaptive gating. A coarse decoder predicts a base motion trajectory, which is augmented by audio-conditioned
stochastic residuals in the expression subspace while retaining the coarse pose parameters. Evaluations on ViCo and L2L show improvements over the evaluated baselines across multiple motion-quality metrics. Additional analyses examine delay sensitivity, gate behavior, and blink dynamics. Finally, deployment on an Ameca humanoid
robot and a perceptual user study demonstrate the applicability of the generated behavior to physical embodiment. Code: https://github.com/lipzh5/REALM Demo: https://youtu.be/Tf5mpd5S8VQ
Read the full abstract ▾
Robotics
SwingRL: Adaptive Observation Reinforcement Learning with World-Model Prediction for Cable-Suspended Hoisting Control
Guangming Wang, Xiaoyu Zhang, Yucheng Xin et al. (+8 more)
Cable-suspended hoisting is widely used to move heavy or bulky payloads that cannot be handled conveniently by rigid pick-and-place systems, for example in crane-assisted construction. Robotic hoisting using flexible cables is challenging because payload motion is underactuated, external disturbances vary, and delayed or lost visual observations can make the perceived payload state stale at control execution. These effects are particularly critical during precise insertion of a suspended payload's sockets onto rebar pins, which is a very common task in construction environments. We present SwingRL, a residual reinforcement-learning (RL) framework that combines an age-aware world model, a classical anti-swing prior, and a recurrent residual policy to address two coupled problems: stale feedback and uncertain dynamics. The world model propagates the newest received payload observation to the current control step using the executed commands, providing a time-aligned state estimate under delayed and lossy sensing. The prior supplies nominal tracking and swing damping. The residual policy learns bounded corrections to the prior rather than the complete control law, compensating for system-parameter variation, external disturbances, and remaining state-estimation errors. We evaluate SwingRL against classical and learning-based baselines across a cumulative difficulty ladder covering system-parameter variation, wind disturbance, degraded sensing, and strong gusts. Under the most difficult setting, SwingRL achieves 69.5% strict and 77.3% broad success, exceeding all baselines by at least 60 percentage points, respectively. World-model ablations support the role of time-aligned state estimation in maintaining insertion success as observation loss increases. Finally, without real-
robot fine-tuning, SwingRL achieves 90% success on the physical rig.
Read the full abstract ▾
All authors Guangming Wang, Xiaoyu Zhang, Yucheng Xin, Wanli Ma, Jiucai Liu, Yunxiang Ma, Joe Ingham, Haibing Wu, Yixiong Jing, Olaf Wysocki, Brian Sheil
Robotics
Is Online Interaction Necessary for Recovery? A Minimalist Approach to Robust Planning via Perturbation
Bumgeun Park and Donghwan Lee
Behavior cloning (BC) is vulnerable to
covariate shift during closed-loop execution, where small prediction or execution errors can drive the
robot toward states poorly covered by the demonstration data. We focus on action-sequence planning, where a policy predicts a finite-horizon sequence of actions as a reference trajectory for
robot execution. Existing approaches to
covariate shift often rely on collecting additional corrective demonstrations, requiring further environment interaction and access to an expert or reference policy. We propose Perturbation-Augmented Recovery Supervision (PARS), a simple training approach for improving recovery using only existing demonstrations. PARS perturbs the
robot's proprioceptive state and anchors only the later portion of the predicted trajectory to the original demonstration, leaving the earlier portion free to generate corrective motion and recover toward the demonstrated behavior within the planning horizon. Unlike conventional input-noise augmentation, which preserves the original supervision target over the entire prediction horizon, PARS explicitly provides trajectory-level supervision for recovery from perturbed states. PARS requires neither additional environment interaction nor expert queries and can be instantiated across different action-sequence policy classes with only minor modifications to their BC objectives. Experiments on 51 RLBench manipulation tasks with flow-based,
transformer-based, and
diffusion-based policies demonstrate that PARS improves robustness to
covariate shift during closed-loop execution.
Read the full abstract ▾