Robotics · Cross-listed from cs.CV
RadarGSOcc: Doppler-Aware 3D Semantic Occupancy Prediction Using 4D Radar Tensor with Geometric-Semantic Feature Extraction
Christofel Rio Goenawan, Dong-Hee Paek and Seung-Hyun Kong
3D semantic occupancy prediction is crucial for autonomous driving, but camera-based methods degrade in low-light conditions and
LiDAR-based methods are sensitive to adverse weather. 4D radar is more robust, yet its sparsity and the high cost of radar tensor processing limit its use. We propose Radar
Geometric-Semantic Occupancy Prediction (RadarGSOcc), an efficient framework that directly processes 4D radar tensors by separately extracting
geometric and semantic features, including Doppler information, with a deformable
attention-based
geometric refinement module as its key component. We also introduce the K-Radar 3DSemOcc
benchmark, the first large-scale
benchmark for 3D semantic occupancy prediction using 4D radar tensors, built on the K-Radar
dataset with refined
LiDAR-based annotations. RadarGSOcc predicts 15 classes within a 51.2m range and achieves 38.2% IoU, 7.6% mIoU, and 25.5% weighted mIoU, outperforming RadarOcc by 17% relative (25.5 vs. 21.9) in weighted mIoU while using fewer parameters and lower computation. It substantially improves truck prediction while trading off car accuracy, outperforms camera-based approaches, and narrows the gap with
LiDAR-based methods. To assess robustness without per-frame ground truth in adverse weather, we further propose a novel proxy metric, Environment-Based KL Divergence (EBKL), under which 4D radar proves a reliable and efficient modality for 3D semantic occupancy prediction. The
dataset and code will be publicly available.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
Model-Based Online Decision Making via Generative Trajectory Planning
Haldun Balim, Yilun Du and Na Li
Online decision-making faces a tension between efficient inference and long-horizon
reasoning. Value-based methods rely on short-horizon bootstrapping that limits the propagation of long-term information, while trajectory optimization methods plan explicitly but must solve a high-dimensional optimization problem at every decision step. We propose Generative Trajectory Planning (GTP), a model-based framework that addresses this tension by learning a generative model as a proposal distribution over action sequences. Instead of optimizing sequences from scratch, GTP samples candidate sequences from a
diffusion prior and refines them using learned dynamics, reward, and value models, enabling efficient trajectory-level
reasoning. Across continuous control and long-horizon manipulation tasks, GTP matches or exceeds state-of-the-art model-free,
diffusion-policy, and model-based baselines, and remains stable in regimes where strong baselines degrade. Our results suggest that generative models are best understood not as policies, but as structured proposal distributions for planning, offering a new perspective on how to integrate generative modeling with model-based decision-making.
Read the full abstract ▾
Robotics
Perception Beyond the Visible: Incompleteness-Aware Scene Understanding for Robot Autonomy
Rohit Mohan
Reliable
robot autonomy requires
reasoning about the world from partial and noisy observations. However, many perception systems model only visible scene regions, rely on fixed semantic categories, and assume reliable sensor measurements. These assumptions break under occlusion, novel object encounters, and sensor degradation. In this paper, I present a research
vision for incompleteness-aware scene understanding. I organize this
vision around three fundamental challenges: incomplete evidence, incomplete knowledge, and incomplete sensor reliability. I discuss my research contributions toward addressing these challenges through amodal perception, semantic occupancy forecasting, open-world scene understanding, label-efficient learning, and uncertainty-guided multi-modal fusion. Building on these contributions, I outline future research directions toward belief-centric 4D scene representations. These representations aim to jointly model geometry, semantics, motion, and uncertainty. I also discuss active sensing and safe adaptation strategies that allow
robots to reduce uncertainty through interaction. The goal is to develop calibrated and actionable scene representations that support reliable autonomy in dynamic, open-world environments.
Read the full abstract ▾
Robotics · Cross-listed from cs.CL
Known-Voice Cloning and the Limits of Personality Transfer for Character Design
Johanna Magdalena Kuch, Maud van Lier, Daksitha Senel Withanage Don et al. (+3 more)
Recent advances in zero-shot voice cloning systems make it possible to reproduce a recognizable voice from a short recording and attach it to arbitrary virtual characters. Prior work suggests that familiar vocal identity can carry over associated personality impressions to new targets such as
robots or virtual characters. However, these works have mainly examined familiarity and personalization effects rather than directly testing whether a cloned familiar voice systematically transfers personality impressions to an embodied virtual character. Such transference raises the concern that cloned voices could be used to manipulate people into engaging with virtual characters as capable social actors. Across two studies, we investigated whether personality impressions transfer when a virtual character speaks with the cloned voice of a familiar person. Study 1 examined this question in a real-world university setting using the cloned voice of a lecturer familiar to the participating students. Study 2 examined the same question online using voices of social media content creators and members of their communities. Drawing on the concepts of identity activation, trait transfer, and person reconstruction, we show that evidence of personality transfer was limited in both studies. Observed effects were selective and did not generalize across traits or study contexts. In the online study in particular, recognizable voice identity cues activated expectations about the speaker but did not reconstruct the broader person impression. Qualitative responses further indicated that judgments were frequently shaped by perceived artificiality and by whether voice and appearance fit together. Our findings moderate common concerns that cloned voices straightforwardly manipulate people into perceiving artificial characters as capable social actors.
Read the full abstract ▾
All authors Johanna Magdalena Kuch, Maud van Lier, Daksitha Senel Withanage Don, Patrick Gebhard, Tanja Schneeberger, Elisabeth André
Robotics
ForkCast: Predictive Recovery for Vision-Language-Action Policies via Counterfactual Futures
Chenru Wen, Huajian Zeng, Haokai Ding et al. (+5 more)
Vision-
language-action (VLA) policies perform manipulation tasks using camera images and
language instructions, but object slip or visual occlusion can disrupt execution. Recovery is typically learned during policy training or delegated to a monitor that triggers a scripted repair after detecting a failure. We introduce ForkCast, a predictive recovery framework that compares counterfactual futures to determine whether and how to intervene in a frozen VLA policy. At each decision point, ForkCast evaluates three candidates: executing the policy's nominal action chunk (Commit), adjusting the camera viewpoint (Reobserve), and replanning the arm trajectory (Replan). A candidate-conditioned world model forecasts the latent outcome of each candidate at a common future horizon, and a shared task-progress head scores the predicted outcomes. By evaluating the policy's own continuation alongside the repair candidates, a single comparison based on task progress determines both whether and how to intervene, without a failure detector, an intervention threshold, or updates to the VLA backbone. On five simulated RoboTwin 2.0 tasks with injected occlusion and object slip, ForkCast increases pooled success from 13.3% for the frozen $\pi_{0.5}$ policy to 30.0% and transfers to two additional VLA backbones without updating ForkCast. After adaptation on real-
robot candidate rollouts, ForkCast also improves success over the base policy under both clean and visibility-limited conditions. Code will be released to benefit the research community.
Read the full abstract ▾
All authors Chenru Wen, Huajian Zeng, Haokai Ding, Jian Kang, Anqing Duan, Dawei Zhou, Dezhen Song, Xingxing Zuo
Robotics
Grid or Nodes? Probing Object Memory in World Models Under Partial Observability
Justin McMillen and Yasin Yilmaz
A
robot's world model must remember objects that leave the camera's view. FloWM, a learned world model, keeps them in a spatial grid of features that it moves with five built-in velocities. A node
memory instead keeps each object's position and velocity as numbers. We compare them in FloWM's own block world, with an oracle-fed Kalman tracker as the node
memory. How the grid is read changes its score: a single-
cell probe finds 66% of out-of-view blocks, while FloWM's own probe 89%. FloWM also scores 8 points higher in view although no frame is shown, which we attribute to its write-back of its own predictions. Out of view, the tracker reaches FloWM's range in 14KB against 5.6MB, with no probe. On diagonal motion, outside its five velocities, FloWM finds 11% of blocks even after fine-tuning, while the unchanged tracker keeps 93%. Faster blocks defeat both until one more allowed speed takes the tracker to 94%, where FloWM's built-in shifts would need 25 copies of its grid. A learned node
memory must still replace the oracles and handle velocity under noise, long gaps, and the bounce.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
LLM2R: From Large Language Models to 3D Reassemblers
Zhouhan Zhong, Qiuhong Shen, Xingyi Yang et al. (+2 more)
3D part assembly aims to reconstruct a complete shape from a set of unassembled parts by predicting a rigid SE(3) transformation for each part. Because the placement of a part is rarely determined by its own geometry, the task demands
reasoning over the entire part set, and existing methods, which rely on pairwise relations or iterative generation and verification, scale poorly to objects with many parts. We present LLM2R, which casts part assembly as sequence modeling with a pretrained decoder-only large
language model (LLM): a Point-BERT encoder and a Q-Former compress each part into a block of
geometric tokens, the blocks of all parts are concatenated, and a causal LLM adapted with Low-Rank Adaptation (LoRA) processes the sequence in one forward pass. Causal
attention, however, lets the hidden states of a part observe only the parts that precede it. LLM2R therefore appends a block of learnable summary
tokens after the last part: these
tokens follow every part, so their states aggregate the whole assembly at a constant
token cost, and a part-aware pose head fuses each part's states with them before predicting its rotation and translation. One architecture thus handles semantic and
geometric assembly and predicts all poses in a single non-autoregressive pass. LLM2R achieves the highest part accuracy and success rate on all three PartNet categories, the best rotation error, translation error, and part accuracy on PartNeXt, and the lowest translation error and fastest inference on Breaking Bad, where only a fracture-specialized method is more accurate. Its advantage grows with the number of parts: at 100 parts it exceeds GARF by 33.5 points in part accuracy on Breaking Bad and Assembler by 29.1 points on PartNet, and it transfers zero-shot to unseen categories with more than 16 points higher part accuracy than the strongest of three baselines.
Read the full abstract ▾
All authors Zhouhan Zhong, Qiuhong Shen, Xingyi Yang, Jinyi Xu, Xinchao Wang
Robotics
ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models
Changyeon Kim, Kimin Lee, Yashraj Narang et al. (+1 more)
Reinforcement learning (RL) post-training improves
robot foundation models (RFMs) beyond their demonstrations, but real-world deployment demands more: learning new tasks efficiently while retaining previously acquired behaviors. Existing continual RL methods on top of VLAs mitigate forgetting but rely on on-policy updates with limited forward transfer, whereas off-policy methods improve single-task sample efficiency without addressing learner persistence across tasks. We propose ACCRUE, a continual off-policy RL framework that carries lightweight actor and critic networks across tasks, allowing each stage to reuse behavioral and value knowledge acquired through prior RL. The inherited critic adapts only on current-task transitions and guides current-task actor learning, while prior behavior is preserved by distilling the actor toward cached targets on a small number of retained episodes. Fixed visual features keep the inherited learner and its replay compatible with base-policy updates. Across contact-rich assembly sequences in simulation and on real-
robot hardware, ACCRUE accelerates subsequent-task learning with near-zero forgetting and significantly reduces real-
robot training time by 31.0% to 68.5%.
Read the full abstract ▾
All authors Changyeon Kim, Kimin Lee, Yashraj Narang, Yijie Guo
Robotics
WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction
Haomin Wang, Anqiong Ji, Ke Wang et al. (+4 more)
Vision-
language-action (VLA) policies excel at generalist robotic manipulation but generate actions without explicitly modeling their environmental consequences. World action models (WAMs) incorporate future prediction, yet most rely on costly
video generation, while latent alternatives either use predicted futures only as auxiliary targets or infer latent actions from suboptimal visual transitions through costly large-scale
video pretraining. We introduce WAVE-0 (World-model-Augmented VLA Execution), a framework that bridges VLAs and WAMs through a lightweight latent world model. By connecting the semantic understanding of pretrained VLAs with explicit future prediction, WAVE-0 allows anticipated action consequences to directly guide action generation. Built on a pretrained VLA, its 257M world model translates VLM-generated latent action plans into predicted future representations that condition the action expert, unifying latent planning, future prediction, and action generation. The world model is trained directly on ground-truth
robot actions with a counterfactual ranking objective, requiring neither pixel reconstruction nor large-scale
video pretraining. WAVE-0 achieves success rates of 91.47% on RoboTwin 2.0 and 98.7% on LIBERO, along with 88.0%, 61.4%, and 75.0% on LIBERO-Plus, LIBERO-Pro, and LIBERO-Para, respectively. On real-world tasks, it improves the success rate of $\pi_{0.5}$ from 38.3% to 73.3%. These results demonstrate strong manipulation performance and generalization across diverse distribution shifts.
Read the full abstract ▾
All authors Haomin Wang, Anqiong Ji, Ke Wang, Zhongshan Liu, Ru Ying, Ran Zheng, Zhipeng Zhang
Robotics
Behavioural self-monitoring for autonomous earth observation agents: The attribute reliability monitor (ARM)
Alejandro D. Mousist
Language and
vision language models can support autonomous systems, but changing operating conditions may compromise the reliability of their semantic outputs. This work introduces the Attribute Reliability Monitor (ARM), a framework that accumulates evidence of suspicion over time to assess individual semantic attributes. ARM accommodates different detectors and temporal monitoring methods, while leaving intervention policies to the consuming system. An initial implementation was evaluated using a specialized
vision language model developed for satellite thermal infrared imagery. Monitoring followed a black box approach, using only observable responses without access to internal model states, confidence scores, or external reference answers. Under controlled input degradation, disagreement between responses provided evidence for selective suspension of attributes. The results illustrate how external behavioral monitoring can inform continued use of model outputs, while distinguishing operational suspicion from semantic correctness.
Read the full abstract ▾
Robotics
APTree: Extended Behavior Trees for Adaptive and Scalable Automated Robotic Task Planning in Construction
Shermin Sherkat, Thomas Wortmann and Andreas Wortmann
Task planning for
robots in off-site construction prefabrication is primarily manual, leading to planning inefficiencies and reluctance to adopt robotic solutions. Existing approaches provide limited support for scalable, modular automated task planning and localized runtime replanning. This paper proposes APTree, a model-driven framework that integrates automated task planning with modular task execution using behavior trees as the execution model. APTree repeatedly applies and coordinates heterogeneous symbolic planners to automatically generate and compose modular task plans at runtime. To support this approach, we develop a domain-specific
language for the formal specification and validation of task plans, together with a compatible execution engine. We evaluate APTree on the real-world livMatS Biomimetic Shell pavilion, where it models the construction robotic fabrication domain, detects injected specification and planning errors at design time and runtime, and automatically generates large-scale task plans by coordinating three symbolic planners across numerous planning modules. A machine-checked formal analysis further shows that composed execution and local replanning are safe under explicitly stated assumptions, including planner soundness and non-interference of generated plans, providing practitioners with explicit conditions for dependable plan composition. APTree thus enables construction practitioners to leverage automated task planning without requiring detailed knowledge of symbolic planning techniques or APTree's internal implementation.
Read the full abstract ▾
Robotics
RTISSLAM: In-air Imaging Sonar SLAM using a Robust Pose Graph and Occupancy Grid Mapping
Jan Steckel
Sonar was one of the first sensors with which mobile
robots built maps, and in-air sonar keeps working where cameras and
lidar fail, in darkness, smoke, dust and next to glass. Classical sonar SLAM, however, relied on
geometric features or occupancy grids built from sparse range readings, and both struggle with the specular reflections that dominate indoor sonar. In this letter, we introduce RTISSLAM, a novel sonar-only SLAM system for an embedded 3D imaging sonar, which combines the robust pose graph of BatSLAM 2.0 as back-end with an occupancy grid as map. Energyscapes from delay-multiply-and-sum beamforming with a range-wise CFAR serve as place descriptors for sequence-verified loop closures, acoustic odometry calibrates the wheel odometry online, recognized revisits are registered metrically, and the grid is stored per keyframe and anchored to the pose-graph nodes, so that it follows every loop closure. In simulated worlds with specular walls, RTISSLAM reduces the trajectory error of the wheel odometry from \SI{8.2}{\meter} to \SI{0.31}{\meter} in an office, while
lidar-style grid SLAM on the same sonar data fails.
Read the full abstract ▾
Robotics
MemoryVLN: From Ego-Aware Memory Representation to Cross-Context Memory Reuse in Vision-Language Navigation
Xueyao Wei, Yifei Wang, Delun Li et al. (+4 more)
Large
vision-
language models (LVLMs) have shown strong potential for embodied
vision-
language navigation (VLN), yet their performance degrades on long-horizon trajectories due to context-length limits, redundant historical observations, and unstable spatial
alignment across time. These issues make it difficult to both compress long navigation histories and reliably reuse past spatial experience. To address these limitations, we propose MemoryVLN, a unified
memory-driven navigation system built around
memory construction and management. During
memory construction, Long Sequence Compression employs forgetting-curve-inspired temporal sampling to preserve dense recent evidence and sparse yet representative distant observations under a limited
token budget. Ego-Aware
Memory Representation further integrates 2D semantic
tokens, 3D
geometric tokens, and 6D
agent poses, allowing each
memory to encode both what was observed and where it was observed. During
memory management, Hierarchical
Memory Bank maintains in-episode and context-level
memories through cross-context warm-start and confidence-controlled write-back.
Memory Enhancement then retrieves the Top-$K$ relevant
memories according to the current navigation state and fuses them with the current visual
tokens. Together, these mechanisms transform raw navigation history into compact, spatially grounded, selectively retrievable, and reusable navigation experience. Extensive experiments on R2R-CE and RxR-CE demonstrate consistent improvements over the state-of-the-art baselines by a substantial margin (i.e., 2% and 6.5% in successful rate).
Read the full abstract ▾
All authors Xueyao Wei, Yifei Wang, Delun Li, Shaoan Wang, Junzhi Yu, Wenjie Song, Xingyu Chen
Robotics · Cross-listed from cs.AI
Thinker-WM: A Unified World Model with Shared Multimodal Representations for Embodied Intelligence
Thinker Team Ubtech:, Peng Gao, Jichao Jiao et al. (+8 more)
World models have emerged as a promising paradigm for embodied AI, yet existing systems often remain fragmented in the modalities they model and the functions they support. In particular, visual dynamics,
geometric structure, and executable action are commonly coupled through separate interfaces or downstream modules rather than being jointly represented and evolved within a shared dynamics model. In this work, we present Thinker-WM, a unified
multimodal world model that jointly models and generates RGB, metric depth, and
robot action through a shared intermediate Unified Dynamics
Transformer with modality-specific encoders and decoders. The model performs
multimodal unification at the representation level: appearance, geometry, and action are transformed into a common dynamic representation and jointly evolved by shared parameters, while their distinct output structures are preserved. The same parameterization supports multiple functions, including joint
video-action generation, world prediction, policy inference, action-conditioned generation,
video-to-action generation, and inverse dynamics. To examine whether the shared representation captures action-related information, we introduce a lightweight motion probe that shows the frozen visual representation contains recoverable motion information. Thinker-WM is trained on a heterogeneous corpus comprising approximately 20K hours of egocentric
video, 10K hours of real-
robot data, and 1K hours of simulation data, including approximately 4K hours of real-
robot data with aligned depth supervision. Extensive experiments demonstrate strong performance across embodied world-model evaluations. Thinker-WM ranks first on EWMBench, WorldOlympiad and WorldArena, while controlled studies further demonstrate the benefits of action conditioning,
geometric interaction, and multi-function training within the shared representation.
Read the full abstract ▾
All authors Thinker Team Ubtech:, Peng Gao, Jichao Jiao, Chen Liao, Daqin Luo, Yiming Ma, Baiyu Pan, Hailin Shi, Jiyuan Wang, Xiangming Xu, Xiongwei Zhao
Robotics
PACT: Posture-Aligned Co-Design Technique for Hand Configuration and Retargeting in Multi-Finger Imitation Learning
Koyo Sasazaki, Satoshi Funabashi, Pranav Ponnivalavan et al. (+3 more)
The fidelity of teleoperated demonstrations used for imitation learning is determined by how faithfully a
robot hand can replicate the operator's posture. The posture retargeting law that the
robot will execute is typically selected only once the hardware has been finalized, and conventional hand designs often neglect human-like grasping postures. We introduce PACT, an optimization based on human grasping postures that jointly optimizes the mechanical parameters of a 17-servo
robot hand (link lengths, joint offsets, mounting angles, and base placements) and its nonlinear posture retargeter (a linear human-to-
robot joint map, per-joint posture-IK weights, and damping). The optimization enforces full-finger correspondence (MCP, PIP, DIP, and tip of each finger) to human poses from the DexYCB
dataset, and is constrained by servo interference, assembly clearances, and flat-hand protrusion, so the resulting design can be physically assembled. We fabricate the PACT Hand and validate it through teleoperation experiments and an imitation-learning demonstration. The hand reproduces all human hand keypoints with a kinematic RMSE of 3.9 mm, compared to 12.8 mm for the LEAP Hand baseline, and keeps all fingertips within 10 mm on every grasping posture, while the baseline achieves this for only 21.6% of the poses. Under matched per-joint torque limits, it lifts twice the baseline payload, 2 kg against 1 kg, and deforms a held paper cup seven times less, reaching a deformation level comparable to a human hand. It also achieves an 86% grasping success rate on YCB daily objects, compared to 64% for the baseline. Finally, as a demonstration, an imitation-learning policy trained on demonstrations collected with the PACT Hand achieved a 75% success rate at held-out placements in a can pick-and-place task. Project page: https://pact-hand-2027.github.io
Read the full abstract ▾
All authors Koyo Sasazaki, Satoshi Funabashi, Pranav Ponnivalavan, Alexander Schmitz, Tetsuya Ogata, Shigeki Sugano
Robotics
CALM: Decentralized Control of Cooperative Aerial Manipulation with Multi-Agent Reinforcement Learning
Jiaxi Huo, Petre Ricioppo and Fausto Francesco Lizzio
Cooperative aerial manipulation is difficult under wind, payload motion, and noisy goal broadcasts because multiple Unmanned Aerial Vehicles (
UAVs) are coupled through a shared payload. We present CALM (Cooperative Aerial Learned Manipulation), a decentralized controller for aerial manipulators carrying a basket while tracking a moving ground
robot. CALM combines a parameter-shared Multi-
Agent Reinforcement Learning (MARL) policy, trained using Multi-
Agent Proximal Policy Optimization (MAPPO), with goal estimation, per-
UAV Extended State Observers (ESOs), basket-mass estimation, and anticipatory formation-yaw feedforward. The anticipatory formation-yaw feedforward uses estimated goal motion to rotate the formation before position error develops, reducing basket-yaw lag without adding net commanded force. A disturbance-conditioned critic supports centralized training while execution remains decentralized. Across held-out trajectories, CALM achieves lower mean position and orientation errors than all evaluated baselines, including the matched arm baseline (7.68 versus 10.95 cm and 3.3 versus 4.7$^\circ$), the cable-suspended realization, and two model-based controllers. Under unseen full-information-path degradation, CALM remains statistically comparable or superior to the arm baseline.
Videos are available at https://petreric.github.io/CALM/.
Read the full abstract ▾
Robotics · Cross-listed from eess.IV
Augmenting Real-time Visualization in Minimally Invasive Surgery with Particle Filtering and Depth-Aware Occlusion Handling
Anuska Sarkar, Partha Acharya, Vishnu Nandury et al. (+2 more)
We present a real-time, hybrid framework for occlusion handling and depth-aware tracking in minimally invasive surgeries (MIS). Occlusions caused by surgical instruments often obscure the region of interest (ROI), affecting surgical precision. To address this, our approach integrates enhanced You Only Look Once (YOLO) based detection, Particle Filtering (PF), and monocular depth estimation using Multiple Depth Estimation Accuracy with Single Network (MiDaS). Occlusion is detected using a distance-based threshold between instruments and ROI. Experimental results demonstrate robust instrument detection with a confidence score of 0.85, effective occlusion event handling, and accurate depth prediction with an MAE of 0.34 cm and RMSE of 0.53 cm. Independent phantom-based validation against stereo depth measurements yielded an MAE of 0.65 cm and RMSE of 0.66 cm. The framework further achieved an occlusion detection accuracy of 81.95% while maintaining real-time performance at 38.7 FPS. Preliminary surgeon feedback indicates that the proposed visualization framework is clinically relevant and holds promise for future intraoperative image-guided navigation and decision support in MIS.
Read the full abstract ▾
All authors Anuska Sarkar, Partha Acharya, Vishnu Nandury, Kaustav Basu, Subhamoy Mandal
Robotics
Intent-guided Adaptive Stiffening for Visuomotor Policies under Unknown Resistance
Francesco Cufino, Carmela Mariniello, Mario Selvaggio et al. (+2 more)
Manipulation tasks such as unscrewing a cap or sliding a part in contact with a surface require imposing force along a high-friction direction: the
robot must overcome a resistance rather than yield to it. Generative visuomotor policies predict the motion for such tasks but not the authority needed to realize it, and that authority is bounded by the impedance controller beneath them: when the resistance exceeds what a compliant controller can exert under a bounded tracking error, the task is unattainable however accurate the prediction is. The required amount cannot be anticipated either, since it depends on a contact state the observation does not reveal. In this paper, we propose a residual stiffening layer placed underneath an unmodified policy. Resistance is recognized, without force sensing, from the mismatch between the motion the policy commands and the motion the
robot realizes, and a four-mode supervisor raises the stiffness along a single direction read off the policy's own action chunk, retains it while the task progresses and releases it afterwards. The layer needs no compliance labels, no retraining and no re-collected demonstrations. On two real tasks, unscrewing a bottle cap and pushing a T-nut along a T-slot groove against non-homogeneous friction, evaluated at six resistance levels of which two were withheld from the demonstrations, the layer raises the success rate from 52% to 90% while producing 61% less peak off-axis force than a uniformly stiff controller at the same resistance.
Read the full abstract ▾
All authors Francesco Cufino, Carmela Mariniello, Mario Selvaggio, Andreas Sochopoulos, Fabio Ruggiero
Robotics
Benchmarking Interaction-Oriented Aerial Mobility across Heterogeneous Environments
Zhibo Duan, Hanlin Kong, Tianran Ma et al. (+8 more)
Aerial mobile manipulation requires unmanned aerial vehicles (
UAVs) to traverse heterogeneous environments such as indoor and outdoor domains, and connect navigation with subsequent physical interaction within the same trajectories. Existing aerial
benchmarks are typically confined to single-environment settings, limiting evaluation of
agents across continuous environmental transitions. When interaction is evaluated,
agents are often initialized near the target, leaving the transition from navigation to interaction largely unassessed. This work introduces CrossFly, a
benchmark for long-horizon interaction-oriented aerial mobility across heterogeneous environments. CrossFly contains 15K trajectories across more than 20 reconstructed scenes under a unified simulation and evaluation framework, covering Indoor, Outdoor, and Indoor$\leftrightarrow$Outdoor trajectories. It further introduces Interaction Readiness Rate (IRR) to assess whether terminal navigation states provide suitable conditions for downstream interaction without assuming a specific manipulation policy. Experiments with five representative aerial
vision-
language-action (VLA)
agents and the proposed CrossFly-
Agent show that Indoor$\leftrightarrow$Outdoor settings are more challenging than single-environment settings, with the highest success rate reaching 9.7% on unseen scenes. A downstream manipulation case study further validates the relevance of interaction-oriented arrival to subsequent physical interaction. We hope CrossFly helps advance aerial embodied intelligence toward unified mobility and physical interaction in heterogeneous environments.
Read the full abstract ▾
All authors Zhibo Duan, Hanlin Kong, Tianran Ma, Yuze Wu, Yuxuan Xu, Kenghou Hoi, Changheng Xie, Xin Zhou, Jianzhe Gao, Fei Gao, Wenguan Wang
Robotics
DR-VLA: Learning Human-Like Quadruped Navigation from Egocentric Web Videos via Domain-Invariant Scene Representations
Jun Tian, Yingjie Niu, Ruo Chen et al. (+2 more)
Quadruped
robots performing outdoor deliveries must interpret complex scenes, anticipate changes, and navigate in socially compliant ways.
Robot demonstrations are costly to collect at the scale needed for generalization. Egocentric web
videos offer abundant human navigation experience, but differences in viewpoint and motion make this experience difficult to transfer to
robots. We introduce DR-VLA, a Mixture-of-
Transformers framework that transfers human navigation experience through a domain-invariant scene representation (DISR). The framework combines
vision-
language understanding and latent world modeling with a flow-matching action expert. We introduce EgoLynx, comprising 10,000 hours of curated egocentric web
video (EgoLynx-Web) and 2,000 clips of teleoperated
robot navigation collected during real-world deliveries (EgoLynx-
Robot). We pretrain the understanding and world experts on EgoLynx-Web without action labels, then adapt the model to
robot observations and train the action expert on EgoLynx-
Robot to predict executable trajectories. We also propose the Quadruped Navigation Score (QNS), a pseudo-closed-loop metric combining safety, instruction compliance, comfort, and progress. On the held-out EgoLynx-
Robot test split, DR-VLA improves QNS by approximately 5% and reduces open-loop final heading error (FHE) by 14.6%, each relative to the best competing result for that metric. Ablations show that web-
video pretraining improves generalization and that the model's key components contribute to performance. As a future-prediction target, DISR outperforms five generic visual representations and yields performance close to task-specific 3D perception without dedicated
geometric labels or point clouds for target construction. Trained separately on NAVSIM v1 without web pretraining, DR-VLA achieves 92.1 PDMS, suggesting that the framework also applies to autonomous driving.
Read the full abstract ▾
All authors Jun Tian, Yingjie Niu, Ruo Chen, Hongyi Lin, Chao Li
Robotics
Visual Sevoing from Lines
Nicolas Andreff, Bernard Espiau and Radu Horaud
In this work, we present a fundamentally new approach to visual servoing using lines. It is based on a theoretical and geometrical study of the main line representations which allowed us to define a new representation, the so-called binormalized Plucker coordinates. They are particularly well suited to visual servoing. Indeed, they allow the definition of a proper image line
alignment concept. The control law which realizes such an
alignment has moreover several properties: partial decoupling between rotation and translation, analytical inversion of the motion equations and global asymptotic stability conditions. This control law was validated both in simulation and experimentally in the specific case of an orthogonal trihedron.
Read the full abstract ▾
Robotics
Let the Mind Conceive, Let the Hand Achieve: Following Multimodal Instructions for Long-Horizon Robotic Manipulation
Youpeng Wen, Gang Wang, Tianyu Huang et al. (+5 more)
A
robot following a multi-step instruction needs to act on the right objects in the right order, even as the scene changes. It must also maintain awareness of task progress and determine when to advance. To address these challenges, we introduce Orchestra, an
agent harness that separates task planning and execution management from action generation. Orchestra decomposes a complete
multimodal instruction into ordered subtasks, tracks completed steps, and updates the locations of relevant objects and destinations during execution. It maintains a shared context containing the active subtask, spatial grounding, and recent observations. An action policy uses this context to generate actions, while a separately trained progress evaluator estimates when to advance to the next subtask. To evaluate whether
robots trained on atomic instructions and short trajectories can execute longer subtask sequences, we introduce a
benchmark named RMBench-Compose. Orchestra achieves a mean success rate of $68.22\%$ across nine RMBench tasks, outperforming Mem-0 by $26.22$ percentage points. On RMBench-Compose, it achieves an Average Success Length (ASL) of $7.35$ subtasks, $2.46$ times that of HarnessVLA. Across four real-world multi-step tasks, it improves mean ASL by $25.0\%$ over Orchestra using the base $\pi_{0.5}$ policy. These results show that Orchestra helps
robots carry out longer sequences of subtasks while preserving the requested subtask order and acting on the specified objects. Our code and project page are available at https://youpengwen.github.io/Orchestra.
Read the full abstract ▾
All authors Youpeng Wen, Gang Wang, Tianyu Huang, Weiwei Zhang, Zhijian Huang, Wei Chen, Xieyuanli Chen, Yunhui Liu
Robotics
CoSMO: Conformal Semantic Mapping for Open-World Safe Robot Navigation
Shuqi Wang, Yiwei Zheng, Pingrui Zhang et al. (+2 more)
Safe autonomous navigation requires not only
geometric obstacle avoidance but also semantic awareness. Different object categories demand different safety clearances, and the set of relevant categories may expand as the
robot explores novel environments.
Vision-
language models (VLMs) offer rich semantic representations for building maps of unknown scenes, yet their confidence scores are poorly calibrated, making raw predictions unreliable for safety-critical planning. Conformal prediction (CP) can provide coverage guarantees, but existing methods assume a fixed label set and stationary data, limiting their use in open-world settings. We propose Conformal Semantic Mapping for Open-World Safe
Robot Navigation (CoSMO), a semantic safety mapping framework that transforms VLM
embeddings into calibrated coverage sets and safety-aware obstacle dilations over a dynamically expanding hazard vocabulary, even under distribution shift. Experiments on the Matterport3D (MP3D)
dataset show that CoSMO reduces semantic collisions by 61.6% relative to a geometry-only baseline, while maintaining formal coverage guarantees even under delayed expert feedback. Real-world experiments on a quadruped
robot further demonstrate generalization by achieving long-run empirical coverage of 90.73% against the 90% target. Project website: https://yangming911.github.io/cosmo-page/.
Read the full abstract ▾
All authors Shuqi Wang, Yiwei Zheng, Pingrui Zhang, Yue Gao, Xiang Yin
Robotics
I2O2I-Nav: A Benchmark with a Modular Agent for Long-Horizon Indoor-to-Outdoor-to-Indoor Navigation
Zhengjie Shu, Wanyu Li, Di Xiao et al. (+11 more)
Service and delivery
robots often need to complete long-horizon navigation missions spanning indoor and outdoor environments. Existing embodied navigation
benchmarks primarily evaluate isolated stages or partial cross-domain journeys, providing limited support for assessing how heterogeneous navigation capabilities compose under continuous execution. We introduce I2O2I-Nav, a physics-enabled
benchmark of 20 real-world composite scenes spanning over 115,000 square meters across multiple buildings and floors, with aligned 3DGS rendering and collision geometry. Its three-level evaluation progresses from independent primitives to two-stage compositions and complete four-stage I2O2I missions. Each stage starts from its predecessor's actual endpoint, while stage-conditional metrics track completion and localize failures along the mission. We further propose the MODULAR-
Agent that coordinates frozen navigation policies through completion verification, stage handoffs, and bounded recovery. Evaluation across 1,374 episodes highlights the difficulty of continuous composition. The MODULAR-
Agent achieves 40.5-76% success on individual primitives, but full I2O2I success remains low. Compared with a fixed policy chain using the same underlying policies, it more than doubles success on both Indoor-to-Outdoor (I2O) and Outdoor-to-Indoor (O2I), while improving full-mission success from 1.15% to 4.02%. Stage-wise analysis shows that the MODULAR-
Agent enables more missions to reach the final object-navigation stage, where conditional success remains low, revealing challenges in both long-horizon stage execution and completion decisions.
Read the full abstract ▾
All authors Zhengjie Shu, Wanyu Li, Di Xiao, Lingxin Jiang, Lai Jiang, Kaidi Hu, Xincheng Tang, Wenkang Hu, Haoran Wang, Jinchuan Zhang, Chang Chen, Wei Li, Jia Pan, Ruigang Yang
Robotics
Cross-Embodiment Affordance Transfer for Robotic Manipulation
Sawera Yaseen, Serge Thill and Rekha Raja
Robotic manipulation systems commonly learn object affordances from human-annotated
datasets, which reflect interaction possibilities for a human hand rather than the physical constraints of a specific
robot. For example, a surface region of an object identified as graspable by a human annotator may be geometrically unreachable or mechanically infeasible for a
robot, yet this mismatch is rarely addressed explicitly. To tackle this challenge, we proposed a framework that translates human affordance annotations into
robot-executable affordance predictions conditioned on the physical properties of the
robot. The network predicts which surface regions of an object are physically executable by the
robot, given a 3D point cloud of an object and a descriptor encoding the
robot's geometry and constraints. The model was trained using human-annotated affordances together with robotic execution outcomes from multiple sampled approach trials in simulation, where execution outcomes determined which regions are executable for the
robot. The evaluation was performed on a household objects
dataset using two parallel-jaw gripper configurations on a UR5e
robot arm. The framework outperformed a model that does not receive information about the
robot. The experimental results demonstrated that affordance prediction must be conditioned on the
robot and that human annotations alone are insufficient for learning
robot-executable affordance representations.
Read the full abstract ▾
Robotics
OneDemo: Unified Policy Learning and Data Generation from a Single Demonstration
Ying Qian, Jianshu Hu, Han Fang et al. (+2 more)
Learning
robot policies from demonstrations is often limited by the cost of collecting diverse, high-quality expert data. While an appealing alternative is to expand a single successful demonstration into a diverse set of successful trajectories, existing augmentation methods typically rely on manually designed transformation rules or complex trajectory planning, requiring substantial task-specific engineering and human effort. We introduce OneDemo, a reinforcement learning framework that learns to edit a single successful demonstration for new task configurations. We formulate demonstration editing as a structured reinforcement learning problem, in which an editor learns, from sparse task-success rewards, both when to transition between object-centric trajectory segments and how to geometrically adapt each segment. The learned editor serves both as a task policy and as a reusable demonstration generator. Across 15 Meta-World tasks, OneDemo reaches a 92.7% success rate within 53.7k environment interactions on average. Across 9 DexMimicGen tasks, it generates successful demonstrations with a 98.4% average success rate, and BC-RNN policies trained on the generated data achieve 99.6% average success at matched
dataset scale. These results show that learning to edit a single successful demonstration provides a structured and effective basis for both policy learning and scalable demonstration generation.
Read the full abstract ▾
All authors Ying Qian, Jianshu Hu, Han Fang, Shuguang Li, Yutong Ban
Robotics · Cross-listed from cs.SE
Assessing the Quality of Concrete Scenarios Generated from Scenario Classes for Automated Driving
Lina Putze, Thies de Graaff, Jan Steffen Becker et al. (+1 more)
A scenario-based development approach has become state of the art for automated driving systems. Once a set of scenario classes covering the operational design domain is available, concrete scenarios are generated from these scenario classes, e.g., to be used for hazard analysis and risk assessment or for the system's verification and validation. For such applications, a high diversity among the generated concrete scenarios is paramount. The question arises how the quality of collections of concrete scenarios can be assessed in such a setting. In this work, we aggregate scenario distance measures into so-called quality functions that quantify the diversity of concrete scenario collections in the context of automated driving. In particular, we consider the case where scenario classes are formalized as constraint systems and the associated concrete scenario space is not explicitly known. Based on a set of desired properties we construct candidates for such quality functions and give proofs for their property fulfillment. To show the practical relevance of quality functions, we perform a variety of experiments in the context of an abstract scenario-based virtual verification toolchain for automated driving systems. More specifically, we use quality functions to compare the effectiveness of variation methods for sampling, examine the choice of different scenario distance measures, and investigate the effects of temporal aspects on concrete scenario collection quality.
Read the full abstract ▾
All authors Lina Putze, Thies de Graaff, Jan Steffen Becker, Christian Neurohr
Robotics
Beyond Planning: Differentiable Control-in-the-loop End-to-End Autonomous Driving
Zhidong Wang, Zirui Li, Zitong Shan et al. (+6 more)
End-to-end autonomous driving has achieved remarkable progress by planning trajectories directly from raw sensor data. Yet the step from planning to control still relies on hand-tuned tracking controllers that block the gradient path from vehicle motion back to the network. This gap stems from non-differentiable vehicle dynamics, aliased supervision from coarsely sampled expert labels, rate mismatch between high-rate control and low-rate re-planning, and vehicle model discrepancies between training and deployment. This paper proposes DIVA (Differentiable Identified Vehicle dynamics for end-to-end Autonomous driving) to close this gap. An identified actuator-level vehicle model is embedded in the training loop of the end-to-end driving network, forming a differentiable chain from perception to vehicle states. A learning-based control decoder replaces the external controller and is supervised on the rolled-out vehicle states, bypassing the aliased control-input labels. A rate-consistent control-input parameterization reconciles the heterogeneous sampling rates and enables asynchronous execution with a 3.8 ms control-only cycle. On-policy re-identification and an online disturbance observer address training-time distribution shift and deployment-time model mismatch respectively. On the closed-loop Bench2Drive
benchmark, DIVA achieves a driving score of 86.7, surpassing the widely adopted state-of-the-art approach by 3.3 points while nearly halving mean jerk and reducing brake switching by 64%, and maintains superior path tracking accuracy under significant vehicle model mismatch at deployment.
Read the full abstract ▾
All authors Zhidong Wang, Zirui Li, Zitong Shan, Zhiqi Mao, Qihang Huang, Zhan Chen, Jingsong Liang, Han Yu, Chen Lv
Robotics
RESTA: Time-Scaling Augmentation for Data-Efficient Robot Imitation Learning
Yunpeng Jiang, Jianshu Hu, Han Fang et al. (+3 more)
Robot manipulation policies trained through imitation learning commonly rely on costly demonstrations recorded in one task direction and at one temporal resolution. We propose Reversal and Stride-based Trajectory Augmentation (RESTA), which exploits time-scaling data augmentation to improve data efficiency through trajectory reversal and temporal resampling. Trajectory-reversal augmentation reverses and relabels demonstrations between paired tasks with opposite objectives, reusing recorded rollouts without additional environment interaction. Because contact-rich manipulation is not exactly reversible, we treat reversal as an approximate paired-task prior rather than a physical
symmetry. Temporal-stride augmentation varies observation and action sampling strides and conditions the policy on the resulting offsets, enabling fixed or randomized temporal schedules. Experiments across 16 paired bimanual manipulation tasks show that trajectory reversal is particularly effective in the low-data regime and that temporal-stride augmentation improves success under randomized temporal schedules. With 25 and 50 independently collected demonstrations per task, RESTA achieves 78.4% and 85.2% average success, respectively, matching or exceeding baselines trained with twice as many demonstrations (77.3% and 85.4%). These results show that RESTA can improve data efficiency in
robot imitation learning.
Read the full abstract ▾
All authors Yunpeng Jiang, Jianshu Hu, Han Fang, Zhiyuan Guan, Paul Weng, Yutong Ban
Robotics
Force Sensor-Based Picking Pattern Analysis and Two-Jaw Gripper Design for Cherry Tomato Robotic Harvesting in Greenhouse
Rajmeet Singh, Manveen Kaur, Yusuf Adamu et al. (+1 more)
Robotic harvesting of cherry tomatoes in greenhouses remains challenging due to dense fruit clusters, delicate skin, and the need for gentle yet effective detachment. This study presents a force-data-driven framework for autonomous cherry tomato harvesting that integrates six-axis force/torque sensing, a bio-inspired compliant gripper, and deployment on a mobile robotic platform. A wearable ATI six-axis force/torque sensor mounted on the index finger captured kinetic signatures of three elemental picking patterns (gripping, twisting, and lateral pulling) across 150 manual trials (50 per pattern) in laboratory and greenhouse settings. Gripping required a peak normal force of 3.92 and 0.45 N, twisting produced a dominant torque of 19.20 2.10 Nmm about the pedicel axis with minimal truss disturbance, and pure lateral pulling alone was insufficient for reliable detachment. These measurements informed a novel fingertip-localised Finray gripper, with compliant TPU 95A Finray geometry restricted to the fruit-contact region and a rigid PLA body, emulating the dual-stiffness human fingertip. A Dynamixel XM430-W350 servomotor under current-based torque control applies the measured gripping force and a 45{\deg} wrist twist. Finite element analysis showed a maximum von Mises stress of 0.24 MPa under the 3.92 N load, below the bruising threshold of cherry tomato skin. Integrated on a UR5 arm and Clearpath Husky A200 base, the system was validated in NVIDIA Isaac Sim and in 50 autonomous trials at the SILAL commercial greenhouse, Abu Dhabi, UAE. The hybrid grip-twist-pull strategy achieved an 86.0% picking success rate, 2.3% fruit damage, and an 8.4 s cycle time per fruit, outperforming
vacuum (66.3%), rotating (70.1%), clamping (76.0%), and suction-cutting (52.4%) end-effectors. These results demonstrate preliminary feasibility under the tested facility and cultivar conditions
Read the full abstract ▾
All authors Rajmeet Singh, Manveen Kaur, Yusuf Adamu, Irfan Hussain