Robotics · Cross-listed from cs.CV
Safety-Aligned 3D Object Detection: Single-Vehicle, Cooperative, and End-to-End Perspectives
Brian Hsuan-Cheng Liao, Chih-Hong Cheng, Hasan Esen et al. (+1 more)
Perception plays a central role in connected and autonomous vehicles (CAVs), underpinning not only conventional modular driving stacks, but also cooperative perception systems and recent end-to-end driving models. While deep learning has greatly improved perception performance, its statistical nature makes perfect predictions difficult to attain. Meanwhile, standard training objectives and evaluation
benchmarks treat all perception errors equally, even though only a subset is safety-critical. In this paper, we investigate safety-aligned evaluation and optimization for 3D object detection that explicitly characterize high-impact errors. Building on our previously proposed safety-oriented metric, NDS-USC, and safety-aware loss function, EC-IoU, we make three contributions. First, we present an expanded study of single-vehicle 3D object detection models across diverse neural network architectures and sensing modalities, showing that gains under standard metrics such as mAP and NDS may not translate to safety-oriented criteria represented by NDS-USC. With EC-IoU, we reaffirm the benefit of safety-aware fine-tuning for improving safety-critical detection performance. Second, we conduct an ego-centric, safety-oriented evaluation of AV-infrastructure cooperative object detection models, underscoring its superiority over vehicle-only models and demonstrating a safety impact analysis that illustrates the potential contribution of cooperative models to "
Vision Zero." Third, we integrate EC-IoU into SparseDrive and show that safety-aware perception hardening can reduce collision rate by nearly 30% and improve system-level safety directly in an end-to-end perception-to-planning framework. Overall, our results indicate that safety-aligned perception evaluation and optimization offer a practical path toward enhancing CAV safety across single-vehicle, cooperative, and end-to-end autonomy settings.
Read the full abstract ▾
All authors Brian Hsuan-Cheng Liao, Chih-Hong Cheng, Hasan Esen, Alois Knoll
Robotics · Cross-listed from cs.CV
VitaTouch: Property-Aware Vision-Tactile-Language Model for Robotic Quality Inspection in Manufacturing
Junyi Zong, Qingxuan Jia, Meixian Shi et al. (+5 more)
Quality inspection in smart manufacturing requires identifying intrinsic material and surface properties beyond visible geometry, yet
vision-only methods remain vulnerable to occlusion and reflection. We propose VitaTouch, a property-aware
vision-tactile-
language model for material-property inference and natural-
language attribute description. VitaTouch uses modality-specific encoders and a dual Q-Former to extract
language-relevant visual and tactile features, which are compressed into prefix
tokens for a large
language model. We align each modality with text and explicitly couple
vision and touch through contrastive learning. We also construct VitaSet, a
multimodal dataset with 186 objects, 52k images, and 5.1k human-verified instruction-answer pairs. VitaTouch achieves the best performance on HCT and the overall TVL
benchmark, while remaining competitive on SSVTP. On VitaSet, it reaches 88.89% hardness accuracy, 75.13% roughness accuracy, and 54.81% descriptor recall; the material-description task further achieves a peak semantic similarity of 0.9009. With LoRA-based fine-tuning, VitaTouch attains 100.0%, 96.0%, and 92.0% accuracy for 2-, 3-, and 5-category defect recognition, respectively, and delivers 94.0% closed-loop recognition accuracy and 94.0% end-to-end sorting success in 100 laboratory robotic trials. More details are available at the project page: https://vitatouch.github.io/
Read the full abstract ▾
All authors Junyi Zong, Qingxuan Jia, Meixian Shi, Tong Li, Jiayuan Li, Zihang Lv, Gang Chen, Fang Deng
Robotics
From Impact to Insight: Dynamics-Aware Proprioceptive Terrain Sensing on Granular Media
Yifeng Zhang, Yue Wu, Jake Futterman et al. (+5 more)
Robots that traverse natural terrain must interpret contact forces generated under highly dynamic conditions. However, most terrain characterization approaches rely on quasi-static assumptions that neglect velocity- and acceleration-dependent effects arising during impact and rapid stance transitions. In this work, we investigate granular terrain interaction during high-speed hopping and develop a physics-based framework for dynamic terrain characterization using proprioceptive sensing alone. Through controlled hopping experiments with systematically varied impact speed and leg compliance, our measurements reveal that quasi-static based assumptions lead to large discrepancies in granular terrain property estimation during high-speed hopping, particularly upon touchdown and controller-induced stiffness transitions. Velocity-dependent drag alone cannot explain these discrepancies. Instead, acceleration-dependent added-mass effects - associated with grain entrainment beneath the foot - dominate transient force responses. We integrate this force decomposition with a momentum-observer-based
estimator that compensates for rigid-body inertia and
gravity, and introduce an acceleration-aware weighted regression to account for increased force variance during high-acceleration events. Together, these methods enable consistent recovery of granular resistance across locomotion conditions, closely matching linear-actuator ground truth. Our results demonstrate that accurate terrain inference during high-speed locomotion requires explicit treatment of acceleration-dependent granular effects, and provide a foundation for
robots to characterize complex deformable terrain during dynamic exploration of terrestrial and planetary environments.
Read the full abstract ▾
All authors Yifeng Zhang, Yue Wu, Jake Futterman, Jacob Meseha, Eduardo Rosales, Irie Cooper, J. Diego Caporale, Feifei Qian
Robotics
Tune to Learn: How Controller Gains Shape Robot Policy Learning
Antonia Bronars, Younghyo Park and Pulkit Agrawal
Position controllers have become the dominant interface for executing learned manipulation policies. Yet a critical design decision remains understudied: how should we choose controller gains for policy learning? The conventional wisdom is to select gains based on desired task compliance or stiffness. However, this logic breaks down when controllers are paired with state-conditioned policies: effective stiffness emerges from the interplay between learned reactions and control dynamics, not from gains alone. We argue that gain selection should instead be guided by learnability: how amenable different gain settings are to the learning
algorithm in use. In this work, we systematically investigate how position controller gains affect three core components of modern
robot learning pipelines: behavior cloning, reinforcement learning from scratch, and sim-to-real transfer. Through extensive experiments across multiple tasks and
robot embodiments, we find that: (1) behavior cloning benefits from compliant and overdamped gain regimes, (2) reinforcement learning can succeed across all gain regimes given compatible hyperparameter tuning, and (3) sim-to-real transfer is harmed by stiff and overdamped gain regimes. These findings reveal that optimal gain selection depends not on the desired task behavior, but on the learning paradigm employed. Project website: https://younghyopark.me/tune-to-learn
Read the full abstract ▾
Robotics
Adaptive Learned State Estimation based on KalmanNet
Arian Mehrfard, Bharanidhar Duraisamy, Stefan Haag et al. (+2 more)
Hybrid state
estimators that combine model-based Kalman filtering with learned components have shown promise on simulated data, yet their performance on real-world automotive data remains insufficient. In this work we present Adaptive Multi-modal KalmanNet (AM-KNet), an advancement of KalmanNet tailored to the multi-sensor autonomous driving setting. AM-KNet introduces sensor-specific measurement modules that enable the network to learn the distinct noise characteristics of radar,
lidar, and camera independently. A hypernetwork with context modulation conditions the filter on target type, motion state, and relative pose, allowing adaptation to diverse traffic scenarios. We further incorporate a covariance estimation branch based on the Josephs form and supervise it through negative log-likelihood losses on both the estimation error and the innovation. A comprehensive, component-wise loss function encodes physical priors on sensor reliability, target class, motion state, and measurement flow consistency. AM-KNet is trained and evaluated on the nuScenes and View-of-Delft
datasets. The results demonstrate improved estimation accuracy and tracking stability compared to the base KalmanNet, narrowing the performance gap with classical
Bayesian filters on real-world automotive data.
Read the full abstract ▾
All authors Arian Mehrfard, Bharanidhar Duraisamy, Stefan Haag, Florian Geiss, Mirko Mählisch
Robotics
F2F-AP: Flow-to-Future Asynchronous Policy for Real-time Dynamic Manipulation
Haoyu Wei, Xiuwei Xu, Ziyang Cheng et al. (+5 more)
Asynchronous inference has emerged as a prevalent paradigm in robotic manipulation, achieving significant progress in ensuring trajectory smoothness and efficiency. However, a systemic challenge remains unresolved, as inherent latency causes generated actions to inevitably lag behind the real-time environment. This issue is particularly exacerbated in dynamic scenarios, where such temporal misalignment severely compromises the policy's ability to interpret and react to rapidly evolving surroundings. In this paper, we propose a novel framework that leverages predicted object flow to synthesize future observations, incorporating a flow-based contrastive learning objective to align the visual feature representations of predicted observations with ground-truth future states. Empowered by this anticipated visual context, our asynchronous policy gains the capacity for proactive planning and motion, enabling it to explicitly compensate for latency and robustly execute manipulation tasks involving actively moving objects. Experimental results demonstrate that our approach significantly enhances responsiveness and success rates in complex dynamic manipulation tasks.
Read the full abstract ▾
All authors Haoyu Wei, Xiuwei Xu, Ziyang Cheng, Hang Yin, Angyuan Ma, Jie Zhou, Jiwen Lu, Bingyao Yu
Robotics
Backup-Based Safety Filters: A Comparative Review of Backup CBF, Model Predictive Shielding, and gatekeeper
Taekyung Kim, Aswin D. Menon, Akshunn Trivedi et al. (+1 more)
This paper revisits three backup-based safety filters – Backup Control Barrier Functions (Backup CBF), Model Predictive Shielding (MPS), and gatekeeper – through a unified comparative framework. Using a common safety-filter abstraction and shared notation, we make explicit both their common backup-policy structure and their key algorithmic differences. We compare the three methods through their filter-inactive sets, i.e., the states where the nominal policy is left unchanged. In particular, we show that MPS is a special case of gatekeeper, and we further relate gatekeeper to the interior of the Backup CBF inactive set within the implicit safe set. This unified view also highlights a key source of conservatism in backup-based safety filters: safety is often evaluated through the feasibility of a backup maneuver, rather than through the nominal policy's continued safe execution. The paper is intended as a compact tutorial and review that clarifies the theoretical connections and differences among these methods.
Read the full abstract ▾
All authors Taekyung Kim, Aswin D. Menon, Akshunn Trivedi, Dimitra Panagou
Robotics
A Dynamic Toolkit for Transmission Characteristics of Precision Reducers with Explicit Contact Geometry
Jiacheng Miao, Chao Liu, Qiliang Wang et al. (+2 more)
Precision reducers couple contact geometry, bearing support, structural deformation, and loading history. This paper presents a dynamic toolkit connecting distributed local contacts to the complete mechanical reaction path. Work-conjugate maps transfer displacement, reaction, and tangent contributions between contacts and rigid or reduced coordinates, with explicit allocation of contact and body elasticity. An implicit generalized-alpha solution distinguishes trial evaluations from accepted history. Contact records then support performance protocols and configured
geometric or constitutive feedback. The numerical studies focus on mechanical coupling and pressure recovery. An idealized annular housing retains physical interfaces while its structural coordinates are reduced. Two reductions with similar static errors have cross-port response errors of 24.975 and 0.312 percent over the same frequency band relative to a common parent model. In a shared-pin example, a 20 micrometer radial displacement of one wheel changes the load on a second, fixed wheel by approximately 75 N. Removing cross-station compliance removes this incremental transfer on the tested sleeve-seating branch. A double-wheel cycloidal assembly relates torsional branch response to aggregate contact-load variation and normalized pressure fields. The pressure maxima remain sensitive to resolution despite small discrete force residuals. Further formulations specify how motion and contact records support precision, vibration, heat, wear, and durability models with their required inputs. The framework separates model representation and numerical resolution while retaining common definitions of motion, force, and observation.
Read the full abstract ▾
All authors Jiacheng Miao, Chao Liu, Qiliang Wang, Yunhui Guan, Weidong He
Robotics
Xueying Li, Feng Lyu, Hao Wu et al. (+3 more)
Training-free
Vision-
Language Navigation (VLN)
agents powered by foundation models can follow instructions and explore 3D environments. However, existing approaches rely on greedy frontier selection and passive spatial
memory, leading to inefficient behaviors such as local oscillation and redundant revisiting. We argue that this stems from a lack of metacognitive capabilities: the
agent cannot monitor its exploration progress, diagnose strategy failures, or adapt accordingly. To address this, we propose MetaNav, a metacognitive
navigation agent integrating spatial
memory, history-aware planning, and reflective correction. Spatial
memory builds a persistent 3D semantic map. History-aware planning penalizes revisiting to improve efficiency. Reflective correction detects stagnation and uses an LLM to generate corrective rules that guide future frontier selection. Experiments on GOAT-Bench, HM3D-OVON, and A-EQA show that MetaNav achieves state-of-the-art performance while reducing VLM queries by 20.7%, demonstrating that metacognitive
reasoning significantly improves robustness and efficiency.
Read the full abstract ▾
All authors Xueying Li, Feng Lyu, Hao Wu, Mingliu Liu, Jia-Nan Liu, Guozi Liu
Robotics
Deep Neural Network Based Roadwork Detection for Autonomous Driving
Sebastian Wullrich, Nicolai Steinke and Daniel Goehring
Road construction sites create major challenges for both autonomous vehicles and human drivers due to their highly dynamic and heterogeneous nature. This paper presents a real-time system that detects and localizes roadworks by combining a YOLO neural network with
LiDAR data. The system identifies individual roadwork objects while driving, merges them into coherent construction sites and records their outlines in world coordinates. The model training was based on an adapted US
dataset and a new
dataset collected from test drives with a prototype vehicle in Berlin, Germany. Evaluations on real-world road construction sites showed a localization accuracy below 0.5 m. The system can support traffic authorities with up-to-date roadwork data and could enable autonomous vehicles to navigate construction sites more safely in the future.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
Model-Based Reinforcement Learning for Control under Time-Varying Dynamics
Klemens Iten, Bruce Lee, Chenhao Li et al. (+3 more)
Learning-based control methods typically assume stationary system dynamics, an assumption often violated in real-world systems due to drift, wear, or changing operating conditions. We study reinforcement learning for control under time-varying dynamics. We consider a continual model-based reinforcement learning setting in which an
agent repeatedly learns and controls a dynamical system whose transition dynamics evolve across episodes. We analyze the problem using Gaussian process dynamics models under frequentist variation-budget assumptions. Our analysis shows that persistent non-stationarity requires explicitly limiting the influence of outdated data to maintain calibrated uncertainty and meaningful dynamic regret guarantees. Motivated by these insights, we propose a practical optimistic model-based reinforcement learning
algorithm with adaptive data buffer mechanisms and demonstrate improved performance on continuous control
benchmarks with non-stationary dynamics.
Read the full abstract ▾
All authors Klemens Iten, Bruce Lee, Chenhao Li, Lenart Treven, Andreas Krause, Bhavya Sukhija
Robotics
A virtual-variable-length method for robust inverse kinematics of multi-segment continuum robots
Weiting Feng, Federico Renda, Yunjie Yang et al. (+1 more)
This paper proposes a new, robust method to solve the inverse kinematics (IK) of multi-segment continuum manipulators. Conventional Jacobian-based solvers, especially when initialized from neutral/rest configurations, often exhibit slow convergence and, in certain conditions, may fail to converge (deadlock). The Virtual-Variable-Length (VVL) method proposed here introduces fictitious variations of segments' length during the solution iteration, conferring virtual axial degrees of freedom that alleviate adverse behaviors and constraints, thus enabling or accelerating convergence. Comprehensive numerical experiments were conducted to compare the VVL method against
benchmark Jacobian-based and Damped Least Square IK solvers. Across more than $1.8\times 10^6$ randomized trials covering manipulators with two to seven segments, the proposed approach achieved up to a 20$\%$ increase in convergence success rate over the
benchmark and a 40-80$\%$ reduction in average iteration count under equivalent accuracy thresholds ($10^{-4}-10^{-8}$). While deadlocks are not restricted to workspace boundaries and may occur at arbitrary poses, our empirical study identifies boundary-proximal configurations as a frequent cause of failed convergence and the VVL method mitigates such occurrences over a statistical sample of test cases.
Read the full abstract ▾
All authors Weiting Feng, Federico Renda, Yunjie Yang, Francesco Giorgio-Serchi
Robotics · Cross-listed from cs.CV
Qiyao Zhang, Shuhua Zheng, Jianli Sun et al. (+6 more)
Embodied visual tracking is crucial for Unmanned Aerial Vehicles (
UAVs) executing complex real-world tasks. In dynamic urban scenarios with complex semantic requirements,
Vision-
Language-Action (VLA) models show great promise due to their cross-modal fusion and continuous action generation capabilities. To
benchmark multimodal tracking in such environments, we construct a dedicated evaluation
benchmark and a large-scale
dataset encompassing over 890K frames, 176 tasks, and 85 diverse objects. Furthermore, to address temporal feature redundancy and the lack of spatial
geometric priors in existing VLA models, we propose an improved VLA tracking model,
UAV-Track VLA. Built upon the $\pi_{0.5}$ architecture, our model introduces a temporal compression net to efficiently capture inter-frame dynamics. Additionally, a parallel dual-branch decoder comprising a spatial-aware auxiliary grounding head and a flow matching action expert is designed to decouple cross-modal features and generate fine-grained continuous actions. Systematic experiments in the CARLA simulator validate the superior end-to-end performance of our method. Notably, in challenging long-distance pedestrian tracking tasks,
UAV-Track VLA achieves a 61.76% success rate and 269.65 average tracking frames, significantly outperforming existing baselines. Furthermore, it demonstrates robust zero-shot generalization in unseen environments and reduces single-step inference latency by 33.4% (to 0.0571s) compared to the original $\pi_{0.5}$, enabling highly efficient, real-time
UAV control. Data samples and demonstration
videos are available at: https://github.com/Hub-Tian/
UAV-Track_VLA.
Read the full abstract ▾
All authors Qiyao Zhang, Shuhua Zheng, Jianli Sun, Chengxiang Li, Xianke Wu, Zihan Song, Zhiyong Cui, Yisheng Lv, Yonglin Tian
Robotics · Cross-listed from cs.CV
UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
Yongkang Li, Lijun Zhou, Sixu Yan et al. (+11 more)
Vision-
Language-Action (VLA) models have recently emerged in autonomous driving, with the promise of leveraging rich world knowledge to improve the cognitive capabilities of driving systems. However, adapting such models for driving tasks currently faces a critical dilemma between spatial perception and semantic
reasoning. Consequently, existing VLA systems are forced into suboptimal compromises: directly adopting 2D
Vision-
Language Models yields limited spatial perception, whereas enhancing them with 3D spatial representations often impairs the native
reasoning capacity of VLMs. We argue that this dilemma largely stems from the coupled optimization of spatial perception and semantic
reasoning within shared model parameters. To overcome this, we propose UniDriveVLA, a Unified Driving
Vision-
Language-Action model based on Mixture-of-
Transformers that addresses the perception-
reasoning conflict via expert decoupling. Specifically, it comprises three experts for driving understanding, scene perception, and action planning, which are coordinated through masked joint
attention. In addition, we combine a sparse perception paradigm with a three-stage progressive training strategy to improve spatial perception while maintaining semantic
reasoning capability. Extensive experiments show that UniDriveVLA achieves state-of-the-art performance in open-loop evaluation on nuScenes and closed-loop evaluation on Bench2Drive. Moreover, it demonstrates strong performance across a broad range of perception, prediction, and understanding tasks, including 3D detection, online mapping, motion
forecasting, and driving-oriented VQA, highlighting its broad applicability as a unified model for autonomous driving. Code and model have been released at https://github.com/xiaomi-research/unidrivevla
Read the full abstract ▾
All authors Yongkang Li, Lijun Zhou, Sixu Yan, Bencheng Liao, Tianyi Yan, Kaixin Xiong, Long Chen, Hongwei Xie, Bing Wang, Guang Chen, Hangjun Ye, Wenyu Liu, Haiyang Sun, Xinggang Wang
Robotics
PRO-SPECT: Probabilistically Safe Scalable Planning for Energy-Aware Coordinated UAV-UGV Teams in Stochastic Environments
Roger Fowler, Cahit Ikbal Er, Benjamin Johnsenberg et al. (+1 more)
We consider energy-aware planning for an unmanned aerial vehicle (
UAV) and unmanned ground vehicle (UGV) team operating in a
stochastic environment. The
UAV must visit a set of air points in minimum time while respecting energy constraints, relying on the UGV as a mobile charging station. Unlike prior work that assumed deterministic travel times or used fixed robustness margins, we model travel times as random variables and bound the probability of failure (energy depletion) across the entire mission to a user-specified risk level. We formulate the problem as a Mixed-Integer Program and propose PRO-SPECT, a
polynomial-time
algorithm that generates risk-bounded plans. The
algorithm supports both offline planning and online re-planning, enabling the team to adapt to disturbances while preserving the risk bound. We provide theoretical results on solution feasibility and time complexity. We also demonstrate the performance of our method via numerical comparisons and simulations.
Read the full abstract ▾
All authors Roger Fowler, Cahit Ikbal Er, Benjamin Johnsenberg, Yasin Yazicioglu
Robotics
ROS 2-Based LiDAR Perception Framework for Mobile Robots in Dynamic Production Environments, Utilizing Synthetic Data Generation, Transformation-Equivariant 3D Detection and Multi-Object Tracking
Lukas Bergs, Tan Chung, Marmik Thakkar et al. (+4 more)
Adaptive
robots in dynamic production environments require robust perception capabilities, including 6D pose estimation and multi-object tracking. To address limitations in real-world data dependency, noise robustness, and spatiotemporal consistency, a
LiDAR framework based on the
Robot Operating System integrating a synthetic-data-trained Transformation-Equivariant 3D Detection with multi-object-tracking leveraging center poses is proposed. Validated across 72 scenarios with motion capture technology, overall results yield an Intersection over Union of 62.6% for standalone pose estimation, rising to 83.12% with multi-object-tracking integration. Our
LiDAR-based framework achieves 91.12% of Higher Order Tracking Accuracy, advancing robustness and versatility of
LiDAR-based perception systems for industrial mobile manipulators.
Read the full abstract ▾
All authors Lukas Bergs, Tan Chung, Marmik Thakkar, Alexander Moriz, Amon Göppert, Chinnawut Nantabut, Robert Schmitt
Robotics
Cross-Modal Visuo-Tactile Object Perception
Anirvan Dutta, Simone Tasciotti, Claudia Cusseddu et al. (+6 more)
Estimating physical properties is critical for safe and efficient autonomous robotic manipulation, particularly during contact-rich interactions. In such settings,
vision and tactile sensing provide complementary information about object geometry, pose, inertia, stiffness, and contact dynamics, such as stick-slip behavior. However, these properties are only indirectly observable and cannot always be modeled precisely (e.g., deformation in non-rigid objects coupled with nonlinear contact friction), making the estimation problem inherently complex and requiring sustained exploitation of visuo-tactile sensory information during action. Existing visuo-tactile perception frameworks have primarily emphasized forceful sensor fusion or static cross-modal
alignment, with limited consideration of how uncertainty and beliefs about object properties evolve over time. Inspired by human multi-sensory perception and active inference, we propose the Cross-Modal Latent Filter (CMLF) to learn a structured, causal latent state-space of physical object properties. CMLF supports bidirectional transfer of cross-modal priors between
vision and touch and integrates sensory evidence through a
Bayesian inference process that evolves over time. Real-world robotic experiments demonstrate that CMLF improves the efficiency and robustness of latent physical properties estimation under uncertainty compared to baseline approaches. Beyond performance gains, the model exhibits perceptual coupling phenomena analogous to those observed in humans, including susceptibility to cross-modal illusions and similar trajectories in learning cross-sensory associations. Together, these results constitutes a significant step toward generalizable, robust and physically consistent cross-modal integration for robotic multi-sensory perception.
Read the full abstract ▾
All authors Anirvan Dutta, Simone Tasciotti, Claudia Cusseddu, Ang Li, Panayiota Poirazi, Julijana Gjorgjieva, Etienne Burdet, Patrick van der Smagt, Mohsen Kaboli
Robotics
HyVGGT-VO: Hybrid Dense Visual Odometry With Classical VO and Feed-Forward Model
Junxiang Pang, Lipu Zhou, Baojie Chen et al. (+1 more)
Dense visual odometry (VO), which provides pose estimation and dense 3D reconstruction, serves as the cornerstone for applications ranging from robotics to augmented reality. Recently, feed-forward models have demonstrated remarkable capabilities in dense mapping. However, when these models are used in dense visual SLAM systems, their keyframe-based processing often leads to sparse pose outputs, limiting continuous per-frame pose estimation. In contrast, traditional sparse methods provide high-frequency pose outputs but lack the capability for dense reconstruction. To address these limitations, we propose HyVGGT-VO, a novel framework that combines the computational efficiency of sparse VO with the dense reconstruction capabilities of feed-forward model. To the best of our knowledge, this is the first work to tightly couple a traditional VO frontend with VGGT. Specifically, we design an adaptive hybrid tracking frontend that dynamically switches between traditional optical flow and the VGGT model to ensure robustness. Furthermore, we introduce a hierarchical optimization framework that jointly refines VO poses and the scale of VGGT predictions to ensure global scale consistency. Under the odometry setting, compared with VGGT-SLAM 2.0, HyVGGT-VO reduces the average ATE by approximately 91% on EuRoC MAV and 20% on TUM RGB-D, and decreases the average relative translation and rotation errors by 46% and 92% on KITTI, respectively, while achieving 16.1 Hz pose output and 3.2 Hz dense-mapping updates with 6.5 GB of
memory on an 8 GB laptop GPU.Our code is available at https://geneta2580.github.io/HyVGGT-VO.io.
Read the full abstract ▾
All authors Junxiang Pang, Lipu Zhou, Baojie Chen, Zhenzhong Wei
Robotics · Cross-listed from cs.CV
CompassAD: Intent-Driven 3D Affordance Grounding in Functionally Competing Objects
Jingliang Li, Jindou Jia, Tuo An et al. (+7 more)
When told to "cut the cake," a
robot must choose the knife over nearby scissors, despite both objects affording the same cutting function. In real-world scenes, multiple objects may share identical affordances, yet only one is appropriate under the given task context. We call such cases confusing pairs. However, existing 3D affordance methods largely sidestep this challenge by evaluating isolated single objects, often with explicit category names provided in the query. We formalize Intent-Driven Confusable Affordance Grounding, a new 3D affordance setting that requires predicting a per-point affordance mask on the correct object within a multi-object point cloud, conditioned on implicit natural
language intent. To study this problem, we construct CompassAD, the first
benchmark centered on implicit intent in confusing multi-object compositions. It comprises 30 confusing object pairs spanning 16 affordance types, 6,422 compositions, and 88K+ query-answer pairs. Furthermore, we propose CompassNet, a framework that incorporates two dedicated modules tailored to this task. Instance-bounded Cross Injection (ICI) constrains
language-geometry
alignment within object boundaries to prevent cross-object semantic leakage. Bi-level Contrastive Refinement (BCR) enforces discrimination at both
geometric-group and point levels, sharpening distinctions between target and confusable surfaces. Extensive experiments demonstrate state-of-the-art results on both seen and unseen queries, and deployment on a robotic manipulator confirms effective transfer to real-world grasping in confusing multi-object compositions.
Read the full abstract ▾
All authors Jingliang Li, Jindou Jia, Tuo An, Chuhao Zhou, Xiangyu Chen, Shilin Shan, Boyu Ma, Bofan Lyu, Gen Li, Jianfei Yang
Robotics
O-ConNet: Geometry-Aware End-to-End Inference of Over-Constrained Spatial Mechanisms
Haoyu Sun, Meng Zhao, Tianhao Wang et al. (+1 more)
Deep learning has shown strong potential for scientific discovery, but its ability to model macroscopic rigid-body kinematic constraints remains underexplored. We study this problem on spatial over-constrained mechanisms and propose O-ConNet, an end-to-end framework that infers mechanism structural parameters from only three sparse reachable points while reconstructing the full motion trajectory, without explicitly solving constraint equations during inference. On a self-constructed Bennett 4R
dataset of 42,860 valid samples, O-ConNet achieves Param-MAE 0.276 +/- 0.077 and Traj-MAE 0.145 +/- 0.018 (mean +/- std over 10 runs), outperforming the strongest sequence baseline (LSTM-Seq2Seq) by 65.1 percent and 88.2 percent, respectively. These results suggest that end-to-end learning can capture closed-loop
geometric structure and provide a practical route for inverse design of spatial over-constrained mechanisms under extremely sparse observations.
Read the full abstract ▾
All authors Haoyu Sun, Meng Zhao, Tianhao Wang, Jianxu Wu
Robotics
Bridging Discrete Planning and Continuous Execution for Redundant Robot
Teng Yan, Yue Yu, Yihan Liu et al. (+1 more)
Voxel-grid reinforcement learning is widely adopted for path planning in redundant manipulators due to its simplicity and reproducibility. However, direct execution through point-wise numerical inverse kinematics on 7-DoF arms often yields step-size jitter, abrupt joint transitions, and instability near singular configurations. This work proposes a bridging framework between discrete planning and continuous execution without modifying the discrete planner itself. On the planning side, step-normalized 26-neighbor Cartesian actions and a
geometric tie-breaking mechanism are introduced to suppress unnecessary turns and eliminate step-size oscillations. On the execution side, a task-priority damped least-squares (TP-DLS) inverse kinematics layer is implemented. This layer treats end-effector position as a primary task, while posture and joint centering are handled as subordinate tasks projected into the null space, combined with trust-region clipping and joint velocity constraints. On a 7-DoF manipulator in random sparse, medium, and dense environments, this bridge raises planning success in dense scenes from about 0.58 to 1.00, shortens representative path length from roughly 1.53 m to 1.10 m, and while keeping end-effector error below 1 mm, reduces peak joint accelerations by over an order of magnitude, substantially improving the continuous execution quality of voxel-based RL paths on redundant manipulators.
Read the full abstract ▾
All authors Teng Yan, Yue Yu, Yihan Liu, Bingzhuo Zhong
Robotics
Integrated Identification of Collaborative Robots for Robot Assisted 3D Printing Processes
Alessandro Dimauro, Davide Tebaldi, Fabio Pini et al. (+2 more)
In recent years, the integration of additive manufacturing (AM) and industrial robotics has opened new perspectives for the production of complex components, particularly in the automotive sector.
Robot-assisted additive manufacturing processes overcome the dimensional and kinematic limitations of traditional Cartesian systems, enabling non-planar deposition and greater
geometric flexibility. However, the increasing dynamic complexity of robotic manipulators introduces challenges related to precision, control, and error prediction. This work proposes a model-based approach equipped with an integrated identification procedure of the system's parameters, including the
robot, the actuators and the controllers. We show that the integrated modeling procedure allows to obtain a reliable dynamic model even in the presence of sensory and programming limitations typical of collaborative
robots. The manipulator's dynamic model is identified through an integrated five step methodology: starting with
geometric and inertial analysis, followed by friction and controller parameters identification, all the way to the remaining parameters identification. The proposed procedure intrinsically ensures the physical consistency of the identified parameters. The identification approach is validated on a real world case study involving a 6-Degrees-Of-Freedom (DoFs) collaborative
robot used in a thermoplastic extrusion process. The very good matching between the experimental results given by actual
robot and those given by the identified model shows the potential enhancement of precision, control, and error prediction in
Robot Assisted 3D Printing Processes.
Read the full abstract ▾
All authors Alessandro Dimauro, Davide Tebaldi, Fabio Pini, Luigi Biagiotti, Francesco Leali
Robotics · Cross-listed from cs.LG
World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry
Yuejiang Liu, Fan Feng, Lingjing Kong et al. (+6 more)
General-purpose world models promise scalable policy evaluation, optimization, and planning, yet achieving the required level of robustness remains challenging. Unlike policy learning which primarily focuses on optimal actions, a world model needs to be reliable over a vast space of suboptimal actions, which are often underrepresented in action-labeled
robot interactions. To address this challenge, we propose World Action Verifier (WAV), a framework that enables world models to identify their own prediction errors and self-improve. The key idea is to decompose action-conditioned state prediction into two independently verifiable factors: state plausibility and action reachability. We show that verifying these factors is significantly more tractable than direct forward prediction due to two underlying asymmetries: the broader availability of action-free data and the lower dimensionality of action-relevant features. Leveraging these asymmetries, we augment a world model with (i) a diverse subgoal generator obtained from
video corpora and (ii) a sparse inverse model that infers actions from a subset of state features. By enforcing cycle consistency among proposed subgoals, inferred actions, and forward rollouts, WAV provides an effective verification mechanism in under-explored regimes, where existing methods often fail. Across nine tasks spanning MiniGrid, RoboMimic, and ManiSkill, our method achieves 2x higher sample efficiency while improving downstream policy performance by over 22%.
Read the full abstract ▾
All authors Yuejiang Liu, Fan Feng, Lingjing Kong, Weifeng Lu, Jinzhou Tang, Kun Zhang, Kevin Murphy, Chelsea Finn, Yilun Du
Robotics · Cross-listed from cs.CV
Ego-Grounding for Personalized Question-Answering in Egocentric Videos
Junbin Xiao, Shenglang Zhang, Pengxiang Zhu et al. (+1 more)
We present the first systematic analysis of
multimodal large
language models (MLLMs) in personalized question-answering requiring ego-grounding - the ability to understand the camera-wearer in
egocentric videos. To this end, we introduce MyEgo, the first
egocentric VideoQA
dataset designed to evaluate MLLMs' ability to understand, remember, and reason about the camera wearer. MyEgo comprises 541 long
videos and 5K personalized questions asking about "my things", "my activities", and "my past". Benchmarking reveals that competitive MLLMs across variants, including open-source vs. proprietary, thinking vs. non-thinking, small vs. large scales all struggle on MyEgo. Top closed- and open-source models (e.g., GPT-5 and Qwen3-VL) achieve only~46% and 36% accuracy, trailing human performance by near 40% and 50% respectively. Surprisingly, neither explicit
reasoning nor model scaling yield consistent improvements. Models improve when relevant evidence is explicitly provided, but gains drop over time, indicating limitations in tracking and remembering "me" and "my past". These findings collectively highlight the crucial role of ego-grounding and long-range
memory in enabling personalized QA in
egocentric videos. We hope MyEgo and our analyses catalyze further progress in these areas for
egocentric personalized assistance. Data and code are available at https://github.com/Ryougetsu3606/MyEgo
Read the full abstract ▾
All authors Junbin Xiao, Shenglang Zhang, Pengxiang Zhu, Angela Yao
Robotics · Cross-listed from cs.CV
On Learning Spatial Structure from Pre-Beamforming Per-Antenna Range-Doppler Radar Measurements
George Sebastian, Philipp Berthold, Bianca Forkel et al. (+2 more)
Automotive radar perception pipelines commonly construct angle-domain representations via beamforming before applying learning-based models. This work instead investigates a representational question: can meaningful spatial structure be learned directly from pre-beamforming per-
antenna range-Doppler (RD) measurements? Experiments are conducted on a 6-TX x 8-RX (48 virtual
antennas) commodity automotive radar employing an A/B chirp-sequence frequency-modulated continuous-wave (CS-FMCW) transmit scheme, in which the effective transmit aperture varies between chirps (single-TX vs multi-TX), enabling controlled analyses of chirp-dependent transmit configurations. We operate on pre-beamforming per-
antenna RD tensors using a dual-chirp shared-weight encoder trained in an end-to-end, fully data-driven manner, and evaluate spatial recoverability using bird's-eye-view (BEV) occupancy as a
geometric probe rather than a performance-driven objective. Supervision is visibility-aware and cross-modal, derived from
LiDAR with explicit modeling of the radar field-of-view and occlusion-aware
LiDAR observability via ray-based visibility. Through analyses of signal properties, transmit configurations (A-only, B-only, and A+B), receive aperture, and range-Doppler structure, together with physics-aligned baselines, we investigate the factors influencing spatial recoverability. The results indicate that meaningful spatial structure is recoverable from pre-beamforming per-
antenna RD tensors under the studied A/B CS-FMCW radar configuration through learned spatial mixing, without relying on hand-crafted signal-processing stages.
Read the full abstract ▾
All authors George Sebastian, Philipp Berthold, Bianca Forkel, Leon Pohl, Mirko Maehlisch
Robotics · Cross-listed from eess.SY
Global Geometry of Orthogonal Foliations of Signed-Quadratic Systems
Antonio Franchi
This work formalizes the differential
topology of redundancy resolution for systems governed by signed-quadratic actuation maps. By analyzing the minimally redundant case, the global
topology of the continuous fiber bundle defining the nonlinear actuation null-space is established. The distribution orthogonal to these fibers is proven to be globally integrable and governed by an exact logarithmic potential field. This field foliates the actuator space, inducing a structural stratification of all orthants into transverse layers whose combinatorial sizes follow a strictly binomial progression. Within these layers, adjacent orthants are continuously connected via lower-dimensional strata termed reciprocal hinges, while the layers themselves are separated by boundary hyperplanes, or portals, that act as global sections of the fibers. This partition formally distinguishes extremal and transitional layers, which exhibit fundamentally distinct fiber
topologies and foliation properties. Exploiting this
geometric framework, we prove that the orthogonal
manifolds within the extremal orthants form a global diffeomorphism to the entire unbounded task space. This establishes the theoretical existence of globally smooth right-inverses that permanently confine the system to a single orthant, guaranteeing the absolute avoidance of kinematic singularities. While motivated by the physical actuation maps of multirotor and marine vehicles, the results provide a strictly foundational topological classification of signed-quadratic surjective systems.
Read the full abstract ▾
Robotics
Posterior Optimization with Clipped Objective for Bridging Efficiency and Stability in Generative Policy Learning
Yuhui Chen, Haoran Li, Zhennan Jiang et al. (+4 more)
Expressive generative models have advanced robotic manipulation by capturing complex, multi-modal action distributions over temporally extended trajectories. However, fine-tuning these policies via RL remains challenging due to instability and sample inefficiency. We introduce Posterior Optimization with Clipped Objective (POCO), a principled RL framework that formulates policy improvement as a posterior inference problem tailored for temporal action chunks. Through an Expectation-Maximization procedure, POCO distills a reward-weighted implicit posterior into the policy without likelihood estimation. Furthermore, POCO adopts an offline-to-online paradigm that anchors online exploration to pre-trained priors, and its model-agnostic design scales to fine-tune large VLA models without architectural modifications. Evaluations across 7 simulation
benchmarks and 4 contact-rich real-world tasks demonstrate that POCO prevents catastrophic policy collapse, outperforms SOTA baselines, and achieves a 96.7% success rate on real-world tasks.
Videos are available at our project website https://cccedric.github.io/poco/.
Read the full abstract ▾
All authors Yuhui Chen, Haoran Li, Zhennan Jiang, Yuxing Qin, Yuxuan Wan, Weiheng Liu, Dongbin Zhao
Robotics
Preferential Bayesian Optimization with Crash Feedback
Johanna Menn, David Stenger and Sebastian Trimpe
Bayesian optimization is a popular black-box optimization method for parameter learning in control and robotics. It typically requires an objective function that reflects the user's optimization goal. However, in practical applications, this objective function is often inaccessible due to complex or unmeasurable performance metrics. Preferential
Bayesian optimization (PBO) overcomes this limitation by leveraging human feedback through pairwise comparisons, eliminating the need for explicit performance quantification. When applying PBO to hardware systems, such as in quadcopter control, crashes can cause time-consuming experimental resets, wear and tear, or otherwise undesired outcomes. Standard PBO methods cannot incorporate feedback from such crashed experiments, resulting in the exploration of parameters that frequently lead to experimental crashes. We thus introduce CrashPBO, a user-friendly mechanism that enables users to both express preferences and report crashes during the optimization process. Benchmarking on synthetic functions shows that this mechanism reduces crashes by 63% and increases data efficiency. Through experiments on three robotics platforms, we demonstrate the wide applicability and transferability of CrashPBO, highlighting that it provides a flexible, user-friendly framework for parameter learning with human feedback on preferences and crashes.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning
Yang Zhou, Xiaofeng Wang, Hao Shao et al. (+8 more)
Recently, world-action models (WAM) have emerged to bridge
vision-
language-action (VLA) models and world models, unifying their
reasoning and instruction-following capabilities and spatio-temporal world modeling. However, existing WAM approaches often focus on modeling 2D appearance or latent representations, with limited
geometric grounding-an essential element for
embodied systems operating in the physical world. We present DriveDreamer-Policy, a unified driving world-action model that integrates depth generation, future
video generation, and motion planning within a single modular architecture. The model employs a large
language model to process
language instructions, multi-view images, and actions, followed by three lightweight generators that produce depth, future
video, and actions. By learning a geometry-aware world representation and using it to guide both future prediction and planning within a unified framework, the proposed model produces more coherent imagined futures and more informed driving actions, while maintaining modularity and controllable latency. Experiments on the Navsim v1 and v2
benchmarks demonstrate that DriveDreamer-Policy achieves strong performance on both closed-loop planning and world generation tasks. In particular, our model reaches 89.2 PDMS on Navsim v1 and 88.7 EPDMS on Navsim v2, outperforming existing world-model-based approaches while producing higher-quality future
video and depth predictions. Ablation studies further show that explicit depth learning provides complementary benefits to
video imagination and improves planning robustness.
Read the full abstract ▾
All authors Yang Zhou, Xiaofeng Wang, Hao Shao, Letian Wang, Guosheng Zhao, Jiangnan Shao, Jiagang Zhu, Tingdong Yu, Zheng Zhu, Guan Huang, Steven L. Waslander
Robotics
Realistic Lip Motion Generation Based on 3D Dynamic Viseme and Coarticulation Modeling for Human-Robot Interaction
Sheng Li, Jingcheng Huang and Min Li
Realistic lip synchronization is essential for the natural human-
robot non-verbal interaction of
humanoid robots. Motivated by this need, this paper presents a lip motion generation framework based on 3D dynamic viseme and coarticulation modeling. By analyzing Chinese pronunciation theory, a 3D dynamic viseme library is constructed based on the ARKit standard, which offers coherent prior trajectories of lips. To resolve motion conflicts within continuous
speech streams, a coarticulation mechanism is developed by incorporating initial-final (Shengmu-Yunmu) decoupling and energy modulation. After developing a strategy to retarget high-dimensional spatial lip motion to a 14-DOF lip actuation system of a
humanoid head platform, the efficiency and accuracy of the proposed architecture is experimentally validated and demonstrated with quantitative ablation experiments using the metrics of the Pearson Correlation Coefficient (PCC) and the Mean Absolute Jerk (MAJ). This research offers a lightweight, efficient, and highly practical paradigm for the
speech-driven lip motion generation of
humanoid robots. The 3D dynamic viseme library and real-world deployment
videos are available at {https://github.com/yuesheng21/Phoneme-to-Lip-14DOF}
Read the full abstract ▾