Robotics
UltraDexGrasp: Learning Universal Dexterous Grasping for Bimanual Robots with Synthetic Data
Sizhe Yang, Yiman Xie, Zhixuan Liang et al. (+4 more)
Grasping is a fundamental capability for
robots to interact with the physical world. Humans, equipped with two hands, autonomously select appropriate grasp strategies based on the shape, size, and weight of objects, enabling robust grasping and subsequent manipulation. In contrast, current robotic grasping remains limited, particularly in multi-strategy settings. Although substantial efforts have targeted parallel-gripper and single-hand grasping, dexterous grasping for bimanual
robots remains underexplored, with data being a primary bottleneck. Achieving physically plausible and geometrically conforming grasps that can withstand external wrenches poses significant challenges. To address these issues, we introduce UltraDexGrasp, a framework for universal dexterous grasping with bimanual
robots. The proposed data-generation pipeline integrates optimization-based grasp synthesis with planning-based demonstration generation, yielding high-quality and diverse trajectories across multiple grasp strategies. With this framework, we curate UltraDexGrasp-20M, a large-scale, multi-strategy grasp
dataset comprising 20 million frames across 1,000 objects. Based on UltraDexGrasp-20M, we further develop a simple yet effective grasp policy that takes point clouds as input, aggregates scene features via unidirectional
attention, and predicts control commands. Trained exclusively on synthetic data, the policy achieves robust zero-shot sim-to-real transfer and consistently succeeds on novel objects with varied shapes, sizes, and weights, attaining an average success rate of 81.2% in real-world universal dexterous grasping. To facilitate future research on grasping with bimanual
robots, we open-source the data generation pipeline at https://github.com/InternRobotics/UltraDexGrasp.
Read the full abstract ▾
All authors Sizhe Yang, Yiman Xie, Zhixuan Liang, Yang Tian, Jia Zeng, Dahua Lin, Jiangmiao Pang
Robotics
Constraint-Free Static Modeling of Continuum Parallel Robot
Lingxiao Xun, Matyas Diezinger, Azad Artinian et al. (+2 more)
Continuum parallel
robots (CPR) combine rigid actuation mechanisms with multiple elastic rods in a closed-loop
topology, making forward statics challenging when rigid–continuum junctions are enforced by explicit kinematic constraints. Such constraint-based formulations typically introduce additional algebraic variables and complicate both numerical solution and downstream control. This paper presents a
geometric exact, configuration-based and constraint-free static model of CPR that remains valid under geometrically nonlinear, large-deformation and large-rotation conditions. Connectivity constraints are eliminated by kinematic
embedding, yielding a reduced unconstrained problem. Each rod of CPR is discretized by nodal poses on SE(3), while the element-wise strain field is reconstructed through a linear strain parameterization. A fourth-order Magnus approximation yields an explicit and geometrically consistent mapping between element end poses and the strain. Rigid attachments at the motor-driven base and the end-effector platforms are handled through kinematic
embeddings. Based on total potential energy and virtual work, we derive assembly-ready residuals and explicit Newton tangents, and solve the resulting nonlinear
equilibrium equations using a Riemannian Newton iteration on the product
manifold. Experiments on a three-servomotor, six-rod prototype validate the model by showing good agreement between simulation and measurements for both unloaded motions and externally loaded cases.
Read the full abstract ▾
All authors Lingxiao Xun, Matyas Diezinger, Azad Artinian, Guillaume Laurent, Brahim Tamadazte
Robotics
Latent Policy Steering through One-Step Flow Policies
Hokyun Im, Andrey Kolobov, Jianlong Fu et al. (+1 more)
Offline reinforcement learning (RL) allows
robots to learn from offline
datasets without risky exploration. Yet, offline RL's performance often hinges on a brittle trade-off between (1) return maximization, which can push policies outside the
dataset support, and (2) behavioral constraints, which typically require sensitive hyperparameter tuning. Latent steering offers a structural way to stay within the
dataset support during RL, but existing offline adaptations commonly approximate action values using latent-space critics learned via indirect
distillation, which can lose information and hinder convergence. We propose Latent Policy Steering (LPS), which enables high-fidelity latent policy improvement by backpropagating original-action-space Q-gradients through a differentiable one-step MeanFlow policy to update a latent-action-space actor. By eliminating proxy latent critics, LPS allows an original-action-space critic to guide end-to-end latent-space optimization, while the one-step MeanFlow policy serves as a behavior-constrained generative prior. This decoupling yields a robust method that works out-of-the-box with minimal tuning. Across OGBench and real-world robotic tasks, LPS achieves state-of-the-art performance and consistently outperforms behavioral cloning and strong latent steering baselines.
Read the full abstract ▾
All authors Hokyun Im, Andrey Kolobov, Jianlong Fu, Youngwoon Lee
Robotics
Online Self-Training for Co-Adaptation in Hierarchical Diffusion Policies
Clemence Grislain, Mathilde Kappel, Olivier Sigaud et al. (+1 more)
Hierarchical policies decompose
language-conditioned long-horizon robotic manipulation into a high-level planner and a low-level controller. However, effective coordination between HL and LL requires that both components operate on compatible subgoal distributions. We propose ORCHID, a self-training framework that enables stable online improvement of hierarchical
diffusion policies by aligning planning and control through iterative refinement. By filtering policy samples via environment feedback, ORCHID identifies trajectories where the planner and controller are jointly successful and distills them back into both modules via supervised learning. This process induces a bidirectional co-adaptation: the planner grounds its subgoals in the actual reaching capabilities of the controller, while the controller specializes in the trajectory structures the planner produces. By relying on supervised
distillation of filtered on-policy samples, ORCHID avoids the instability typical of online hierarchical gradient-based RL training with
diffusion models. On the CALVIN
benchmark, ORCHID allows a lightweight, initially weak model to outperform pure offline methods, including a
Vision-
Language-Action model twice its size.
Read the full abstract ▾
All authors Clemence Grislain, Mathilde Kappel, Olivier Sigaud, Mohamed Chetouani
Robotics
From Code to Road: A Vehicle-in-the-Loop and Digital Twin-Based Framework for Central Car Server Testing in Autonomous Driving
Chengdong Wu, Sven Kirchner, Nils Purschke et al. (+9 more)
Simulation is one of the most essential parts in the development stage of automotive software. However, purely virtual simulations often struggle to accurately capture all real-world factors due to limitations in modeling. To address this challenge, this work presents a test framework for automotive software on the centralized E/E architecture, which is a central car server in our case, based on Vehicle-in-the-Loop (ViL) and digital twin technology. The framework couples a physical test vehicle on a dynamometer test bench with its synchronized virtual counterpart in a simulation environment. Our approach provides a safe, reproducible, realistic, and cost-effective platform for validating autonomous driving
algorithms with a centralized architecture. This test method eliminates the need to test individual physical ECUs and their communication protocols separately. In contrast to traditional ViL methods, the proposed framework runs the full autonomous driving software directly on the vehicle hardware after the simulation process, eliminating flashing and intermediate layers while enabling seamless virtual-physical integration and accurately reflecting centralized E/E behavior. In addition, incorporating mixed testing in both simulated and physical environments reduces the need for full hardware integration during the early stages of automotive development. Experimental case studies demonstrate the effectiveness of the framework in different test scenarios. These findings highlight the potential to reduce development and integration efforts for testing autonomous driving pipelines in the future.
Read the full abstract ▾
All authors Chengdong Wu, Sven Kirchner, Nils Purschke, Axel Torschmied, Norbert Kroth, Yinglei Song, André Schamschurko, Erik Leo Haß, Kuo-Yi Chao, Yi Zhang, Nenad Petrovic, Alois C. Knoll
Robotics
Curve-Induced Dynamical Systems on Riemannian Manifolds and Lie Groups
Saray Bakker, Martin Schonger, Tobias Löw et al. (+2 more)
Deploying
robots in household environments requires safe, adaptable, and interpretable behaviors that respect the
geometric structure of tasks. Often represented on Lie groups and Riemannian
manifolds, this includes poses on SE(3) or symmetric positive definite
matrices encoding stiffness or damping
matrices. In this context, dynamical system-based approaches offer a natural framework for generating such behavior, providing stability and convergence while remaining responsive to changes in the environment. We introduce Curve-induced Dynamical systems on Smooth
Manifolds (CDSM), a real-time framework for constructing dynamical systems directly on Riemannian
manifolds and Lie groups. The proposed approach constructs a nominal curve on the
manifold, and generates a dynamical system which combines a tangential component that drives motion along the curve and a normal component that attracts the state toward the curve. We provide a stability analysis of the resulting dynamical system and validate the method quantitatively. On an S2
benchmark, CDSM demonstrates improved trajectory accuracy, reduced path deviation, and faster generation and query times compared to state-of-the-art methods. Finally, we demonstrate the practical applicability of the framework on both a robotic manipulator, where poses on SE(3) and damping
matrices on SPD(n) are adapted online, and a mobile manipulator.
Read the full abstract ▾
All authors Saray Bakker, Martin Schonger, Tobias Löw, Javier Alonso-Mora, Sylvain Calinon
Robotics
Rethinking the Role of Collaborative Robots in Rehabilitation
Vivek Gupte, Shalutha Rajapakshe and Emmanuel Senft
Current research on collaborative
robots (cobots) in physical rehabilitation largely focuses on repeated motion training for people undergoing physical therapy (PuPT), even though these sessions include phases that could benefit from robotic collaboration and assistance. Meanwhile, access to physical therapy remains limited for people with disabilities and chronic illnesses. Cobots could support both PuPT and therapists, and improve access to therapy, yet their broader potential remains underexplored. We propose extending the scope of cobots by imagining their role in assisting therapists and PuPT before, during, and after a therapy session. We discuss how cobot assistance may lift access barriers by promoting ability-based therapy design and helping therapists manage their time and effort. Finally, we highlight challenges to realizing these roles, including advancing user-state understanding, ensuring safety, and integrating cobots into therapists' workflow. This view opens new research questions and opportunities to draw from the HRI community's advances in assistive robotics.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Digital Twin Driven Textile Classification and Foreign Object Recognition in Automated Sorting Systems
Serkan Ergun, Tobias Mitterer and Hubert Zangl
The increasing demand for sustainable textile recycling requires robust automation solutions capable of handling deformable garments and detecting foreign objects in cluttered environments. This work presents a digital twin driven robotic sorting system that integrates grasp prediction, multi modal perception, and semantic
reasoning for real world textile classification. A dual arm robotic
cell equipped with RGBD sensing, capacitive tactile feedback, and collision-aware motion planning autonomously separates garments from an unsorted basket, transfers them to an inspection zone, and classifies them using state of the art Visual
Language Models (VLMs). We
benchmark nine VLM s from five model families on a
dataset of 223 inspection scenarios comprising shirts, socks, trousers, underwear, foreign objects (including garments outside of the aforementioned classes), and empty scenes. The evaluation assesses per class accuracy,
hallucination behavior, and computational performance under practical hardware constraints. Results show that the Qwen model family achieves the highest overall accuracy (up to 87.9 %), with strong foreign object detection performance, while lighter models such as Gemma3 offer competitive speed accuracy trade offs for edge deployment. A digital twin combined with MoveIt enables collision aware path planning and integrates segmented 3D point clouds of inspected garments into the virtual environment for improved manipulation reliability. The presented system demonstrates the feasibility of combining semantic VLM
reasoning with conventional grasp detection and digital twin technology for scalable, autonomous textile sorting in realistic industrial settings.
Read the full abstract ▾
Robotics
Critic in the Loop: A Tri-System VLA Framework for Robust Long-Horizon Manipulation
Pengfei Yi, Yingjie Ma, Wenjiang Xu et al. (+4 more)
Balancing high-level semantic
reasoning with low-level reactive control remains a core challenge in visual robotic manipulation. While
Vision-
Language Models (VLMs) excel at cognitive planning, their inference latency precludes real-time execution. Conversely, fast
Vision-
Language-Action (VLA) models often lack the semantic depth required for complex, long-horizon tasks. To bridge this gap, we introduce Critic in the Loop, an adaptive hierarchical framework driven by dynamic VLM-Expert scheduling. At its core is a bionic Tri-System architecture comprising a VLM
brain for global
reasoning, a VLA cerebellum for reactive execution, and a lightweight visual Critic. By continuously monitoring the workspace, the Critic dynamically routes control authority. It sustains rapid closed-loop execution via the VLA for routine subtasks, and adaptively triggers the VLM for replanning upon detecting execution anomalies such as task stagnation or failures. Furthermore, our architecture seamlessly integrates human-inspired rules to intuitively break infinite retry loops. This visually-grounded scheduling minimizes expensive VLM queries, while substantially enhancing system robustness and autonomy in out-of-distribution (OOD) scenarios. Comprehensive experiments on challenging, long-horizon manipulation
benchmarks reveal that our approach achieves state-of-the-art performance.
Read the full abstract ▾
All authors Pengfei Yi, Yingjie Ma, Wenjiang Xu, Yanan Hao, Shuai Gan, Wanting Li, Shanlin Zhong
Robotics
Lifelong Language-Conditioned Robotic Manipulation Learning
Xudong Wang, Zebin Han, Zhiyu Liu et al. (+5 more)
Traditional
language-conditioned manipulation
agent sequential adaptation to new manipulation skills leads to catastrophic forgetting of old skills, limiting dynamic scene practical deployment. In this paper, we propose SkillsCrafter, a novel robotic manipulation framework designed to continually learn multiple skills while reducing catastrophic forgetting of old skills. Specifically, we propose a Manipulation Skills Adaptation to retain the old skills knowledge while inheriting the shared knowledge between new and old skills to facilitate learning of new skills. Meanwhile, we perform the singular value decomposition on the diverse skill instructions to obtain common skill semantic subspace projection
matrices, thereby recording the essential semantic space of skills. To achieve forget-less and generalization manipulation, we propose a Skills Specialization Aggregation to compute inter-skills similarity in skill semantic subspaces, achieving aggregation of the previously learned skill knowledge for any new or unknown skill. Extensive experiments demonstrate the effectiveness and superiority of our proposed SkillsCrafter.
Read the full abstract ▾
All authors Xudong Wang, Zebin Han, Zhiyu Liu, Gan Li, Jiahua Dong, Baichen Liu, Lianqing Liu, Zhi Han
Robotics · Cross-listed from cs.CV
Act, Think or Abstain: Complexity-Aware Adaptive Inference for Vision-Language-Action Models
Riccardo Andrea Izzo, Gianluca Bardaro and Matteo Matteucci
Current research on
Vision-
Language-Action (VLA) models predominantly focuses on enhancing generalization through
reasoning techniques. While effective, these improvements increase computational complexity and inference latency. Furthermore, these mechanisms are typically applied indiscriminately, wasting resources on trivial tasks while failing to provide the uncertainty estimation necessary to prevent catastrophic failure on out-of-distribution scenarios. Inspired by human cognition, we propose an adaptive framework that dynamically routes VLA execution based on the complexity of the perceived state. Our approach transforms the VLA's
vision-
language backbone into an active detection tool by projecting latent
embeddings into a set of parametric and non-parametric
estimators. This allows the system to execute known tasks immediately (Act), reason about ambiguous scenarios (Think), and preemptively halt execution when encountering physical or semantic anomalies (Abstain). We find that a Gaussian Mixture Model fitted to fused
vision-
language embeddings provides the most reliable task-complexity signal, combining visual novelty with instruction context and cross-modal compatibility. Evaluated on the LIBERO and LIBERO-PRO
benchmarks as well as on a real
robot, our fused configuration achieves up to 87.5% F1-score across two VLA backbones (SmolVLA and $\pi_0$), retains 83% with as little as 5% of training data, and surpasses state-of-the-art failure detectors.
Read the full abstract ▾
Robotics
SeedPolicy: Horizon Scaling via Self-Evolving Diffusion Policy for Robot Manipulation
Youqiang Gui, Yuxuan Zhou, Shen Cheng et al. (+4 more)
Imitation Learning (IL) enables
robots to acquire manipulation skills from expert demonstrations.
Diffusion Policy (DP) models multi-modal expert behaviors but degrades when naively increasing stacked observation horizons, limiting long-horizon manipulation. We propose Self-Evolving Gated
Attention (SEGA), a temporal module that maintains a time-evolving latent state via gated
attention, enabling efficient recurrent updates that accumulate long-term context into a compact latent representation while filtering irrelevant temporal information. Integrating SEGA into DP yields Self-Evolving
Diffusion Policy (SeedPolicy), which resolves the temporal modeling bottleneck and extends the effective temporal horizon with moderate overhead. On the RoboTwin 2.0
benchmark with 50 manipulation tasks, SeedPolicy outperforms DP and other IL baselines. Averaged across both CNN and
Transformer backbones, SeedPolicy achieves 36.8% relative improvement in clean settings and 169% relative improvement in randomized challenging settings over the DP. Compared to
vision-
language-action models such as RDT with 1.2B parameters, SeedPolicy achieves stronger performance in the clean setting with one to two orders of magnitude fewer parameters, demonstrating strong efficiency. These results establish SeedPolicy as a state-of-the-art imitation learning method for long-horizon robotic manipulation. Code is available at: https://github.com/Youqiang-Gui/SeedPolicy.
Read the full abstract ▾
All authors Youqiang Gui, Yuxuan Zhou, Shen Cheng, Xinyang Yuan, Haoqiang Fan, Peng Cheng, Shuaicheng Liu
Robotics · Cross-listed from cs.LG
Decoupling Task and Behavior: A Two-Stage Reward Curriculum in Reinforcement Learning for Robotics
Kilian Freitag, Knut Åkesson and Morteza Haghir Chehreghani
Deep Reinforcement Learning is a promising tool for robotic control, yet practical application is often hindered by the difficulty of designing effective reward functions. Real-world tasks typically require optimizing multiple objectives simultaneously, necessitating precise tuning of their weights to learn a policy with the desired characteristics. To address this, we propose a two-stage reward curriculum where we decouple task-specific objectives from behavioral terms. In our method, we first train the
agent on a simplified task-only reward function to ensure effective exploration before introducing the full reward that includes auxiliary behavior-related terms such as energy efficiency. Further, we analyze various transition strategies and demonstrate that reusing samples between phases is critical for training stability. We validate our approach on the DeepMind Control Suite, ManiSkill3, and a mobile
robot environment, modified to include auxiliary behavioral objectives. Our method proves to be simple yet effective, substantially outperforming baselines trained directly on the full reward while exhibiting higher robustness to specific reward weightings.
Read the full abstract ▾
Robotics
SPIRIT: Perceptive Shared Autonomy for Robust Robotic Manipulation under Deep Learning Uncertainty
Jongseok Lee, Ribin Balachandran, Harsimran Singh et al. (+6 more)
Deep learning (DL) has enabled impressive advances in robotic perception, yet its limited robustness and lack of interpretability hinder reliable deployment in safety critical applications. We propose a concept termed perceptive shared autonomy, in which uncertainty estimates from DL based perception are used to regulate the level of autonomy. Specifically, when the
robot's perception is confident, semi-autonomous manipulation is enabled to improve performance; when uncertainty increases, control transitions to haptic teleoperation for maintaining robustness. In this way, high-performing but uninterpretable DL methods can be integrated safely into robotic systems. A key technical enabler is an uncertainty aware DL based point cloud registration approach based on the so called Neural Tangent Kernels (NTK). We evaluate perceptive shared autonomy on challenging aerial manipulation tasks through a user study of 15 participants and realization of mock-up industrial scenarios, demonstrating reliable robotic manipulation despite failures in DL based perception. The resulting system, named SPIRIT, improves both manipulation performance and system reliability. SPIRIT was selected as a finalist of a major industrial innovation award.
Read the full abstract ▾
All authors Jongseok Lee, Ribin Balachandran, Harsimran Singh, Jianxiang Feng, Hrishik Mishra, Marco De Stefano, Rudolph Triebel, Alin Albu-Schaeffer, Konstantin Kondak
Robotics
GaussTwin: Unified Simulation and Correction with Gaussian Splatting for Robotic Digital Twins
Yichen Cai, Paul Jansonnie, Cristiana de Farias et al. (+2 more)
Digital twins promise to enhance robotic manipulation by maintaining a consistent link between real-world perception and simulation. However, most existing systems struggle with the lack of a unified model, complex dynamic interactions, and the real-to-sim gap, which limits downstream applications such as model predictive control. Thus, we propose GaussTwin, a real-time digital twin that combines position-based dynamics with discrete Cosserat rod formulations for physically grounded simulation, and Gaussian splatting for efficient rendering and visual correction. By anchoring Gaussians to physical primitives and enforcing coherent SE(3) updates driven by photometric error and
segmentation masks, GaussTwin achieves stable prediction-correction while preserving physical fidelity. Through experiments in both simulation and on a Franka Research 3 platform, we show that GaussTwin consistently improves tracking accuracy and robustness compared to shape-matching and rigid-only baselines, while also enabling downstream tasks such as push-based planning. These results highlight GaussTwin as a step toward unified, physically meaningful digital twins that can support closed-loop robotic interaction and learning.
Read the full abstract ▾
All authors Yichen Cai, Paul Jansonnie, Cristiana de Farias, Oleg Arenz, Jan Peters
Robotics
AIM-SLAM: Dense Monocular SLAM via Adaptive and Informative Multi-View Keyframe Prioritization with Foundation Model
Jinwoo Jeon, Dong-Uk Seo, Eungchang Mason Lee et al. (+1 more)
Recent advances in
geometric foundation models have emerged as a promising alternative for addressing the challenge of dense reconstruction in
monocular visual simultaneous localization and mapping (SLAM). Although
geometric foundation models enable SLAM to leverage variable input views, the previous methods remain confined to two-view pairs or fixed-length inputs without sufficient deliberation of
geometric context for view selection. To tackle this problem, we propose AIM-SLAM, a dense
monocular SLAM framework that exploits an adaptive and informative multi-view keyframe prioritization with dense pointmap predictions from visual geometry grounded
transformer (VGGT). Specifically, we introduce the selective information- and
geometric-aware multi-view adaptation (SIGMA) module, which employs voxel overlap and information gain to retrieve a candidate set of keyframes and adaptively determine its size. Furthermore, we formulate a joint multi-view Sim(3) optimization that enforces consistent
alignment across selected views, substantially improving pose estimation accuracy. The effectiveness of AIM-SLAM is demonstrated on real-world
datasets, where it achieves state-of-the-art pose estimation performance and accurate dense reconstruction results. Our system supports ROS integration, with code is available at https://aimslam.github.io/.
Read the full abstract ▾
All authors Jinwoo Jeon, Dong-Uk Seo, Eungchang Mason Lee, Hyun Myung
Robotics
VinePT-Map: Pole-Trunk Semantic Mapping for Resilient Autonomous Robotics in Vineyards
Giorgio Audrito, Mauro Martini, Alessandro Navone et al. (+2 more)
Reliable long-term deployment of autonomous
robots in agricultural environments remains challenging due to perceptual aliasing, seasonal variability, and the dynamic nature of crop canopies. Vineyards, characterized by repetitive row structures and significant visual changes across phenological stages, represent a pivotal field challenge, limiting the robustness of conventional feature-based localization and mapping approaches. This paper introduces VinePT-Map, a semantic mapping framework that leverages vine trunks and support poles as persistent structural landmarks to enable season-agnostic and resilient
robot localization. The proposed method formulates the mapping problem as a factor graph, integrating GPS, IMU, and RGB-D observations through robust geometrical constraints that exploit vineyard structure. An efficient perception pipeline based on instance
segmentation and tracking, combined with a clustering filter for outlier rejection and pose refinement, enables accurate landmark detection using low-cost sensors and onboard computation. To validate the pipeline, we present a multi-season
dataset for trunk and pole
segmentation and tracking. Extensive field experiments conducted across diverse seasons demonstrate the robustness and accuracy of the proposed approach, highlighting its suitability for long-term autonomous operation in agricultural environments.
Read the full abstract ▾
All authors Giorgio Audrito, Mauro Martini, Alessandro Navone, Giorgia Galluzzo, Marcello Chiaberge
Robotics · Cross-listed from cs.CV
CoIn3D: Revisiting Configuration-Invariant Multi-Camera 3D Object Detection
Zhaonian Kuang, Rui Ding, Haotian Wang et al. (+3 more)
Multi-camera 3D object detection (MC3D) has attracted increasing
attention with the growing deployment of multi-sensor physical
agents, such as
robots and autonomous vehicles. However, MC3D models still struggle to generalize to unseen platforms with new multi-camera configurations. Current solutions simply employ a meta-camera for unified representation but lack comprehensive consideration. In this paper, we revisit this issue and identify that the devil lies in spatial prior discrepancies across source and target configurations, including different intrinsics, extrinsics, and array layouts. To address this, we propose CoIn3D, a generalizable MC3D framework that enables strong transferability from source configurations to unseen target ones. CoIn3D explicitly incorporates all identified spatial priors into both feature
embedding and image observation through spatial-aware feature modulation (SFM) and camera-aware data augmentation (CDA), respectively. SFM enriches feature space by integrating four spatial representations, such as focal length, ground depth, ground gradient, and Plücker coordinate. CDA improves observation diversity under various configurations via a training-free dynamic novel-view image synthesis scheme. Extensive experiments demonstrate that CoIn3D achieves strong cross-configuration performance on landmark
datasets such as NuScenes, Waymo, and Lyft, under three dominant MC3D paradigms represented by BEVDepth, BEVFormer, and PETR.
Read the full abstract ▾
All authors Zhaonian Kuang, Rui Ding, Haotian Wang, Xinhu Zheng, Meng Yang, Gang Hua
Robotics
Direct Contact-Tolerant Motion Planning With Vision Language Models
He Li, Jian Sun, Chengyang Li et al. (+4 more)
Navigation in cluttered environments often requires
robots to tolerate contact with movable or deformable objects to maintain efficiency. Existing contact-tolerant motion planning (CTMP) methods rely on indirect spatial representations (e.g., prebuilt map, obstacle set), resulting in inaccuracies and a lack of adaptiveness to environmental uncertainties. To address this issue, we propose a direct contact-tolerant (DCT) planner, which integrates
vision-
language models (VLMs) into direct point perception and
navigation, including two key components. The first one is VLM point cloud partitioner (VPP), which performs contact-tolerance
reasoning in image space using VLM, caches inference masks, propagates them across frames using odometry, and projects them onto the current scan to generate a contact-aware point cloud. The second innovation is VPP guided
navigation (VGN), which formulates CTMP as a perception-to-control optimization problem under direct contact-aware point cloud constraints, which is further solved by a specialized deep neural network (DNN). We implement DCT in Isaac Sim and a real car-like
robot, demonstrating that DCT achieves robust and efficient
navigation in cluttered environments with movable obstacles, outperforming representative baselines across diverse metrics. The code is available at: https://github.com/ChrisLeeUM/DCT.
Read the full abstract ▾
All authors He Li, Jian Sun, Chengyang Li, Guoliang Li, Qiyu Ruan, Shuai Wang, Chengzhong Xu
Robotics
Observer Design for Augmented Reality-based Teleoperation of Soft Robots
Jorge Francisco García-Samartín, Iago López Pérez, Emirhan Yolcu et al. (+2 more)
Although virtual and augmented reality are gaining traction as teleoperation tools for various types of
robots, including manipulators and mobile
robots, they are not being used for soft
robots. The inherent difficulties of modelling soft
robots mean that combining accurate and computationally efficient representations is very challenging. This paper presents an augmented reality interface for teleoperating these devices. The developed system consists of Microsoft HoloLens 2 glasses and a central computer responsible for calculations. Validation is performed on PETER, a highly modular pneumatic manipulator. Using data collected from sensors, the computer estimates the
robot's position based on the physics of the virtual reality programme. Errors obtained are on the order of 5% of the
robot's length, demonstrating that augmented reality facilitates operator interaction with soft manipulators and can be integrated into the control loop.
Read the full abstract ▾
All authors Jorge Francisco García-Samartín, Iago López Pérez, Emirhan Yolcu, Jaime del Cerro, Antonio Barrientos
Robotics · Cross-listed from cs.CV
Person Detection and Tracking from an Overhead Crane LiDAR
Nilusha Jayawickrama, Henrik Toikka and Risto Ojala
This paper investigates person detection and tracking in an industrial indoor workspace using a
LiDAR mounted on an overhead crane. The overhead viewpoint introduces a strong domain shift from common vehicle-centric
LiDAR benchmarks, and limited availability of suitable public training data. Henceforth, we curate a site-specific overhead
LiDAR dataset with 3D human bounding-box annotations and adapt selected candidate 3D detectors under a unified training and evaluation protocol. We further integrate lightweight tracking-by-detection using AB3DMOT and SimpleTrack to maintain person identities over time. Detection performance is reported with distance-sliced evaluation to quantify the practical operating envelope of the sensing setup. The best adapted detector configurations achieve average precision (AP) up to 0.84 within a 5.0 m horizontal radius, increasing to 0.97 at 1.0 m, with VoxelNeXt and SECOND emerging as the most reliable backbones across this range. The acquired results contribute in bridging the domain gap between standard driving
datasets and overhead sensing for person detection and tracking. We also report latency measurements, highlighting practical real-time feasibility. Finally, we release our
dataset and implementations in GitHub to support further research
Read the full abstract ▾
Robotics
Integrated cooperative localization of heterogeneous measurement swarm: A unified data-driven method
Kunrui Ze, Wei Wang, Guibin Sun et al. (+3 more)
The cooperative localization (CL) problem in heterogeneous robotic systems with different measurement capabilities is investigated in this work. In practice, heterogeneous sensors lead to directed and sparse measurement
topologies, whereas most existing CL approaches rely on multilateral localization with restrictive multi-neighbor
geometric requirements. To overcome this limitation, we enable pairwise relative localization (RL) between neighboring
robots using only mutual measurement and odometry information. A unified data-driven adaptive RL
estimator is first developed to handle heterogeneous and unidirectional measurements. Based on the convergent RL estimates, a distributed pose-coupling CL strategy is then designed, which guarantees CL under a weakly connected directed measurement
topology, representing the least restrictive condition among existing results. The proposed method is independent of specific control tasks and is validated through a formation control application and real-world experiments.
Read the full abstract ▾
All authors Kunrui Ze, Wei Wang, Guibin Sun, Jiaqi Yan, Kexin Liu, Jinhu Lü
Robotics
U-OBCA: Uncertainty-Aware Optimization-Based Collision Avoidance via Wasserstein Distributionally Robust Chance Constraints
Zehao Wang, Yuxuan Tang, Han Zhang et al. (+2 more)
Uncertainties arising from localization error, trajectory prediction errors of the moving obstacles and environmental disturbances pose significant challenges to
robot's safe
navigation. Existing uncertainty-aware planners often approximate polygon-shaped
robots and obstacles using simple
geometric primitives such as circles or ellipses. Though computationally convenient, these approximations substantially shrink the feasible space, leading to overly conservative trajectories and even planning failure in narrow environments. In addition, many such methods rely on specific assumptions about noise distributions, which may not hold in practice and thus limit their performance guarantees. To address these limitations, we extend the Optimization-Based Collision Avoidance (OBCA) framework to an uncertainty-aware formulation, termed U-OBCA. The proposed method explicitly accounts for the collision risk between polygon-shaped
robots and obstacles by formulating OBCA-based chance constraints, and hence avoiding
geometric simplifications and reducing unnecessary conservatism. These probabilistic constraints are further tightened into deterministic nonlinear constraints under mild distributional assumptions, which can be solved efficiently by standard numerical optimization solvers. The proposed approach is validated through theoretical analysis, numerical simulations and real-world experiments. The results demonstrate that U-OBCA significantly mitigates the conservatism in trajectory planning and achieves higher
navigation efficiency compared to existing baseline methods, particularly in narrow and cluttered environments.
Read the full abstract ▾
All authors Zehao Wang, Yuxuan Tang, Han Zhang, Jingchuan Wang, Weidong Chen
Robotics
Beyond the Patch: Exploring Vulnerabilities of Visuomotor Policies via Viewpoint-Consistent 3D Adversarial Object
Chanmi Lee, Minsung Yoon, Woojae Kim et al. (+2 more)
Neural network-based visuomotor policies enable
robots to perform manipulation tasks but remain susceptible to perceptual attacks. For example, conventional 2D
adversarial patches are effective under fixed-camera setups, where appearance is relatively consistent; however, their efficacy often diminishes under dynamic viewpoints from moving cameras, such as wrist-mounted setups, due to perspective distortions. To proactively investigate potential vulnerabilities beyond 2D patches, this work proposes a viewpoint-consistent
adversarial texture optimization method for 3D objects through differentiable rendering. As optimization strategies, we employ Expectation over Transformation (EOT) with a Coarse-to-Fine (C2F) curriculum, exploiting distance-dependent frequency characteristics to induce textures effective across varying camera-object distances. We further integrate saliency-guided perturbations to redirect policy
attention and design a targeted loss that persistently drives
robots toward
adversarial objects. Our comprehensive experiments show that the proposed method is effective under various environmental conditions, while confirming its black-box transferability and real-world applicability.
Read the full abstract ▾
All authors Chanmi Lee, Minsung Yoon, Woojae Kim, Sebin Lee, Sung-eui Yoon
Robotics
VPWEM: Non-Markovian Visuomotor Policy with Working and Episodic Memory
Yuheng Lei, Zhixuan Liang, Hongyuan Zhang et al. (+1 more)
Imitation learning from human demonstrations has achieved significant success in robotic control, yet most visuomotor policies still condition on single-step observations or short-context histories, making them struggle with non-Markovian tasks that require long-term
memory. Simply enlarging the context window incurs substantial computational and
memory costs and encourages overfitting to spurious correlations, leading to catastrophic failures under distribution shift and violating real-time constraints in robotic systems. By contrast, humans can compress important past experiences into long-term
memories and exploit them to solve tasks throughout their lifetime. In this paper, we propose VPWEM, a non-Markovian visuomotor policy equipped with working and episodic
memories. VPWEM retains a sliding window of recent observation
embeddings as short-term working
memory, and introduces a
Transformer-based contextual
memory compressor that recursively converts out-of-window observations into a fixed number of episodic
memory embeddings. The compressor uses self-
attention over a cache of past summary
embeddings and cross-
attention over a cache of historical observations, and is trained jointly with the policy. We instantiate VPWEM on
diffusion policies to exploit both short-term and episode-wide information for action generation with nearly constant
memory and computation per step. Experiments demonstrate that VPWEM outperforms state-of-the-art baselines including
diffusion policies and
vision-
language-action (VLA) models by more than 20% on the
memory-intensive manipulation tasks in MIKASA and achieves an average 5% improvement on the mobile manipulation
benchmark MoMaRT. Code is available at https://github.com/HarryLui98/code_vpwem.
Read the full abstract ▾
All authors Yuheng Lei, Zhixuan Liang, Hongyuan Zhang, Ping Luo
Robotics · Cross-listed from cs.AI
Causally Robust Reward Learning from Reason-Augmented Preference Feedback
Minjune Hwang, Yigit Korkmaz, Daniel Seita et al. (+1 more)
Preference-based reward learning is widely used for shaping
agent behavior to match a user's preference, yet its sparse binary feedback makes it especially vulnerable to causal confusion. The learned reward often latches onto spurious features that merely co-occur with preferred trajectories during training, collapsing when those correlations disappear or reverse at test time. We introduce ReCouPLe, a lightweight framework that uses natural
language rationales to provide the missing causal signal. Each rationale is treated as a guiding projection axis in an
embedding space, training the model to score trajectories based on features aligned with that axis while de-emphasizing context that is unrelated to the stated reason. Because the same rationales (e.g., "avoids collisions", "completes the task faster") can appear across multiple tasks, ReCouPLe naturally reuses the same causal direction whenever tasks share semantics, and transfers preference knowledge to novel tasks without extra data or
language-model fine-tuning. Our learned reward model can ground preferences on the articulated reason, aligning better with user intent and generalizing beyond spurious features. ReCouPLe outperforms baselines by up to 1.5x in reward accuracy under distribution shifts, and 2x in downstream policy performance in novel tasks. We have released our code at https://github.com/mj-hwang/ReCouPLe
Read the full abstract ▾
All authors Minjune Hwang, Yigit Korkmaz, Daniel Seita, Erdem B{\i}y{\i}k
Robotics
Hyperbolic Multiview Pretraining for Robotic Manipulation
Jin Yang, Ping Wei, Yixin Chen et al. (+1 more)
3D-aware visual pretraining has proven effective in improving the performance of downstream robotic manipulation tasks. However, existing methods are constrained to Euclidean
embedding spaces, whose flat geometry limits their ability to model structural relations among
embeddings. As a result, they struggle to learn structured
embeddings that are essential for robust spatial perception in robotic applications. To this end, we propose HyperMVP, a self-supervised framework for \underline{Hyper}bolic \underline{M}ulti\underline{V}iew \underline{P}retraining.
Hyperbolic space offers
geometric properties well suited for capturing structural relations. Methodologically, we extend the masked autoencoder paradigm and design a GeoLink encoder to learn multiview
hyperbolic representations. The pretrained encoder is then finetuned with visuomotor policies on manipulation tasks. In addition, we introduce 3D-MOV, a large-scale
dataset comprising multiple types of 3D point clouds to support pretraining. We evaluate HyperMVP on COLOSSEUM, RLBench, and real-world scenarios, where it consistently outperforms strong baselines across diverse tasks and perturbation settings. Our results highlight the potential of 3D-aware pretraining in a non-Euclidean space for learning robust and generalizable robotic manipulation policies.
Read the full abstract ▾
All authors Jin Yang, Ping Wei, Yixin Chen, Nanning Zheng
Robotics
Task-Relevant and Irrelevant Region-Aware Augmentation for Generalizable Vision-Based Imitation Learning in Agricultural Manipulation
Shun Hattori, Hikaru Sasaki, Takumi Hachimine et al. (+2 more)
Vision-based imitation learning has shown promise for robotic manipulation; however, its generalization remains limited in practical agricultural tasks. This limitation stems from scarce demonstration data and substantial visual domain gaps caused by i) crop-specific appearance diversity and ii) background variations. To address this limitation, we propose Dual-Region Augmentation for Imitation Learning (DRAIL), a region-aware augmentation framework designed for generalizable
vision-based imitation learning in agricultural manipulation. DRAIL explicitly separates visual observations into task-relevant and task-irrelevant regions. The task-relevant region is augmented in a domain-knowledge-driven manner to preserve essential visual characteristics, while the task-irrelevant region is aggressively randomized to suppress spurious background correlations. By jointly handling both sources of visual variation, DRAIL promotes learning policies that rely on task-essential features rather than incidental visual cues. We evaluate DRAIL on
diffusion policy-based visuomotor controllers through
robot experiments on artificial vegetable harvesting and real lettuce defective leaf picking preparation tasks. The results show consistent improvements in success rates under unseen visual conditions compared to baseline methods. Further
attention analysis and representation generalization metrics indicate that the learned policies rely more on task-essential visual features, resulting in enhanced robustness and generalization.
Read the full abstract ▾
All authors Shun Hattori, Hikaru Sasaki, Takumi Hachimine, Yusuke Mizutani, Takamitsu Matsubara
Robotics
On the Strengths and Weaknesses of Data for Open-set Embodied Assistance
Pradyumna Tambwekar, Andrew Silva, Deepak Gopinath et al. (+3 more)
Embodied foundation models are increasingly performant in real-world domains such as robotics or autonomous driving. These models are often deployed in interactive or assistive settings, where it is important that these assistive models generalize to new users and new tasks. Diverse interactive data generation offers a promising avenue for providing data-efficient generalization capabilities for interactive
embodied foundation models. In this paper, we investigate the generalization capabilities of a
multimodal foundation model fine-tuned on diverse interactive assistance data in a synthetic domain. We explore generalization along two axes: a) assistance with unseen categories of user behavior and b) providing guidance in new configurations not encountered during training. We study a broad capability called Open-Set Corrective Assistance, in which the model needs to inspect lengthy user behavior and provide assistance through either corrective actions or
language-based feedback. This task remains unsolved in prior work, which typically assumes closed corrective categories or relies on external planners, making it a challenging testbed for evaluating the limits of assistive data. To support this task, we generate synthetic assistive
datasets in Overcooked and fine-tune a LLaMA-based model to evaluate generalization to novel tasks and user behaviors. Our approach provides key insights into the nature of assistive
datasets required to enable open-set assistive intelligence. In particular, we show that performant models benefit from
datasets that cover different aspects of assistance, including
multimodal grounding, defect inference, and exposure to diverse scenarios.
Read the full abstract ▾
All authors Pradyumna Tambwekar, Andrew Silva, Deepak Gopinath, Jonathan DeCastro, Xiongyi Cui, Guy Rosman
Robotics · Cross-listed from cs.LG
Diffusion Policy through Conditional Proximal Policy Optimization
Ben Liu, Shunpeng Yang and Hua Chen
Reinforcement learning (RL) has been extensively employed in a wide range of decision-making problems, such as games and robotics. Recently,
diffusion policies have shown strong potential in modeling multi-modal behaviors, enabling more diverse and flexible action generation compared to the conventional Gaussian policy. Despite various attempts to combine RL with
diffusion, a key challenge is the difficulty of computing action log-likelihood under the
diffusion model. This greatly hinders the direct application of
diffusion policies in on-policy reinforcement learning. Most existing methods calculate or approximate the log-likelihood through the entire denoising process in the
diffusion model, which can be
memory- and computationally inefficient. To overcome this challenge, we propose a novel and efficient method to train a
diffusion policy in an on-policy setting that requires only evaluating a simple Gaussian probability. This is achieved by aligning the policy iteration with the
diffusion process, which is a distinct paradigm compared to previous work. Moreover, our formulation can naturally handle
entropy regularization, which is often difficult to incorporate into
diffusion policies. Experiments demonstrate that the proposed method produces
multimodal policy behaviors and achieves superior performance on a variety of
benchmark tasks in both IsaacLab and MuJoCo Playground.
Read the full abstract ▾