Robotics
Invascal: Inverse-Vacuity Self-Calibration for Uncertainty-Aware LiDAR Range-View Semantic Segmentation
Kerim Turacan, Hannes Reichert, Andrei Bolandut et al. (+1 more)
LiDAR semantic
segmentation is a core perception capability for autonomous vehicles and mobile
robots. However, safe operation also depends on knowing when predictions are unreliable. Existing approaches typically rely on softmax confidence, which is often miscalibrated and overconfident, while stronger uncertainty estimates from Monte Carlo dropout or ensembles are often computationally expensive for real-time use. To this end, we introduce a novel, architecture-agnostic uncertainty-aware Adapter Head. It decomposes the prediction into a Preference Head for class ranking and a Strength Head that refines uncertainty assessment, thereby enabling a principled construction of evidential Dirichlet representations. Building on this design, we propose our inverse-vacuity self-calibration objective (Invascal), which directly supervises the strength signal to produce reliable and well-calibrated uncertainty estimates while preventing runaway evidence growth. We evaluate our framework across multiple
LiDAR datasets and backbone architectures. We compare against deterministic training, Monte Carlo dropout and ensembles, and prior evidential methods. Our approach consistently improves uncertainty calibration over traditional deterministic methods with minimal computational overhead. At the same time, it preserves competitive
segmentation accuracy, where prior evidential methods often suffer performance degradation.
Read the full abstract ▾
All authors Kerim Turacan, Hannes Reichert, Andrei Bolandut, Konrad Doll
Robotics · Cross-listed from cs.CV
GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation
Kaichen Zhou, Yuzhen Chen, Fangneng Zhan et al. (+8 more)
Video world models can generate realistic futures from a single instruction, but they often fail to track the same physical points consistently across time. As a result, the generated
videos appear plausible, yet lack the physical grounding required for reliable action execution, such as
robot manipulation. We present GEM-4D, a geometry-grounded
video world model that resolves this limitation by injecting dense 4D correspondence supervision distilled from a pretrained geometry foundation model into the
video generative backbone during training. This supervision enables the model to jointly capture appearance and
geometric structure while retaining a single-stream architecture with no additional inference cost. We further introduce an inverse dynamics module that converts correspondence-consistent
video rollouts into executable
robot trajectories, enabling direct deployment in both real-world and simulated manipulation. GEM-4D achieves state-of-the-art performance on both
video prediction and
geometric consistency across both simulation and realistic scenarios and improves real-world manipulation success from 61% to 81%. Additional results are available at https://gem-4d.github.io/.
Read the full abstract ▾
All authors Kaichen Zhou, Yuzhen Chen, Fangneng Zhan, Hang Hua, Grace Chen, Xinhai Chang, Ao Qu, Yilun Du, Zhuang Liu, Paul Pu Liang, Mengyu Wang
Robotics
Safe and Steerable Geometric Motion Policies for Robotic Dexterous Manipulation
Albert Wu, Riccardo Bonalli, Thomas Lew et al. (+1 more)
Robotic dexterous manipulation requires continuously reconciling objectives and constraints defined on heterogeneous
geometric spaces: a
robot controlled on a $\mathbb{R}^7$ configuration
manifold may need to track end effector poses on $\mathrm{SE}(3)$ while satisfying obstacle avoidance margins in $\mathbb{R}$. We present Safe Pullback Bundle Dynamical Systems (SafePBDS), a geometrically consistent framework that computes optimal, certifiably safe configuration
manifold accelerations from objectives and safety requirements on arbitrary task
manifolds. SafePBDS builds on prior work that combines predefined task
manifold dynamical systems to produce autonomous motion. Its first innovation is a pullback control barrier function construction, which converts task
manifold safety conditions into linear constraints on configuration
manifold accelerations. The second innovation is a task
manifold action interface that allows a high-level policy to inject low dimensional residual motions; zero input recovers the autonomous behavior, while safety is preserved under arbitrary inputs. This lets high-level policies efficiently steer exploration while leaving precise motion to the autonomous behavior. We validate SafePBDS in simulation and on a 23-DOF Franka Panda-Allegro Hand platform. On dexterous grasping, SafePBDS achieves a $92.5\%$ success rate across 20 household objects and 120 trials. Using the action interface, the method can exclude any one of the four fingers during grasping via a one-dimensional action, achieving $94.4\%$ 3-finger grasp success across 3 objects and 36 trials. The efficient planning and safety guarantee of SafePBDS also enables the first model-based, fully actuated palm-down in-hand reorientation, exceeding $360^\circ$ of yaw rotation in both directions under varying object weight and wrist motion. Demo
video and details: https://tml.stanford.edu/safe-pbds
Read the full abstract ▾
All authors Albert Wu, Riccardo Bonalli, Thomas Lew, C. Karen Liu
Robotics · Cross-listed from cs.LG
stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation
Lucas Maes, Quentin Le Lidec, Luiz Facury et al. (+9 more)
World models are central to building
agents that can reason, plan, and generalize beyond their training data. However, research on world models is currently fragmented, with disparate codebases, data pipelines, and evaluation protocols hindering reproducibility and fair comparison. Current practice is further limited by three key bottlenecks: fragile one-off codebases, slow
video data loading, and the lack of standardized generalization
benchmarks. We present stable-worldmodel (swm), an open-source platform for standardized and reproducible world modeling research and evaluation. It delivers (1) a high-performance Lance-based data layer with native support and conversion tools for MP4, HDF5, and LeRobot
datasets, (2) clean, well-tested implementations of modern world model baselines and planning solvers, and (3) a broad suite of environments and tasks extended with controllable visual,
geometric, and physical factors of variation for systematic in-silico evaluation of dynamics understanding, control performance, representation quality, and out-of-distribution generalization. By unifying the full pipeline under a single, scalable framework, swm dramatically reduces research overhead and accelerates trustworthy progress toward reliable world models.
Read the full abstract ▾
All authors Lucas Maes, Quentin Le Lidec, Luiz Facury, Nassim Massaudi, Ayush Chaurasia, Francesco Capuano, Richard Gao, Taj Gillin, Dan Haramati, Damien Scieur, Yann LeCun, Randall Balestriero
Robotics · Cross-listed from cs.CV
SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matching
Xuefei Sun, Xujia Zhang, Brendan Crowe et al. (+2 more)
Zero-shot 3D visual grounding requires localizing objects in unstructured environments from free-form natural
language. Recent
vision-
language model (VLM) approaches achieve promising results but rely on view-dependent
reasoning or implicit representations, limiting spatial consistency and interpretability for compositional queries. We propose SceneGraphGrounder, a framework that reformulates 3D grounding as structured graph matching over a reconstructed 3D scene graph. To enable this formulation, we introduce a visual marker prompting strategy that enables a VLM to infer object-object relationships from 2D views, which are subsequently lifted into a persistent 3D scene graph encoding both spatial and semantic relations. Given a query, we construct a query graph and perform constrained
alignment with the scene graph, ensuring multi-view consistency and interpretable
reasoning. Experiments on the ScanRefer
benchmark demonstrate that our method achieves competitive performance among zero-shot approaches, using only RGB-D inputs. We further validate our framework through real-world deployment on a mobile
robot, demonstrating robust spatial
reasoning in long-horizon physical environments. We will make our code publicly available upon acceptance.
Read the full abstract ▾
All authors Xuefei Sun, Xujia Zhang, Brendan Crowe, Doncey Albin, Christoffer Heckman
Robotics · Cross-listed from cs.CV
Improving 3D Labeling in Self-Driving by Inferring Vehicle Information using Vision Language Models
Steven Chen, Shivesh Khaitan and Nemanja Djuric
We present an approach to improve 3D vehicle labeling in self-driving applications through zero-shot inference of vehicle information, leveraging Vehicle Make and Model Recognition (VMMR) methods. The proposed approach utilizes a
Vision Language Model (VLM) to both infer a vehicle's make, model, and generation from image crops, and output accurate 3D bounding box dimensions to seed manual labeling. We evaluate the impact of iterative
prompt engineering and the choice of different VLMs on both vehicle bounding box inference and make/model/generation recognition. When compared to strong baselines, the proposed approach not only shows high accuracy, but also excels in mitigating specific failure modes where VLMs provide better dimensions than initial
lidar-aided human annotated labels (e.g., in cases of significant vehicle occlusion). Experiments on both public and proprietary data strongly suggest that our conclusions are generalizable across different labelers and
datasets. The results demonstrate that integrating VLMs into the labeling process can reduce manual labeling time while increasing label quality.
Read the full abstract ▾
Robotics
Learning Altruistic Collaboration in Heterogeneous Multi-Team Systems
Riwa Karam, Ruoyu Lin, Brooks A. Butler et al. (+1 more)
This paper studies heterogeneous multi-team collaboration through dynamic
robot allocation, where the
robots themselves are treated as transferable resources. Leveraging Hamilton's rule from ecology as an altruistic decision-making mechanism, we propose a multi-team collaborative resource allocation framework with heterogeneous capabilities, transfer costs, and capability-dependent contributions. The resulting general heterogeneous resource assignment problem is combinatorial and is shown to be NP-hard. To address scalability, we develop a graph neural network policy under centralized training and decentralized execution that approximates the altruistic allocations based on Hamilton's rule. The model operates over the team interaction graph and predicts
robot-level transfer decisions and subsequent
robot-to-team assignments. The proposed approach is evaluated in a fire-fighting scenario against exact optimization and multi-step simulations, quantifying allocation quality, decision computation, and fire suppression. Five paired physical testbed trials with 15
robots compare learned reallocation with fixed team membership; the GNN reaches the modeled fire stopping threshold in all five trials, compared with two for static allocation.
Read the full abstract ▾
All authors Riwa Karam, Ruoyu Lin, Brooks A. Butler, Magnus Egerstedt
Robotics
Mind the Gaps: Multi-Robot Feedback-Driven Ergodic Coverage in Unknown Environments
Thales Costa Silva and Nora Ayanian
In this work, we address the problem of multi-
robot adaptive coverage, where teams of
robots perform dynamic sampling by continuously adjusting their positions to collect data in an environment. This task can be challenging, particularly when
robots must be efficiently allocated to new sampling locations over time. Ergodic search methods optimize
robot trajectories by ensuring that the
robots' time-averaged spatial distribution aligns with the spatial distribution of environmental information. While these methods promote effective exploration provided a target distribution, they often fail to account for unknown prior distributions of the environment. To overcome this limitation, we propose an adaptive coverage strategy that utilizes real-time feedback from an environmental model to adjust
robot sampling behavior in response to unknown conditions. Our approach enhances traditional ergodic trajectory optimization by constructing a target spatial information distribution based on parametric models of the environment, which are updated online. This strategy assumes that the environment is either static or changes slowly compared to the
robot's motion. Our framework allows
robots to dynamically prioritize regions of high interest, improving coverage efficiency, synthesizing effective control policies for individual
agents, and optimizing resource use in settings with unknown prior distributions. We validate our approach through simulations, demonstrating its effectiveness in enhancing coverage and resource allocation.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
AVI-HT: Adaptive Vision-IMU Fusion for 3D Hand Tracking
Ziyi Kou, Ankit Kumar, Mia Huang et al. (+4 more)
We present AVI-HT, an adaptive visual-IMU fusion approach for tracking 3D hand poses by jointly modeling the
egocentric image with on-glove 6-DoF IMU signals. AVI-HT achieves significantly improved accuracy and availability, particularly in hand-object interaction (HOI) scenarios involving heavy visual occlusion. Two complementary ingredients underpin its success: (1) synchronized multi-modal training data pairing on-body
vision-IMU sensor streams with ground-truth 3D hand poses from a motion-capture system, and (2) a cross-sensor deep
attention mechanism that adaptively modulates the trust assigned to the
vision and individual IMU sensors. To evaluate AVI-HT in real-world settings, we conduct extensive experiments on our DexGloveHOI
dataset that consists of 100K+ pairwise
vision-IMU samples with synchronized 3D annotated poses, in which users manipulate a variety of objects during daily tasks. We compare against multiple single- and multi-modal tracking approaches under two hand models (UmeTrack, MANO). The results show that AVI-HT reduces mean keypoint error by 16.1% and its wrist-aligned variant by 24.2% over the baselines. Ablation studies further reveal the per-finger contribution of IMU sensors across activity types, and the model's sensitivity to IMU noise and temporal misalignment in
vision-IMU fusion.
Read the full abstract ▾
All authors Ziyi Kou, Ankit Kumar, Mia Huang, Taylor Niehues, Vatsal Mehta, Ergys Ristani, Li Guan
Robotics
PGDG: Physically Grounded Data Generation for Robust Bimanual Policy Learning from a Single Demonstration
Cunxi Dai, Haoran Chang, Aditya Nisal et al. (+5 more)
Behavior cloning for contact-rich bimanual manipulation remains challenging because diverse demonstrations are expensive to collect, and even small disturbances can push the system into off-
manifold states where no recovery supervision is available. We propose PGDG, a data generation framework with zero-shot curation that expands a single demonstration into a compact
dataset of physically plausible, successful, and diverse recovery behaviors without additional human labeling. PGDG iterates between a physics-grounded sampler and a
dataset curator, where the curator selects informative, non-redundant, and recoverable behaviors to update the sampling distribution toward under-covered recovery modes, and the sampler draws physically plausible rollout candidates from this updated distribution and retains successful trajectories. To further improve data quality, PGDG applies short-horizon sampling-based control to relabel selected risky states with corrective actions. Across four bimanual manipulation tasks, PGDG consistently outperforms spatial-only augmentation in both simulation and zero-shot real-world transfer. On RotateBox-Pitch, success improves from 38% to 93% in simulation and from 35% to 82% in the real world. PGDG also enables effective foundation models fine-tuning such as GR00T, increasing success from 46% to 77%. Additional results are available in our website: https://cunxid.github.io/PGDG/.
Read the full abstract ▾
All authors Cunxi Dai, Haoran Chang, Aditya Nisal, Rahul Kumar, Guofei Chen, Tao Chen, Yuzhe Qin, Guanya Shi
Robotics
Gait-Level Motion Design and Evaluation Framework for Grasp-Based Dynamic Locomotion in Microgravity
Chaerim Moon, Joohyung Kim and Justin K. Yim
Locomotion in microgravity often relies on sparsely and irregularly arranged anchors, motivating grasp-based mobility with multiple limbs. In this setting, dynamic traversal requires consecutive anchored interactions under coupled dynamic and kinematic constraints, yet the effects of gait-level motion design on locomotion feasibility and performance remain insufficiently understood. This paper formulates the feasibility and performance objectives for grasp-based dynamic locomotion and develops a gait-level parameter-metric framework that relates motion parameters to corresponding evaluation metrics. A physics-based simulation study instantiates the framework across two quadruped morphologies in randomized three-dimensional anchor environments. Controlled variations in gait-level parameters reveal broadly consistent effects on contact support, motion-induced loading, kinematic feasibility, actuation demand, and traversal time across the two
robot realizations. These findings suggest that the investigated gait-level parameters provide an interpretable basis for analyzing feasibility and performance trade-offs.
Read the full abstract ▾
Robotics
Closed-Loop Sim-to-Real Reinforcement Learning for Deformable Microfiber Shape Control
Alessandro Amici, Houari Bettahar, Veeti Jaakkola et al. (+1 more)
Autonomous contact-based micromanipulation is challenging because surface and interfacial interactions at the microscale are difficult to model accurately, limiting the use of conventional model-based control and sim-to-real learning. We present a closed-loop sim-to-real reinforcement learning (RL) approach for microfiber shape control on a surface. The central idea is to train
geometric shape regulation in a simplified frictionless simulator and rely on real-time visual feedback during deployment to iteratively correct the observed effects of unmodeled surface interactions. An RL policy trained entirely in simulation is transferred directly to a physical dual-gripper micromanipulation system operating at 40 Hz, without retraining or domain adaptation. Using silk microfibers as a testbed, the policy achieves a mean point-wise shape error of 270 $\pm$ 80 $\mu$m across twenty-four diverse initial configurations. Across nine specimens covering all combinations of three fiber diameters (50, 80, and 120 $\mu$m) and three manipulated lengths (10 mm, 15mm, and 20 mm), the same policy achieves sub-millimeter final shape error without any retraining or retuning. These results show that a policy learned in a simplified simulator can achieve repeatable real-world microfiber shape regulation under surface contact, provided that the task-relevant effects of the sim-to-real mismatch remain observable and correctable within the closed feedback loop.
Read the full abstract ▾
All authors Alessandro Amici, Houari Bettahar, Veeti Jaakkola, Quan Zhou
Robotics
Distributed Multi-Coverage for Robot Swarms
Mariem Guitouni and Aaron T. Becker
Autonomous drone swarms deployed for surveillance, environmental monitoring, and infrastructure inspection must maintain reliable coverage of critical assets despite
robot failures. This requires multicoverage: each asset must be observed by multiple
robots for redundancy, with coverage requirements varying by asset importance. While recent work has solved the centralized problem optimally using integer programming, practical deployments face constraints that demand distributed solutions:
robots operate with limited communication ranges, onboard computation restricts global planning, and partial system failures must not cause mission abort. We present a distributed multicoverage
algorithm for
robot swarms operating with local sensing, local communication, and no global coordination.
Read the full abstract ▾
Robotics
Flying Together: Human-Guided Immersive Shared Control for Aerial Robot Teams in Unknown Environments
Lou De Bel-Air, Luca Morando, Ruitao Chen et al. (+7 more)
While autonomous multi-
robots can achieve safe and coordinated
navigation, they often struggle to adapt to unforeseen conditions and to capture operator-driven objectives in unstructured environments. We present a Virtual Reality (VR)-based shared control framework for teams of drones operating in constrained and unknown environments, enabling real-time, user-guided exploration. At the core of our approach is a novel, user-guided motion-primitive-based planner that computes continuous, collision-free trajectories while continuously integrating operator input. This planner is coupled with an admittance controller, allowing the operator to flexibly influence team behavior and guide drones toward regions of interest that autonomous planners may overlook. The system supports mixed-reality operations with both physical and simulated drones, and implements a bilateral VR-based interface, allowing the operator to guide the
robot team via migration points while receiving immediate visual feedback of the team state. Experimental results show that shared control improves obstacle avoidance, maintains inter-
agent spacing, and reduces operator effort, demonstrating the feasibility and advantages of immersive, human-in-the-loop multi-
robot navigation.
Read the full abstract ▾
All authors Lou De Bel-Air, Luca Morando, Ruitao Chen, Keru Wang, Benjamin Jarvis, Charbel Toumieh, Yang Zhou, Ken Perlin, Dario Floreano, Giuseppe Loianno
Robotics · Cross-listed from cs.CV
PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objects
Ziang Cao, Yinghao Liu, Haitian Li et al. (+5 more)
Simulation-ready physical 3D assets have emerged as a promising direction owing to their broad applicability in downstream tasks. However, most existing 3D generation methods either neglect physical properties or are limited to a single asset category, e.g., rigid, deformable, or articulated objects. To address these limitations, we introduce PhysX-Omni, a unified framework for simulation-ready physical 3D generation across diverse asset types. Specifically, we develop a novel and efficient geometry representation tailored for
Vision-
Language Models, which directly encodes high-resolution 3D structures without compression, significantly improving generation performance. In addition, we construct the first general simulation-ready 3D
dataset, PhysXVerse, covering diverse indoor and outdoor categories. Furthermore, to comprehensively and flexibly evaluate both generative and understanding capabilities in the wild, we propose PhysX-Bench, which encompasses six key attributes: geometry, absolute scale, material, affordance, kinematics, and function description. Extensive experiments with conventional metrics and PhysX-Bench show that PhysX-Omni performs strongly in both generation and understanding. Moreover, additional studies further validate the potential of PhysX-Omni for applications in simulation-ready scene generation and robotic policy learning. We believe PhysX-Omni can significantly advance a wide range of downstream applications, particularly in
embodied AI and physics-based simulation.
Read the full abstract ▾
All authors Ziang Cao, Yinghao Liu, Haitian Li, Runmao Yao, Fangzhou Hong, Zhaoxi Chen, Liang Pan, Ziwei Liu
Robotics
HITL-D: Human In The Loop Diffusion Assisted Shared Control
Riley Zilka, Sergey Khlynovskiy, Allie Wang et al. (+1 more)
Autonomous manipulation systems have achieved remarkable capabilities, yet the integration of human expertise with
diffusion-based policies in shared control remains relatively unexplored. In this paper, we propose Human-In-The-Loop
Diffusion (HITL-D), a shared control framework that enhances user performance in multi-step, insertion, and fine manipulation tasks. HITL-D leverages a novel combination of
diffusion-based policies and human control to provide autonomous end effector orientation updates conditioned on a scene point cloud and the Cartesian position of the end effector. This approach reduces the number of joystick control axes required, thereby lowering mental workload. In a multi-task user study with 12 participants, HITL-D reduced average task completion times by 40%, decreased perceived workload by 37%, and improved Likert-scale ratings for independence, intuitiveness, and confidence compared to traditional teleoperation methods. These results demonstrate that HITL-D effectively integrates human expertise with autonomous assistance, improving both objective and subjective aspects of teleoperation.
Read the full abstract ▾
All authors Riley Zilka, Sergey Khlynovskiy, Allie Wang, Martin Jagersand
Robotics
Lost in Fog: Sensor Perturbations Expose Reasoning Fragility in Driving VLAs
Abhinaw Priyadershi and Jelena Frtunikj
Interpretable autonomous driving planners depend not only on generating explanations, but also on those explanations remaining reliable under real-world sensor degradation. In this paper we present a controlled perturbation study of
Vision-
Language-Action (VLA) robustness in autonomous driving, evaluating Alpamayo R1 (10B parameters) across 1,996 scenarios under eight sensor perturbations (Gaussian noise at four intensities, two lighting extremes, and two fog levels; ${\sim}18{,}000$ inference trials). We find that
reasoning consistency is a high-fidelity indicator of trajectory reliability: when Chain-of-Causation (CoC) explanations change after perturbation, trajectory deviation spikes $5.3{\times}$ (21.8m vs 4.1m), with $r\!=\!0.99$ across attack types and $r_{pb}\!=\!0.53$ per-sample (Cohen's $d\!=\!1.12$). A controlled ablation provides evidence that enabling CoC generation is associated with improved trajectory accuracy (11.8% on average across conditions; $p < 0.0001$) under matched inference settings. Over the tested noise range ($\sigma \in \{10, 30, 50, 70\}$), degradation is approximately linear ($R^2\!=\!0.957$), while standard input preprocessing defenses provide only marginal relief. Together, these results establish CoC consistency as a quantitative proxy for planning safety and motivate
reasoning-based runtime monitoring for safer VLA deployment.
Read the full abstract ▾
Robotics · Cross-listed from eess.SY
Fully Actuated Manifold Constraint Based Output Feedback Control for Input-Constrained Uncertain Nonlinear Systems
Dianrui Mu, Changchun Hua, Yafeng Li et al. (+2 more)
This paper presents a low-complexity, model-free, output-feedback controller for a class of unknown time-varying nonlinear systems with unknown input constraints. The controller achieves the preset control accuracy when the actuator is not saturated and maintains flexible control accuracy after actuator saturation. This result extends existing constraint control methods for linear
manifolds to a more general form, including the construction of nonlinear
manifolds and various types of constraints, thereby achieving preset control accuracy within finite or fixed time. Additionally, flexible control under unknown saturation is achieved through the construction of an error-driven flexible constraint. Finally, second-order and higher-order control examples and simulations are provided.
Read the full abstract ▾
All authors Dianrui Mu, Changchun Hua, Yafeng Li, Jiannan Chen, Rao Wei
Robotics
roto 2.0: The Robot Tactile Olympiad
Elle Miller, Jayaram Reddy, Ayush Deshmukh et al. (+4 more)
Tactile-based reinforcement learning (RL) is currently hindered by fragmented research and a focus on over-saturated orientation tasks. We introduce v2 of the
Robot Tactile Olympiad (roto 2.0), a GPU-parallelised
benchmark designed to standardise tactile-based RL across four distinct robotic morphologies (16-DOF to 24-DOF). Unlike prior
benchmarks, roto focuses on end-to-end "blind" manipulation, utilising only proprioception and tactile sensing without state information or
distillation. We demonstrate a significant performance leap, with our blind
agents achieving 13 Baoding ball rotations in 10 seconds, an order of magnitude faster than current state-of-the-art speeds. By open-sourcing our environments and robustly tuned baselines, we reduce the barrier to entry and enable researchers to prioritise fundamental algorithmic challenges over tedious RL tuning. Website: https://elle-miller.github.io/roto/
Read the full abstract ▾
All authors Elle Miller, Jayaram Reddy, Ayush Deshmukh, Trevor McInroe, David Abel, Oisin Mac Aodha, Sethu Vijayakumar
Robotics
PointACT: Vision-Language-Action Models with Multi-Scale Point-Action Interaction
Shizhe Chen, Paul Pacaud and Cordelia Schmid
Vision-
Language-Action (VLA) models have shown strong potential for general-purpose robotic manipulation by leveraging large pretrained
vision-
language backbones. However, most existing VLAs rely primarily on 2D visual representations, which limit their ability to reason about fine-grained geometry and spatial grounding - capabilities that are essential for precise and robust manipulation in 3D environments. In this paper, we propose PointACT, a dual-system 3D-aware VLA policy that integrates hierarchical 3D point cloud representations directly into the action decoding process. PointACT employs a multi-scale point-action interaction mechanism with efficient bottleneck window self-
attention, enabling evolving action
tokens to densely attend to both local
geometric detail and global scene structure. We evaluate PointACT on the LIBERO and RLBench
benchmarks and systematically compare it against monolithic and dual-system VLA baselines, including variants augmented with point cloud inputs. PointACT achieves consistent improvements across both
benchmarks, increasing success rates by 10% on the challenging RLBench-10Tasks suite over state-of-the-art pretrained VLAs, with even larger gains when the
vision-
language backbone is frozen and the action expert is trained from scratch. Extensive ablation studies demonstrate that tightly coupling hierarchical 3D geometry with pretrained 2D semantic representations is critical for robust and spatially grounded
robot control. Our results also highlight the promise of pretrained 3D representations for 3D-aware VLA policies.
Read the full abstract ▾
Robotics
MC-Risk: Multi-Component Risk Fields for Risk Identification and Motion Planning
Maximilian Link, Yingjie Xu, Yingbai Hu et al. (+1 more)
We present MC-Risk, a planner-aligned, multi-component risk field on a bird's-eye-view grid that yields early, calibrated, and class-aware risk localization. MC-Risk linearly composes three interpretable modules: (i) a motorized-
agent field that fuses a black-box
multimodal trajectory predictor with an analytic Gaussian-
torus construction whose lateral width grows with speed/
curvature and whose height attenuates with look-ahead; (ii) a VRU risk field that replaces isotropic pedestrian blobs with a forward-biased anisotropic kernel aligned to heading and speed; and (iii) a road penalty field that exploits full HD-map
topology, imposing an off-road penalty and lane-aware risk exposure for same/opposite directions. We conduct, to our knowledge, the first standardized quantitative evaluation of a risk-field formulation on RiskBench's collision subset. MC-Risk attains the best overall risk localization and the earliest hazard indication. Finally, we demonstrate a plug-and-play planning interface by using the field as an MPC cost density, enabling risk-aware trajectory generation without additional training.
Read the full abstract ▾
All authors Maximilian Link, Yingjie Xu, Yingbai Hu, Yinlong Liu
Robotics
From swept contact to pose: Probe-aware registration via complementary-shape docking
Chen Chen, Yunwen Li, Yifan Xu et al. (+5 more)
Accurate registration between a prior model and the real scene is essential for high-precision robotic manipulation, yet optical methods suffer from long calibration chains, line-of-sight constraints, and fabrication errors. We propose a calibration-free alternative that reformulates contact registration as complementary-shape docking between the object and the probe's swept volume, explicitly accounting for probe geometry and leveraging both contact and non-contact evidence. Our solver integrates a global-to-local search via 3D FFT correlation over low-discrepancy SO(3) samples, then followed by continuous SE(3) refinement using Lie-
algebra updates and analytic contact sensitivities. This pipeline yields efficient exploration and metric-grade convergence without fragile point correspondences. Simulation across free-form meshes achieved sub-0.04 mm and sub-0.4{\deg} accuracy and robustness to pose noise and contact loss. On a tooth-preparation
robot, our method attained 0.42 mm and 3.75{\deg}, outperforming an optical tracker registration while requiring no external sensors. These results demonstrate a practical and precise registration strategy for
surgical and industrial
robots.
Read the full abstract ▾
All authors Chen Chen, Yunwen Li, Yifan Xu, Xiangjie Yan, Chang Shu, Jianxia Hou, Shiji Song, Xiang Li
Robotics · Cross-listed from cs.CV
Closed Loop Dynamic Driving Data Mixture for Real-Synthetic Co-Training
Hongzhi Ruan, Pei Liu, Weiliang Ma et al. (+5 more)
Data scaling is fundamental to modern deep learning, and grows increasingly critical as autonomous driving shifts to end-to-end learning. Real-world driving data is expensive to annotate and scene-biased, making real-synthetic co-training with near-infinite synthetic data a promising direction. However, naively incorporating all available synthetic data is inefficient and leads to distribution shifts, and optimizing data mixture under practical training budgets remains a critical yet under-explored problem. In this sense, we claim that the mixture of training data requires clear guidance in terms of scene types and quantities. Particularly in this work, we conceptualize the data mixture approximately as a dynamic optimization process that iteratively adjusts the training data mixture to maximize model performance, guided by closed-loop evaluation feedback, and propose AutoScale, a fully automated closed-loop data engine unifying scene representation, data mixture optimization and
retrieval, as well as model training and evaluation. Specifically, we propose Graph Regularized AutoEncoder (Graph-RAE) for driving scene representations, introduce Cluster-aware Gradient Ascent (Cluster-GA) for cluster-wise importance estimation and reweighting, and perform cluster-guided vector
retrieval to select high-value samples. Experiments on NavSim demonstrate that AutoScale outperforms vanilla co-training and cross-domain baselines, achieving better performance with fewer synthetic samples under constrained budgets.
Read the full abstract ▾
All authors Hongzhi Ruan, Pei Liu, Weiliang Ma, Zhengning Li, Xueyang Zhang, Jun Ma, Dan Xu, Kun Zhan
Robotics
Learning Robust Dexterous In-Hand Manipulation from Joint Sensors with Proprioceptive Transformer
Senlan Yao, Chenyu Yang, Jaehoon Kim et al. (+2 more)
In-hand object manipulation is a fundamental yet challenging capability for dexterous
robots. Despite significant progress in dexterous manipulation, existing approaches rely heavily on
vision or tactile sensing to track object states, while joint sensing – the most readily available modality on any robotic hand – remains largely overlooked, particularly for tendon-driven hands. In this paper, we study how far joint sensing alone can go by asking: (i) whether motor encoders or direct joint sensing provides better proprioceptive feedback, (ii) how to extract environment information from joint measurements, and (iii) whether joint-only control can achieve competitive real-world performance without external perception. We present the Proprioceptive
Transformer (PT), an exteroceptive-free approach for continuous cube rotation on a tendon-driven dexterous hand that uses only joint sensing feedback. A teacher policy is first trained via reinforcement learning with privileged object information, then distilled into PT, which operates solely on joint position and velocity histories. The
Transformer architecture effectively extracts implicit object state information from temporal patterns in joint sensor readings. Experiments on the real ORCA hand show that our approach achieves 3.1x higher rotation speed than baselines. We also demonstrate that our PT achieves a 23.4% lower RMSE for cube position estimation than the MLP baseline, indicating superior extraction of exteroceptive information from proprioceptive sources.
Read the full abstract ▾
All authors Senlan Yao, Chenyu Yang, Jaehoon Kim, Aristotelis Sympetheros, Robert K. Katzschmann
Robotics · Cross-listed from cs.CV
Hyper-V2X: Hypernetworks for Estimating Epistemic and Aleatoric Uncertainty in Cooperative Bird's-Eye-View Semantic Segmentation
Abhishek Dinkar Jagtap, Sanath Tiptur Sadashivaiah and Andreas Festag
Cooperative perception enabled by Vehicle-to-Everything (V2X) communication enhances autonomous driving safety by creating a unified environmental representation through shared sensory data. While recent works have advanced multi-
agent fusion for improved perception, uncertainty quantification in such cooperative frameworks remains largely unexplored. This paper introduces Hyper-V2X, a hypernetwork-based framework for estimating both epistemic and aleatoric uncertainties in V2X-based perception. Specifically, we propose a partial weight generation scheme and V2X context
embedding module that conditions a
Bayesian hypernetwork on fused multi-
agent features to generate weight distributions for
stochastic Bird's-Eye-View (BEV)
segmentation. Unlike existing deterministic BEV models, Hyper-V2X enables efficient uncertainty estimation with little computation overhead. Our approach is architecture-agnostic, and can be seamlessly integrating with modern cooperative backbones such as CoBEVT. Experiments on the OPV2V
benchmark demonstrate that Hyper-V2X provides accurate, well-calibrated uncertainty estimates and improves overall perception reliability. Our code and
benchmark are publicly available under an open-source license: https://github.com/abhishekjagtap1/Hyper-V2X
Read the full abstract ▾
Robotics
Learning Structural Latent Points for Efficient Visual Representations in Robotic Manipulation
Yicheng Jiang, Jiaxu Wang, Junhao He et al. (+8 more)
Current 3D-aware pretraining methods for
embodied perception and manipulation are largely built on differentiable rendering frameworks, producing either fully implicit neural fields or fully explicit
geometric primitives. Implicit representations, while expressive, lack explicit structural cues, whereas explicit ones preserve geometry but suffer from resolution limits and weak generalization. To address these limitations, we propose a novel pretraining framework that learns a hybrid representation-structural latent points. Specifically, we insert a point-wise latent variational autoencoder into the latent space of a point-cloud autoencoder, jointly regularizing point-wise features and coordinates toward a Gaussian prior. The resulting compact latent preserves coarse structural tendencies, which do not encode precise geometry but capture richer rough shape and semantic information, effectively combining the expressiveness of implicit representations with the structural priors of explicit ones. In addition, informed by shared design choices in prior work, we develop a streamlined, efficient 3DGS-based rendering pipeline that is deliberately kept lightweight, improving efficiency while leaving greater representational capacity to the front-end latent module. Extensive evaluations on RLBench, ManiSkill2, and a real-
robot platform demonstrate consistent gains in task success, sample efficiency, and robustness to viewpoint and scene variations over strong baselines. Ablation studies further confirm that each component of our framework is critical to overall performance.
Read the full abstract ▾
All authors Yicheng Jiang, Jiaxu Wang, Junhao He, Zesen Gan, Junhao Li, Qiang Zhang, Jingkai Sun, Jiahang Cao, Mingyuan Sun, Xiangyu Yue, Qiming Shao
Robotics
DiffCVaR: Reinforcement Learning for Risk Adaptation via Differentiable CVaR Barrier Functions
Xinyi Wang, Taekyung Kim, Bardh Hoxha et al. (+2 more)
Planning through crowded environments under uncertain obstacle motions remains difficult, as
stochastic interactions often induce overly conservative behavior or reduced efficiency. To address this challenge, we propose an end-to-end risk adaptation framework for crowd
navigation under obstacle-motion uncertainty modeled by a Gaussian mixture model. The framework combines reinforcement learning~(RL) with a differentiable quadratic-program safety layer based on Conditional Value-at-Risk~(CVaR) barrier functions, jointly learning nominal control input, risk level, and safety margin and enforcing explicit probabilistic safety constraints. This design enables context-aware adaptation, promoting efficient behavior while invoking caution only when necessary. We conduct extensive evaluations in dynamic, uncertain, and crowded environments across varying obstacle densities and
robot models, and further assess generalization under three out-of-distribution cases. Comparisons across optimization-based, RL-based, and integrated RL and optimization methods are provided, and the proposed method is shown to deliver the strongest overall performance in safety, efficiency, and generalization under uncertainty.
Read the full abstract ▾
All authors Xinyi Wang, Taekyung Kim, Bardh Hoxha, Georgios Fainekos, Dimitra Panagou
Robotics
To Select or not to Select, that is the Question: Distilling Robot Skill Prediction into a Small Ensemble
Haechan Mark Bong, Simon Roy, Euhid Aman et al. (+1 more)
As
robot fleets become more heterogeneous, including
humanoids, rovers, quadrupeds, and drones, selecting the right
robot for a task becomes a core systems problem. We study
robot skill prediction: mapping a natural-
language task description to the physical capabilities required to execute it, such as fly, wheels, legs, surface water, under water and hands. Since labelled data that maps natural-
language task descriptions to
robot's physical capabilities does not exist, we construct a synthetic task-to-skill
dataset using LLM-assisted generation and targeted label
auditing. Trained on this data, a ~133M-parameter ensemble of two fine-tuned sentence encoders (mpnet + MiniLM) reaches 83.5% task-to-skill matching on a stratified 200 task
dataset, outperforming Kimi K2 (1T MoE) at 72.0%, GPT-OSS-120B at 71.5%, and Llama-4-Scout-17B at 69.0% under the same zero-shot
prompt. These results suggest that, for fixed
robot skill taxonomies, small specialized models trained on synthetic data can outperform much larger general-purpose LLMs for fleet-level task routing.
Read the full abstract ▾
All authors Haechan Mark Bong, Simon Roy, Euhid Aman, Giovanni Beltrame
Robotics
A Terrain-Adaptive epsilon-Constraint MPC for Uneven Terrain Kinodynamic Planning
Otobong Jerome, Geesara Kalathunga and Tiago Nascimento
Kinodynamic planning for car-like vehicles on uneven terrain requires simultaneously optimizing competing objectives such as path efficiency and pose stability. This work presents an adaptive epsilon-constraint method integrated into a Model Predictive Control (MPC) framework, where the epsilon bounds are dynamically adjusted based on terrain descriptors to explore the Pareto front in real time. To capture vehicle-terrain dynamics, we develop a semi-parametric model combining analytical vehicle dynamics with a Sparse Gaussian Process (SGP) trained on the same terrain descriptors. The proposed epsilon-MPC is evaluated against MPPI and GAKD baselines, achieving a 94%
navigation success rate while reducing maximum orientation deviation by 24% and improving multi-objective trade-off quality by 23%.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Comparative Analysis of Military Detection Using Drone Imagery Across Multiple Visual Spectrums
Sourov Roy Shuvo, Prajwal Panth, Rajesh Chowdhury et al. (+3 more)
In modern warfare, drones are becoming an essential part of intelligence gathering and carrying out precise attacks in different kinds of hostile environments. Their ability to operate in real-time and hostile environments from a safe distance makes them invaluable for surveillance and military operations. The KIIT-MiTA
dataset is comprised of images of different military scenarios taken from drones, and these provide a foundation for detecting military objects, but it does not take into account the various types of real-world scenarios. With that in mind, to evaluate how the models are performing under varying conditions, four different types of
datasets are created: Gray Scale, Thermal
Vision, Night
Vision, and Obscura
Vision. These simulate the real-world environments such as low visibility, heat-based imagery, and nighttime conditions. The YOLOv11-small model is trained and used to detect objects across diverse settings. This research boosts the performance and reliability of drone-based operations by contributing to the development of advanced detection systems in both defensive and offensive missions.
Read the full abstract ▾
All authors Sourov Roy Shuvo, Prajwal Panth, Rajesh Chowdhury, Sorup Chakraborty, Sudip Chakrabarty, Prasant Kumar Pattnaik