Robotics
When Does Test-Time Physical Diagnosis Pay? A Frozen Policy Buys Evidence It Never Reads
Zhengshu Zhang
When a
robot faces unfamiliar physical conditions, a common approach is to collect evidence about what changed and adapt. For such diagnosis to improve behavior, six ordered empirical conditions must hold: a meaningful reference, identifiability of the physical condition, use of the acquired evidence, decision value, selection value over a fixed alternative, and safe realization. We test this chain in controlled and public environments. It holds end to end in our controlled environments. After transfer to unseen mechanisms, however, it breaks at evidence use. On decisions requiring the full trace, the frozen decoder does not change its choice. A linear model using only trace increments recovers the correct choice on mechanisms excluded from fitting, showing that the trace is informative but unused. The failure is concentrated at the richest evidence level: those decisions fall to chance, while decisions settled with lower-cost evidence remain correct, a split hidden by aggregate accuracy. The same chain can fail at other links in public environments. Successful physical identification therefore guarantees neither evidence use nor useful adaptation; evaluation should identify where the chain breaks rather than rely on recovery accuracy or aggregate performance alone.
Read the full abstract ▾
Robotics · Cross-listed from cs.NI
Toward Mission-Critical ISAC: Reliable Energy-Aware Coordination in UAV Swarms
Masoud Shokrnezhad, Tarik Taleb and Amir Javadpour
Unmanned aerial vehicle (
UAV) swarms deployed in mission-critical applications must simultaneously track a mobile aerial target and maintain reliable data links. However, active integrated sensing and communication (
ISAC) operation imposes a dual energy burden on propulsion and transmission, threatening mission continuity through premature battery depletion. In this paper, we propose a two-tier
UAV swarm architecture in which mission
UAVs (MUAVs) execute cooperative
ISAC for mobile aerial target tracking while dedicated charging
UAVs (CUAVs), equipped with solar harvesting panels, replenish low-battery MUAVs via aerial
UAV-to-
UAV wireless power transfer (WPT). We formulate the joint minimization of the cooperative posterior Cramér-Rao bound (PCRB) over MUAV trajectories, per-slot sensing-communication time splits, WPT scheduling and admission, and CUAV rendezvous trajectories, subject to minimum uplink rate, dual-tier energy causality, WPT proximity, collision-avoidance, and speed constraints, yielding a non-convex mixed-integer program (MIP) that, to the best of our knowledge, is the first to jointly couple cooperative
ISAC sensing quality with aerial WPT and dual-tier energy management. To solve it efficiently, we propose Receding-Horizon Alternating Optimization (RHAO), a four-block per-slot
algorithm that decomposes the problem into: charging admission via the Hungarian
algorithm, MUAV trajectory and time-split via successive convex approximation (SCA), CUAV rendezvous, and WPT power allocation, with monotone convergence guarantees. Simulation results demonstrate that RHAO reduces the mean PCRB by 16.8 times over a fixed-time-split baseline and 5.4 times over a static-trajectory scheme, while the aerial WPT subsystem sustains all MUAVs above the energy-critical threshold throughout the full mission horizon.
Read the full abstract ▾
Robotics
The Neverwhere Visual Parkour Benchmark Suite
Ziyu Chen, Henghui Bao, Haoran Chang et al. (+12 more)
State-of-the-art visual locomotion controllers are increasingly capable at handling complex visual environments, making evaluating their real-world performance before deployment increasingly difficult. This work intends to narrow this train/evaluation gap by developing a collection of hyper-photo-realistic, closed-loop evaluation environments - The Neverwhere
Benchmark Suite - comprised of over sixty 3D Gaussian Splatting reconstructions of urban indoor and outdoor scenes. Our goal is to encourage large-scale and reproducible
robot evaluation by making it easier to create and integrate Gaussian splats-based reconstructions into simulated continuous testing setups. We also underscore the potential pitfalls of relying exclusively on 3D Gaussian-generated data for training, by providing policy checkpoints trained over multiple Neverwhere scenes and their performance when evaluated in novel scenes. Our analysis illustrates the necessity of sourcing diverse data to ensure performance. Code and data are available on the project page: https://ziyc.github.io/neverwhere-bench/.
Read the full abstract ▾
All authors Ziyu Chen, Henghui Bao, Haoran Chang, Alan Yu, Ran Choi, Kai McClennen, Gio Huh, Kevin Yang, Ri-Zhao Qiu, Yajvan Ravan, John J. Leonard, Xiaolong Wang, Phillip Isola, Ge Yang, Yue Wang
Robotics
XRoboToolKit-T: Teleoperation with High Stability and Precision with Tactile Sensing for Contact-rich Manipulation
Xiwen Dengxiong, Xueting Wang, Ke Jing et al. (+2 more)
Collecting high-quality
robot data for contact-rich manipulation tasks is essential for enabling
robots to acquire real-world skills. However, existing data collection solutions often lack the capability to obtain stable and high-frequency tactile feedback, limiting their effectiveness in contact-rich manipulation scenarios. In this work, we propose a versatile teleoperation system with tactile-driven assistance to enable high-frequency and stable contact-rich manipulation. The proposed XRoboToolKit-T teleoperation system incorporates a tactile-informed force control architecture, designed to ensure both stable and precise force control in contact-rich manipulation during teleoperation. The stabilizer haptic module rapidly analyzes the normal force distribution and infers pseudo shear force, enabling real-time tactile-based assistance during manipulation. The refiner haptic module integrates a
vision-
language-action model to predict and refine manipulation actions based on tactile sensing data and task descriptions. We apply the proposed teleoperation system to challenging contact-rich manipulation tasks, including grasping a deformable rubber pipette for
liquid transfer and inserting a medical syringe into a vascular training pad, to demonstrate the effectiveness of tactile-informed force control. Furthermore, the system achieves higher data collection efficiency and improved manipulation stability compared to state-of-the-art teleoperation without tactile assistance.
Read the full abstract ▾
All authors Xiwen Dengxiong, Xueting Wang, Ke Jing, Rui Li, Yunbo Zhang
Robotics · Cross-listed from physics.optics
A Programmable Optics Cloud Laboratory
Sachin Vaidya, Caio Silva, Seou Choi et al. (+2 more)
Laboratory automation can improve experimental throughput, accessibility, and reproducibility, but many robotic laboratory systems remain difficult to reconfigure. This challenge is especially pronounced in free-space optics, where experiments are built from heterogeneous components, require precise
alignment, and are frequently rearranged as experimental goals change. In this work, we present the Programmable Infrastructure for Cloud Optics (PICO), a robotic cloud-laboratory architecture designed to make reconfigurable optical experiments easier to program, operate, and reproduce. PICO provides a common domain-specific abstraction and software layer through which experimental configurations and actions can be controlled across different user interfaces. This enables the same physical laboratory to support remote interactive use, scripted experiments, autonomous routines, and features such as version control. We implement PICO on a robotic free-space optics platform and demonstrate it through an experimental case study.
Read the full abstract ▾
All authors Sachin Vaidya, Caio Silva, Seou Choi, Joshua Chen, Marin Soljačić
Robotics
Collision-Aware Humanoid Whole-Body Control under Imperfect Tracking Targets
Mohitvishnu S. Gadde, Ashish Malik, Pranay Dugar et al. (+3 more)
Humanoid
robots often execute motion commands through whole-body controllers (WBCs) that track targets while maintaining balance and stability. However, most WBCs are blind to scene geometry, which can lead to collisions from imperfect target motions that are geometrically unsafe due to perception, planning, or teleoperation errors. We propose RECAL, a
Robot–Environment Cross-
Attention Layer that wraps a blind WBC to trade off target tracking against collision avoidance using external scene geometry. RECAL supports collision-aware tracking of floating-base and end-effector commands, including collision avoidance for held objects. It represents the
robot, held objects, and environment as point clouds, using cross-
attention between
robot/object points and the environment to produce geometry-aware control features. In simulation, RECAL improves collision avoidance while preserving target-tracking performance across frozen-arm and adaptive-arm locomotion, object-carrying, and standing-manipulation scenarios relative to alternative geometry-aware WBC architectures. We further demonstrate the controller on a real Digit V3 humanoid
robot.
Read the full abstract ▾
All authors Mohitvishnu S. Gadde, Ashish Malik, Pranay Dugar, Aayam Kumar Shrestha, Fuxin Li, Alan Fern
Robotics
Geometry vs Structure: Graph-Based Diagnostics for LiDAR Point-Cloud Simulation Fidelity
Ghazal Farhani and Taufiq Rahman
Digital twins provide a scalable and cost-effective complement to real-world testing for validating autonomous-driving and advanced driver-assistance system (ADAS) sensor pipelines. However, quantifying their fidelity remains challenging, particularly for 3D
LiDAR point clouds, where conventional
geometric metrics may overlook important structural discrepancies. We present a graph-based framework for evaluating the structural fidelity of simulated
LiDAR point clouds against real-world scans. While scan-level metrics such as Chamfer distance capture point-wise
geometric similarity, they do not explicitly represent connectivity,
topology, or object-level organization. Our framework constructs graphs from real and simulated point clouds, applies Louvain community detection to identify spatially coherent subgraphs, and matches corresponding communities using centroid proximity. For each matched pair, we compute $r_\lambda$, a bounded graph-spectral metric motivated by Weyl's inequality, and compare it with density-aware Chamfer distance (CDC) as a
geometric baseline. Controlled perturbation experiments demonstrate that $r_\lambda$ is invariant to rigid transformations and robust to sensor noise while remaining sensitive to structural deformation. We evaluate the framework on 50 paired real and simulated
LiDAR scans acquired using a Velodyne VLP-32C sensor and CARLA, respectively. The
dataset contains more than 1,000 matched communities across four representative classes: vehicles, vegetation, trees, and building walls. The results show that
geometric and structural measures capture complementary aspects of simulation fidelity, supporting graph-spectral analysis as an additional diagnostic layer for validating digital twins in ADAS and autonomous-driving applications.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
Autonomous Droplet Navigation via Model-Based Reinforcement Learning
Rajneesh Anand and Mayuresh V. Kothare
Precise manipulation of
liquid droplets underpins lab-on-a-chip platforms for diagnostics, chemical synthesis, and biological assays. Yet autonomous droplet transport through confined geometries of varying complexity remains an open challenge. Droplets exhibit contact-angle hysteresis, deformability, and capillary pinning, which make their response to actuation nonlinear and history dependent, that classical controllers and pre-programmed trajectories cannot cope in multi-turn environments. Here we demonstrate autonomous navigation of a
liquid droplet through geometries of increasing complexity on a
gravity driven (Labyrinth) platform using model-based reinforcement learning. A thin silicone oil film reduces contact-line pinning while two-axis tilt supplies the gravitational driving force, and an overhead camera tracks the droplet in real time. An offline-trained policy discovers effective tilt strategies from limited physical interaction data, without simulation or analytical droplet models. The system operates under partial observability, as oil-film thickness, instantaneous contact angle, and droplet deformation state remain hidden from the controller. Despite these challenges, the learned policy achieves reliable navigation across straight, right-angle, and curved-arc paths, including outside-corner geometries. We further demonstrate that a policy trained on a simpler geometry transfers to complex ones, succeeding zero-shot on right-angle and staircase paths and reaching full success on a curved arc with a fifth of the training data. The findings suggest promising avenues for enabling droplet based microfluidic systems to serve as intelligent chemical laboratories.
Read the full abstract ▾
Robotics
UDAV: Uncertainty-Driven Adaptive VLM Waypoint Planner
Ghazal Farhani and Shabnam Shabani
Vision-
language models (VLMs) can generate routes directly from aerial imagery for off-road navigation, but their predictions provide no indication of reliability. We present UDAV, an Uncertainty-Driven Adaptive VLM Waypoint Planner for
UAV-guided UGV navigation. UDAV draws multiple
stochastic trajectory predictions, selects their medoid as a self-consistent nominal route, and estimates predictive uncertainty from their spatial dispersion. When the maximum uncertainty across interior waypoints exceeds a threshold, UDAV invokes a reconsideration stage; otherwise, it returns the medoid directly. We evaluate UDAV on 400 held-out trajectory queries from two
UAV flights.
Stochastic medoid selection reduces the mean average displacement error (ADE) from 147.4 pixels for a deterministic prediction to 115.9 pixels. The complete planner achieves a mean ADE of 110.4 pixels, a 25.1% reduction relative to deterministic planning, while producing valid trajectories for all queries. UDAV also yields the lowest 90th- and 95th-percentile errors among all evaluated configurations, including a higher-budget K=10 consensus baseline. Relative to the K=5 medoid, UDAV reduces these errors from 225.3 and 326.0 pixels to 199.0 and 290.8 pixels, respectively. These results demonstrate that
stochastic VLM predictions provide both a stronger nominal route and an actionable uncertainty signal for selectively mitigating large planning errors.
Read the full abstract ▾
Robotics
Auto-HSI: Personalized human control of a robot swarm on demand by using LLMs for online automatic code generation
Alessandro Nazzari, Nathan Cerisara, Dorian Tonnis et al. (+5 more)
This paper presents Auto-HSI, a method for generating personalized human-swarm interaction (HSI) interfaces on demand. The objective is to enable untrained operators to use natural
language descriptions and gesture demonstrations to explain how they want the
robots to collectively behave in response to their gestures. Based on these inputs, the code should automatically be generated for personalized state machines that will control the
robots as desired, in response to the desired gesture inputs. In the developed Auto-HSI prototype, the generated code produces a personalized interface for centralized control using one- and two-handed gestures, enabling a user to teleoperate the
robots' motion, formation shape, and shape deformation. We test the gesture tracking and code generation components of Auto-HSI against performance
benchmarks. We then test the full Auto-HSI prototype in “live” operation experiments, in which real human operators centrally control 50 simulated
robots in a physics-based simulator, under nominal and noisy conditions. In these experiments,
robots are teleoperated to: score a goal, traverse a maze that requires shape deformation, and score two simultaneous goals by splitting into two groups. We also demonstrate a real human operator making live updates to their personalized Auto-HSI interface during operation (in simulation). Finally, we demonstrate live operation of real
robots.
Read the full abstract ▾
All authors Alessandro Nazzari, Nathan Cerisara, Dorian Tonnis, Raina Zakir, Lorenzo Labarile, Weixu Zhu, Marco Dorigo, Mary Katherine Heinrich
Robotics
ManiSkillFormer: Demonstration-Free Compositional Manipulation via Geometric Contracts and Agentic Skill Graph
Peiqi Yu, Mosam Dabhi, Shangtao Li et al. (+3 more)
Adapting robotic manipulation to new objects and tasks often requires additional demonstrations or manual engineering. Reusable manipulation skills can reduce this effort, but adapting these skills to new scenes remains challenging. We present ManiSkillFormer, a framework for demonstration-free and compositional manipulation that connects perception and action through explicit
geometric contracts. Building on reusable skill schemas, LLM
agents generate contracts specifying the geometry primitives required by each skill, together with corresponding motion templates for semantic objects and task contexts. These contracts guide a perception module to ground task-relevant 3D geometry from observations, which is then used to instantiate reusable motion templates in a skill library. We evaluate ManiSkillFormer on a dual-arm
robot across demonstration-free pick-and-place with 30 instances from 8 object categories, functional manipulation including unscrewing, pouring, pressing, and folding, and three long-horizon tasks. ManiSkillFormer achieves an average success rate of 88.97% for pick-and-place, 75.00% for functional manipulation, and completion rates of 50–80% across the long-horizon tasks, outperforming the evaluated baselines and two ablated pipelines. These results demonstrate the potential of explicit
geometric contracts to support skill reuse and composition across objects and tasks without per-object policy fine-tuning or additional
robot demonstrations. The project website is at https://patricia1019.github.io/ManiSkillFormer/.
Read the full abstract ▾
All authors Peiqi Yu, Mosam Dabhi, Shangtao Li, Bowei Li, Laszlo Jeni, Changliu Liu
Robotics
ConGraspXL: Controllable Constraint-Conditioned Dexterous Grasping Motion Synthesis
Hui Zhang, Mirko Meboldt and Jie Song
Dexterous grasping is usually conducted for specific tasks, leading to heterogeneous constraints such as specific approach directions, desired contact regions, specified wrist trajectories, and functional hand poses. Our previous work, GraspXL, achieves scalable grasping motion synthesis for diverse objects and hand morphologies, while lacking controllability for synthesis under such various task-driven constraints. In this paper, we propose ConGraspXL, which extends GraspXL with controllable constraint-conditioned grasp motion synthesis that accommodates diverse task-driven constraints and their combinations. We introduce a hierarchical constraint formulation, enable flexible constraint composition with a masked residual interface, and improve control precision with dynamic hand centers and feed-forward wrist guidance. Without losing the strong generalization capabilities of GraspXL, ConGraspXL enables precise and flexible controllability for various individual constraints and their combinations, providing a plug-and-play low-level grasp controller for downstream applications such as whole-body grasp completion, functional grasping, and human-motion imitation.
Read the full abstract ▾
Robotics
Tendon-Driven Continuum Robot with Modular Stiffness and In-Situ Self Pose Estimation
Guo Ning Sue, Zheng Cao, Junzhe Hu et al. (+5 more)
Continuum
robots enable smooth shape morphing and safe interaction in confined environments. However, most existing systems are task-specific and depend on external sensing infrastructure, limiting their adaptability and real-world deployment. This paper presents a self-contained modular continuum robotic platform that combines mechanical reconfigurability with onboard pose estimation. The
robot is constructed from interchangeable continuum joints with analytically precomputed stiffness, allowing rapid assembly and direct programming of the
robot shape. Proprioceptive sensing is achieved using
magnetic sensors and a modular learning-based framework, where a single model is trained per joint and reused across configurations. The system is experimentally validated in real world, demonstrating self-sensing capabilities and adaptation without external tracking.
Read the full abstract ▾
All authors Guo Ning Sue, Zheng Cao, Junzhe Hu, Xiangyun Bu, David Quinn, Tiancheng Wu, Zackory Erickson, Carmel Majidi
Robotics
Occupancy Network-Guided Autonomous Robotic Partial Nephrectomy
Ethan Kilmer, Pit Henrich, Jiawei Ge et al. (+13 more)
Autonomous soft-tissue cancer surgery has been limited to interventions on organ surfaces, because current systems cannot perceive and adapt to anatomy once it deforms or is cut. We introduce the first
vision-guided autonomous system capable of performing complete tumor resections for partial nephrectomy. Our system integrates conditional occupancy networks, trained entirely in a physics-based simulation, that infer full 3-D anatomy (tumor, margin tissue, and kidney) from single-view partial point clouds. These occupancy networks maintain intraoperative tracking even as tissue is cut and deformed, enabling adaptive planning and execution. The surgical platform combines a depth camera for capturing surface point clouds, dual robotic arms for electrosurgical cutting and
vacuum-based tissue manipulation, and an autonomous control strategy for tumor resection. In patient-derived hydrogel phantoms under an open partial nephrectomy setting, the
robot performed eight consecutive autonomous tumor resections comprising 77 electrosurgical cuts, with all cuts achieving negative surgical margins and 1.61 $\pm$ 0.48 mm mean absolute margin error. This work demonstrates, for the first time, a foundation for supervised autonomous closed-loop, imaging-driven, margin-negative tumor removal in phantoms.
Read the full abstract ▾
All authors Ethan Kilmer, Pit Henrich, Jiawei Ge, Paul M. Scheikl, Laura Connolly, Soum D. Lokeshwar, Joseph Chen, Justin D. Opfermann, Kaitlyn Kumar, Lauren Shepard, Ahmed Ghazi, Nirmish Singla, Richard J. Cha, Kevin Cleary, Franziska Mathis-Ullrich, Axel Krieger
Robotics
Structure-Preserving Quantum Circuit Architectures for Robot Kinematics
Andrea Morghen, Pierluigi Arpenti, Roberto Schiattarella et al. (+2 more)
Structured spatial data require
quantum encodings that preserve
geometric relations, expose measurable observables, and remain implementable on finite-depth hardware. This work introduces a
quantum representation and
circuit architecture for rigid-body transformations and specializes it to Denavit–Hartenberg kinematics of serial open-chain manipulators. Each translational contribution is factorized into a classical metric magnitude and a signed unit direction encoded by a single-
qubit Bloch vector, while parameterized rotations reproduce the ordered propagation of frame directions. A selector register prepares probabilities proportional to the contribution magnitudes, and the reduced state of a designated readout
qubit encodes their normalized weighted sum. The retained classical scale then reconstructs the metric end-effector position. Two additional readout
qubits encode terminal-frame axes, providing a compact and geometrically interpretable pose interface. At the ideal expectation-value level, measured Pauli observables reproduce the corresponding classical kinematic quantities. Alternative
circuit architectures realize the same representation with different tradeoffs in
qubit count,
circuit depth, controlled operations, and measurement requirements. Validation on a serial manipulator yields numerically negligible position and orientation reconstruction errors under ideal simulation. Finite-shot simulations, noisy executions, transpilation analysis, and a hardware demonstration further characterize statistical error, noise sensitivity, and implementation overhead without asserting computational advantage.
Read the full abstract ▾
All authors Andrea Morghen, Pierluigi Arpenti, Roberto Schiattarella, Giovanni Acampora, Bruno Siciliano
Robotics
ResSafe: Learning Safety Filtering with Residual Reinforcement Learning for Humanoids
Gechen Qu, Tong Zhang, Bike Zhang et al. (+4 more)
Safe control of humanoid
robots remains challenging due to their high-dimensional dynamics, contact-rich interactions, and sensitivity to disturbances. Although reinforcement learning has enabled effective locomotion and motion tracking, learned policies can still generate unsafe actions that lead to instability or falls. In this work, we propose residual reinforcement learning as an implicit safety-filtering mechanism for safe humanoid control. Instead of relying on a single nominal policy to simultaneously balance performance, safety, and robustness, we decouple performance and safety. The nominal policy focuses solely on task performance, while a residual policy learns safety corrections. This decoupling leads to a better performance–safety Pareto trade-off and avoids the need for careful tuning of multiple competing reward terms within a single policy training. We show that the residual policy can act as an implicit safety filter.
Read the full abstract ▾
All authors Gechen Qu, Tong Zhang, Bike Zhang, Yen-Jen Wang, Koushil Sreenath, Claire Tomlin, Jason Jangho Choi
Robotics
MessyMem: Learning-from-Doing Memory for Mobile Manipulation
Anuva Banwasi, William Muckelroy III, Priya Sundaresan et al. (+3 more)
Mobile manipulators deployed across many rooms and visits should improve with experience: after discovering that a cabinet is locked or finding an object in a drawer, the
robot should reuse that knowledge rather than start each task from scratch. Yet today's
robots often treat each task as new: compact scene representations omit interaction-derived knowledge, raw
video histories are difficult to query, and VLM planners reason at inference time without persistently updating what the
robot knows. We present MessyMem, a persistent
memory system that enables mobile manipulators to learn from experience and reuse that knowledge across future tasks. It maintains a spatially grounded 3D scene graph of objects and locations, augments it with properties and outcomes learned through interaction, and links visual observations for fine-grained recall. We evaluate MessyMem in simulation and on a real mobile manipulator. In a continuous 25-task simulation spanning over 3 hours, MessyMem achieves 80.0% task progress, outperforming the strongest ablation by 14.8 percentage points and the strongest external baseline by 28.9 points, while retrieving task-relevant evidence from thousands of stored keyframes and over an hour into the past.
Read the full abstract ▾
All authors Anuva Banwasi, William Muckelroy III, Priya Sundaresan, Linfeng Zhao, Jeannette Bohg, Cherie Ho
Robotics
Beyond Single-Axis Testing: Paired Evaluation of Compound Robustness in Vision-Language-Action Policies
Hiroki Sawada and Shunichi Kasahara
Vision-
language-action policies are typically evaluated one perturbation at a time, providing a useful diagnosis of their sensitivity to individual distribution shifts. Real-world deployment, however, may involve several shifts simultaneously, and it remains unclear how these individual robustness measurements compose. We ask whether compound robustness can be inferred from single-axis evaluations. We introduce LIBERO-CTRL, a six-axis
benchmark that pairs each initial state across single-axis conditions and a matched simultaneous condition. This design reveals two opposing outcome changes that aggregate success rates cannot distinguish: emergent failures, where all single-axis rollouts succeed but the simultaneous rollout fails, and compensated successes, where at least one single-axis rollout fails but the simultaneous rollout succeeds. Because one transition decreases compound success while the other increases it, they can cancel, making aggregate compound performance appear consistent with single-axis measurements even when individual outcomes differ substantially. These opposing transitions can largely cancel in aggregate: even when the difference between the two transition rates is not statistically distinguishable from zero, as many as 29.0% of matched initial states still change outcome. Across six policies and three severity levels, such outcome changes reach 34.5% in the most affected condition. The relative prevalence of the two transitions varies across policies and severities, while the transition rates remain similar under independent re-evaluation of
stochastic policies. Compound robustness therefore cannot be characterized from aggregate single-axis success rates alone; matched per-instance evaluation is needed to reveal how joint perturbations alter behavior.
Read the full abstract ▾
Robotics
Touch2Trace: Tactile-Driven Imitation Learning for Dexterous Cable Tracing
Matteo Grimaldi, David Klee, Ziling Chen et al. (+5 more)
Dexterous manipulation of deformable objects demands continuous fingertip-level regulation of pressure, friction, and incipient slip. We study one of the most challenging cases: dexterous cable tracing, feeding a cable through the hand with repeated pinch-and-curl motions of the thumb and index finger. We introduce Touch2Trace, a tactile-driven imitation-learning system for this task, and provide, to our knowledge, the first systematic real-world characterization of how encoder pretraining, control rate, temporal context, and spatial resolution each shape policy performance. The winning learning recipe combines a tactile encoder pretrained for a custom 32 x 32 piezoresistive sensor (TacV5) via self-supervised learning with a lightweight
transformer policy trained on teleoperated demonstrations via behavior cloning, deployed at 60 Hz on a Tesollo DG-5F hand. Tactile feedback without
vision or explicit cable-state estimation significantly improves tracing performance versus a proprioception-only baseline: from 0.2 cm to 20.1 cm mean distance and 0% to 93% success rate, with zero-shot transfer to unseen cables and routing conditions. The results quantify the influence of key parameters in tactile-driven systems for reliable dexterous deformable object manipulation.
Read the full abstract ▾
All authors Matteo Grimaldi, David Klee, Ziling Chen, Tong Jian, Wonju Lee, Wenjie Lu, Tao Yu, Saleh Nabi
Robotics
SlipSense: Multimodal Tactile Learning for Low-Latency and Generalized Slip Detection
Tong Jian, Aditya Thurvas Senthil Kumar, Xinyi Li et al. (+7 more)
Slip detection is fundamental to dexterous manipulation, yet existing systems often lack precise characterization of detection latency and cross-platform generalization. We present SlipSense, a
multimodal tactile slip-detection framework built on TacV5, a compact sensor integrating a $32 \times 32$ piezoresistive array operating at 240 Hz and a 3-axis MEMS accelerometer operating at 8 kHz. The piezoresistive array captures spatial pressure distributions, while the accelerometer captures friction-induced vibrations, providing complementary slip cues. The framework performs modality-specific encoding, intra-sensor fusion, and cross-modal
attention with causal temporal prediction at 240 Hz. Experiments on a
dataset of 1.4 million frames spanning 37 objects demonstrate the complementarity of the two modalities. SlipSense achieves 96.7% Macro F1 with a false-positive rate below 1.6%, detecting 76% of slip events within 23.1 ms. When trained solely on UMI data, SlipSense generalizes zero-shot to a Tesollo dexterous hand, transferring across unseen objects, distinct sensor units, and robotic platforms without retraining.
Read the full abstract ▾
All authors Tong Jian, Aditya Thurvas Senthil Kumar, Xinyi Li, Ziling Chen, Tianyu Dai, Ali Sengul, Matteo Grimaldi, Wenjie Lu, Saleh Nabi, Tao Yu
Robotics
Goal-Oriented Communications for Physical AI: Design and Testbed
Shutong Chen, Wenkai Zhang, Adnan Aijaz et al. (+2 more)
Physical AI relies on frequently-updated, latency-sensitive
video stream to perceive, reason, and interact with the physical world, resulting in strict latency requirements with much higher data volumes that existing 5G networks cannot support. Goal-oriented communication (GoC) offers as a promising approach to solve this challenge by transmitting only task-relevant semantic representations. However, existing GoC frameworks were mainly evaluated in the simulations while their effectiveness has never been validated in a practical deployment of physical AI application. In this work, we develop an end-to-end GoC testbed for Physical AI, which connects a PiPER
robot arm equipped with an RGB-D camera and a 5G modem to an NVIDIA Jetson AGX Orin edge server through a 5G OpenAirInterface network. We propose and implement three GoC frameworks that transmit 3D bounding boxes, 2D scene graphs, and 3D scene graphs, as three types of semantic representations, respectively. They share the common functional modules designed for closed-loop Physical AI applications, including semantic extraction, full stack 5G transmission,
language model inference, digital twin validation, and robotic control. Extensive experiments on our testbed show that our GoC frameworks reduce the task completion time by up to 52.6% and improve task success probability by up to 45%, compared to the traditional framework that periodically transmits the raw image data. These results validate the practical effectiveness of our GoC framework and pave the way for efficient and reliable Physical AI applications over future 6G networks. Project website: https://sites.google.com/view/goc-physical-ai-testbed.
Read the full abstract ▾
All authors Shutong Chen, Wenkai Zhang, Adnan Aijaz, Miao Guo, Yansha Deng
Robotics
WLA$^3$: World Latent Action Modeling for Semantics, Dynamics, and Kinematics
Peidong Liu, Zhiyuan Xiang, Mingyang Li et al. (+4 more)
Scaling generalist policy models with heterogeneous data is limited by the lack of unified, low-noise action supervision. Human egocentric
videos are abundant, but only a small fraction comes with high-quality hand-action labels. Observed world transitions offer a common source of action-related supervision across data sources. We introduce WLA$^3$ (World Latent Action Modeling for Semantics, Dynamics, and Kinematics), a unified generalist policy model framework built around representations learned by a World Latent Action Model (WLAM). WLAM first learns how
multimodal world states change over a local interval, encoding synchronized camera views and available embodiment-state changes into a compact local latent action and a richer transition feature. Reconstruction from partial modalities and consistency across overlapping windows encourage robust transition representations. WLA$^3$ reuses them across semantics, dynamics, and kinematics: local latent actions support action-sensitive physical-dynamics modeling, segment-level features directly supervise the VLM through a Semantic Latent Aggregate (SLA), and an action expert jointly predicts latent actions together with embodiment-specific
robot controls. Human
videos provide scalable transition supervision, while
robot trajectories ground the shared representation in executable native controls. On LARYBench, the final 32D latent action reaches 67.89% average classification accuracy. WLA$^3$ achieves 81.9% average success across six real-
robot tasks versus 66.2% for $\pi_{0.5}$. Performance improves as generalist policy model mid-training data scales, and human
videos support human-to-
robot transfer. Project page can be found at https://wla-3.github.io/.
Read the full abstract ▾
All authors Peidong Liu, Zhiyuan Xiang, Mingyang Li, Wenhao Li, Jiale Zhang, Jiahao Sun, Jiawei Li
Robotics
DuctAM: A Duct-Assisted Quadrotor-Based Aerial Manipulator Enabling High-Force Push-and-Pull Interactions
Yi Wang, Rui Jin, Xinhang Xu et al. (+4 more)
Uncrewed Aerial Manipulators (UAMs) extend the capabilities of Uncrewed Aerial Vehicles (
UAVs) from perception to physical interaction. Among various aerial interactions, push-and-pull operations are fundamental manipulation primitives that require sustained horizontal forces while maintaining stable flight. In this paper, we propose DuctAM, a compact aerial manipulation platform that enhances horizontal force capability for push-and-pull interactions using two ducted fans integrated along the quadrotor interaction axis. An attitude-force decoupled control scheme enables controllable horizontal forces without requiring large attitude changes. Extensive real-world experiments are conducted to validate the DuctAM. Figure-eight trajectory tracking experiments demonstrate stable flight and accurate motion control in both quad and duct modes. Force-measurement experiments quantify the decoupled longitudinal force capability of DuctAM. Finally, representative push-and-pull interaction tasks, including cart pushing, door closing, and drawer opening, verify the practical effectiveness of DuctAM. The results show that DuctAM achieves significantly improved horizontal interaction force capability while maintaining stable flight compared with conventional
UAVs.
Read the full abstract ▾
All authors Yi Wang, Rui Jin, Xinhang Xu, Haotian Jin, Ruiyang Liu, Yizhuo Yang, Lihua Xie
Robotics
Uncertainty-Guided Sparse Refinement for Action Chunking Transformer Policies
Chenyang Wang, Yuntian Wang, Xiaoxiong Yang et al. (+3 more)
Learning chunk-based visuomotor policies for long-horizon
robot manipulation remains challenging. Recent action-chunking methods have shown promising performance by predicting temporally extended action sequences. However, their failures are often dominated by prediction errors at a small number of critical timesteps rather than uniformly poor predictions across the entire action chunk, making uniform refinement inefficient and insufficiently targeted. To address this bottleneck, we propose Uncertainty-Guided Refinement (UGR), a sparse refinement framework for chunk-based visuomotor policies. Specifically, UGR follows a coarse-to-refine design: it first predicts a full action chunk, estimates per-step temporal uncertainty from the coarse hidden states, and applies residual correction only to the most uncertain timesteps selected by a binary mask. The uncertainty branch is decoupled from the coarse action predictor, enabling clean attribution of the refinement gains to uncertainty-guided correction rather than additional predictor capacity. Extensive experiments on five dual-arm manipulation tasks from the RoboTwin
benchmark show that UGR achieves the best success rate on four tasks, improves over the ACT baseline by up to 13% absolute, and outperforms both full-chunk and position-agnostic block refinement in ablation studies.
Read the full abstract ▾
All authors Chenyang Wang, Yuntian Wang, Xiaoxiong Yang, Dingde Jiang, Siao Liu, Yang Liu
Robotics
JEPLO: Joint-Embedding Predictive Learning for LiDAR-Based Legged Locomotion
Qihao Yuan, Yixuan Qiu, Ziyu Cao et al. (+2 more)
Light detection and ranging (
LiDAR) remains less explored than RGB-D sensing for perceptive legged locomotion, and existing
LiDAR-based approaches often rely on explicit mapping. We present JEPLO (Joint-
Embedding Predictive learning for legged LOcomotion), a single-stage learning framework for mapping-free,
LiDAR-based perceptive locomotion for legged
robots. We introduce a proprio-exteroceptive JEPA (PE-JEPA) world model to learn predictive egocentric terrain representations from onboard observations, including raw
LiDAR scans. A concurrent JEPA-teacher-student (CJTS) pipeline is further proposed to train a locomotion policy informed by JEPA latent representations in simulation using deep reinforcement learning with a simple reward formulation. The framework achieves successful sim-to-real transfer, enabling omnidirectional traversal of diverse terrains, including long staircases and high boxes, with lightweight onboard computation. Evaluations demonstrate greater robustness than existing perceptive locomotion frameworks, particularly under degraded perception caused by occlusion, sparsity and noise. Further analysis validates JEPLO's ability to retain task-relevant information under these challenging conditions. We open-source our implementation, experimental
datasets, and hardware setup designs https://github.com/ASIG-X/JEPLO.
Read the full abstract ▾
All authors Qihao Yuan, Yixuan Qiu, Ziyu Cao, Ming Cao, Kailai Li
Robotics
Bench2Dex: Benchmarking Visuo-Tactile Bimanual Dexterous Manipulation Across Dexterous Hands
Zhenjie Yang, Yideng Zhang, Dongjie Zhang et al. (+21 more)
Tactile sensing provides contact information that can be difficult to infer from
vision alone, but tactile hardware for dexterous hands has not converged to a common design. Dexterous hands differ in finger structure, contact surfaces, and sensor layouts, while simulated tactile signals still differ from measurements produced by physical sensors. These factors make it difficult to study visuo-tactile manipulation across diverse dexterous hands within a consistent experimental setting. We present Bench2Dex, a simulation
benchmark for visuo-tactile bimanual manipulation across 12 dexterous hands. We adapt existing
robot models with a shared simulated tactile interface that converts local contact geometry into image-like tactile observations. The interface provides a consistent observation format across different hand morphologies without attempting to reproduce the output of a specific physical tactile sensor. Bench2Dex includes 26 bimanual manipulation tasks that involve tool use, articulated-object interaction, and multi-stage manipulation, together with about 1.3K human-teleoperated demonstrations. The
benchmark provides synchronized visual, tactile, proprioceptive, action, and object-state observations, together with executable task metrics. For robustness, we group seven perturbation types into invariance axis, where the correct action does not change, and equivariance axis, where the correct action changes together with the perturbation. We evaluate ACT,
Diffusion Policy, pi0.5, and GR00T N1.5 on Bench2Dex and report their performance and failure modes. Bench2Dex is meant as a platform for studying visuo-tactile learning across dexterous hands. It does not assume that simulated tactile observations can replace real tactile sensing; it offers a shared setting for
algorithm development while tactile hardware and simulation models are still evolving.
Read the full abstract ▾
All authors Zhenjie Yang, Yideng Zhang, Dongjie Zhang, Chenyu Jiang, Xianshuai Liu, Yufeng Li, Zuhao Ge, Xingyu Jiao, Zheng Zhang, Kaiyu He, He Wang, Yuwen Zhong, Yi Deng, Muyun Jiang, Xianliang Huang, Haisheng Su, Donghang Zhang, Jian Zhang, Xue Yang, Hongyang Li, Zuxuan Wu, Yu-Gang Jiang, Xiaosong Jia, Junchi Yan
Robotics
Continuous Manifold-Decomposed Impedance Retargeting for Contact-Rich Imitation Learning
Jiahao Liu, Kento Kawaharazuka, Tasuku Makabe et al. (+1 more)
CMDIR extends
Manifold-Decomposed Impedance Retargeting (MDIR) to transform fixed-impedance demonstrations into continuous variable-impedance controllers, which can also serve as structured supervision for imitation learning. Continuous Task-
Manifold Impedance Representation (TMIR) pairs an evolving task frame with controller instructions. Demo-relative Compromise dynamics retain moving-basis transport and control/physical metric mismatch, yielding displacement, reaction-impulse, and perturbation-sensitivity criteria. Quality-to-Fast automatically compiles a solver structure from development paths within a predefined finite space, re-instantiates that structure for each demonstration, and certifies the resulting candidate by multi-resolution evaluation. Across 225 retargeted-controller trials in three real contact tasks, full CMDIR improves mean task-proxy retention and reduces mean pose deviation, force fluctuation, and peak force relative to discrete MDIR. FastMPO achieves a $5.8$–$9.4\times$ speedup over C-MPO with comparable closed-loop outcomes. Downstream experiments demonstrate learnability of the complete TMIR supervision interface; lower force fluctuation and peak force are observed among successful executions, while completion reliability remains uneven across tasks and environments.
Read the full abstract ▾
All authors Jiahao Liu, Kento Kawaharazuka, Tasuku Makabe, Kei Okada