Robotics
Event-based Heterogeneous Information Processing for Online Vision-based Obstacle Detection and Localization
Reza Ahmadvand, Sarah Safura Sharif and Yaser Mike Banad
This paper introduces a novel framework for robotic
vision-based
navigation that integrates Hybrid Neural Networks (HNNs) with Spiking Neural Network (SNN)-based filtering to enhance situational awareness for unmodeled obstacle detection and localization. By leveraging the complementary strengths of Artificial Neural Networks (ANNs) and SNNs, the system achieves both accurate environmental understanding and fast, energy-efficient processing. The proposed architecture employs a dual-pathway approach: an ANN component processes static spatial features at low frequency, while an SNN component handles dynamic, event-based sensor data in real time. Unlike conventional hybrid architectures that rely on domain conversion mechanisms, our system incorporates a pre-developed SNN-based filter that directly utilizes spike-encoded inputs for localization and state estimation. Detected anomalies are validated using contextual information from the ANN pathway and continuously tracked to support anticipatory
navigation strategies. Simulation results demonstrate that the proposed method offers acceptable detection accuracy while maintaining computational efficiency close to SNN-only implementations, which operate at a fraction of the resource cost. This framework represents a significant advancement in neuromorphic
navigation systems for
robots operating in unpredictable and dynamic environments.
Read the full abstract ▾
Robotics
Robustness and Resilience Evaluation of Eco-Driving Strategies at Signalized Intersections
Zhaohui Liang, Chengyuan Ma, Keke Long et al. (+1 more)
Eco-driving strategies have demonstrated substantial potential for improving energy efficiency and reducing emissions, especially at signalized intersections. However, evaluations of eco-driving methods typically rely on simplified simulation or experimental conditions, where certain assumptions are made to manage complexity and experimental control. This study introduces a unified framework to evaluate eco-driving strategies through the lens of two complementary criteria: control robustness and environmental resilience. We define formal indicators that quantify performance degradation caused by internal execution variability and external environmental disturbances, respectively. These indicators are then applied to assess multiple eco-driving controllers through real-world vehicle experiments. The results reveal key tradeoffs between tracking accuracy and adaptability, showing that optimization-based controllers offer more consistent performance across varying disturbance levels, while analytical controllers may perform comparably under nominal conditions but exhibit greater sensitivity to execution and timing variability.
Read the full abstract ▾
All authors Zhaohui Liang, Chengyuan Ma, Keke Long, Xiaopeng Li
Robotics
CLEAR: A Semantic-Geometric Terrain Abstraction for Large-Scale Unstructured Environments
Pranay Meshram, Charuvahan Adhivarahan, Ehsan Tarkesh Esfahani et al. (+3 more)
Long-horizon
navigation in unstructured environments demands terrain abstractions that scale to tens of square kilometers while preserving semantic and
geometric structure. Grids scale poorly, and quadtrees misalign with terrain boundaries. Neither encodes terrain semantics essential for traversability-aware planning, yielding infeasible or inefficient paths for autonomous ground vehicles operating over more than 10 square kilometers. CLEAR (Connected Landcover Elevation Abstract Representation) is a reusable terrain abstraction framework for large-scale planning that produces convex, semantically aligned regions encoded as a terrain-aware graph. Evaluated on digital terrain maps spanning 9 to 100 square kilometers with physics-based simulation, CLEAR reduces per-query planning by 2x to 19.6x over unabstracted raw-grid A-
star after one-time abstraction, with 6.7 percent cost overhead. In physics-based simulation, CLEAR delivers 5 to 8.6 percent shorter executed paths and 100 percent task completion across all maps, compared with 90 to 100 percent for Quadtree. These results hold against AMRA-
star, a strong anytime multi-resolution planner, and generalize to a learned elevation-driven cost, demonstrating CLEAR's scalability and cost-model independence as a reusable planning layer.
Read the full abstract ▾
All authors Pranay Meshram, Charuvahan Adhivarahan, Ehsan Tarkesh Esfahani, Souma Chowdhury, Chen Wang, Karthik Dantu
Robotics · Cross-listed from cs.HC
Towards Natural Language Environment: Understanding Seamless Natural-Language-Based Human-Multi-Robot Interactions
Ziyi Liu, Xinyi Wang, Shao-Kang Hsia et al. (+5 more)
As multiple
robots are expected to coexist in future households, natural
language is increasingly envisioned as a primary medium for human-
robot and
robot-
robot communication. This paper introduces the concept of a Natural
Language Environment (NLE), defined as an interaction space in which humans and multiple heterogeneous
robots coordinate primarily through natural
language. Rather than proposing a deployable system, this work aims to explore the design space of such environments. We first synthesize prior work on
language-based human-
robot interaction to derive a preliminary design space for NLEs. We then conduct a role-playing study in virtual reality to investigate how people conceptualize, negotiate, and coordinate human-multi-
robot interactions within this imagined environment. Based on qualitative and quantitative analysis, we refine the preliminary design space and derive design implications that highlight key tensions and opportunities around task coordination dominance,
robot autonomy, and
robot personality in Natural
Language Environments.
Read the full abstract ▾
All authors Ziyi Liu, Xinyi Wang, Shao-Kang Hsia, Chenfei Zhu, Zhengzhe Zhu, Xiyun Hu, Anastasia Kouvaras Ostrowski, Karthik Ramani
Robotics
Autonomous Navigation at the Nano-Scale: Algorithms, Architectures, and Constraints
Mahmud S. Zango and Jianglin Lan
Autonomous
navigation for nano-scale unmanned aerial vehicles (nano-
UAVs) is governed by extreme Size, Weight, and Power (SWaP) constraints (with the weight < 50 g and sub-100 mW onboard processor), distinguishing it fundamentally from standard robotic paradigms. This review synthesizes the state-of-the-art in sensing, computing, and control architectures designed specifically for these sub- 100mW computational envelopes. We critically analyse the transition from classical geometry-based methods to emerging "Edge AI" paradigms, including quantized deep neural networks deployed on ultra-low-power System-on-Chips (SoCs) and neuromorphic event-based control. Beyond
algorithms, we evaluate the hardware-software co-design requisite for autonomy, covering advancements in dense optical flow, optimized Simultaneous Localization and Mapping (SLAM), and learning-based flight control. While significant progress has been observed in visual
navigation and relative pose estimation, our analysis reveals persistent gaps in long-term endurance, robust obstacle avoidance in dynamic environments, and the "Sim-to-Real" transfer of reinforcement learning policies. This survey provides a roadmap for bridging these gaps, advocating for hybrid architectures that fuse lightweight classical control with data-driven perception to enable fully autonomous, agile nano-
UAVs in GPS-denied environments.
Read the full abstract ▾
Robotics
Diffusion-based Inverse Model of a Distributed Tactile Sensor for Object Pose Estimation
Ante Marić, Giammarco Caroleo, Alessandro Albini et al. (+3 more)
Tactile sensing provides a promising sensing modality for object pose estimation in manipulation settings where visual information is limited due to occlusion or environmental effects. However, efficiently leveraging tactile data for estimation remains a challenge due to partial observability, with single observations corresponding to multiple possible contact configurations. This limits conventional estimation approaches largely tailored to
vision. We propose to address these challenges by learning an inverse tactile sensor model using denoising
diffusion. The model is conditioned on tactile observations from a distributed tactile sensor and trained in simulation using a
geometric sensor model based on signed distance fields. Contact constraints are enforced during inference through single-step projection using distance and gradient information from the signed distance field. For online pose estimation, we integrate the inverse model with a particle filter through a proposal scheme that combines generated hypotheses with particles from the prior belief. Our approach is validated in simulated and real-world planar pose estimation settings, without access to visual data or tight initial pose priors. We further evaluate robustness to unmodeled contact and sensor dynamics for pose tracking in a box-pushing scenario. Compared to local sampling baselines, the inverse sensor model improves sampling efficiency and estimation accuracy while preserving
multimodal beliefs across objects with varying tactile discriminability.
Read the full abstract ▾
All authors Ante Marić, Giammarco Caroleo, Alessandro Albini, Julius Jankowski, Perla Maiolino, Sylvain Calinon
Robotics
MATTERIX: toward a digital twin for robotics-assisted chemistry laboratory automation
Kourosh Darvish, Arjun Sohal, Abhijoy Mandal et al. (+20 more)
Accelerated materials discovery is critical for addressing global challenges. However, developing new laboratory workflows relies heavily on real-world experimental trials, and this can hinder scalability because of the need for numerous physical make-and-test iterations. Here we present MATTERIX, a multiscale, graphics processing unit-accelerated robotic simulation framework designed to create high-fidelity digital twins of chemistry laboratories, thus accelerating workflow development. This multiscale digital twin simulates robotic physical manipulation, powder and
liquid dynamics, device functionalities, heat transfer and basic chemical reaction kinetics. This is enabled by integrating realistic physics simulation and photorealistic rendering with a modular graphics processing unit-accelerated semantics engine, which models logical states and continuous behaviors to simulate chemistry workflows across different levels of abstraction. MATTERIX streamlines the creation of digital twin environments through open-source asset libraries and interfaces, while enabling flexible workflow design via hierarchical plan definition and a modular skill library that incorporates learning-based methods. Our approach demonstrates sim-to-real transfer in robotic chemistry setups, reducing reliance on costly real-world experiments and enabling the testing of hypothetical automated workflows in silico. The project website is available at https://accelerationconsortium.github.io/Matterix/ .
Read the full abstract ▾
All authors Kourosh Darvish, Arjun Sohal, Abhijoy Mandal, Hatem Fakhruldeen, Nikola Radulov, Zhengxue Zhou, Satheeshkumar Veeramani, Joshua Choi, Sijie Han, Brayden Zhang, Jeeyeoun Chae, Alex Wright, Yijie Wang, Hossein Darvish, Yuchi Zhao, Gary Tom, Han Hao, Miroslav Bogdanovic, Gabriella Pizzuto, Andrew I. Cooper, Alán Aspuru-Guzik, Florian Shkurti, Animesh Garg
Robotics
Active Informative Planning for UAV-based Weed Mapping using Discrete Gaussian Process Representations
Jacob Swindell, Marija Popović and Riccardo Polvara
Accurate agricultural weed mapping using unmanned aerial vehicles (
UAVs) is crucial for precision farming. While traditional methods rely on rigid, pre-defined flight paths and intensive offline processing, informative path planning (IPP) offers a way to collect data adaptively where it is most needed. Gaussian process (GP) mapping provides a continuous model of weed distribution with built-in uncertainty. However, GPs must be discretised for practical use in autonomous planning. Many discretisation techniques exist, but the impact of discrete representation choice remains poorly understood. This paper investigates how different discrete GP representations influence both mapping quality and mission-level performance in
UAV-based weed mapping. Considering a
UAV equipped with a downward-facing camera, we implement a receding-horizon IPP strategy that selects sampling locations based on the map uncertainty, travel cost, and coverage penalties. We investigate multiple discretisation strategies for representing the GP posterior and use their induced map partitions to generate candidate viewpoints for planning. Experiments on real-world weed distributions show that representation choice significantly affects exploration behaviour and efficiency. Overall, our results demonstrate that discretisation is not only a representational detail but a key design choice that shapes planning dynamics, coverage efficiency, and computational load in online
UAV weed mapping.
Read the full abstract ▾
Robotics
Helical Tendon-Driven Continuum Robot with Programmable Follow-the-Leader Operation
Behnam Moradkhani, Raghav Sankaranarayanan, Pejman Kheradmand et al. (+4 more)
Spinal cord stimulation (SCS) is widely used for pain management and has recently shown promise in promoting functional recovery following spinal cord injury. Effective activation of motor
neurons requires accurate placement of SCS leads in the ventral or lateral epidural space, where corticospinal and rubrospinal motor fibers are located, a task that remains challenging with conventional manual steering. This study presents a static modeling framework for ExoNav, a steerable robotic tool designed to enable precise
navigation to ventral and lateral epidural targets. A Cosserat rod formulation is employed to relate tendon actuation forces to the
robot shape, and a simulation environment is developed based on this model, incorporating
gravity as an external load. Experimental validation across four prototypes yields tip position root mean square errors (RMSEs) of 1.76 mm, 2.33 mm, 2.18 mm, and 1.33 mm, corresponding to 2.3 , 3.1, 3.4, and 2.1 percent of the
robot length, respectively. Owing to its helical deformation under actuation, ExoNav enables follow-the-leader (FTL) motion when combined with insertion and rotation degrees of freedom, which is demonstrated in both simulation and experiments. The proposed simulation computes optimal tendon tensions to track desired FTL paths under
gravity-induced deformation. Three FTL experiments show repeatable end-effector
alignment with a maximum RMSE of 3.75 mm (less than 5 percent of the
robot length). Finally, phantom experiments demonstrate successful teleoperated
navigation to ventral and lateral spinal cord targets, as well as the dorsal root ganglia, highlighting ExoNav potential for both motor function recovery and pain management.
Read the full abstract ▾
All authors Behnam Moradkhani, Raghav Sankaranarayanan, Pejman Kheradmand, Harshith Jella, Nicholas Ahn, Ajmal Zemmar, Yash Chitalia
Robotics
LLM-VLM Fusion Framework for Autonomous Maritime Port Inspection using a Heterogeneous UAV-USV System
Muhayy Ud Din, Waseem Akram, Ahsan B. Bakht et al. (+1 more)
Maritime port inspection plays a critical role in ensuring safety, regulatory compliance, and operational efficiency in complex maritime environments. However, existing inspection methods often rely on manual operations and conventional computer
vision techniques that lack scalability and contextual understanding. This study introduces a novel integrated engineering framework that utilizes the synergy between Large
Language Models (LLMs) and
Vision Language Models (VLMs) to enable autonomous maritime port inspection using cooperative aerial and surface robotic platforms. The proposed framework replaces traditional state-machine mission planners with LLM-driven symbolic planning and improved perception pipelines through VLM-based semantic inspection, enabling context-aware and adaptive monitoring. The LLM module translates natural
language mission instructions into executable symbolic plans with dependency graphs that encode operational constraints and ensure safe
UAV-USV coordination. Meanwhile, the VLM module performs real-time semantic inspection and compliance assessment, generating structured reports with contextual
reasoning. The framework was validated using the extended MBZIRC Maritime Simulator with realistic port infrastructure and further assessed through real-world robotic inspection trials. The lightweight on-board design ensures suitability for resource-constrained maritime platforms, advancing the development of intelligent, autonomous inspection systems. Project resources (code and
videos) can be found here: https://github.com/Muhayyuddin/llm-vlm-fusion-port-inspection
Read the full abstract ▾
All authors Muhayy Ud Din, Waseem Akram, Ahsan B. Bakht, Irfan Hussain
Robotics
Exploiting Light To Enhance The Endurance and Navigation of Lighter-Than-Air Micro-Drones
Harry Huang, Talia Xu and Marco Zúñiga Zamalloa
Micro-Unmanned Aerial Vehicles (
UAVs) are rapidly expanding into tasks from inventory to environmental sensing, yet their short endurance and unreliable
navigation in GPS-denied spaces limit deployment. Lighter-Than-Air (LTA) drones offer an energy-efficient alternative: they use a helium envelope to provide buoyancy, which enables near-zero-power drain during hovering and much longer operation. LTAs are promising, but their design is complex, and they lack integrated solutions to enable sustained autonomous operations and
navigation with simple, low-infrastructure. We propose a compact, self-sustaining LTA drone that uses light for both energy harvesting and
navigation. Our contributions are threefold: (i) a high-fidelity simulation framework to analyze LTA aerodynamics and select a stable, efficient configuration; (ii) a framework to integrate
solar cells on the envelope to provide net-positive energy; and (iii) a point-and-go
navigation system with three light-seeking
algorithms operating on a single light beacon. Our LTA-analysis, together with the integrated
solar panels, not only saves energy while flying, but also enables sustainable operation: providing 1 minute of flying time for every 4 minutes of energy harvesting, under illuminations of 80klux. We also demonstrate robust single-beacon
navigation towards a light source that can be up to 7m away, in indoor and outdoor environments, even with moderate winds. The resulting system indicates a plausible path toward persistent, autonomous operation for indoor and outdoor monitoring. More broadly, this work provides a practical pathway for translating the promise of LTA drones into a persistent, self-sustaining aerial system.
Read the full abstract ▾
Robotics
Static Is Not Enough: A Comparative Study of VR and SpaceMouse in Static and Dynamic Teleoperation Tasks
Yijun Zhou, Muhan Hou and Kim Baraka
Imitation learning relies on high-quality demonstrations, and teleoperation is a primary way to collect them, making teleoperation interface choice crucial for the data. Prior work mainly focused on static tasks, i.e., discrete, segmented motions, yet demonstrations also include dynamic tasks requiring reactive control. As dynamic tasks impose fundamentally different interface demands, insights from static-task evaluations cannot generalize. To address this gap, we conduct a within-subjects study comparing a VR controller and a SpaceMouse across two static and two dynamic tasks ($N=25$). We assess success rate, task duration, cumulative success, alongside NASA-TLX, SUS, and open-ended feedback. Results show statistically significant advantages for VR: higher success rates, particularly on dynamic tasks, shorter successful execution times across tasks, and earlier successes across attempts, with significantly lower workload and higher usability. As existing VR teleoperation systems are rarely open-source or suited for dynamic tasks, we release our VR interface to fill this gap.
Read the full abstract ▾
Robotics
Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalization
Hao Luo, Ye Wang, Wanpeng Zhang et al. (+9 more)
We introduce Being-H0.5, a foundational
Vision-
Language-Action (VLA) model designed for robust cross-embodiment generalization across diverse robotic platforms. While existing VLAs often struggle with morphological heterogeneity and data scarcity, we propose a human-centric learning paradigm that treats human interaction traces as a universal "mother tongue" for physical interaction. To support this, we present UniHand-2.0, the largest
embodied pre-training recipe to date, comprising over 35,000 hours of
multimodal data across 30 distinct robotic embodiments. Our approach introduces a Unified Action Space that maps heterogeneous
robot controls into semantically aligned slots, enabling low-resource
robots to bootstrap skills from human data and high-resource platforms. Built upon this human-centric foundation, we design a unified sequential modeling and multi-task pre-training paradigm to bridge human demonstrations and robotic execution. Architecturally, Being-H0.5 utilizes a Mixture-of-
Transformers design featuring a novel Mixture-of-Flow (MoF) framework to decouple shared motor primitives from specialized embodiment-specific experts. Finally, to make cross-embodiment policies stable in the real world, we introduce
Manifold-Preserving Gating for robustness under sensory shift and Universal Async Chunking to universalize chunked control across embodiments with different latency and control profiles. We empirically demonstrate that Being-H0.5 achieves state-of-the-art results on simulated
benchmarks, such as LIBERO (98.9%) and RoboCasa (53.9%), while also exhibiting strong cross-embodiment capabilities on five robotic platforms.
Read the full abstract ▾
All authors Hao Luo, Ye Wang, Wanpeng Zhang, Sipeng Zheng, Ziheng Xi, Chaoyi Xu, Haiweng Xu, Haoqi Yuan, Chi Zhang, Yiqing Wang, Yicheng Feng, Zongqing Lu
Robotics
Imitation learning-based spacecraft rendezvous and docking method with Expert Demonstration
Shibo Shao, Dong Zhou, Guanghui Sun et al. (+2 more)
Existing spacecraft rendezvous and docking control methods largely rely on predefined dynamic models and often exhibit limited robustness in realistic on-orbit environments. To address this issue, this paper proposes an Imitation Learning-based spacecraft rendezvous and docking control framework (IL-SRD) that directly learns control policies from expert demonstrations, thereby reducing dependence on accurate modeling. We propose an anchored decoder target mechanism, which conditions the decoder queries on state-related anchors to explicitly constrain the control generation process. This mechanism enforces physically consistent control
evolution and effectively suppresses implausible action deviations in sequential prediction, enabling reliable six-degree-of-freedom (6-DOF) rendezvous and docking control. To further enhance stability, a temporal aggregation mechanism is incorporated to mitigate error accumulation caused by the sequential prediction nature of
Transformer-based models, where small inaccuracies at each time step can propagate and amplify over long horizons. Extensive simulation results demonstrate that the proposed IL-SRD framework achieves accurate and energy-efficient model-free rendezvous and docking control. Robustness evaluations further confirm its capability to maintain competitive performance under significant unknown disturbances. The source code is available at https://github.com/Dongzhou-1996/IL-SRD.
Read the full abstract ▾
All authors Shibo Shao, Dong Zhou, Guanghui Sun, Liwen Zhang, Mingxuan Jiang
Robotics
Active Inference-Driven World Modeling for Adaptive UAV Swarm Trajectory Design
Kaleem Arshid, Ali Krayani, Lucio Marcenaro et al. (+2 more)
This paper proposes an Active Inference-based framework for autonomous trajectory design in
UAV swarms. The method integrates probabilistic
reasoning and self-learning to enable distributed mission allocation, route ordering, and motion planning. Expert trajectories generated using a Genetic
Algorithm with Repulsion Forces (GA-RF) are employed to train a hierarchical World Model capturing swarm behavior across mission, route, and motion levels. During online operation,
UAVs infer actions by minimizing divergence between current beliefs and model-predicted states, enabling adaptive responses to dynamic environments. Simulation results show faster convergence, higher stability, and safer
navigation than Q-Learning, demonstrating the scalability and cognitive grounding of the proposed framework for intelligent
UAV swarm control.
Read the full abstract ▾
All authors Kaleem Arshid, Ali Krayani, Lucio Marcenaro, David Martin Gomez, Carlo Regazzoni
Robotics
ForeDiffusion: Foresight-Conditioned Diffusion Policy via Future View Construction for Robot Manipulation
Weize Xie, Yi Ding, Ying He et al. (+5 more)
Diffusion strategies have advanced visual motor control by progressively denoising high-dimensional action sequences, providing a promising method for
robot manipulation. However, as task complexity increases, the success rate of existing baseline models decreases considerably. Analysis indicates that current
diffusion strategies are confronted with two limitations. First, these strategies only rely on short-term observations as conditions. Second, the training objective remains limited to a single denoising loss, which leads to error accumulation and causes grasping deviations. To address these limitations, this paper proposes Foresight-Conditioned
Diffusion (ForeDiffusion), by injecting the predicted future view representation into the
diffusion process. As a result, the policy is guided to be forward-looking, enabling it to correct trajectory deviations. Following this design, ForeDiffusion employs a dual loss mechanism, combining the traditional denoising loss and the consistency loss of future observations, to achieve the unified optimization. Extensive evaluation on the Adroit suite and the MetaWorld
benchmark demonstrates that ForeDiffusion achieves an average success rate of 80% for the overall task, significantly outperforming the existing mainstream
diffusion methods by 23% in complex tasks, while maintaining more stable performance across the entire tasks.
Read the full abstract ▾
All authors Weize Xie, Yi Ding, Ying He, Leilei Wang, Binwen Bai, Zheyi Zhao, Chenyang Wang, F. Richard Yu
Robotics
Dynamic Hand Gesture Recognition for Robot Manipulator Tasks
Dharmendra Sharma, Peeyush Thakur, Sandeep Gupta et al. (+2 more)
This paper proposes a novel approach to recognizing dynamic hand gestures facilitating seamless interaction between humans and
robots. Here, each
robot manipulator task is assigned a specific gesture. There may be several such tasks, hence, several gestures. These gestures may be prone to several dynamic variations. All such variations for different gestures shown to the
robot are accurately recognized in real-time using the proposed unsupervised model based on the Gaussian Mixture model. The accuracy during training and real-time testing prove the efficacy of this methodology.
Read the full abstract ▾
All authors Dharmendra Sharma, Peeyush Thakur, Sandeep Gupta, Narendra Kumar Dhar, Laxmidhar Behera
Robotics
PlannerRFT: Reinforcing Diffusion Planners through Closed-Loop and Sample-Efficient Fine-Tuning
Hongchen Li, Tianyu Li, Jiazhi Yang et al. (+10 more)
Diffusion-based planners have emerged as a promising approach for human-like trajectory generation in autonomous driving. Recent works incorporate reinforcement fine-tuning to enhance the robustness of
diffusion planners through reward-oriented optimization in a generation-evaluation loop. However, they struggle to generate multi-modal, scenario-adaptive trajectories, hindering the exploitation efficiency of informative rewards during fine-tuning. To resolve this, we propose PlannerRFT, a sample-efficient reinforcement fine-tuning framework for
diffusion-based planners. PlannerRFT adopts a dual-branch optimization that simultaneously refines the trajectory distribution and adaptively guides the denoising process toward more promising exploration, without altering the original inference pipeline. To support parallel learning at scale, we develop nuMax, an optimized simulator that achieves 10 times faster rollout compared to native nuPlan. Extensive experiments shows that PlannerRFT yields state-of-the-art performance with distinct behaviors emerging during the learning process.
Read the full abstract ▾
All authors Hongchen Li, Tianyu Li, Jiazhi Yang, Haochen Tian, Caojun Wang, Lei Shi, Mingyang Shang, Zengrong Lin, Gaoqiang Wu, Zhihui Hao, Xianpeng Lang, Jia Hu, Hongyang Li
Robotics
Sparse ActionGen: Accelerating Diffusion Policy with Real-time Pruning
Kangye Ji, Jianbo Zhou, Yuan Meng et al. (+3 more)
Diffusion Policy has dominated action generation due to its strong capabilities for modeling multi-modal action distributions, but its multi-step denoising processes make it impractical for real-time visuomotor control. Existing caching-based acceleration methods typically rely on $\textit{static}$ schedules that fail to adapt to the $\textit{dynamics}$ of
robot-environment interactions, thereby leading to suboptimal performance. In this paper, we propose $\underline{\textbf{S}}$parse $\underline{\textbf{A}}$ction$\underline{\textbf{G}}$en ($\textbf{SAG}$) for extremely sparse action generation. To accommodate the iterative interactions, SAG customizes a rollout-adaptive prune-then-reuse mechanism that first identifies prunable computations globally and then reuses cached activations to substitute them during action
diffusion. To capture the rollout dynamics, SAG parameterizes an observation-conditioned
diffusion pruner for environment-aware adaptation and instantiates it with a highly parameter- and inference-efficient design for real-time prediction. Furthermore, SAG introduces a one-for-all reusing strategy that reuses activations across both timesteps and blocks in a zig-zag manner, minimizing the global redundancy. Extensive experiments on multiple robotic
benchmarks demonstrate that SAG achieves up to 4$\times$ generation speedup without sacrificing performance. Project Page: https://sparse-actiongen.github.io.
Read the full abstract ▾
All authors Kangye Ji, Jianbo Zhou, Yuan Meng, Ye Li, Hanyun Cui, Zhi Wang