Robotics
RL-Augmented MPC for Non-Gaited Legged and Hybrid Locomotion
Andrea Patrizi, Carlo Rizzardo, Arturo Laurenzi et al. (+3 more)
We propose a contact-explicit hierarchical architecture coupling Reinforcement Learning (RL) and Model Predictive Control (MPC), where a high-level RL
agent provides gait and
navigation commands to a low-level locomotion MPC. This offloads the combinatorial burden of contact timing from the MPC by learning acyclic gaits through trial and error in simulation. We show that only a minimal set of rewards and limited tuning are required to obtain effective policies. We validate the architecture in simulation across robotic platforms spanning 50 kg to 120 kg and different MPC implementations, observing the emergence of acyclic gaits and timing adaptations in flat-terrain legged and hybrid locomotion, and further demonstrating extensibility to non-flat terrains. Across all platforms, we achieve zero-shot sim-to-sim transfer without domain randomization, and we further demonstrate zero-shot sim-to-real transfer without domain randomization on Centauro, our 120 kg wheeled-legged
humanoid robot. We make our software framework and evaluation results publicly available at https://github.com/AndrePatri/AugMPC.
Read the full abstract ▾
All authors Andrea Patrizi, Carlo Rizzardo, Arturo Laurenzi, Francesco Ruscelli, Luca Rossini, Nikos G. Tsagarakis
Robotics
FG-CLTP: Fine-Grained Contrastive Language Tactile Pretraining for Robotic Manipulation
Wenxuan Ma, Chaofan Zhang, Yinghao Cai et al. (+3 more)
Recent advancements in integrating tactile sensing into
vision-
language-action (VLA) models have demonstrated transformative potential for robotic perception. However, existing tactile representations predominantly rely on qualitative descriptors (e.g., texture), neglecting quantitative contact states such as force magnitude, contact geometry, and principal axis orientation, which are indispensable for fine-grained manipulation. To bridge this gap, we propose FG-CLTP, a fine-grained contrastive
language tactile pretraining framework. We first introduce a novel
dataset comprising over 100k tactile 3D point cloud-
language pairs that explicitly capture multidimensional contact states from the sensor's perspective. We then implement a discretized numerical tokenization mechanism to achieve quantitative-semantic
alignment, effectively injecting explicit physical metrics into the
multimodal feature space. The proposed FG-CLTP model yields a 95.9% classification accuracy and reduces the regression error (MAE) by 52.6% compared to state-of-the-art methods. Furthermore, the integration of 3D point cloud representations establishes a sensor-agnostic foundation with a minimal sim-to-real gap of 3.5%. Building upon this fine-grained representation, we develop a 3D tactile-
language-action (3D-TLA) architecture driven by a flow matching policy to enable
multimodal reasoning and control. Extensive experiments demonstrate that our framework significantly outperforms strong baselines in contact-rich manipulation tasks, providing a robust and generalizable foundation for tactile-
language-action models.
Read the full abstract ▾
All authors Wenxuan Ma, Chaofan Zhang, Yinghao Cai, Guocai Yao, Shaowei Cui, Shuo Wang
Robotics
GRACE: A Unified 2D Multi-Robot Path Planning Simulator & Benchmark for Grid, Roadmap, And Continuous Environments
Chuanlong Zang, Anna Mannucci, Isabelle Barz et al. (+3 more)
Advancing Multi-
Agent Pathfinding (MAPF) and Multi-
Robot Motion Planning (MRMP) requires platforms that enable transparent, reproducible comparisons across modeling choices. Existing tools either scale under simplifying assumptions (grids, homogeneous
agents) or offer higher fidelity with less comparable instrumentation. We present GRACE, a unified 2D simulator+
benchmark that instantiates the same task at multiple abstraction levels (grid, roadmap, continuous) via explicit, reproducible operators and a common evaluation protocol. Our empirical results on public maps and representative planners enable commensurate comparisons on a shared instance set. Furthermore, we quantify the expected representation-fidelity trade-offs (MRMP solves instances at higher fidelity but lower speed, while grid/roadmap planners scale farther). By consolidating representation, execution, and evaluation, GRACE thereby aims to make cross-representation studies more comparable and provides a means to advance multi-
robot planning research and its translation to practice.
Read the full abstract ▾
All authors Chuanlong Zang, Anna Mannucci, Isabelle Barz, Philipp Schillinger, Florian Lier, Wolfgang Hönig
Robotics
Semantic Landmark Particle Filter for Robot Localisation in Vineyards
Rajitha de Silva, Jonathan Cox, James R. Heselden et al. (+3 more)
Reliable localisation in vineyards is hindered by row-level perceptual aliasing: parallel crop rows produce nearly identical
LiDAR observations, causing geometry-only and
vision-based SLAM systems to converge towards incorrect corridors, particularly during headland transitions. We present a Semantic Landmark Particle Filter (SLPF) that integrates trunk and pole landmark detections with 2D
LiDAR within a probabilistic localisation framework. Detected trunks are converted into semantic walls, forming structural row boundaries embedded in the measurement model to improve discrimination between adjacent rows. GNSS is incorporated as a lightweight prior that stabilises localisation when semantic observations are sparse. Field experiments in a 10-row vineyard demonstrate consistent improvements over geometry-only (AMCL),
vision-based (RTAB-Map), and GNSS baselines. Compared to AMCL, SLPF reduces Absolute Pose Error by 22% and 65% across two traversal directions; relative to a NoisyGNSS baseline, APE decreases by 65% and 61%. Row correctness improves from 0.67 to 0.73, while mean cross-track error decreases from 1.40 m to 1.26 m. These results show that
embedding row-level structural semantics within the measurement model enables robust localisation in highly repetitive outdoor agricultural environments.
Read the full abstract ▾
All authors Rajitha de Silva, Jonathan Cox, James R. Heselden, Marija Popović, Cesar Cadena, Riccardo Polvara
Robotics · Cross-listed from cs.DS
Sublinear-Time Reconfiguration of Programmable Matter with Joint Movements
Manish Kumar, Othon Michail, Andreas Padalkin et al. (+1 more)
We study centralized reconfiguration problems for
geometric amoebot structures. A set of $n$ amoebots occupy nodes on the triangular grid and can reconfigure via expansion and contraction operations. We focus on the joint movement extension, where amoebots may expand and contract in parallel, enabling coordinated motion of larger substructures. Prior work introduced this extension and analyzed reconfiguration under additional assumptions such as metamodules. In contrast, we investigate the intrinsic dynamics of reconfiguration without such assumptions by restricting
attention to centralized
algorithms, leaving distributed solutions for future work. We study the reconfiguration problem between two classes of amoebot structures $A$ and $B$: For every structure $S\in A$, the goal is to compute a schedule that reconfigures $S$ into some structure $S'\in B$. Our focus is on sublinear-time
algorithms. We affirmatively answer the open problem by Padalkin et al. (Auton.
Robots, 2025) whether a within-the-model sublinear-time universal reconfiguration
algorithm is possible, by proving that any structure can be reconfigured into a canonical line-segment structure in $O(\sqrt{n}\log n)$ rounds. Additionally, we give a constant-time
algorithm for reconfiguring any spiral structure into a line segment. These results are enabled by new constant-time primitives that facilitate efficient parallel movement. Our findings demonstrate that the joint movement model supports sublinear reconfiguration without auxiliary assumptions. A central open question is whether universal reconfiguration within this model can be achieved in polylogarithmic or even constant time.
Read the full abstract ▾
All authors Manish Kumar, Othon Michail, Andreas Padalkin, Christian Scheideler
Robotics
ASTER: Attitude-aware Suspended-payload Quadrotor Traversal via Efficient Reinforcement Learning
Dongcheng Cao, Jin Zhou and Shuo Li
Agile maneuvering of the quadrotor cable-suspended system is significantly hindered by its non-smooth hybrid dynamics. While model-free Reinforcement Learning (RL) circumvents explicit differentiation of complex models, achieving attitude-constrained or inverted flight remains an open challenge due to the extreme reward sparsity under strict orientation requirements. This paper presents ASTER, a robust RL framework that achieves, to our knowledge, the first successful autonomous inverted flight for the cable-suspended system. We propose hybrid-dynamics-informed state seeding (HDSS), an initialization strategy that back-propagates target configurations through physics-consistent kinematic inversions across both taut and slack cable phases. HDSS enables the policy to discover aggressive maneuvers that are unreachable via standard exploration. Extensive simulations and real-world experiments demonstrate remarkable agility, precise attitude
alignment, and robust zero-shot sim-to-real transfer across complex trajectories.
Read the full abstract ▾
Robotics
MAVEN: A Meta-Reinforcement Learning Framework for Varying-Dynamics Expertise in Agile Quadrotor Maneuvers
Jin Zhou, Dongcheng Cao, Xian Wang et al. (+1 more)
Reinforcement learning (RL) has emerged as a powerful paradigm for achieving online agile
navigation with quadrotors. Despite this success, policies trained via standard RL typically fail to generalize across significant dynamic variations, exhibiting a critical lack of adaptability. This work introduces MAVEN, a meta-RL framework that enables a single policy to achieve robust end-to-end
navigation across a wide range of quadrotor dynamics. Our approach features a novel predictive context encoder, which learns to infer a latent representation of the system dynamics from interaction history. We demonstrate our method in agile waypoint traversal tasks under two challenging scenarios: large variations in quadrotor mass and severe single-rotor thrust loss. We leverage a GPU-vectorized simulator to distribute tasks across thousands of parallel environments, overcoming the long training times of meta-RL to converge in less than an hour. Through extensive experiments in both simulation and the real world, we validate that MAVEN achieves superior adaptation and agility. The policy successfully executes zero-shot sim-to-real transfer, demonstrating robust online adaptation by performing high-speed maneuvers despite mass variations of up to 66.7% and single-rotor thrust losses as severe as 70%.
Read the full abstract ▾
All authors Jin Zhou, Dongcheng Cao, Xian Wang, Shuo Li
Robotics
FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Model
Xiaoxu Xu, Hao Li, Jinhui Ye et al. (+7 more)
Predictive foresight is important to intelligent
embodied agents. Since the motor execution of a
robot is intrinsically constrained by its visual perception of environmental geometry, effectively anticipating the future requires capturing this tightly coupled visuomotor interplay. While recent
vision-
language-action models attempt to incorporate future guidance, they struggle with this joint modeling. Existing explicit methods divert capacity to task-irrelevant visual details, whereas implicit methods relying on sparse frame pairs disrupt temporal continuity. By heavily relying on visual reconstruction, these methods become visually dominated, entangling static scene context with dynamic action intent. We argue that effective joint visuomotor predictive modeling requires both temporal continuity and visually-conditioned supervision decoupling. To this end, we propose FutureVLA, featuring a novel Joint Visuomotor Predictive Architecture. FutureVLA is designed to extract joint visuomotor
embeddings by first decoupling visual and motor information, and then jointly encoding generalized physical priors. Specifically, in the pretraining stage, we leverage heterogeneous manipulation
datasets and introduce a Joint Visuomotor Gating mechanism to structurally separate visual state preservation from temporal action modeling. It allows the motor stream to focus on continuous physical dynamics while explicitly querying visual
tokens for environmental constraints, yielding highly generalizable joint visuomotor
embeddings. Subsequently, in the post-training stage, we employ a latent
embeddings alignment strategy, enabling diverse downstream VLA models to internalize these temporal priors without modifying their inference architectures. Extensive experiments demonstrate that FutureVLA consistently improves VLA frameworks.
Read the full abstract ▾
All authors Xiaoxu Xu, Hao Li, Jinhui Ye, Yilun Chen, Jia Zeng, Xinyi Chen, Linning Xu, Dahua Lin, Weixin Li, Jiangmiao Pang
Robotics
Parallel-in-Time Nonlinear Optimal Control via GPU-native Sequential Convex Programming
Yilin Zou, Zhong Zhang, Maxime Robic et al. (+1 more)
Real-time solution of nonlinear optimal control problems remains challenging on embedded robotic hardware, where conventional solvers often rely on global sparse linear
algebra or sequential recursions that are difficult to map efficiently to massively parallel processors. This paper presents ucenter, a GPU-native Sequential Convex Programming (SCP) framework for nonlinear optimal control. At each SCP iteration, nonlinear dynamics are linearized around a nominal trajectory, and the resulting convexified subproblem is solved by a consensus Alternating Direction Method of Multipliers (ADMM) scheme. The temporal splitting replaces global sparse Karush-Kuhn-Tucker factorizations with independent per-node dense solves, closed-form dynamic consistency updates, and analytical projections onto convex constraint sets. Both the outer SCP loop and the inner ADMM subproblem are executed entirely on the GPU, enabling efficient optimization. The proposed solver is evaluated on quadrotor obstacle avoidance and Mars powered descent problems using an NVIDIA Jetson AGX Orin edge platform. Benchmarking against a CPU-parallel iLQR baseline in randomized environments reveals that the GPU implementation achieves over 100 Hz batched planning throughput, a 4.1x speedup, and a 51% reduction in energy consumption, while consistently maintaining low nonlinear dynamics defects. The framework exposes reusable GPU-parallel optimization primitives that can be specialized to a wide variety of complex nonlinear optimal control settings, as demonstrated by the scenario-based robust MPC and batched Monte Carlo generation tasks.
Read the full abstract ▾
All authors Yilin Zou, Zhong Zhang, Maxime Robic, Fanghua Jiang
Robotics
MapGCLR: Geospatial Contrastive Learning of Representations for Online Vectorized HD Map Construction
Jonas Merkert, Alexander Blumberg, Jan-Hendrik Pauls et al. (+1 more)
Autonomous vehicles rely on map information to understand the world around them. However, the creation and maintenance of offline high-definition (HD) maps remains costly. A more scalable alternative lies in online HD map construction, which only requires map annotations at training time. To further reduce the need for annotating vast training labels, self-supervised training provides an alternative. This work focuses on improving the latent birds-eye-view (BEV) feature grid representation within a vectorized online HD map construction model by enforcing geospatial consistency between overlapping BEV feature grids as part of a contrastive loss function. To ensure geospatial overlap for contrastive pairs, we introduce an approach to analyze the overlap between traversals within a given
dataset and generate subsidiary
dataset splits following adjustable multi-traversal requirements. We train the same model supervised using a reduced set of single-traversal labeled data and self-supervised on a broader unlabeled set of data following our multi-traversal requirements, effectively implementing a semi-supervised approach. Our approach outperforms the supervised baseline across the board, both quantitatively in terms of the downstream tasks vectorized map perception performance and qualitatively in terms of
segmentation in the principal component analysis (PCA) visualization of the BEV feature space.
Read the full abstract ▾
All authors Jonas Merkert, Alexander Blumberg, Jan-Hendrik Pauls, Christoph Stiller
Robotics
OnFly: Onboard Zero-Shot Aerial Vision-Language Navigation toward Safety and Efficiency
Guiyong Zheng, Yueting Ban, Mingjie Zhang et al. (+2 more)
Aerial
vision-
language navigation (AVLN) enables
UAVs to follow natural-
language instructions in complex 3D environments. However, existing zero-shot AVLN methods often suffer from unstable single-stream
Vision-
Language Model decision-making, unreliable long-horizon progress monitoring, and a trade-off between safety and efficiency. We propose OnFly, a fully onboard, real-time framework for zero-shot AVLN. OnFly adopts a shared-perception dual-
agent architecture that decouples high-frequency target generation from low-frequency progress monitoring, thereby stabilizing decision-making. It further employs a hybrid keyframe-recent-frame
memory to preserve global trajectory context while maintaining KV-cache prefix stability, enabling reliable long-horizon monitoring with termination and recovery signals. In addition, a semantic-
geometric verifier refines VLM-predicted targets for instruction consistency and
geometric safety using VLM features and depth cues, while a receding-horizon planner generates optimized collision-free trajectories under
geometric safety constraints, improving both safety and efficiency. In simulation, OnFly improves task success from 26.4% to 67.8%, compared with the strongest state-of-the-art baseline, while fully onboard real-world flights validate its feasibility for real-time deployment. The code will be released at https://github.com/Robotics-
STAR-Lab/OnFly
Read the full abstract ▾
All authors Guiyong Zheng, Yueting Ban, Mingjie Zhang, Juepeng Zheng, Boyu Zhou
Robotics
Cybo-Waiter: A Physical Agentic Framework for Humanoid Whole-Body Locomotion-Manipulation
Peng Ren, Haoyang Ge, Chuan Qi et al. (+5 more)
Robots are increasingly expected to execute open ended natural
language requests in human environments, which demands reliable long horizon execution under partial observability. This is especially challenging for
humanoids because locomotion and manipulation are tightly coupled through stance, reachability, and balance. We present a
humanoid agent framework that turns VLM plans into verifiable task programs and closes the loop with multi object 3D
geometric supervision. A VLM planner compiles each instruction into a typed JSON sequence of subtasks with explicit predicate based preconditions and success conditions. Using SAM3 and RGB-D, we ground all task relevant entities in 3D, estimate object centroids and extents, and evaluate predicates over stable frames to obtain condition level diagnostics. The supervisor uses these diagnostics to verify subtask completion and to provide condition-level feedback for progression and replanning. We execute each subtask by coordinating
humanoid locomotion and whole-body manipulation, selecting feasible motion primitives under reachability and balance constraints. Experiments on tabletop manipulation and long horizon
humanoid loco manipulation tasks show improved robustness from multi object grounding, temporal stability, and recovery driven replanning.
Read the full abstract ▾
All authors Peng Ren, Haoyang Ge, Chuan Qi, Cong Huang, Hong Li, Jiang Zhao, Pei Chi, Kai Chen
Robotics
Dynamic Modeling and Attitude Control of a Reaction-Wheel-Based Low-Gravity Bipedal Hopper
Shriram Hari, M Venkata Sai Nikhil and R Prasanth Kumar
Planetary bodies characterized by low gravitational acceleration, such as the Moon and near-Earth asteroids, impose unique locomotion constraints due to diminished contact forces and extended airborne intervals. Among traversal strategies, hopping locomotion offers high energy efficiency but is prone to mid-flight attitude instability caused by asymmetric thrust generation and uneven terrain interactions. This paper presents an underactuated bipedal hopping
robot that employs an internal reaction wheel to regulate body posture during the ballistic flight phase. The system is modeled as a gyrostat, enabling analysis of the dynamic coupling between torso rotation and reaction wheel momentum. The locomotion cycle comprises three phases: a leg-driven propulsive jump, mid-air attitude stabilization via an active momentum exchange controller, and a shock-absorbing landing. A reduced-order model is developed to capture the critical coupling between torso rotation and reaction wheel dynamics. The proposed framework is evaluated in MuJoCo-based simulations under lunar
gravity conditions (g = 1.625 m/s^2). Results demonstrate that activation of the reaction wheel controller reduces peak mid-air angular deviation by more than 65% and constrains landing attitude error to within 3.5 degrees at touchdown. Additionally, actuator saturation per hop cycle is reduced, ensuring sufficient control authority. Overall, the approach significantly mitigates in-flight attitude excursions and enables consistent upright landings, providing a practical and control-efficient solution for locomotion on irregular extraterrestrial terrains.
Read the full abstract ▾
Robotics
STM32-Based Smart Waste Bin for Hygienic Disposal Using Embedded Sensing and Automated Control
Mohammed Aman Bhuiyan, Aritra Islam Saswato, Md. Misbah Khan et al. (+3 more)
The increasing demand for hygienic and contactless solutions in public and private environments has encouraged the development of automated systems for everyday applications. This paper presents the design and implementation of a motion- sensing automatic waste bin using an STM32 microcontroller, ultrasonic sensors, and a servo motor. The system detects user presence through ultrasonic sensing and automatically opens the bin lid using a servo motor controlled by the microcontroller. An additional ultrasonic sensor is used to monitor the internal waste level of the bin, while an OLED display provides real- time feedback regarding system status. The proposed system offers a low-cost, reliable, and easily deployable solution for touch-free waste disposal. Experimental evaluation demonstrates fast response time, stable sensing performance, and smooth mechanical operation. The system can be effectively deployed in homes, educational institutions, hospitals, and public facilities to improve hygiene and user convenience.
Read the full abstract ▾
All authors Mohammed Aman Bhuiyan, Aritra Islam Saswato, Md. Misbah Khan, Anish Paul, Ahmed Faizul Haque Dhrubo, Mohammad Abdul Qayum
Robotics
Interleaving Scheduling and Motion Planning with Incremental Learning of Symbolic Space-Time Motion Abstractions
Elisa Tosello, Arthur Bit-Monnot, Davide Lusuardi et al. (+2 more)
Task and Motion Planning combines high-level task sequencing (what to do) with low-level motion planning (how to do it) to generate feasible, collision-free execution plans. However, in many real-world domains, such as automated warehouses, tasks are predefined, shifting the challenge to if, when, and how to execute them safely and efficiently under resource, time and motion constraints. In this paper, we formalize this as the Scheduling and Motion Planning problem for multi-object
navigation in shared workspaces. We propose a novel solution framework that interleaves off-the-shelf schedulers and motion planners in an incremental learning loop. The scheduler generates candidate plans, while the motion planner checks feasibility and returns symbolic feedback, i.e., spatial conflicts and timing adjustments, to guide the scheduler towards motion-feasible solutions. We validate our proposal on logistics and job-shop scheduling
benchmarks augmented with motion tasks, using state-of-the-art schedulers and sampling-based motion planners. Our results show the effectiveness of our framework in generating valid plans under complex temporal and spatial constraints, where synchronized motion is critical.
Read the full abstract ▾
All authors Elisa Tosello, Arthur Bit-Monnot, Davide Lusuardi, Alessandro Valentini, Andrea Micheli
Robotics
AdaClearGrasp: Learning Adaptive Clearing for Zero-Shot Robust Dexterous Grasping in Densely Cluttered Environments
Zixuan Chen, Wenquan Zhang, Jing Fang et al. (+7 more)
In densely cluttered environments, physical interference, visual occlusions, and unstable contacts often cause direct dexterous grasping to fail, while aggressive singulation strategies may compromise safety. Enabling
robots to adaptively decide whether to clear surrounding objects or directly grasp the target is therefore crucial for robust manipulation. We propose AdaClearGrasp, a closed-loop decision-execution framework for adaptive clearing and zero-shot dexterous grasping in densely cluttered environments. The framework formulates manipulation as a controllable high-level decision process that determines whether to directly grasp the target or first clear surrounding objects. A pretrained
vision-
language model (VLM) interprets visual observations and
language task descriptions to reason about grasp interference and generate a high-level planning skeleton, which invokes structured atomic skills through a unified action interface. For dexterous grasping, we train a reinforcement learning policy with a relative hand-object distance representation, enabling zero-shot generalization across diverse object geometries and physical properties. During execution, visual feedback monitors outcomes and triggers replanning upon failures, forming a closed-loop correction mechanism. To evaluate
language-conditioned dexterous grasping in clutter, we introduce Clutter-Bench, the first simulation
benchmark with graded clutter complexity. It includes seven target objects across three clutter levels, yielding 210 task scenarios. We further perform sim-to-real experiments on three objects under three clutter levels (18 scenarios). Results demonstrate that AdaClearGrasp significantly improves grasp success rates in densely cluttered environments. For more
videos and code, please visit our project website: https://chenzixuan99.github.io/adaclear-grasp.github.io/.
Read the full abstract ▾
All authors Zixuan Chen, Wenquan Zhang, Jing Fang, Ruiming Zeng, Zhixuan Xu, Yiwen Hou, Xinke Wang, Jieqi Shi, Jing Huo, Yang Gao
Robotics
Learning Bimanual Cloth Manipulation with Vision-based Tactile Sensing via Single Robotic Arm
Dongmyoung Lee, Wei Chen, Xiaoshuai Chen et al. (+2 more)
Robotic cloth manipulation remains challenging due to the high-dimensional state space of fabrics, their deformable nature, and frequent occlusions that limit
vision-based sensing. Although dual-arm systems can mitigate some of these issues, they increase hardware and control complexity. This paper presents Touch G.O.G., a compact
vision-based tactile gripper and perception/control framework for single-arm bimanual cloth manipulation. The proposed framework combines three key components: (1) a novel gripper design and control strategy for in-gripper cloth sliding with a single
robot arm, (2) a
Vision Foundation Model-backboned
Vision Transformer pipeline for cloth part classification (PC-Net) and edge pose estimation (PE-Net) using real and synthetic tactile images, and (3) an encoder-decoder synthetic data generator (SD-Net) that reduces manual annotation by producing high-fidelity tactile images. Experiments show 96% accuracy in distinguishing edges, corners, interior regions, and grasp failures, together with sub-millimeter edge localization and 4.5{\deg} orientation error. Real-world results demonstrate reliable cloth unfolding, even for crumpled fabrics, using only a single robotic arm. These results highlight Touch G.O.G. as a compact and cost-effective solution for deformable object manipulation.
Read the full abstract ▾
All authors Dongmyoung Lee, Wei Chen, Xiaoshuai Chen, Rui Zong, Petar Kormushev
Robotics
Recover to Predict: Progressive Retrospective Learning for Variable-Length Trajectory Prediction
Hao Zhou, Lu Qi, Jason Li et al. (+5 more)
Trajectory prediction is critical for autonomous driving, enabling safe and efficient planning in dense, dynamic traffic. Most existing methods optimize prediction accuracy under fixed-length observations. However, real-world driving often yields variable-length, incomplete observations, posing a challenge to these methods. A common strategy is to directly map features from incomplete observations to those from complete ones. This one-shot mapping, however, struggles to learn accurate representations for short trajectories due to significant information gaps. To address this issue, we propose a Progressive Retrospective Framework (PRF), which gradually aligns features from incomplete observations with those from complete ones via a cascade of retrospective units. Each unit consists of a Retrospective
Distillation Module (RDM) and a Retrospective Prediction Module (RPM), where RDM distills features and RPM recovers previous timesteps using the distilled features. Moreover, we propose a Rolling-Start Training Strategy (RSTS) that enhances data efficiency during PRF training. PRF is plug-and-play with existing methods. Extensive experiments on
datasets Argoverse 2 and Argoverse 1 demonstrate the effectiveness of PRF. Code is available at https://github.com/zhouhao94/PRF.
Read the full abstract ▾
All authors Hao Zhou, Lu Qi, Jason Li, Jie Zhang, Yi Liu, Xu Yang, Mingyu Fan, Fei Luo
Robotics · Cross-listed from cs.CV
Need for Speed: Zero-Shot Depth Completion with Single-Step Diffusion
Jakub Gregorek, Paraskevas Pegios, Nando Metzger et al. (+3 more)
We introduce Marigold-SSD, a single-step, late-fusion depth completion framework that leverages strong
diffusion priors while eliminating the costly test-time optimization typically associated with
diffusion-based methods. By shifting computational burden from inference to finetuning, our approach enables efficient and robust 3D perception under real-world latency constraints. Marigold-SSD achieves significantly faster inference with a training cost of only 4.5 GPU days. We evaluate our method across four indoor and two outdoor
benchmarks, demonstrating strong cross-domain generalization and zero-shot performance compared to existing depth completion approaches. Our approach significantly narrows the efficiency gap between
diffusion-based and discriminative models. Finally, we challenge common evaluation protocols by analyzing performance under varying input sparsity levels. Page: https://dtu-pas.github.io/marigold-ssd/
Read the full abstract ▾
All authors Jakub Gregorek, Paraskevas Pegios, Nando Metzger, Konrad Schindler, Theodora Kontogianni, Lazaros Nalpantidis
Robotics
Safety-critical Control Under Partial Observability: Reach-Avoid POMDP meets Belief Space Control
Matti Vahs, Joris Verhagen and Jana Tumova
Partially Observable Markov Decision Processes (POMDPs) provide a principled framework for
robot decision-making under uncertainty. Solving reach-avoid POMDPs, however, requires coordinating three distinct behaviors: goal reaching, safety, and active information gathering to reduce uncertainty. Existing online POMDP solvers attempt to address all three within a single belief tree search, but this unified approach struggles with the conflicting time scales inherent to these objectives. We propose a layered, certificate-based control architecture that operates directly in belief space, decoupling goal reaching, information gathering, and safety into modular components. We introduce Belief Control Lyapunov Functions (BCLFs) that formalize information gathering as a Lyapunov convergence problem in belief space, and show how they can be learned via reinforcement learning. For safety, we develop Belief Control Barrier Functions (BCBFs) that leverage conformal prediction to provide probabilistic safety guarantees over finite horizons. The resulting control synthesis reduces to lightweight quadratic programs solvable in real time, even for non-Gaussian belief representations with dimension $>10^4$. Experiments in simulation and on a space-robotics platform demonstrate real-time performance and improved safety and task success compared to state-of-the-art constrained POMDP solvers.
Read the full abstract ▾
Robotics
TacLoc: Global Tactile Localization on Objects from a Registration Perspective
Zirui Zhang, Boyang Zhang, Fumin Zhang et al. (+1 more)
Pose estimation is essential for robotic manipulation, particularly when visual perception is occluded during gripper-object interactions. Existing tactile-based methods generally rely on tactile simulation or pre-trained models, which limits their generalizability and efficiency. In this study, we propose TacLoc, a novel tactile localization framework that formulates the problem as a one-shot point cloud registration task. TacLoc introduces a graph-theoretic partial-to-full registration method, leveraging dense point clouds and surface normals from tactile sensing for efficient and accurate pose estimation. Without requiring rendered data or pre-trained models, TacLoc achieves improved performance through normal-guided graph
pruning and a hypothesis-and-verification pipeline. TacLoc is evaluated extensively on the YCB
dataset. We further demonstrate TacLoc on real-world objects across two different visual-tactile sensors.
Read the full abstract ▾
All authors Zirui Zhang, Boyang Zhang, Fumin Zhang, Huan Yin
Robotics
BinWalker: Development and Field Evaluation of a Quadruped Manipulator Platform for Sustainable Litter Collection
Giulio Turrisi, Angelo Bratta, Giovanni Minelli et al. (+4 more)
Litter pollution represents a growing environmental problem affecting natural and urban ecosystems worldwide. Waste discarded in public spaces often accumulates in areas that are difficult to access, such as uneven terrains, coastal environments, parks, and roadside vegetation. Over time, these materials degrade and release harmful substances, including toxic chemicals and microplastics, which can contaminate soil and water and pose serious threats to wildlife and human health. Despite increasing awareness of the problem, litter collection is still largely performed manually by human operators, making large-scale cleanup operations labor-intensive, time-consuming, and costly. Robotic solutions have the potential to support and partially automate environmental cleanup tasks. In this work, we present a quadruped robotic system designed for autonomous litter collection in challenging outdoor scenarios. The
robot combines the mobility advantages of legged locomotion with a manipulation system consisting of a robotic arm and an onboard litter container. This configuration enables the
robot to detect, grasp, and store litter items while navigating through uneven terrains. The proposed system aims to demonstrate the feasibility of integrating perception, locomotion, and manipulation on a legged robotic platform for environmental cleanup tasks. Experimental evaluations conducted in outdoor scenarios highlight the effectiveness of the approach and its potential for assisting large-scale litter removal operations in environments that are difficult to reach with traditional robotic platforms. The code associated with this work can be found at: https://github.com/iit-DLSLab/trash-collection-isaaclab.
Read the full abstract ▾
All authors Giulio Turrisi, Angelo Bratta, Giovanni Minelli, Gabriel Fischer Abati, Amir H. Rad, João Carlos Virgolino Soares, Claudio Semini
Robotics · Cross-listed from cs.LG
Muscle Synergy Priors Enhance Biomechanical Fidelity in Predictive Musculoskeletal Locomotion Simulation
Ilseung Park, Eunsik Choi, Jangwhan Ahn et al. (+1 more)
Human locomotion emerges from high-dimensional neuromuscular control, making predictive musculoskeletal simulation challenging. We present a physiology-informed reinforcement-learning framework that constrains control using muscle synergies. We extracted a low-dimensional synergy basis from inverse musculoskeletal analyses of a small set of overground walking trials and used it as the action space for a muscle-driven three-dimensional model trained across variable speeds, slopes and uneven terrain. The resulting controller generated stable gait from 0.7-1.8 m/s and on $\pm$ 6$^{\circ}$ grades and reproduced condition-dependent modulation of joint angles, joint moments and ground reaction forces. Compared with an unconstrained controller, synergy-constrained control reduced non-physiological knee kinematics and kept knee moment profiles within the experimental envelope. Across conditions, simulated vertical ground reaction forces correlated strongly with human measurements, and muscle-activation timing largely fell within inter-subject variability. These results show that
embedding neurophysiological structure into reinforcement learning can improve biomechanical fidelity and generalization in predictive human locomotion simulation with limited experimental data.
Read the full abstract ▾
All authors Ilseung Park, Eunsik Choi, Jangwhan Ahn, Jooeun Ahn
Robotics
DepthCache: Depth-Guided Training-Free Visual Token Merging for Vision-Language-Action Model Inference
Yuquan Li, Lianjie Ma, Han Ding et al. (+1 more)
Vision-
Language-Action (VLA) models enable generalist robotic manipulation but suffer from high inference latency. This bottleneck stems from the massive number of visual
tokens processed by large
language backbones. Existing methods either prune or merge
tokens uniformly, degrading the spatial
reasoning essential for robotic control. We present DepthCache, a training-free framework that leverages depth as a structural prior for visual
token compression. It partitions observations into depth-based regions and applies spatially differentiated merge ratios, preserving the near-field workspace while compressing the distant background. To exploit temporal redundancy, DepthCache distributes the merging process across consecutive frames, ensuring consistent representations while reducing per-step computation. A motion-adaptive pipeline further optimizes auxiliary view compression based on end-effector dynamics. The framework requires no model modification, generalizing across diverse VLA architectures. On the LIBERO
benchmark, DepthCache achieves up to 1.28x inference speedup with less than 1% average success rate degradation across three VLA models (pi_0.5, OpenVLA, GR00T), whereas
pruning and merging baselines incur 4–24% degradation at comparable compression. Real-world experiments on a physical manipulator demonstrate that DepthCache enables faster task throughput and more responsive closed-loop control in latency-sensitive scenarios.
Read the full abstract ▾
All authors Yuquan Li, Lianjie Ma, Han Ding, Lijun Zhu
Robotics
SUBTA: A Framework for Supported User-Guided Bimanual Teleoperation in Structured Assembly
Xiao Liu, Prakash Baskaran, Songpo Li et al. (+4 more)
In human-
robot collaboration, shared autonomy enhances human performance through precise, intuitive support. Effective robotic assistance requires accurately inferring human intentions and understanding task structures to determine optimal support timing and methods. In this paper, we present SUBTA, a supported teleoperation system for bimanual assembly that couples learned intention estimation, scene-graph task planning, and context-dependent motion assists. We validate our approach through a user study (N=12) comparing standard teleoperation, motion-support only, and SUBTA. Linear mixed-effects analysis revealed that SUBTA significantly outperformed standard teleoperation in position accuracy (p<0.001, d=1.18) and orientation accuracy (p<0.001, d=1.75), while reducing mental demand (p=0.002, d=1.34). Post-experiment ratings indicate clearer, more trustworthy visual feedback and predictable interventions in SUBTA. The results demonstrate that SUBTA greatly improves both effectiveness and user experience in teleoperation.
Read the full abstract ▾
All authors Xiao Liu, Prakash Baskaran, Songpo Li, Simon Manschitz, Wei Ma, Dirk Ruiken, Soshi Iba
Robotics
FAR-Dex: Few-shot Data Augmentation and Adaptive Residual Policy Refinement for Dexterous Manipulation
Yushan Bai, Fulin Chen, Hongzheng Sun et al. (+3 more)
Achieving human-like dexterous manipulation through the collaboration of multi-fingered hands with robotic arms remains a longstanding challenge in robotics, primarily due to the scarcity of high-quality demonstrations and the complexity of high-dimensional action spaces. To address these challenges, we propose FAR-Dex, a hierarchical framework that integrates few-shot data augmentation with adaptive residual refinement to enable robust and precise arm-hand coordination in dexterous tasks. First, FAR-DexGen leverages the IsaacLab simulator to generate diverse and physically constrained trajectories from a few demonstrations, providing a data foundation for policy training. Second, FAR-DexRes introduces an adaptive residual module that refines policies by combining multi-step trajectory segments with observation features, thereby enhancing accuracy and robustness in manipulation scenarios. Experiments in both simulation and real-world demonstrate that FAR-Dex improves data quality by 13.4% and task success rates by 7% over state-of-the-art methods. It further achieves over 80% success in real-world tasks, enabling fine-grained dexterous manipulation with strong positional generalization.
Read the full abstract ▾
All authors Yushan Bai, Fulin Chen, Hongzheng Sun, Yuchuang Tong, En Li, Zhengtao Zhang
Robotics
DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control
Teli Ma, Jia Zheng, Zifan Wang et al. (+4 more)
Vision-
Language-Action (VLA) models have emerged as a promising paradigm for
robot learning, but their representations are still largely inherited from static image-text pretraining, leaving physical dynamics to be learned from comparatively limited action data. Generative
video models, by contrast, encode rich spatiotemporal structure and implicit physics, making them a compelling foundation for robotic manipulation. But their potentials are not fully explored in the literature. To bridge the gap, we introduce DiT4DiT, an end-to-end
Video-Action Model that couples a
video Diffusion Transformer with an action
Diffusion Transformer in a unified cascaded framework. Instead of relying on reconstructed future frames, DiT4DiT extracts intermediate denoising features from the
video generation process and uses them as temporally grounded conditions for action prediction. We further propose a dual flow-matching objective with decoupled timesteps and noise scales for
video prediction, hidden-state extraction, and action inference, enabling coherent joint training of both modules. Across simulation and real-world
benchmarks, DiT4DiT achieves state-of-the-art results, reaching average success rates of 98.6% on LIBERO and 50.8% on RoboCasa GR1 while using substantially less training data. On the Unitree G1
robot, it also delivers superior real-world performance and strong zero-shot generalization. Importantly, DiT4DiT improves sample efficiency by over 10x and speeds up convergence by up to 7x, demonstrating that
video generation can serve as an effective scaling proxy for
robot policy learning. We release code and models at https://dit4dit.github.io/.
Read the full abstract ▾
All authors Teli Ma, Jia Zheng, Zifan Wang, Chunli Jiang, Andy Cui, Junwei Liang, Shuo Yang
Robotics
KnowDiffuser: A Knowledge-Guided Diffusion Planner with LLM Reasoning
Fan Ding, Xuewen Luo, Fengze Yang et al. (+4 more)
Recent advancements in
Language Models (LMs) have demonstrated strong semantic
reasoning capabilities, enabling their application in high-level decision-making for autonomous driving (AD). However, LMs operate over discrete
token spaces and lack the ability to generate continuous, physically feasible trajectories required for motion planning. Meanwhile,
diffusion models have proven effective at generating reliable and dynamically consistent trajectories, but often lack semantic interpretability and
alignment with scene-level understanding. To address these limitations, we propose KnowDiffuser, a knowledge-guided motion planning framework that tightly integrates the semantic understanding of
language models with the generative power of
diffusion models. The framework employs a
language model to infer context-aware meta-actions from structured scene representations, which are then mapped to prior trajectories that anchor the subsequent denoising process. A two-stage truncated denoising mechanism refines these trajectories efficiently, preserving both semantic
alignment and physical feasibility. Experiments on the nuPlan
benchmark demonstrate that KnowDiffuser significantly outperforms existing planners in both open-loop and closed-loop evaluations, establishing a robust and interpretable framework that effectively bridges the semantic-to-physical gap in AD systems.
Read the full abstract ▾
All authors Fan Ding, Xuewen Luo, Fengze Yang, Bo Yu, HwaHui Tew, Ganesh Krishnasamy, Junn Yong Loo
Robotics
AsyncMDE: Real-Time Monocular Depth Estimation via Asynchronous Spatial Memory
Lianjie Ma, Yuquan Li, Bingzheng Jiang et al. (+3 more)
Foundation-model-based
monocular depth estimation offers a viable alternative to active sensors for
robot perception, yet its computational cost often prohibits deployment on edge platforms. Existing methods perform independent per-frame inference, wasting the substantial computational redundancy between adjacent viewpoints in continuous
robot operation. This paper presents AsyncMDE, an asynchronous depth perception system consisting of a frozen foundation model and a lightweight fast path that amortizes the foundation model's computational cost over time. The foundation model periodically produces high-quality spatial features in the background, while the lightweight fast path runs asynchronously in the foreground, fusing cached
memory with current observations through complementary fusion, outputting depth estimates, and autoregressively updating
memory. This enables cross-frame feature reuse with bounded accuracy degradation. With 3.83M trainable fast-path parameters and a 97.5M frozen slow path, AsyncMDE's fast path operates at 237 FPS on an RTX 4090, recovering 77% of the accuracy gap to the foundation model. Across indoor static, dynamic, and synthetic extreme-motion
benchmarks, AsyncMDE degrades predictably and reaches 161 FPS fast-path inference on a TensorRT-optimized Jetson AGX Orin, supporting real-time edge deployment.
Read the full abstract ▾
All authors Lianjie Ma, Yuquan Li, Bingzheng Jiang, Ziming Zhong, Han Ding, Lijun Zhu
Robotics
COHORT: Hybrid RL for Collaborative Large DNN Inference on Multi-Robot Systems Under Real-Time Constraints
Mohammad Saeid Anwar, Anuradha Ravi, Indrajeet Ghosh et al. (+3 more)
Large deep neural networks (DNNs), especially
transformer-based and
multimodal architectures, are computationally demanding and challenging to deploy on resource-constrained edge platforms like field
robots. These challenges intensify in mission-critical scenarios (e.g., disaster response), where
robots must collaborate under tight constraints on bandwidth, latency, and battery life, often without infrastructure or server support. To address these limitations, we present COHORT, a collaborative DNN inference and task-execution framework for multi-
robot systems built on the Robotic Operating System (ROS). COHORT employs a hybrid offline-online reinforcement learning (RL) strategy to dynamically schedule and distribute DNN module execution across
robots. Our key contributions are threefold: (a) Offline RL policy learning combined with Advantage-Weighted Regression (AWR), trained on auction-based task allocation data from heterogeneous DNN workloads across distributed
robots, (b) Online policy adaptation via Multi-
Agent PPO (MAPPO), initialized from the offline policy and fine-tuned in real time, and (c) comprehensive evaluation of COHORT on
vision-
language model (VLM) inference tasks such as CLIP and SAM, analyzing scalability with increasing
robot/workload and robustness under . We
benchmark COHORT against genetic
algorithms and multiple RL baselines. Experimental results demonstrate that COHORT reduces battery consumption by 15.4% and increases GPU utilization by 51.67%, while satisfying frame-rate and deadline constraints 2.55 times of the time.
Read the full abstract ▾
All authors Mohammad Saeid Anwar, Anuradha Ravi, Indrajeet Ghosh, Gaurav Shinde, Carl Busart, Nirmalya Roy