Robotics
Studying the Effects of Robot Intervention on School Shooters in Virtual Reality
Christopher A McClurg and Alan R Wagner
We advance the understanding of robotic intervention in high-risk scenarios by examining their potential to distract and impede a school shooter. To evaluate this concept, we conducted a virtual reality study with 150 university participants role-playing as a school shooter. Within the simulation, an autonomous
robot predicted the shooter's movements and positioned itself strategically to interfere and distract. The strategy the
robot used to approach the shooter was manipulated – either moving directly in front of the shooter (aggressive) or maintaining distance (passive) – and the distraction method, ranging from no additional cues (low), to siren and lights (medium), to siren, lights, and smoke to impair visibility (high). An aggressive, high-distraction
robot reduced the number of victims by 46.6% relative to a no-
robot control. This outcome underscores both the potential of robotic intervention to enhance safety and the pressing ethical questions surrounding their use in school environments.
Read the full abstract ▾
Robotics
Yulin Luo, Chun-Kai Fan, Menghang Dong et al. (+19 more)
Building
robots that can perceive, reason, and act in dynamic, unstructured environments remains a central challenge. Recent
embodied systems often follow a dual-system paradigm, where System 2 performs high-level
reasoning and System 1 handles low-level control. We refer to System 2 as the
embodied brain, the cognitive core for decision-making in manipulation. Although evaluating this
embodied brain is crucial, existing
benchmarks mainly measure execution success or cover only limited aspects of high-level cognition and task realism. We introduce RoboBench, a
benchmark for evaluating
multimodal large
language models (MLLMs) as
embodied brains. RoboBench covers five dimensions: Instruction Comprehension, Perception
Reasoning, Generalized Planning, Affordance Prediction, and Failure Analysis. It spans 14 capabilities, 25 tasks, and 6,092 QA pairs. To improve realism, it draws from large-scale real robotic data and in-house collection across diverse embodiments, attribute-rich objects, multi-view scenes, and
memory-driven
navigation. For planning, RoboBench introduces an MLLM-as-world-simulator framework that assesses whether predicted plans can achieve critical object-state changes under physical and visual constraints, enabling more faithful evaluation of long-horizon
reasoning than symbolic matching. Experiments on 18 state-of-the-art MLLMs reveal persistent limitations in implicit instruction understanding, spatiotemporal
reasoning, cross-scenario planning, fine-grained affordance understanding, and failure diagnosis. We further analyze how
embodied cognitive abilities relate to downstream robotic control. RoboBench offers a comprehensive scaffold for quantifying high-level cognition and guiding next-generation MLLMs toward more robust robotic intelligence.
Read the full abstract ▾
All authors Yulin Luo, Chun-Kai Fan, Menghang Dong, Jiayu Shi, Xiangju Mi, Mengdi Zhao, Bo-Wen Zhang, Cheng Chi, Jiaming Liu, Gaole Dai, Rongyu Zhang, Ruichuan An, Kun Wu, Zhengping Che, Shaoxuan Xie, Guocai Yao, Zhongxia Zhao, Pengwei Wang, Guang Liu, Zhongyuan Wang, Tiejun Huang, Shanghang Zhang
Robotics
SoftMimic: Learning Compliant Whole-body Control from Examples
Gabriel B. Margolis, Michelle Wang, Nolan Fey et al. (+1 more)
We introduce SoftMimic, a framework for learning compliant whole-body control policies for
humanoid robots from example motions. Imitating human motions with reinforcement learning allows
humanoids to quickly learn new skills, but existing methods incentivize stiff control that aggressively corrects deviations from a reference motion, leading to brittle and unsafe behavior when the
robot encounters unexpected contacts. In contrast, SoftMimic enables
robots to respond compliantly to external forces while maintaining balance and posture. Our approach leverages an inverse kinematics solver to generate an augmented
dataset of feasible compliant motions, which we use to train a reinforcement learning policy. By rewarding the policy for matching compliant responses rather than rigidly tracking the reference motion, SoftMimic learns to absorb disturbances and generalize to varied tasks from a single motion clip. We validate our method through simulations and real-world experiments, demonstrating safe and effective interaction with the environment.
Read the full abstract ▾
All authors Gabriel B. Margolis, Michelle Wang, Nolan Fey, Pulkit Agrawal
Robotics
Botany-Bot: Digital Twin Monitoring of Occluded and Underleaf Plant Structures with Gaussian Splats
Simeon Adebola, Chung Min Kim, Justin Kerr et al. (+5 more)
Commercial plant phenotyping systems using fixed cameras cannot perceive many plant details due to leaf occlusion. In this paper, we present Botany-Bot, a system for building detailed "annotated digital twins" of living plants using two stereo cameras, a digital turntable inside a lightbox, an industrial
robot arm, and 3D segmentated Gaussian Splat models. We also present
robot algorithms for manipulating leaves to take high-resolution indexable images of occluded details such as stem buds and the underside/topside of leaves. Results from experiments suggest that Botany-Bot can segment leaves with 90.8% accuracy, detect leaves with 86.2% accuracy, lift/push leaves with 77.9% accuracy, and take detailed overside/underside images with 77.3% accuracy. Code,
videos, and
datasets are available at https://berkeleyautomation.github.io/Botany-Bot/.
Read the full abstract ▾
All authors Simeon Adebola, Chung Min Kim, Justin Kerr, Shuangyu Xie, Prithvi Akella, Jose Luis Susa Rincon, Eugen Solowjow, Ken Goldberg
Robotics
RESample: A Robust Data Augmentation Framework via Exploratory Sampling for Robotic Manipulation
Yuquan Xue, Guanxing Lu, Zhenyu Wu et al. (+4 more)
Vision-
Language-Action (VLA) models have shown strong manipulation capability when trained with large-scale imitation learning
datasets. However, these
datasets that predominantly consist of successful trajectories rarely provide the corrective supervision required when execution deviates from standard demonstrations. During deployment, these physical deviations lead to the distributional shift that drives policy to failure scenarios, yet missing failure recovery data prevents policy from correcting these execution deviations. To address this failure recovery problem, we propose a coverage-guided data augmentation framework RESample to actively supplement demonstration
datasets for failure recovery. Specifically, to guide the augmentation and generate failure modes that possibly appear in the real world, RESample trains a conservative coverage function to identify failure cases that reside within the actual data distribution but are missing in the standard successful demonstrations. Guided by the evaluated coverage discrepancy, we perform exploratory sampling to actively sample exploration behaviors followed by recovery actions, extending the coverage of training data with failure recovery trajectories. With the augmented trajectory, the refined policy, which deviated in real settings, can recover from failure. Experiments on the LIBERO
benchmark and real-world manipulation tasks show that RESample consistently improves policy success rates, achieving up to 12% absolute gain with no more than 20% additional samples.
Read the full abstract ▾
All authors Yuquan Xue, Guanxing Lu, Zhenyu Wu, Chuanrui Zhang, Bofang Jia, Zhengyi Gu, Ziwei Wang
Robotics
MoE-Based Learned Inertial Odometry for Bicycle Localization
Hao Qiao, Yan Wang, Shuo Yang et al. (+3 more)
GNSS suffers from multipath errors in urban canyons, making reliable bicycle localization difficult. Hand-crafted inertial alternatives, such as cycling dead reckoning and nonholonomic constraints, fail to generalize across riders, postures, and road surfaces. This work implements learned inertial odometry for bicycle localization: a neural velocity predictor, trained on multirider, multi-bicycle, and multi-surface IMU data, is tightly coupled with an extended Kalman filter for three-dimensional pose estimation, replacing hand-crafted cycling models with a data-driven representation. To reduce computational overhead, the predictor is realized as a sparsely-gated Mixture of Experts (MoE) network with top-K routing, trained under an alternating parameter-freezing scheme with a per-expert capacity constraint to encourage diverse and balanced specialization.The proposed model attains 0.333 m/s inference error, 9.49 m ATE, and 2.58 m RTE at only 28.76 M FLOPs. This corresponds to approximately 7 times and 9 times fewer FLOPs than the ResMLP and ResNet backbones used in LLIO and TLIO, while maintaining comparable or superior accuracy. The proposed MoE variants further rank first on both ATE and RTE across all evaluation splits, demonstrating strong generalization.
Read the full abstract ▾
All authors Hao Qiao, Yan Wang, Shuo Yang, Xiaoyao Yu, Jian kuang, Xiaoji Niu
Robotics
Intent-Driven LLM Ensemble Planning for Flexible Multi-Robot Disassembly: Demonstration on EV Batteries
Cansu Erdogan, Cesar Alan Contreras, Alireza Rastegarpanah et al. (+2 more)
This paper addresses the problem of planning complex manipulation tasks, in which multiple
robots with different end-effectors and capabilities, informed by computer
vision, must plan and execute concatenated sequences of actions on a variety of objects that can appear in arbitrary positions and configurations in unstructured scenes. We propose an intent-driven planning pipeline which can robustly construct such action sequences with varying degrees of supervisory input from a human using simple
language instructions. The pipeline integrates: (i) perception-to-text scene encoding, (ii) an ensemble of large
language models (LLMs) that generate candidate removal sequences based on the operator's intent, (iii) an LLM-based verifier that enforces formatting and precedence constraints, and (iv) a deterministic consistency filter that rejects hallucinated objects. The pipeline is evaluated on an example task in which two
robot arms work collaboratively to dismantle an Electric Vehicle battery for recycling applications. A variety of components must be grasped and removed in specific sequences, determined by human instructions and/or by task-order feasibility decisions made by the autonomous system. On 200 real scenes with 600 operator
prompts across five component classes, we used metrics of full-sequence correctness and next-task correctness to evaluate and compare five LLM-based planners (including ablation analyses of pipeline components). We also evaluated the LLM-based human interface in terms of time to execution and NASA TLX with human participant experiments. Results indicate that our ensemble-with-verification approach reliably maps operator intent to safe, executable multi-
robot plans while maintaining low user effort.
Read the full abstract ▾
All authors Cansu Erdogan, Cesar Alan Contreras, Alireza Rastegarpanah, Manolis Chiou, Rustam Stolkin
Robotics · Cross-listed from cs.LG
Towards a Practical Understanding of Lagrangian Methods in Safe Reinforcement Learning
Lindsay Spoor, Álvaro Serra-Gómez, Aske Plaat et al. (+1 more)
Safe reinforcement learning addresses constrained optimization problems where maximizing performance must be balanced against safety constraints, and Lagrangian methods are a widely used approach for this purpose. However, the effectiveness of Lagrangian methods depends crucially on the choice of the Lagrange multiplier $\lambda$, which governs the multi-objective trade-off between return and cost. A common practice is to update the multiplier automatically during training. Although this approach is standard in practice, there remains limited empirical evidence on the optimally achievable trade-off between return and cost as a function of $\lambda$, and there is currently no systematic
benchmark comparing automated update mechanisms to this empirical optimum. Therefore, we study (i) the constraint geometry for eight widely used safety tasks and (ii) the previously overlooked constraint-regime sensitivity of different Lagrange multiplier update mechanisms in safe reinforcement learning. Through the lens of multi-objective analysis, we present empirical Pareto frontiers that offer a complete visualization of the trade-off between return and cost in the underlying optimization problem. Our results reveal the highly sensitive nature of $\lambda$ and further show that the restrictiveness of the constraint cost can vary across different cost limits within the same task. This highlights the importance of careful cost limit selection across different regions of cost restrictiveness when evaluating safe reinforcement learning methods. We provide a recommended set of cost limits for each evaluated task and offer an open-source code base: https://github.com/lindsayspoor/Lagrangian_SafeRL.
Read the full abstract ▾
All authors Lindsay Spoor, Álvaro Serra-Gómez, Aske Plaat, Thomas Moerland
Robotics
Distributed Spatial-Temporal Trajectory Optimization for Unmanned-Aerial-Vehicle Swarm
Xiaobo Zheng, Pan Tang, Defu Lin et al. (+1 more)
Swarm trajectory optimization problems are a well-recognized class of multi-
agent optimal control problems with strong nonlinearity. However, the heuristic nature of needing to set the final time for
agents beforehand and the time-consuming limitation of the significant number of iterations prohibit the application of existing methods to large-scale swarm of Unmanned Aerial Vehicles (
UAVs) in practice. In this paper, we propose a spatial-temporal trajectory optimization framework that accomplishes multi-
UAV consensus based on the Alternating Direction Multiplier Method (ADMM) and uses Differential Dynamic Programming (DDP) for fast local planning of individual
UAVs. The introduced framework is a two-level architecture that employs Parameterized DDP (PDDP) as the trajectory optimizer for each
UAV, and ADMM to satisfy the local constraints and accomplish the spatial-temporal parameter consensus among all
UAVs. This results in a fully distributed
algorithm called Distributed Parameterized DDP (D-PDDP). In addition, an adaptive tuning criterion based on the spectral gradient method for the penalty parameter is proposed to reduce the number of algorithmic iterations. Several simulation examples are presented to verify the effectiveness of the proposed
algorithm.
Read the full abstract ▾
All authors Xiaobo Zheng, Pan Tang, Defu Lin, Shaoming He
Robotics
HumanHalo: Safe and Efficient 3D Navigation Among Humans via Minimally Conservative MPC
Simon Schaefer, Helen Oleynikova, Sandra Hirche et al. (+1 more)
Safe and efficient robotic
navigation among humans is essential for integrating
robots into everyday environments. Most existing approaches focus on simplified 2D crowd
navigation and fail to account for the full complexity of human body dynamics beyond root motion. We present HumanHalo, an MPC framework for 3D MAV
navigation among humans that combines theoretical safety guarantees with data-driven models for realistic human motion
forecasting. Our approach introduces a novel reachability-based safety formulation that constrains only the initial control input for safety while modeling its effects over the entire planning horizon, enabling safe yet efficient
navigation. We validate HumanHalo in both simulated experiments using real human trajectories and in the real world, demonstrating its effectiveness across tasks ranging from goal-directed
navigation to visual servoing for human tracking. While we apply our method to MAV in this work, it is generic and can be adapted to other platforms. Our results show that the method preserves safety without excessive conservatism, even under imperfect human prediction and in constrained workspaces, while remaining efficient enough for real-time onboard execution.
Read the full abstract ▾
All authors Simon Schaefer, Helen Oleynikova, Sandra Hirche, Stefan Leutenegger
Robotics · Cross-listed from physics.med-ph
Inverse Optimal Control of Muscle Force Sharing During Pathological Gait
Filip Bečanović, Vincent Bonnet, Kosta Jovanović et al. (+2 more)
Muscle force sharing is typically resolved by minimizing a specific objective function to approximate neural control strategies. An inverse optimal control approach was applied to identify the "best" objective function, among a positive linear combination of basis objective functions, associated with the gait of two post-stroke males, one high-functioning (subject S1) and one low-functioning (subject S2). It was found that the "best" objective function is subject- and leg-specific. No single function works universally well, yet the best options are usually differently weighted combinations of muscle activation- and power-minimization. Subject-specific inverse optimal control models performed best on their respective limbs (RMSE 178/213 N, CC 0.71/0.61 for non-paretic and paretic legs of S1; RMSE 205/165 N, CC 0.88/0.85 for respective legs of S2), but cross-subject generalization was poor, particularly for paretic legs. Moreover, minimizing the root mean square of muscle power emerged as important for paretic limbs, while minimizing activation-based functions dominated for non-paretic limbs. This may suggest different neural control strategies between affected and unaffected sides, possibly altered by the presence of spasticity. Among the 15 considered objective functions commonly used in inverse dynamics-based computations, the root mean square of muscle power was the only one explicitly incorporating muscle velocity, leading to a possible model for spasticity in the paretic limbs. Although this objective function has been rarely used, it may be relevant for modeling pathological gait, such as post-stroke gait.
Read the full abstract ▾
All authors Filip Bečanović, Vincent Bonnet, Kosta Jovanović, Samer Mohammed, Raphaël Dumas
Robotics
Switching Among Feedback-Linearizing Output Sets (Melds): Dwell-Time and Compatibility Guarantees
Mirko Mizzoni, Pieter van Goor, Barbara Bazzana et al. (+1 more)
We study switching among multiple square selections of output functions (melds) drawn from a deck of candidate outputs for nonlinear systems that are static feedback linearizable via outputs. Fixing an operating point, each meld induces a distinct feedback-linearizing coordinate chart defined on a common neighborhood. Switching between melds therefore produces state-dependent coordinate mismatches that are not captured by classical switched-system analyses. We quantify this effect through Lipschitz bounds on the cross-chart maps over a compact safe set and introduce a reference-compatibility constant that measures mismatch among reference families across melds. We derive an explicit dwell-time condition depending on controller decay rates and the compatibility constant, that guarantees exponential decay of the active-output tracking errors between switches, seamless tracking of outputs shared by consecutive melds, and uniform boundedness of the state error within the safe set. A planar 3R manipulator illustrates the results.
Read the full abstract ▾
All authors Mirko Mizzoni, Pieter van Goor, Barbara Bazzana, Antonio Franchi
Robotics
From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priors
Zhengshen Zhang, Hao Li, Yalun Dai et al. (+10 more)
Existing
vision-
language-action (VLA) models act in 3D real-world but are typically built on 2D encoders, leaving a spatial
reasoning gap that limits generalization and adaptability. Recent 3D integration techniques for VLAs either require specialized sensors and transfer poorly across modalities, or inject weak cues that lack geometry and degrade
vision-
language alignment. In this work, we introduce FALCON (From Spatial to Action), a novel paradigm that injects rich 3D spatial
tokens into the action head. FALCON leverages spatial foundation models to deliver strong
geometric priors from RGB alone, and includes an
Embodied Spatial Model that can optionally fuse depth, or pose for higher fidelity when available, without retraining or architectural changes. To preserve
language reasoning, spatial
tokens are consumed by a Spatial-Enhanced Action Head rather than being concatenated into the
vision-
language backbone. These designs enable FALCON to address limitations in spatial representation, modality transferability, and
alignment. In comprehensive evaluations across three simulation
benchmarks and eleven real-world tasks, our proposed FALCON achieves state-of-the-art performance, consistently surpasses competitive baselines, and remains robust under clutter, spatial-
prompt conditioning, and variations in object scale and height.
Read the full abstract ▾
All authors Zhengshen Zhang, Hao Li, Yalun Dai, Zhengbang Zhu, Lei Zhou, Chenchen Liu, Dong Wang, Francis E. H. Tay, Sijin Chen, Ziwei Liu, Yuxiao Liu, Xinghang Li, Pan Zhou
Robotics
Integrating Trustworthy Artificial Intelligence with Energy-Efficient Robotic Arms for Waste Sorting
Halima I. Kure, Jishna Retnakumari, Augustine O. Nwajana et al. (+3 more)
This paper presents a novel methodology that integrates trustworthy artificial intelligence (AI) with an energy-efficient robotic arm for intelligent waste classification and sorting. By utilizing a convolutional neural network (CNN) enhanced through transfer learning with MobileNetV2, the system accurately classifies waste into six categories: plastic, glass, metal, paper, cardboard, and trash. The model achieved a high training accuracy of 99.8% and a validation accuracy of 80.5%, demonstrating strong learning and generalization. A robotic arm simulator is implemented to perform virtual sorting, calculating the energy cost for each action using Euclidean distance to ensure optimal and efficient movement. The framework incorporates key elements of trustworthy AI, such as transparency, robustness,
fairness, and safety, making it a reliable and scalable solution for smart waste management systems in urban settings.
Read the full abstract ▾
All authors Halima I. Kure, Jishna Retnakumari, Augustine O. Nwajana, Umar M. Ismail, Bilyaminu A. Romo, Ehigiator Egho-Promise
Robotics · Cross-listed from cs.AI
Graph Attention-Guided Search for Dense Multi-Agent Pathfinding
Rishabh Jain, Keisuke Okumura, Michael Amir et al. (+1 more)
Finding near-optimal solutions for dense multi-
agent pathfinding (MAPF) problems in real-time remains challenging even for state-of-the-art planners. To this end, we develop a hybrid framework that integrates a learned heuristic derived from MAGAT, a neural MAPF policy with a graph
attention scheme, into a leading search-based
algorithm, LaCAM. While prior work has explored learning-guided search in MAPF, such methods have historically underperformed. In contrast, our approach, termed LaGAT, outperforms both purely search-based and purely learning-based methods in dense scenarios. This is achieved through an enhanced MAGAT architecture, a pre-train-then-fine-tune strategy on maps of interest, and a deadlock detection scheme to account for imperfect neural guidance. Our results demonstrate that, when carefully designed, hybrid search offers a powerful solution for tightly coupled, challenging multi-
agent coordination problems.
Read the full abstract ▾
All authors Rishabh Jain, Keisuke Okumura, Michael Amir, Amanda Prorok
Robotics
Bridging Embodiment Gaps: Deploying Vision-Language-Action Models on Soft Robots
Haochen Su, Cristian Meo, Francesco Stella et al. (+3 more)
Robotic systems are increasingly expected to operate in human-centered, unstructured environments where safety, adaptability, and generalization are essential.
Vision-
Language-Action (VLA) models have been proposed as a
language guided generalized control framework for real
robots. However, their deployment has been limited to conventional serial link manipulators. Coupled by their
rigidity and unpredictability of learning based control, the ability to safely interact with the environment is missing yet critical. In this work, we present the deployment of a VLA model on a soft continuum manipulator to demonstrate autonomous safe human-
robot interaction. We present a structured finetuning and deployment pipeline evaluating two state-of-the-art VLA models (OpenVLA-OFT and $\pi_0$) across representative manipulation tasks, and show while out-of-the-box policies fail due to embodiment mismatch, through targeted finetuning the soft
robot performs equally to the rigid counterpart. Our findings highlight the necessity of finetuning for bridging embodiment gaps, and demonstrate that coupling VLA models with soft
robots enables safe and flexible
embodied AI in human-shared environments.
Read the full abstract ▾
All authors Haochen Su, Cristian Meo, Francesco Stella, Andrea Peirone, Kai Junge, Josie Hughes
Robotics · Cross-listed from cs.CV
M2H: Multi-Task Learning with Efficient Window-Based Cross-Task Attention for Monocular Spatial Perception
U.V.B.L Udugama, George Vosselman and Francesco Nex
Deploying real-time spatial perception on edge devices requires efficient multi-task models that leverage complementary task information while minimizing computational overhead. This paper introduces Multi-Mono-Hydra (M2H), a novel multi-task learning framework designed for semantic
segmentation and depth, edge, and surface normal estimation from a single
monocular image. Unlike conventional approaches that rely on independent single-task models or shared encoder-decoder architectures, M2H introduces a Window-Based Cross-Task
Attention Module that enables structured feature exchange while preserving task-specific details, improving prediction consistency across tasks. Built on a lightweight ViT-based DINOv2 backbone, M2H is optimized for real-time deployment and serves as the foundation for
monocular spatial perception systems supporting 3D scene graph construction in dynamic environments. Comprehensive evaluations show that M2H outperforms state-of-the-art multi-task models on NYUDv2, surpasses single-task depth and semantic baselines on Hypersim, and achieves superior performance on the Cityscapes
dataset, all while maintaining computational efficiency on laptop hardware. Beyond
benchmarks, M2H is validated on real-world data, demonstrating its practicality in spatial perception tasks.
Read the full abstract ▾
Robotics
Interactive Force-Impedance Control
Fan Shao, Satoshi Endo, Sandra Hirche et al. (+1 more)
Human collaboration with
robots requires flexible role adaptation, enabling the
robot to switch between an active leader and a passive follower. Effective role switching depends on accurately estimating human intentions, which is typically achieved through external force analysis, nominal
robot dynamics, or data-driven approaches. However, these methods are primarily effective in contact-sparse environments. When
robots under hybrid or unified force-impedance control physically interact with active humans or non-passive environments, the robotic system may lose passivity and thus compromise safety. To address this challenge, this paper proposes a unified Interactive Force-Impedance Control (IFIC) framework that adapts to interaction power flow, ensuring safe and effortless interaction in contact-rich environments. The proposed control architecture is formulated within a port-
Hamiltonian framework, incorporating both interaction and task control ports, thereby guaranteeing autonomous system passivity. Experiments in both rigid and soft contact scenarios demonstrate that IFIC ensures stable collaboration under active human interaction, reduces contact impact forces and interaction force oscillations.
Read the full abstract ▾
All authors Fan Shao, Satoshi Endo, Sandra Hirche, Fanny Ficuciello
Robotics
DDBot: Differentiable Physics-based Digging Robot for Unknown Granular Materials
Xintong Yang, Minglun Wei, Yu-Kun Lai et al. (+1 more)
Automating the manipulation of granular materials poses significant challenges due to complex contact dynamics, unpredictable material properties, and intricate system states. Existing approaches often fail to achieve efficiency and accuracy in such tasks. To fill the research gap, this article studies the small-scale and high-precision granular material digging task with unknown physical properties. A key scientific problem addressed is the feasibility of applying first-order gradient-based optimization to complex differentiable granular material simulation and overcoming associated numerical instability. A new framework, named differentiable digging
robot (DDBot), is proposed to manipulate granular materials, including sand and soil. Specifically, we equip DDBot with a differentiable physics-based simulator, tailored for granular material manipulation, powered by GPU-accelerated parallel computing and automatic differentiation. DDBot can perform efficient differentiable system identification and high-precision digging skill optimization for unknown granular materials, which is enabled by a differentiable skill-to-action mapping, a task-oriented demonstration method, gradient clipping and line search-based gradient descent. Experimental results show that DDBot can efficiently (converge within 5 to 20 minutes) identify unknown granular material dynamics and optimize digging skills, with high-precision results in zero-shot real-world deployments, highlighting its practicality.
Benchmark results against state-of-the-art baselines also confirm the robustness and efficiency of DDBot in such digging tasks.
Read the full abstract ▾
All authors Xintong Yang, Minglun Wei, Yu-Kun Lai, Ze Ji
Robotics
Video Replanning via Latent Embedding Refinement and Rejection-Based Sampling
Po-Chen Ko, Yu Chan, Yu-Hsiang Fu et al. (+7 more)
Video planning has emerged as a flexible framework for
robot manipulation, in which a generative model predicts a
video of task completion, and a downstream module translates the predicted frames into actions. However, existing methods typically ignore information from past interactions, limiting their ability to adapt to latent physical properties that can only be revealed through trial and error, such as whether a door should be pushed or pulled, or how friction affects object dynamics. When a plan fails, these methods usually replan from scratch without leveraging the information revealed by the failure. To address this limitation, we introduce RELIC, REplanning with Latent
embedding refInement and Candidate rejection, a
video planning framework that adapts to hidden physical properties from test-time failures. RELIC optimizes a latent
embedding that captures the environment's hidden physical properties from interaction
videos and introduces a rejection-based sampling mechanism that filters out hypotheses inconsistent with prior failures. Across eight tasks in two simulation suites, RELIC consistently reduces the number of replanning steps required for success, and linear probes show that its
embedding captures the hidden parameters from the interaction itself rather than from scene appearance. Across four challenging real-world robotic manipulation tasks involving hidden interaction modes, e.g., friction, center of mass, and object mass, RELIC raises the one-shot replanning success rate of a
video planning baseline from 30.0% to 63.8% after a single physical interaction.
Read the full abstract ▾
All authors Po-Chen Ko, Yu Chan, Yu-Hsiang Fu, Hsien-Jeng Yeh, Chu-Rong Chen, Dong-Chen Tsai, Wei-Chiu Ma, Yilun Du, Jiayuan Mao, Shao-Hua Sun
Robotics
Floating-Base Deep Lagrangian Networks
Lucas Schulze, Juliano Decico Negri, Victor Barasuol et al. (+4 more)
Grey-box methods for system identification combine deep learning with physics-informed constraints, capturing complex dependencies while improving out-of-distribution generalization. Despite the growing importance of floating-base systems such as
humanoids and quadrupeds, current grey-box models ignore their specific physical constraints. For instance, the inertia
matrix is not only positive definite but also exhibits branch-induced sparsity and input independence. Moreover, the 6x6 composite spatial inertia of the floating base inherits properties of single-rigid-body inertia
matrices. As we show, this includes the triangle inequality on the
eigenvalues of the composite rotational inertia. To address the lack of physical consistency in deep learning models of floating-base systems, we introduce a parameterization of inertia
matrices that satisfies all these constraints. Inspired by Deep Lagrangian Networks (DeLaN), we train neural networks to predict physically plausible inertia
matrices that minimize inverse dynamics error under Lagrangian mechanics. For evaluation, we collected and released a
dataset on multiple quadrupeds and
humanoids. In these experiments, our Floating-Base Deep Lagrangian Networks (FeLaN) achieve better overall performance on both simulated and real
robots, while providing greater physical interpretability.
Read the full abstract ▾
All authors Lucas Schulze, Juliano Decico Negri, Victor Barasuol, Vivian Suzano Medeiros, Marcelo Becker, Jan Peters, Oleg Arenz
Robotics
High-Level Multi-Robot Trajectory Planning And Spurious Behavior Detection
Fernando Salanova, Jesús Roche, Cristian Mahulea et al. (+1 more)
The reliable execution of high-level missions in multi-
robot systems with heterogeneous
agents, requires robust methods for detecting spurious behaviors. In this paper, we address the challenge of identifying spurious executions of plans specified as a Linear Temporal Logic (LTL) formula, as incorrect task sequences, violations of spatial constraints, timing inconsistencies, or deviations from intended mission semantics. To tackle this, we introduce a structured data generation framework based on the Nets-within-Nets (NWN) paradigm, which coordinates
robot actions with LTL-derived global mission specifications. We further propose a
Transformer-based anomaly detection pipeline that classifies
robot trajectories as normal or anomalous. Experimental evaluations show that our method achieves high accuracy (91.3%) in identifying execution inefficiencies, and demonstrates robust detection capabilities for core mission violations (88.3%) and constraint-based adaptive anomalies (66.8%). An ablation experiment of the
embedding and architecture was carried out, obtaining successful results where our novel proposition performs better than simpler representations.
Read the full abstract ▾
All authors Fernando Salanova, Jesús Roche, Cristian Mahulea, Eduardo Montijano
Robotics
An adaptive hierarchical control framework for quadrupedal robots in planetary exploration
Franek Stark, Rohit Kumar, Shubham Vyas et al. (+6 more)
Planetary exploration missions require
robots capable of navigating extreme and unknown environments. While wheeled rovers have dominated past missions, their mobility is limited to traversable surfaces. Legged
robots, especially quadrupeds, can overcome these limitations by handling uneven, obstacle-rich, and deformable terrains. However, deploying such
robots in unknown conditions is challenging due to the need for environment-specific control, which is infeasible when terrain and
robot parameters are uncertain. This work presents a modular control framework that combines model-based dynamic control with online model adaptation and adaptive footstep planning to address uncertainties in both
robot and terrain properties. The framework includes state estimation for quadrupeds with and without contact sensing, supports runtime reconfiguration, and is integrated into ROS 2 with open-source availability. Its performance was validated on two quadruped platforms, multiple hardware architectures, and in a volcano field test, where the
robot walked over 700 m.
Read the full abstract ▾
All authors Franek Stark, Rohit Kumar, Shubham Vyas, Hannah Isermann, Jonas Haack, Mihaela Popescu, Jakob Middelberg, Dennis Mronga, Frank Kirchner
Robotics
Pole-Image: A Self-Supervised Pole-Anchored Descriptor for Long-Term LiDAR Localization and Map Maintenance
Wuhao Xie and Kanji Tanaka
Long-term autonomy for mobile
robots requires both robust self-localization and reliable map maintenance. Conventional landmark-based methods face a fundamental trade-off between landmarks with high detectability but low distinctiveness (e.g., poles) and those with high distinctiveness but difficult stable detection (e.g., local point cloud structures). This work addresses the challenge of descriptively identifying a unique "signature" (local point cloud) by leveraging a detectable, high-precision "anchor" (like a pole). To solve this, we propose a novel canonical representation, "Pole-Image," as a hybrid method that uses poles as anchors to generate signatures from the surrounding 3D structure. Pole-Image represents a pole-like landmark and its surrounding environment, detected from a
LiDAR point cloud, as a 2D polar coordinate image with the pole itself as the origin. This representation leverages the pole's nature as a high-precision reference point, explicitly encoding the "relative geometry" between the stable pole and the variable surrounding point cloud. The key advantage of pole landmarks is that "detection" is extremely easy. This ease of detection allows the
robot to easily track the same pole, enabling the automatic and large-scale collection of diverse observational data (positive pairs). This data acquisition feasibility makes "Contrastive Learning (CL)" applicable. By applying CL, the model learns a viewpoint-invariant and highly discriminative descriptor. The contributions are twofold: 1) The descriptor overcomes perceptual aliasing, enabling robust self-localization. 2) The high-precision encoding enables high-sensitivity change detection, contributing to map maintenance.
Read the full abstract ▾
Robotics
Performance Evaluation of an Integrated System for Visible Light Communication and Positioning Using an Event Camera
Ryota Soga, Masataka Kobayashi, Tsukasa Shimizu et al. (+4 more)
Event cameras, featuring high temporal resolution and high dynamic range, offer visual sensing capabilities comparable to conventional image sensors while capturing fast-moving objects and handling scenes with extreme lighting contrasts such as tunnel exits. Leveraging these properties, this study proposes a novel self-localization system that integrates visible light communication (VLC) and visible light positioning (VLP) within a single event camera. The system enables a vehicle to estimate its position even in GPS-denied environments, such as tunnels, by using VLC to obtain coordinate information from LED transmitters and VLP to estimate the distance to each transmitter. Multiple LEDs are installed on the transmitter side, each assigned a unique pilot sequence based on Walsh-Hadamard codes. The event camera identifies individual LEDs within its field of view by correlating the received signal with these codes, allowing clear separation and recognition of each light source. This mechanism enables simultaneous high-capacity MISO (multi-input single-output) communication through VLC and precise distance estimation via phase-only correlation (POC) between multiple LED pairs. To the best of our knowledge, this is the first vehicle-mounted system to achieve simultaneous VLC and VLP functionalities using a single event camera. Field experiments were conducted by mounting the system on a vehicle traveling at 30 km/h (8.3 m/s). The results demonstrated robust real-world performance, with a root mean square error (RMSE) of distance estimation within 0.75 m for ranges up to 100 m and a bit error rate (BER) below 0.01 across the same range.
Read the full abstract ▾
All authors Ryota Soga, Masataka Kobayashi, Tsukasa Shimizu, Shintaro Shiba, Quan Kong, Shan Lu, Takaya Yamazato
Robotics
SimpleVSF: VLM-Scoring Fusion for Trajectory Prediction of End-to-End Autonomous Driving
Peiru Zheng, Yun Zhao, Zhan Gong et al. (+2 more)
End-to-end autonomous driving has emerged as a promising paradigm for achieving robust and intelligent driving policies. However, existing end-to-end methods still face significant challenges, such as suboptimal decision-making in complex scenarios. In this paper,we propose SimpleVSF (Simple VLM-Scoring Fusion), a novel framework that enhances end-to-end planning by leveraging the cognitive capabilities of
Vision-
Language Models (VLMs) and advanced trajectory fusion techniques. We utilize the conventional scorers and the novel VLM-enhanced scorers. And we leverage a robust weight fusioner for quantitative aggregation and a powerful VLM-based fusioner for qualitative, context-aware decision-making. As the leading approach in the ICCV 2025 NAVSIM v2 End-to-End Driving Challenge, our SimpleVSF framework demonstrates state-of-the-art performance, achieving a superior balance between safety, comfort, and efficiency.
Read the full abstract ▾
All authors Peiru Zheng, Yun Zhao, Zhan Gong, Hong Zhu, Shaohua Wu
Robotics
OmniVIC: A Self-Improving Variable Impedance Controller with Vision-Language In-Context Learning for Safe Robotic Manipulation
Heng Zhang, Wei-Hsing Huang, Gokhan Solak et al. (+1 more)
We present OmniVIC, a universal variable impedance controller (VIC) enhanced by a
vision language model (VLM), which improves safety and adaptation in any contact-rich robotic manipulation task to enhance safe physical interaction. Traditional VIC have shown advantages when the
robot physically interacts with the environment, but lack generalization in unseen, complex, and unstructured safe interactions in universal task scenarios involving contact or uncertainty. To this end, the proposed OmniVIC interprets task context derived
reasoning from images and natural
language and generates adaptive impedance parameters for a VIC controller. Specifically, the core of OmniVIC is a self-improving
Retrieval-Augmented Generation(RAG) and in-context learning (ICL), where RAG retrieves relevant prior experiences from a structured
memory bank to inform the controller about similar past tasks, and ICL leverages these retrieved examples and the
prompt of current task to query the VLM for generating context-aware and adaptive impedance parameters for the current manipulation scenario. Therefore, a self-improved RAG and ICL guarantee OmniVIC works in universal task scenarios. The impedance parameter regulation is further informed by real-time force/torque feedback to ensure interaction forces remain within safe thresholds. We demonstrate that our method outperforms baselines on a suite of complex contact-rich tasks, both in simulation and on real-world robotic tasks, with improved success rates and reduced force violations. OmniVIC takes a step towards bridging high-level semantic
reasoning and low-level compliant control, enabling safer and more generalizable manipulation. Overall, the average success rate increases from 27% (baseline) to 61.4% (OmniVIC).
Read the full abstract ▾
All authors Heng Zhang, Wei-Hsing Huang, Gokhan Solak, Arash Ajoudani
Robotics
DiffVLA++: Bridging Cognitive Reasoning and End-to-End Driving through Metric-Guided Alignment
Yu Gao, Anqing Jiang, Yiru Wang et al. (+7 more)
Conventional end-to-end (E2E) driving models are effective at generating physically plausible trajectories, but often fail to generalize to long-tail scenarios due to the lack of essential world knowledge to understand and reason about surrounding environments. In contrast,
Vision-
Language-Action (VLA) models leverage world knowledge to handle challenging cases, but their limited 3D
reasoning capability can lead to physically infeasible actions. In this work we introduce DiffVLA++, an enhanced autonomous driving framework that explicitly bridges cognitive
reasoning and E2E planning through metric-guided
alignment. First, we build a VLA module directly generating semantically grounded driving trajectories. Second, we design an E2E module with a dense trajectory vocabulary that ensures physical feasibility. Third, and most critically, we introduce a metric-guided trajectory scorer that guides and aligns the outputs of the VLA and E2E modules, thereby integrating their complementary strengths. The experiment on the ICCV 2025 Autonomous Grand Challenge leaderboard shows that DiffVLA++ achieves EPDMS of 49.12.
Read the full abstract ▾
All authors Yu Gao, Anqing Jiang, Yiru Wang, Wang Jijun, Hao Jiang, Zhigang Sun, Heng Yuwen, Wang Shuo, Hao Zhao, Sun Hao
Robotics
Decentralized Real-Time Planning for Multi-UAV Cooperative Manipulation via Imitation Learning
Shantnav Agarwal, Javier Alonso-Mora and Sihao Sun
Existing approaches for transporting and manipulating cable-suspended loads using multiple
UAVs along reference trajectories typically rely on either centralized control architectures or reliable inter-
agent communication. In this work, we propose a novel machine learning based method for decentralized kinodynamic planning that operates effectively under partial observability and without inter-
agent communication. Our method leverages imitation learning to train a decentralized student policy for each
UAV by imitating a centralized kinodynamic motion planner with access to privileged global observations. The student policy generates smooth trajectories using physics-informed neural networks that respect the derivative relationships in motion. During training, the student policies utilize the full trajectory generated by the teacher policy, leading to improved sample efficiency. Moreover, each student policy can be trained in under two hours on a standard laptop. We validate our method in both simulation and real-world environments to follow an agile reference trajectory, demonstrating performance comparable to that of centralized approaches.
Read the full abstract ▾
Robotics
Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey
Weifan Guan, Qinghao Hu, Aosheng Li et al. (+1 more)
Vision-
Language-Action (VLA) models extend
vision-
language models to
embodied control by mapping natural-
language instructions and visual observations to
robot actions. Despite their capabilities, VLA systems face significant challenges due to their massive computational and
memory demands, which conflict with the constraints of edge platforms such as on-board mobile manipulators that require real-time performance. Addressing this tension has become a central focus of recent research. In light of the growing efforts toward more efficient and scalable VLA systems, this survey provides a systematic review of approaches for improving VLA efficiency, with an emphasis on reducing latency,
memory footprint, and training and inference costs. We categorize existing solutions into four dimensions: model architecture, perception feature, action generation, and training/inference strategies, summarizing representative techniques within each category. Finally, we discuss future trends and open challenges, highlighting directions for advancing efficient
embodied intelligence.
Read the full abstract ▾
All authors Weifan Guan, Qinghao Hu, Aosheng Li, Jian Cheng