Robotics · Cross-listed from eess.SY
Input-to-State Safe Backstepping: Robust Safety-Critical Control with Unmatched Uncertainties
Max H. Cohen, Pio Ong and Aaron D. Ames
Guaranteeing safety in the presence of unmatched disturbances – uncertainties that cannot be directly canceled by the control input – remains a key challenge in nonlinear control. This paper presents a constructive approach to safety-critical control of nonlinear systems with unmatched disturbances. We first present a generalization of the input-to-state safety (ISSf) framework for systems with these uncertainties using the recently developed notion of an Optimal Decay CBF, which provides more flexibility for satisfying the associated Lyapunov-like conditions for safety. From there, we outline a procedure for constructing ISSf-CBFs for two relevant classes of systems with unmatched uncertainties: i) strict-feedback systems; ii) dual-relative-degree systems, which are similar to differentially flat systems. Our theoretical results are illustrated via numerical simulations of an inverted pendulum and planar quadrotor.
Read the full abstract ▾
Robotics · Cross-listed from cs.MA
When Should Agents Coordinate in Differentiable Sequential Decision Problems?
Caleb Probine, Su Ann Low, David Fridovich-Keil et al. (+1 more)
Multi-
robot teams must coordinate to operate effectively. When a team operates in an uncoordinated manner, and
agents choose actions that are only individually optimal, the team's outcome can suffer. However, in many domains, coordination requires costly communication. We explore the value of coordination in a broad class of differentiable motion-planning problems. In particular, we model coordinated behavior as a spectrum: at one extreme,
agents jointly optimize a common team objective, and at the other,
agents make unilaterally optimal decisions given their individual decision variables, i.e., they operate at Nash
equilibria. We then demonstrate that
reasoning about coordination in differentiable motion-planning problems reduces to
reasoning about the second-order properties of
agents' objectives, and we provide
algorithms that use this second-order
reasoning to determine at which times a team of
agents should coordinate.
Read the full abstract ▾
All authors Caleb Probine, Su Ann Low, David Fridovich-Keil, Ufuk Topcu
Robotics
MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstruction
Jung Min Lee, Dohyeok Lee, Seokhun Ju et al. (+5 more)
Latent actions learned from diverse human
videos serve as pseudo-labels for
vision-
language-action (VLA) pretraining, but provide effective supervision only if they remain informative about the underlying ground-truth actions. For effective supervision, latent actions should contain information about the underlying actions even though they are inaccessible. We propose Multi-ViewPoint Latent Action Moel (MVP-LAM), which learns latent actions that are highly informative about ground-truth actions from multi-view
videos. MVP-LAM trains latent actions with a cross-viewpoint reconstruction objective, so that a latent action from one view must explain the future in another view, reducing reliance on viewpoint-specific cues. On Bridge V2, MVP-LAM produces more action-centric latent actions, achieving higher mutual information with ground-truth actions and improved action prediction, including under out-of-distribution evaluation. Finally, pretraining VLAs with MVP-LAM latent actions improves downstream manipulation performance on various
benchmarks. The code and trained checkpoints are available at https://jmsnu.github.io.
Read the full abstract ▾
All authors Jung Min Lee, Dohyeok Lee, Seokhun Ju, Taehyun Cho, Jin Woo Koo, Li Zhao, Sangwoo Hong, Jungwoo Lee
Robotics
Variance-Reduced Model Predictive Path Integral via Quadratic Model Approximation
Fabian Schramm, Franki Nguimatsia Tiofack, Nicolas Perrin-Gilbert et al. (+2 more)
Sampling-based controllers, such as Model Predictive Path Integral (MPPI) methods, offer substantial flexibility but often suffer from high variance and low sample efficiency. To address these challenges, we introduce a hybrid variance-reduced MPPI framework that integrates a prior model into the sampling process. Our key insight is to decompose the objective function into a known approximate model and a residual term. Since the residual captures only the discrepancy between the model and the objective, it typically exhibits a smaller magnitude and lower variance than the original objective. Although this principle applies to general modeling choices, we demonstrate that adopting a quadratic approximation enables the derivation of a closed-form, model-guided prior that effectively concentrates samples in informative regions. Crucially, the framework is agnostic to the source of
geometric information, allowing the quadratic model to be constructed from exact derivatives, structural approximations (e.g., Gauss- or Quasi-Newton), or gradient-free randomized smoothing. We validate the approach on standard optimization
benchmarks, a nonlinear, underactuated cart-pole control task, and a contact-rich manipulation problem with non-smooth dynamics. Across these domains, we achieve faster convergence and superior performance in low-sample regimes compared to standard MPPI. These results suggest that the method can make sample-based control strategies more practical in scenarios where obtaining samples is expensive or limited.
Read the full abstract ▾
All authors Fabian Schramm, Franki Nguimatsia Tiofack, Nicolas Perrin-Gilbert, Marc Toussaint, Justin Carpentier
Robotics
Self-supervised Physics-Informed Manipulation of Deformable Linear Objects with Non-negligible Dynamics
Youyuan Long, Gokhan Solak, Sara Zeynalpour et al. (+2 more)
We address dynamic manipulation of deformable linear objects by presenting SPiD, a physics-informed self-supervised learning framework that couples an accurate deformable object model with an augmented self-supervised training strategy. On the modeling side, we extend a mass-spring model to more accurately capture object dynamics while remaining lightweight enough for high-throughput rollouts during self-supervised learning. On the learning side, we train a neural controller using a task-oriented cost, enabling end-to-end optimization through interaction with the differentiable object model. In addition, we propose a self-supervised DAgger variant that detects distribution shift during deployment and performs offline self-correction to further enhance robustness without expert supervision. We evaluate our method primarily on the rope stabilization task, where a
robot must bring a swinging rope to rest as quickly and smoothly as possible. Extensive experiments in both simulation and the real world demonstrate that the proposed controller achieves fast and smooth rope stabilization, generalizing across unseen initial states, rope lengths, masses, non-uniform mass distributions, and external disturbances. Additionally, we develop an affordable markerless rope perception method and demonstrate that our controller maintains performance with noisy and low-frequency state updates. Furthermore, we demonstrate the generality of the framework by extending it to the rope trajectory tracking task. Overall, SPiD offers a data-efficient, robust, and physically grounded framework for dynamic manipulation of deformable linear objects, featuring strong sim-to-real generalization.
Read the full abstract ▾
All authors Youyuan Long, Gokhan Solak, Sara Zeynalpour, Heng Zhang, Arash Ajoudani
Robotics
Human-in-the-Loop Failure Recovery with Adaptive Task Allocation
Lorena Maria Genua, Nikita Boguslavskii and Zhi Li
Since the recent Covid-19 pandemic, mobile manipulators and
humanoid assistive
robots with higher levels of autonomy have increasingly been adopted for patient care and living assistance. Despite advancements in autonomy, these
robots often struggle to perform reliably in dynamic and unstructured environments and require human intervention to recover from failures. Effective human-
robot collaboration is essential to enable
robots to receive assistance from the most competent operator, in order to reduce their workload and minimize disruptions in task execution. In this paper, we propose an adaptive method for allocating robotic failures to human operators (ARFA). Our proposed approach models the capabilities of human operators, and continuously updates these beliefs based on their actual performance for failure recovery. For every failure to be resolved, a reward function calculates expected outcomes based on operator capabilities and historical data, task urgency, and current workload distribution. The failure is then assigned to the operator with the highest expected reward. Our simulations and user studies show that ARFA outperforms random allocation, significantly reducing
robot idle time, improving overall system performance, and leading to a more distributed workload among operators.
Read the full abstract ▾
Robotics
Multi-Player, Multi-Strategy Quantum Game Model for Interaction-Aware Decision-Making in Automated Driving
Karim Essalmi, Fernando Garrido and Fawzi Nashashibi
Although significant progress has been made in decision-making for automated driving, challenges remain for deployment in the real world. One challenge lies in addressing interaction-awareness. Most existing approaches oversimplify interactions between the ego vehicle and surrounding
agents, and often neglect interactions among the
agents themselves. A common solution is to model these interactions using classical game theory. However, its formulation assumes rational players, whereas human behavior is frequently uncertain or irrational. To address these challenges, we propose the
Quantum Game Decision-Making (QGDM) model, a novel framework that combines classical game theory with
quantum mechanics principles (such as superposition,
entanglement, and interference) to tackle multi-player, multi-strategy decision-making problems. To the best of our knowledge, this is one of the first studies to apply
quantum game theory to decision-making for automated driving. QGDM runs in real time on a standard computer, without requiring
quantum hardware. We evaluate QGDM in simulation across various scenarios, including roundabouts, merging, and highways, and compare its performance with multiple baseline methods. Results show that QGDM significantly improves success rates and reduces collision rates compared to classical approaches, particularly in scenarios with high interaction.
Read the full abstract ▾
Robotics · Cross-listed from cs.SE
Formal Evidence Generation for Assurance Cases for Robotic Software Models
Fang Yan, Simon Foster, Ana Cavalcanti et al. (+2 more)
Robotics and Autonomous Systems are increasingly deployed in safety-critical domains, so that demonstrating their safety is essential. Assurance Cases (ACs) provide structured arguments supported by evidence, but generating and maintaining this evidence is labour-intensive, error-prone, and difficult to keep consistent as systems evolve. We present a model-based approach to systematically generating AC evidence by
embedding formal verification into the assurance workflow. The approach addresses three challenges: systematically deriving formal assertions from natural
language requirements using templates, orchestrating multiple formal verification tools to handle diverse property types, and integrating formal evidence production into the workflow. Leveraging RoboChart, a domain-specific modelling
language with formal semantics, we combine model checking and
theorem proving in our approach. Structured requirements are automatically transformed into formal assertions using predefined templates, and verification results are automatically integrated as evidence. Case studies demonstrate the effectiveness of our approach.
Read the full abstract ▾
All authors Fang Yan, Simon Foster, Ana Cavalcanti, Ibrahim Habli, James Baxter
Robotics
AffordanceGrasp-R1:Leveraging Reasoning-Based Affordance Segmentation with Reinforcement Learning for Robotic Grasping
Dingyi Zhou, Mu He, Zhuowei Fang et al. (+4 more)
We introduce AffordanceGrasp-R1, a
reasoning-driven affordance
segmentation framework for robotic grasping that combines a chain-of-thought (CoT) cold-start strategy with reinforcement learning to enhance deduction and spatial grounding. In addition, we redesign the grasping pipeline to be more context-aware by generating grasp candidates from the global scene point cloud and subsequently filtering them using instruction-conditioned affordance masks. Extensive experiments demonstrate that AffordanceGrasp-R1 consistently outperforms state-of-the-art (SOTA) methods on
benchmark datasets, and real-world robotic grasping evaluations further validate its robustness and generalization under complex
language-conditioned manipulation scenarios.
Read the full abstract ▾
All authors Dingyi Zhou, Mu He, Zhuowei Fang, Xiangtong Yao, Yinlong Liu, Alois Knoll, Hu Cao
Robotics
Robot Programming with Augmented Reality: The Role of Spatial Ability
Nicolas Leins, Muriel Fischer, Malte Teichmann et al. (+2 more)
Programming a
robot arm requires users to interpret coordinate frames, joint rotations, and trajectories that are not directly visible. Augmented reality (AR) can make these spatial relations visible, but its benefits may depend on users' spatial ability. We conducted a randomized between-subjects experiment ($N=71$) in which participants learned to program a physical UR5e
robot using either conventional teach-pendant controls with PDF instructions or a head-mounted AR interface that displayed joints, coordinate frames, and waypoints. We measured users' spatial ability with the Mental Rotation Test and assessed subjective cognitive load and system usability. Overall, AR did not significantly improve cognitive load or usability compared with conventional instruction. However, exploratory analyses revealed a compensatory effect: spatial ability predicted higher usability and lower extraneous cognitive load in the control group, but not in the AR condition, suggesting AR mitigated the disadvantage typically faced by users with lower spatial ability. These findings point to a compensatory function of AR that should be explored to guide the design of personalized AR interfaces for human-
robot interaction.
Read the full abstract ▾
All authors Nicolas Leins, Muriel Fischer, Malte Teichmann, Jana Gonnermann-Müller, Sebastian Pokutta
Robotics
CMR: Contractive Mapping Embeddings for Robust Humanoid Locomotion on Unstructured Terrains
Qixin Zeng, Hongyin Zhang, Shangke Lyu et al. (+3 more)
Robust disturbance rejection remains a longstanding challenge in
humanoid locomotion, particularly on unstructured terrains where sensing is unreliable and model mismatch is pronounced. While perception information, such as height map, enhances terrain awareness, sensor noise and sim-to-real gaps can destabilize policies in practice. In this work, we provide theoretical analysis that bounds the return gap under observation noise, when the induced latent dynamics are contractive. Furthermore, we present Contractive Mapping for Robustness (CMR) framework that maps high-dimensional, disturbance-prone observations into a latent space, where local perturbations are attenuated over time. Specifically, this approach couples contrastive representation learning with Lipschitz regularization to preserve task-relevant geometry while explicitly controlling sensitivity. Notably, the formulation can be incorporated into modern deep reinforcement learning pipelines as an auxiliary loss term with minimal additional technical effort required. Further, our extensive
humanoid experiments show that CMR potently outperforms other locomotion
algorithms under increased noise.
Read the full abstract ▾
All authors Qixin Zeng, Hongyin Zhang, Shangke Lyu, Junxi Jin, Donglin Wang, Chao Huang
Robotics
HetroD: A High-Fidelity Drone Dataset and Benchmark for Autonomous Driving in Heterogeneous Traffic
Yu-Hsiang Chen, Wei-Jer Chang, Christian Kotulla et al. (+7 more)
We present HetroD, a
dataset and
benchmark for developing autonomous driving systems in heterogeneous environments. HetroD targets the critical challenge of navi- gating real-world heterogeneous traffic dominated by vulner- able road users (VRUs), including pedestrians, cyclists, and motorcyclists that interact with vehicles. These mixed
agent types exhibit complex behaviors such as hook turns, lane splitting, and informal right-of-way negotiation. Such behaviors pose significant challenges for autonomous vehicles but remain underrepresented in existing
datasets focused on structured, lane-disciplined traffic. To bridge the gap, we collect a large- scale drone-based
dataset to provide a holistic observation of traffic scenes with centimeter-accurate annotations, HD maps, and traffic signal states. We further develop a modular toolkit for extracting per-
agent scenarios to support downstream task development. In total, the
dataset comprises over 65.4k high- fidelity
agent trajectories, 70% of which are from VRUs. HetroD supports modeling of VRU behaviors in dense, het- erogeneous traffic and provides standardized
benchmarks for
forecasting, planning, and simulation tasks. Evaluation results reveal that state-of-the-art prediction and planning models struggle with the challenges presented by our
dataset: they fail to predict lateral VRU movements, cannot handle unstructured maneuvers, and exhibit limited performance in dense and multi-
agent scenarios, highlighting the need for more robust approaches to heterogeneous traffic. See our project page for more examples: https://hetroddata.github.io/HetroD/
Read the full abstract ▾
All authors Yu-Hsiang Chen, Wei-Jer Chang, Christian Kotulla, Thomas Keutgens, Steffen Runde, Tobias Moers, Christoph Klas, Wei Zhan, Masayoshi Tomizuka, Yi-Ting Chen
Robotics · Cross-listed from cs.AI
CRL-VLA: Continual Vision-Language-Action Learning
Qixin Zeng, Shuo Zhang, Hongyin Zhang et al. (+6 more)
Lifelong learning is critical for
embodied agents in open-world environments, where reinforcement learning fine-tuning has emerged as an important paradigm to enable
Vision-
Language-Action (VLA) models to master dexterous manipulation through environmental interaction. Thus, Continual Reinforcement Learning (CRL) is a promising pathway for deploying VLA models in lifelong robotic scenarios, yet balancing stability (retaining old skills) and plasticity (learning new ones) remains a formidable challenge for existing methods. We introduce CRL-VLA, a framework for continual post-training of VLA models with rigorous theoretical bounds. We derive a unified performance bound linking the stability-plasticity trade-off to goal-conditioned advantage magnitude, scaled by policy divergence. CRL-VLA resolves this dilemma via asymmetric regulation: constraining advantage magnitudes on prior tasks while enabling controlled growth on new tasks. This is realized through a simple but effective dual-critic architecture with novel Goal-Conditioned Value Formulation (GCVF), where a frozen critic anchors semantic consistency and a trainable
estimator drives adaptation. Experiments on the LIBERO
benchmark demonstrate that CRL-VLA effectively harmonizes these conflicting objectives, outperforming baselines in both anti-forgetting and forward adaptation.
Read the full abstract ▾
All authors Qixin Zeng, Shuo Zhang, Hongyin Zhang, Renjie Wang, Han Zhao, Libang Zhao, Runze Li, Donglin Wang, Chao Huang
Robotics
Soft Swing-up: Benchmarking Model-Based Optimal Control for Rigid-Soft Underactuated Systems
Daniele Caradonna, Nikhil Nair, Anup Teejo Mathew et al. (+5 more)
Continuum soft
robots are inherently underactuated and subject to intrinsic input constraints, making dynamic control particularly challenging, especially in hybrid rigid-soft
robots. While most existing methods focus on quasi-static behaviors, dynamic tasks such as swing-up require accurate exploitation of continuum dynamics. This has led to studies on simple low-order template systems that often fail to capture the complexity of real continuum deformations. Model-based optimal control offers a systematic solution; however, its application to rigid-soft
robots is often limited by the computational cost and inaccuracy of numerical differentiation for high-dimensional models. Building on recent advances in the
Geometric Variable Strain model that enable analytical derivatives, this work investigates three optimal control strategies for underactuated soft systems – Direct Collocation, Differential Dynamic Programming, and Nonlinear Model Predictive Control – to perform dynamic swing-up tasks. To address stiff continuum dynamics and constrained actuation, implicit integration schemes and warm-start strategies are employed to improve numerical robustness and computational efficiency. The methods are evaluated in simulation on three Rigid-Soft and high-order soft
benchmark systems – the Soft Cart-Pole, the Soft Pendubot, and the Soft Furuta Pendulum – highlighting their performance and computational trade-offs.
Read the full abstract ▾
All authors Daniele Caradonna, Nikhil Nair, Anup Teejo Mathew, Daniel Feliu Talegón, Imran Afgan, Egidio Falotico, Cosimo Della Santina, Federico Renda
Robotics
ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response
Xiaomeng Zhu, Fengming Zhu, Weijie Zhou et al. (+8 more)
While passive
agents merely follow instructions, proactive
agents align with higher-level objectives, such as assistance and safety by continuously monitoring the environment to determine when and how to act. However, developing proactive
agents is hindered by the lack of specialized resources. To address this, we introduce ProAct-75, a
benchmark designed to train and evaluate proactive
agents across diverse domains, including assistance, maintenance, and safety monitoring. Spanning 75 tasks, our
dataset features 91,581 step-level annotations enriched with explicit task graphs. These graphs encode step dependencies and parallel execution possibilities, providing the structural grounding necessary for complex decision-making. Building on this
benchmark, we propose ProAct-Helper, a reference baseline powered by a
Multimodal Large
Language Model (MLLM) that grounds decision-making in state detection, and leveraging task graphs to enable
entropy-driven heuristic search for action selection, allowing
agents to execute parallel threads independently rather than mirroring the human's next step. Extensive experiments demonstrate that ProAct-Helper outperforms strong closed-source models, improving trigger detection mF1 by 6.21%, saving 0.25 more steps in online one-step decision, and increasing the rate of parallel actions by 15.58%.
Read the full abstract ▾
All authors Xiaomeng Zhu, Fengming Zhu, Weijie Zhou, Ye Tian, Zhenlin Hu, Yufei Huang, Yuchun Guo, Xinyu Wu, Zhengyou Zhang, Fangzhen Lin, Xuantang Xiong
Robotics
Learning-based Initialization of Trajectory Optimization for Path-following Problems of Redundant Manipulators
Minsung Yoon, Mincheul Kang, Daehyung Park et al. (+1 more)
Trajectory optimization (TO) is an efficient tool to generate a redundant manipulator's joint trajectory following a 6-dimensional Cartesian path. The optimization performance largely depends on the quality of initial trajectories. However, the selection of a high-quality initial trajectory is non-trivial and requires a considerable time budget due to the extremely large space of the solution trajectories and the lack of prior knowledge about task constraints in configuration space. To alleviate the issue, we present a learning-based initial trajectory generation method that generates high-quality initial trajectories in a short time budget by adopting example-guided reinforcement learning. In addition, we suggest a null-space projected imitation reward to consider null-space constraints by efficiently learning kinematically feasible motion captured in expert demonstrations. Our statistical evaluation in simulation shows the improved optimality, efficiency, and applicability of TO when we plug in our method's output, compared with three other baselines. We also show the performance improvement and feasibility via real-world experiments with a seven-degree-of-freedom manipulator.
Read the full abstract ▾
All authors Minsung Yoon, Mincheul Kang, Daehyung Park, Sung-Eui Yoon
Robotics
Deep-Learning-Based Control of a Decoupled Two-Segment Continuum Robot for Endoscopic Submucosal Dissection
Yuancheng Shao, Yao Zhang, Jia Gu et al. (+7 more)
Manual endoscopic submucosal dissection (ESD) is technically demanding, and existing single-segment robotic tools offer limited dexterity. These limitations motivate the development of more advanced solutions. To address this, DESectBot, a novel dual segment continuum
robot with a decoupled structure and integrated
surgical forceps, enabling 6 degrees of freedom (DoFs) tip dexterity for improved lesion targeting in ESD, was developed in this work. Deep learning controllers based on gated recurrent units (GRUs) for simultaneous tip position and orientation control, effectively handling the nonlinear coupling between continuum segments, were proposed. The GRU controller was benchmarked against Jacobian based inverse kinematics, model predictive control (MPC), a feedforward neural network (FNN), and a long short-term
memory (LSTM) network. In nested-rectangle and Lissajous trajectory tracking tasks, the GRU achieved the lowest position/orientation RMSEs: 1.11 mm/ 4.62{\deg} and 0.81 mm/ 2.59{\deg}, respectively. For orientation control at a fixed position (four target poses), the GRU attained a mean RMSE of 0.14 mm and 0.72{\deg}, outperforming all alternatives. In a peg transfer task, the GRU achieved a 100% success rate (120 success/120 attempts) with an average transfer time of 11.8s, the STD significantly outperforms novice-controlled systems. Additionally, an ex vivo ESD demonstration grasping, elevating, and resecting tissue as the scalpel completed the cut confirmed that DESectBot provides sufficient stiffness to divide thick gastric mucosa and an operative workspace adequate for large lesions.These results confirm that GRU-based control significantly enhances precision, reliability, and usability in ESD
surgical training scenarios.
Read the full abstract ▾
All authors Yuancheng Shao, Yao Zhang, Jia Gu, Zixi Chen, Di Wu, Yuqiao Chen, Bo Lu, Wenjie Liu, Cesare Stefanini, Peng Qi
Robotics
Enhancing Navigation Efficiency of Quadruped Robots via Leveraging Personal Transportation Platforms
Minsung Yoon and Sung-Eui Yoon
Quadruped
robots face limitations in long-range
navigation efficiency due to their reliance on legs. To ameliorate the limitations, we introduce a Reinforcement Learning-based Active Transporter Riding method (RL-ATR), inspired by humans' utilization of personal transporters, including Segways. The RL-ATR features a transporter riding policy and two state
estimators. The policy devises adequate maneuvering strategies according to transporter-specific control dynamics, while the
estimators resolve sensor ambiguities in non-inertial frames by inferring unobservable
robot and transporter states. Comprehensive evaluations in simulation validate proficient command tracking abilities across various transporter-
robot models and reduced energy consumption compared to legged locomotion. Moreover, we conduct ablation studies to quantify individual component contributions within the RL-ATR. This riding ability could broaden the locomotion modalities of quadruped
robots, potentially expanding the operational range and efficiency.
Read the full abstract ▾
Robotics
PlanTRansformer: Unified Prediction and Planning with Goal-conditioned Transformer
Constantin Selzer and Fabina B. Flohr
Trajectory prediction and planning are fundamental yet disconnected components in autonomous driving. Prediction models forecast surrounding
agent motion under unknown intentions, producing
multimodal distributions, while planning assumes known ego objectives and generates deterministic trajectories. This mismatch creates a critical bottleneck: prediction lacks supervision for
agent intentions, while planning requires this information. Existing prediction models, despite strong benchmarking performance, often remain disconnected from planning constraints such as collision avoidance and dynamic feasibility. We introduce Plan
TRansformer (PTR), a unified Gaussian Mixture
Transformer framework integrating goal-conditioned prediction, dynamic feasibility, interaction awareness, and lane-level
topology reasoning. A teacher-student training strategy progressively masks surrounding
agent commands during training to align with inference conditions where
agent intentions are unavailable. PTR achieves 4.3%/3.5% improvement in marginal/joint mAP compared to the baseline Motion
Transformer (MTR) and 15.5% planning error reduction at 5s horizon compared to GameFormer. The architecture-agnostic design enables application to diverse
Transformer-based prediction models. Project Website: https://github.com/SelzerConst/PlanTRansformer
Read the full abstract ▾
Robotics
Learning-based Adaptive Control of Quadruped Robots for Active Stabilization on Moving Platforms
Minsung Yoon, Heechan Shin, Jeil Jeong et al. (+1 more)
A quadruped
robot faces balancing challenges on a six-degrees-of-freedom moving platform, like subways, buses, airplanes, and yachts, due to independent platform motions and resultant diverse inertia forces on the
robot. To alleviate these challenges, we present the Learning-based Active Stabilization on Moving Platforms (LAS-MP), featuring a self-balancing policy and system state
estimators. The policy adaptively adjusts the
robot's posture in response to the platform's motion. The
estimators infer
robot and platform states based on proprioceptive sensor data. For a systematic training scheme across various platform motions, we introduce platform trajectory generation and scheduling methods. Our evaluation demonstrates superior balancing performance across multiple metrics compared to three baselines. Furthermore, we conduct a detailed analysis of the LAS-MP, including ablation studies and evaluation of the
estimators, to validate the effectiveness of each component.
Read the full abstract ▾
All authors Minsung Yoon, Heechan Shin, Jeil Jeong, Sung-Eui Yoon
Robotics
Manipulation via Force Distribution at Contact
Haegu Lee, Yitaek Kim, Casper Hewson Rask et al. (+1 more)
Efficient and robust trajectories play a crucial role in contact-rich manipulation, which demands accurate mod- eling of object-
robot interactions. Many existing approaches rely on point contact models due to their computational effi- ciency. Simple contact models are computationally efficient but inherently limited for achieving human-like, contact-rich ma- nipulation, as they fail to capture key frictional dynamics and torque generation observed in human manipulation. This study introduces a Force-Distributed Line Contact (FDLC) model in contact-rich manipulation and compares it against conventional point contact models. A bi-level optimization framework is constructed, in which the lower-level solves an optimization problem for contact force computation, and the upper-level optimization applies iLQR for trajectory optimization. Through this framework, the limitations of point contact are demon- strated, and the benefits of the FDLC in generating efficient and robust trajectories are established. The effectiveness of the proposed approach is validated by a box rotating task, demonstrating that FDLC enables trajectories generated via non-uniform force distributions along the contact line, while requiring lower control effort and less motion of the
robot.
Read the full abstract ▾
All authors Haegu Lee, Yitaek Kim, Casper Hewson Rask, Christoffer Sloth
Robotics
RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalization
Songming Liu, Bangguo Li, Kai Ma et al. (+5 more)
Vision-
Language-Action (VLA) models hold promise for generalist robotics but currently struggle with data scarcity, architectural inefficiencies, and the inability to generalize across different hardware platforms. We introduce RDT2, a robotic foundation model built upon a 7B parameter VLM designed to enable zero-shot deployment on novel embodiments for open-vocabulary tasks. To achieve this, we collected one of the largest open-source robotic
datasets–over 10,000 hours of demonstrations in diverse families–using an enhanced, embodiment-agnostic Universal Manipulation Interface (UMI). Our approach employs a novel three-stage training recipe that aligns discrete linguistic knowledge with continuous control via Residual Vector Quantization (RVQ), flow-matching, and
distillation for real-time inference. Consequently, RDT2 becomes one of the first models that simultaneously zero-shot generalizes to unseen objects, scenes, instructions, and even robotic platforms. Besides, it outperforms state-of-the-art baselines in dexterous, long-horizon, and dynamic downstream tasks like playing table tennis. See https://rdt-robotics.github.io/rdt2/ for more information.
Read the full abstract ▾
All authors Songming Liu, Bangguo Li, Kai Ma, Lingxuan Wu, Hengkai Tan, Xiao Ouyang, Hang Su, Jun Zhu
Robotics · Cross-listed from cs.CV
LEVIO: Lightweight Embedded Visual Inertial Odometry for Resource-Constrained Devices
Jonas Kühne, Christian Vogt, Michele Magno et al. (+1 more)
Accurate, infrastructure-less sensor systems for motion tracking are essential for mobile robotics and augmented reality (AR) applications. The most popular state-of-the-art visual-inertial odometry (VIO) systems, however, are too computationally demanding for resource-constrained hardware, such as micro-drones and smart glasses. This work presents LEVIO, a fully featured VIO pipeline optimized for ultra-low-power compute platforms, allowing six-degrees-of-freedom (DoF) real-time sensing. LEVIO incorporates established VIO components such as Oriented FAST and Rotated BRIEF (ORB) feature tracking and bundle adjustment, while emphasizing a computationally efficient architecture with parallelization and low
memory usage to suit embedded microcontrollers and low-power systems-on-chip (SoCs). The paper proposes and details the algorithmic design choices and the hardware-software co-optimization approach, and presents real-time performance on resource-constrained hardware. LEVIO is validated on a parallel-processing ultra-low-power RISC-V SoC, achieving 20 FPS while consuming less than 100 mW, and benchmarked against public VIO
datasets, offering a compelling balance between efficiency and accuracy. To facilitate reproducibility and adoption, the complete implementation is released as open-source.
Read the full abstract ▾
All authors Jonas Kühne, Christian Vogt, Michele Magno, Luca Benini
Robotics
Collision Detection with Analytical Derivatives of Contact Kinematics
Anup Teejo Mathew, Anees Peringal, Daniele Caradonna et al. (+2 more)
Differentiable contact kinematics are essential for gradient-based methods in robotics, yet the mapping from
robot state to contact distance, location, and normal becomes non-smooth in degenerate configurations of shapes with zero or undefined
curvature. We address this inherent limitation by selectively regularizing such geometries into strictly convex implicit representations, restoring uniqueness and smoothness of the contact map. Leveraging this
geometric regularization, we develop iDCOL, an implicit differentiable collision detection and contact kinematics framework. iDCOL represents colliding bodies using strictly convex implicit surfaces and computes collision detection and contact kinematics by solving a fixed-size nonlinear system derived from a
geometric scaling-based convex optimization formulation. By applying the Implicit Function
Theorem to the resulting system residual, we derive analytical derivatives of the contact kinematic quantities. We develop a fast Newton-based solver for iDCOL and provide an open-source C++ implementation of the framework. The robustness of the approach is evaluated through extensive collision simulations and benchmarking, and applicability is demonstrated in gradient-based kinematic path planning and differentiable contact physics, including multi-body rigid collisions and a soft-
robot interaction example.
Read the full abstract ▾
All authors Anup Teejo Mathew, Anees Peringal, Daniele Caradonna, Frederic Boyer, Federico Renda
Robotics
A thin and soft optical tactile sensor for highly sensitive object perception
Yanchen Shen, Kohei Tsuji, Haruto Koizumi et al. (+7 more)
Tactile sensing is crucial in robotics and wearable devices for safe perception and interaction with the environment. Optical tactile sensors have emerged as promising solutions, as they are immune to electromagnetic interference and have high spatial resolution. However, existing optical approaches, particularly
vision-based tactile sensors, rely on complex optical assemblies that involve lenses and cameras, resulting in bulky, rigid, and
alignment-sensitive designs. In this study, we present a thin, compact, and soft optical tactile sensor featuring an
alignment-free configuration. The soft optical sensor operates by capturing deformation-induced changes in speckle patterns generated within a soft silicone material, thereby enabling precise force measurements and texture recognition via machine learning. The experimental results show a root-mean-square error of 40 mN in the force measurement and a classification accuracy of 93.33% over nine classes of textured surfaces, including Mahjong tiles. The proposed speckle-based approach provides a compact, easily fabricated, and mechanically compliant platform that bridges optical sensing with flexible shape-adaptive architectures, thereby demonstrating its potential as a novel tactile-sensing paradigm for soft robotics and wearable haptic interfaces.
Read the full abstract ▾
All authors Yanchen Shen, Kohei Tsuji, Haruto Koizumi, Jiseon Hong, Tomoaki Niiyama, Hiroyuki Kuwabara, Hayato Ishida, Jun Hiramitsu, Mitsuhito Mase, Satoshi Sunada
Robotics
Omnidirectional Solid-State mmWave Radar Perception for UAV Power Line Collision Avoidance
Nicolaj Haarhøj Malle and Emad Ebeid
Detecting and estimating distances to power lines is a challenge for both human
UAV pilots and autonomous systems, which increases the risk of unintended collisions. We present a mmWave radar-based perception system that provides spherical sensing coverage around a small
UAV for robust power line detection and avoidance. The system integrates multiple compact solid-state mmWave radar modules to synthesize an omnidirectional field of view while remaining lightweight. We characterize the sensing behavior of this omnidirectional radar arrangement in power line environments and develop a robust detection-and-avoidance
algorithm tailored to that behavior. Field experiments on real power lines demonstrate reliable detection at ranges up to 10 m, successful avoidance maneuvers at flight speeds upwards of 10 m/s, and detection of wires as thin as 1.2 mm in diameter. These results indicate the approach's suitability as an additional safety layer for both autonomous and manual
UAV flight.
Read the full abstract ▾
Robotics
Depth Completion in Unseen Field Robotics Environments Using Extremely Sparse Depth Measurements
Marco Job, Thomas Stastny, Eleni Kelasidi et al. (+2 more)
Autonomous field
robots operating in unstructured environments require robust perception to ensure safe and reliable operations. Recent advances in
monocular depth estimation have demonstrated the potential of low-cost cameras as depth sensors; however, their adoption in field robotics remains limited due to the absence of reliable scale cues, ambiguous or low-texture conditions, and the scarcity of large-scale
datasets. To address these challenges, we propose a depth completion model that trains on synthetic data and uses extremely sparse measurements from depth sensors to predict dense metric depth in unseen field robotics environments. A synthetic
dataset generation pipeline tailored to field robotics enables the creation of multiple realistic
datasets for training purposes. This
dataset generation approach utilizes textured 3D meshes from Structure from Motion and photorealistic rendering with novel viewpoint synthesis to simulate diverse field robotics scenarios. Our approach achieves an end-to-end latency of 53 ms per frame on a Nvidia Jetson AGX Orin, enabling real-time deployment on embedded platforms. Extensive evaluation demonstrates competitive performance across diverse real-world field robotics scenarios.
Read the full abstract ▾
All authors Marco Job, Thomas Stastny, Eleni Kelasidi, Roland Siegwart, Michael Pantic
Robotics
HUSKY: Humanoid Skateboarding System via Physics-Aware Whole-Body Control
Jinrui Han, Dewei Wang, Chenyun Zhang et al. (+4 more)
While current
humanoid whole-body control frameworks predominantly rely on the static environment assumptions, addressing tasks characterized by high dynamism and complex interactions presents a formidable challenge. In this paper, we address
humanoid skateboarding, a highly challenging task requiring stable dynamic maneuvering on an underactuated wheeled platform. This integrated system is governed by non-holonomic constraints and tightly coupled human-object interactions. Successfully executing this task requires simultaneous mastery of hybrid contact dynamics and robust balance control on a mechanically coupled, dynamically unstable skateboard. To overcome the aforementioned challenges, we propose HUSKY, a learning-based framework that integrates
humanoid-skateboard system modeling and physics-aware whole-body control. We first model the coupling relationship between board tilt and truck steering angles, enabling a principled analysis of system dynamics. Building upon this, HUSKY leverages
Adversarial Motion Priors (AMP) to learn human-like pushing motions and employs a physics-guided, heading-oriented strategy for lean-to-steer behaviors. Moreover, a trajectory-guided mechanism ensures smooth and stable transitions between pushing and steering. Experimental results on the Unitree G1
humanoid platform demonstrate that our framework enables stable and agile maneuvering on skateboards in real-world scenarios. The project page is available on https://husky-
humanoid.github.io/.
Read the full abstract ▾
All authors Jinrui Han, Dewei Wang, Chenyun Zhang, Xinzhe Liu, Ping Luo, Chenjia Bai, Xuelong Li
Robotics
Hierarchical Proportion Models for Motion Generation via Integration of Motion Primitives
Yu-Han Shu, Toshiaki Tsuji and Sho Sakaino
Imitation learning (IL) enables
robots to acquire human-like motion skills from demonstrations, but it still requires extensive high-quality data and retraining to handle complex or long-horizon tasks. To improve data efficiency and adaptability, this study proposes a hierarchical IL framework that integrates motion primitives with proportion-based motion synthesis. The proposed method employs a two-layer architecture, where the upper layer performs long-term planning, while a set of lower-layer models learn individual motion primitives, which are combined according to specific proportions. Three model variants are introduced to explore different trade-offs between learning flexibility, computational cost, and adaptability: a learning-based proportion model, a sampling-based proportion model, and a playback-based proportion model, which differ in how the proportions are determined and whether the upper layer is trainable. Through real-
robot pick-and-place experiments, the proposed models successfully generated complex motions not included in the primitive set. The sampling-based and playback-based proportion models achieved more stable and adaptable motion generation than the standard hierarchical model, demonstrating the effectiveness of proportion-based motion integration for practical
robot learning.
Read the full abstract ▾
Robotics
Estimation of Ground Reaction Forces from Kinematic Data during Locomotion
Gautami Golani, Dong Anh Khoa To, Ananda Sidarta et al. (+5 more)
Ground reaction forces (GRFs) provide fundamental insight into human gait mechanics and are widely used to assess joint loading, limb
symmetry, balance control, and motor function. Despite their clinical relevance, the use of GRF remains underutilised in clinical workflows due to the practical limitations of force plate systems. In this work, we present a force-plate-free approach for estimating GRFs using only marker-based motion capture data. This kinematics only method to estimate and decompose GRF makes it well suited for widespread clinical depolyment. By using kinematics from sixteen body segments, we estimate the centre of mass (CoM) and compute GRFs, which are subsequently decomposed into individual components through a minimization-based approach. Through this framework, we can identify gait stance phases and provide access to clinically meaningful kinetic measures without a dedicated force plate system. Experimental results demonstrate the viability of CoM and GRF estimation based solely on kinematic data, supporting force-plate-free gait analysis.
Read the full abstract ▾
All authors Gautami Golani, Dong Anh Khoa To, Ananda Sidarta, Arun-Kumar Kaliya-Perumal, Oliver Roberts, Lek Syn Lim, Jim Patton, Domenico Campolo