Robotics · Cross-listed from cs.CV
OmniSAT: Compact Action Token, Faster Auto Regression
Huaihai Lyu, Chaofan Chen, Senwei Xie et al. (+4 more)
Existing
Vision-
Language-Action (VLA) models can be broadly categorized into
diffusion-based and auto-regressive (AR) approaches:
diffusion models capture continuous action distributions but rely on computationally heavy iterative denoising. In contrast, AR models enable efficient optimization and flexible sequence construction, making them better suited for large-scale pretraining. To further improve AR efficiency, particularly when action chunks induce extended and high-dimensional sequences, prior work applies
entropy-guided and
token-frequency techniques to shorten the sequence length. However, such compression struggled with poor reconstruction or inefficient compression. Motivated by this, we introduce an Omni Swift Action Tokenizer, which learns a compact, transferable action representation. Specifically, we first normalize value ranges and temporal horizons to obtain a consistent representation with B-Spline encoding. Then, we apply multi-stage residual quantization to the position, rotation, and gripper subspaces, producing compressed discrete
tokens with coarse-to-fine granularity for each part. After pre-training on the large-scale
dataset Droid, the resulting discrete tokenization shortens the training sequence by 6.8$\times$, and lowers the target
entropy. To further explore the potential of OmniSAT, we develop a cross-embodiment learning strategy that builds on the unified action-pattern space and jointly leverages
robot and human demonstrations. It enables scalable auxiliary supervision from heterogeneous
egocentric videos. Across diverse real-
robot and simulation experiments, OmniSAT encompasses higher compression while preserving reconstruction quality, enabling faster AR training convergence and model performance.
Read the full abstract ▾
All authors Huaihai Lyu, Chaofan Chen, Senwei Xie, Pengwei Wang, Xiansheng Chen, Shanghang Zhang, Changsheng Xu
Robotics
GATO: GPU-Accelerated and Batched Trajectory Optimization for Scalable Edge Model Predictive Control
Alexander Du, Emre Adabag, Gabriel Bravo-Palacios et al. (+1 more)
While Model Predictive Control (MPC) delivers strong performance across robotics applications, solving the underlying (batches of) nonlinear trajectory optimization (TO) problems online remains computationally demanding. Existing GPU-accelerated approaches either parallelize single solves, handle large batches at sub-real-time rates, or sacrifice model generality for speed. This leaves a large gap in solver performance for many state-of-the-art MPC applications that require real-time batches of tens to low-hundreds of solves. As such, we present GATO, an open source, GPU-accelerated, batched TO solver co-designed across
algorithm, software, and computational hardware to deliver real-time throughput for these moderate batch size regimes. Our approach leverages a combination of block-, warp-, and thread-level parallelism within and across solves for ultra-high performance. We demonstrate the effectiveness of our approach through a combination of: simulated
benchmarks showing speedups of 18-21x over CPU baselines and 1.4-16x over GPU baselines as batch size increases; case studies highlighting improved disturbance rejection and convergence behavior; and finally a validation on hardware using an industrial manipulator. We open source GATO to support reproducibility and adoption.
Read the full abstract ▾
All authors Alexander Du, Emre Adabag, Gabriel Bravo-Palacios, Brian Plancher
Robotics
Inspection Planning Primitives with Implicit Models
Jingyang You, Hanna Kurniawati and Lashika Medagoda
The aging and increasing complexity of infrastructures make efficient inspection planning more critical in ensuring safety. Thanks to sampling-based motion planning, many inspection planners are fast. However, they often require huge
memory. This is particularly true when the structure under inspection is large and complex, consisting of many struts and pillars of various geometry and sizes. Such structures can be represented efficiently using implicit models, such as neural Signed Distance Functions (SDFs). However, most primitive computations used in sampling-based inspection planner have been designed to work efficiently with explicit environment models, which in turn requires the planner to use explicit environment models or performs frequent transformations between implicit and explicit environment models during planning. This paper proposes a set of primitive computations, called Inspection Planning Primitives with Implicit Models (IPIM), that enable sampling-based inspection planners to entirely use neural SDFs representation during planning. Evaluation on three scenarios, including inspection of a complex real-world structure with over 92M triangular mesh faces, indicates that even a rudimentary sampling-based planner with IPIM can generate inspection trajectories of similar quality to those generated by the state-of-the-art planner, while using up to 70x less
memory than the state-of-the-art inspection planner.
Read the full abstract ▾
Robotics · Cross-listed from cs.HC
IGUANA: Immersive Guidance, Navigation, and Control for Consumer UAV
Victor Victor, Tania Krisanty, Matthew McGinity et al. (+2 more)
As the
markets for unmanned aerial vehicles (
UAVs) and mixed reality (MR) headsets continue to grow, recent research has increasingly explored their integration, which enables more intuitive, immersive, and situationally aware control systems. We present IGUANA, an MR-based immersive guidance,
navigation, and control system for consumer
UAVs. IGUANA introduces three key elements beyond conventional control interfaces: (1) a 3D terrain map interface with draggable waypoint markers and live camera preview for high-level control, (2) a novel spatial control metaphor that uses a virtual ball as a physical analogy for low-level control, and (3) a spatial overlay that helps track the
UAV when it is not visible with the naked eye or visual line of sight is interrupted. We conducted a user study to evaluate our design, both quantitatively and qualitatively, and found that (1) the 3D map interface is intuitive and easy to use, relieving users from manual control and suggesting improved accuracy and consistency with lower perceived workload relative to conventional dual-stick controller, (2) the virtual ball interface is intuitive but limited by the lack of physical feedback, and (3) the spatial overlay is very useful in enhancing the users' situational awareness.
Read the full abstract ▾
All authors Victor Victor, Tania Krisanty, Matthew McGinity, Stefan Gumhold, Uwe Aßmann
Robotics
AVO: Amortized Value Optimization for Contact Mode Switching in Multi-Finger Manipulation
Adam Hung, Fan Yang, Abhinav Kumar et al. (+4 more)
Dexterous manipulation tasks often require switching between different contact modes, such as rolling, sliding, sticking, or non-contact contact modes. When formulating dexterous manipulation tasks as a trajectory optimization problem, a common approach is to decompose these tasks into sub-tasks for each contact mode, which are each solved independently. Optimizing each sub-task independently can limit performance, as optimizing contact points, contact forces, or other variables without information about future sub-tasks can place the system in a state from which it is challenging to make progress on subsequent sub-tasks. Further, optimizing these sub-tasks is very computationally expensive. To address these challenges, we propose Amortized Value Optimization (AVO), which introduces a learned value function that predicts the total future task performance. By incorporating this value function into the cost of the trajectory optimization at each planning step, the value function gradients guide the optimizer toward states that minimize the cost in future sub-tasks. This effectively bridges separately optimized sub-tasks, and accelerates the optimization by reducing the amount of online computation needed. We validate AVO on a screwdriver grasping and turning task in both simulation and real world experiments, and show improved performance even with 50% less computational budget compared to trajectory optimization without the value function.
Read the full abstract ▾
All authors Adam Hung, Fan Yang, Abhinav Kumar, Sergio Aguilera Marinovic, Soshi Iba, Rana Soltani Zarrin, Dmitry Berenson
Robotics
HJCD-IK: GPU-Accelerated Inverse Kinematics through Batched Hybrid Jacobian Coordinate Descent
Cael Yasutake, Andrew H. Liu, Zachary Kingston et al. (+1 more)
Inverse Kinematics (IK) is a core problem in robotics, in which joint configurations are found to achieve a (collision-free) desired end-effector pose. Modern IK solvers face a fundamental trade-off: analytical methods are fast but lack generality, while numerical optimization-based methods are broadly applicable but prone to local minima and high computational costs. To overcome this challenge, we introduce HJCD-IK, a GPU-accelerated, sampling-based hybrid solver. By pairing a novel orientation-aware greedy coordinate descent initialization with Jacobian-based polishing and a parallel collision filter, our method achieves up to order-of-magnitude gains in speed and accuracy over state-of-the-art solvers, consistently finding collision-free solutions on the accuracy-latency Pareto frontier, while producing a diverse distribution of high-quality samples. We validate our solver on a physical Franka manipulator and release our code open-source.
Read the full abstract ▾
All authors Cael Yasutake, Andrew H. Liu, Zachary Kingston, Brian Plancher
Robotics
VeMo: A Lightweight Data-Driven Approach to Model Vehicle Dynamics
Girolamo Oddo, Roberto Nuca and Matteo Parsani
Developing a dynamic model for a high-performance vehicle is a complex problem that requires extensive structural information about the system under analysis. This information is often unavailable to those who did not design the vehicle and represents a typical issue in autonomous driving applications, which are frequently developed on top of existing vehicles; therefore, vehicle models are developed under conditions of information scarcity. This paper proposes a lightweight encoder-decoder model based on Gate Recurrent Unit layers to correlate the vehicle's future state with its past states, measured onboard, and control actions the driver performs. The results demonstrate that the model achieves a maximum mean relative error below 2.6% in extreme dynamic conditions. It also shows good robustness when subject to noisy input data across the interested frequency components. Furthermore, being entirely data-driven and free from physical constraints, the model exhibits physical consistency in the output signals, such as longitudinal and lateral accelerations, yaw rate, and the vehicle's longitudinal velocity.
Read the full abstract ▾
Robotics · Cross-listed from cs.NE
A Rotation-Invariant Embedded Platform for (Neural) Cellular Automata
Dominik Woiwode, Jakob Marten and Bodo Rosenhahn
This paper presents a rotation-invariant embedded platform for simulating (neural) cellular automata (NCA) in modular robotic systems. Inspired by previous work on physical NCA, we introduce key innovations that overcome limitations in prior hardware designs. Our platform features a symmetric, modular structure, enabling seamless connections between
cells regardless of orientation. Additionally, each
cell is battery-powered, allowing it to operate independently and retain its state even when disconnected from the collective. To demonstrate the platform's applicability, we present a novel rotation-invariant NCA model for isotropic shape classification. The proposed system provides a robust foundation for exploring the physical realization of NCA, with potential applications in distributed robotic systems and self-organizing structures. Our implementation, including hardware, software code, a simulator, and a
video, is openly shared at: https://github.com/dwoiwode/embedded_nca
Read the full abstract ▾
Robotics
FLEET: Formal Language-Grounded Scheduling for Heterogeneous Robot Teams
Corban Rivera, Grayson Byrd, Meghan Booker et al. (+6 more)
Coordinating heterogeneous
robot teams from free-form natural-
language instructions is hard.
Language-only planners struggle with long-horizon coordination and
hallucination, while purely formal methods require closed-world models. We present FLEET, a hybrid decentralized framework that turns
language into optimized multi-
robot schedules. An LLM front-end produces (i) a task graph with durations and precedence and (ii) a capability-aware
robot–task fitness
matrix; a formal back-end solves a makespan-minimization problem while the underlying
robots execute their free-form subtasks with agentic closed-loop control. Across multiple free-form
language-guided autonomy coordination
benchmarks, FLEET improves success over state of the art generative planners on two-
agent teams across heterogeneous tasks. Ablations show that mixed integer linear programming (MILP) primarily improves temporal structure, while LLM-derived fitness is decisive for capability-coupled tasks; together they deliver the highest overall performance. We demonstrate the translation to real world challenges with hardware trials using a pair of quadruped
robots with disjoint capabilities.
Read the full abstract ▾
All authors Corban Rivera, Grayson Byrd, Meghan Booker, Bethany Kemp, Allison Gaines, Emma Holmes, James Uplinger, Celso M de Melo, David Handelman
Robotics · Cross-listed from cs.CV
WristWorld: Generating Wrist-Views via 4D World Models for Robotic Manipulation
Zezhong Qian, Xiaowei Chi, Yuming Li et al. (+5 more)
Wrist-view observations are crucial for VLA models as they capture fine-grained hand-object interactions that directly enhance manipulation performance. Yet large-scale
datasets rarely include such recordings, resulting in a substantial gap between abundant anchor views and scarce wrist views. Existing world models cannot bridge this gap, as they require a wrist-view first frame and thus fail to generate wrist-view
videos from anchor views alone. Amid this gap, recent visual geometry models such as VGGT emerge with
geometric and cross-view priors that make it possible to address extreme viewpoint shifts. Inspired by these insights, we propose WristWorld, the first 4D world model that generates wrist-view
videos solely from anchor views. WristWorld operates in two stages: (i) Reconstruction, which extends VGGT and incorporates our Spatial Projection Consistency (SPC) Loss to estimate geometrically consistent wrist-view poses and 4D point clouds; (ii) Generation, which employs our
video generation model to synthesize temporally coherent wrist-view
videos from the reconstructed perspective. Experiments on Droid, Calvin, and Franka Panda demonstrate state-of-the-art
video generation with superior spatial consistency, while also improving VLA performance, raising the average task completion length on Calvin by 3.81% and closing 42.4% of the anchor-wrist view gap.
Read the full abstract ▾
All authors Zezhong Qian, Xiaowei Chi, Yuming Li, Shizun Wang, Zhiyuan Qin, Xiaozhu Ju, Sirui Han, Shanghang Zhang
Robotics
HyPlan: Hybrid Learning-Assisted Planning Under Uncertainty for Safe Autonomous Driving
Donald Pfaffmann, Matthias Klusch and Marcel Steinmetz
We present a novel hybrid learning-assisted planning method, named HyPlan, for solving the collision-free
navigation problem for self-driving cars in partially observable traffic environments. HyPlan combines methods for multi-
agent behavior prediction, deep reinforcement learning with proximal policy optimization and approximated online POMDP planning with heuristic confidence-based vertical
pruning to reduce its execution time without compromising safety of driving. Our experimental performance analysis on the CARLA-CTS2
benchmark of critical traffic scenarios with pedestrians revealed that HyPlan may navigate safer than selected relevant baselines and perform significantly faster than considered alternative online POMDP planners.
Read the full abstract ▾
Robotics
COMPAct: Computational Optimization and Automated Modular design of Planetary Actuators
Aman Singh, Deepak Kapa, Suryank Joshi et al. (+1 more)
The optimal design of robotic actuators is a critical area of research, yet limited
attention has been given to optimizing gearbox parameters and automating actuator CAD. This paper introduces COMPAct: Computational Optimization and Automated Modular Design of Planetary Actuators, a framework that systematically identifies optimal gearbox parameters for a given motor across four gearbox types, single-stage planetary gearbox (SSPG), compound planetary gearbox (CPG), Wolfrom planetary gearbox (WPG), and double-stage planetary gearbox (DSPG). The framework minimizes mass and actuator width while maximizing efficiency, and further automates actuator CAD generation to enable direct 3D printing without manual redesign. Using this framework, optimal gearbox designs are explored across a wide range of gear ratios, providing insights into the suitability of different gearbox types while automatically generating CAD models for all four gearbox types with varying gear ratios and motors. Two actuator types are fabricated and experimentally evaluated through power efficiency, no-load backlash, and transmission stiffness tests. Experimental results indicate that the SSPG actuator achieves a mechanical efficiency of 60-80%, a no-load backlash of 0.59 deg, and a transmission stiffness of 242.7 Nm/rad, while the CPG actuator demonstrates 60% efficiency, 2.6 deg backlash, and a stiffness of 201.6 Nm/rad. CODE: https://github.com/singhaman1750/COMPAct.git
VIDEO: https://youtu.be/etK6anjXag8?si=jFK7HgAPSBy-GnDR
Read the full abstract ▾
All authors Aman Singh, Deepak Kapa, Suryank Joshi, Shishir Kolathaya
Robotics
Yi Han, Enshen Zhou, Shanyu Rong et al. (+6 more)
Vision-
Language Models (VLMs) have shown remarkable capabilities in spatial
reasoning, yet they remain fundamentally limited to qualitative precision and lack the computational precision required for real-world robotics. Current approaches fail to leverage metric cues from depth sensors and camera calibration, instead reducing
geometric problems to pattern recognition tasks that cannot deliver the centimeter-level accuracy essential for robotic manipulation. We present TIGeR (Tool-Integrated
Geometric Reasoning), a novel framework that transforms VLMs from perceptual
estimators to
geometric computers by enabling them to generate and execute precise
geometric computations through external tools. Rather than attempting to internalize complex
geometric operations within neural networks, TIGeR empowers models to recognize
geometric reasoning requirements, synthesize appropriate computational code, and invoke specialized libraries for exact calculations. To support this paradigm, we introduce TIGeR-300K, a comprehensive tool-invocation-oriented
dataset covering point transformations, pose estimation, and spatial compatibility verification, complete with tool invocation sequences and intermediate computations. Through a two-stage training pipeline combining supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT) with our proposed hierarchical reward design, TIGeR achieves SOTA performance on
geometric reasoning benchmarks while demonstrating centimeter-level precision in real-world robotic manipulation tasks.
Read the full abstract ▾
All authors Yi Han, Enshen Zhou, Shanyu Rong, Jingkun An, Pengwei Wang, Zhongyuan Wang, Cheng Chi, Lu Sheng, Shanghang Zhang
Robotics
A Narwhal-Inspired Sensing-to-Control Framework for Small Fixed-Wing Aircraft
Fengze Xie, Xiaozhou Fan, Jacob Schuster et al. (+2 more)
Fixed-wing unmanned aerial vehicles (
UAVs) offer endurance and efficiency but lack low-speed agility due to highly coupled dynamics. We present an end-to-end sensing-to-control pipeline that combines bio-inspired hardware, physics-informed dynamics learning, and convex control allocation. Measuring airflow on a small airframe is difficult because near-body aerodynamics, propeller slipstream, control-surface actuation, and ambient gusts distort pressure signals. Inspired by the narwhal's protruding tusk, we mount in-house multi-hole probes far upstream and complement them with sparse, carefully placed wing pressure sensors for local flow measurement. A data-driven calibration maps probe pressures to airspeed and flow angles. We then learn a control-affine dynamics model using the estimated airspeed/angles and sparse sensors. A soft left/right
symmetry regularizer improves identifiability under partial observability and limits confounding between wing pressures and flaperon inputs. Desired wrenches (forces and moments) are realized by a regularized least-squares allocator that yields smooth, trimmed actuation. Wind-tunnel studies across a wide operating range show that adding wing pressures reduces force-estimation error by 25-30%, the proposed model degrades less under distribution shift (about 12% versus 44% for an unstructured baseline), and force tracking improves with smoother inputs, including a 27% reduction in normal-force RMSE versus a plain affine model and 34% versus an unstructured baseline.
Read the full abstract ▾
All authors Fengze Xie, Xiaozhou Fan, Jacob Schuster, Yisong Yue, Morteza Gharib
Robotics
DPL: Depth-only Perceptive Humanoid Locomotion via Realistic Depth Synthesis and Cross-Attention Terrain Reconstruction
Jingkai Sun, Gang Han, Pihai Sun et al. (+5 more)
Recent advancements in legged
robot perceptive locomotion have shown promising progress. However, terrain-aware
humanoid locomotion remains largely constrained to two paradigms: depth image-based end-to-end learning and elevation map-based methods. The former suffers from limited training efficiency and a significant sim-to-real gap in depth perception, while the latter depends heavily on multiple
vision sensors and localization systems, resulting in latency and reduced robustness. To overcome these challenges, we propose a novel framework that tightly integrates three key components: (1) Terrain-Aware Locomotion Policy with a Blind Backbone, which leverages pre-trained elevation map-based perception to guide reinforcement learning with minimal visual input; (2) Multi-Modality Cross-
Attention Transformer, which reconstructs structured terrain representations from noisy depth images; (3) Realistic Depth Images Synthetic Method, which employs self-occlusion-aware ray casting and noise-aware modeling to synthesize realistic depth observations, achieving over 30% reduction in terrain reconstruction error. This combination enables efficient policy training with limited data and hardware resources, while preserving critical terrain features essential for generalization. We validate our framework on a full-sized
humanoid robot, demonstrating agile and adaptive locomotion across diverse and challenging terrains.
Read the full abstract ▾
All authors Jingkai Sun, Gang Han, Pihai Sun, Wen Zhao, Jiahang Cao, Jiaxu Wang, Qiang Zhang, Yijie Guo
Robotics · Cross-listed from cs.LG
ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problems
Egor Cherepanov, Alexey K. Kovalev and Aleksandr I. Panov
Real-world robotic
agents must act under partial observability and long horizons, where key cues may appear long before they affect decision making. However, most modern approaches rely solely on instantaneous information, without incorporating insights from the past. Standard recurrent or
transformer models struggle with retaining and leveraging long-term dependencies: context windows truncate history, while naive
memory extensions fail under scale and sparsity. We propose ELMUR (External Layer
Memory with Update/Rewrite), a
transformer architecture with structured external
memory. Each layer maintains
memory embeddings, interacts with them via bidirectional cross-
attention, and updates them through an Least Recently Used (LRU)
memory module using replacement or convex blending. ELMUR extends effective horizons up to 100,000 times beyond the
attention window and achieves a 100% success rate on a synthetic T-Maze task with corridors up to one million steps. In POPGym, it outperforms baselines on more than half of the tasks. On MIKASA-Robo sparse-reward manipulation tasks with visual observations, it nearly doubles the performance of strong baselines, achieving the best success rate on 21 out of 23 tasks and improving the aggregate success rate across all tasks by about 70% over the previous best baseline. These results demonstrate that structured, layer-local external
memory offers a simple and scalable approach to decision making under partial observability. Code and project page: https://elmur-paper.github.io/.
Read the full abstract ▾
Robotics
TrackVLA++: Unleashing Reasoning and Memory Capabilities in VLA Models for Embodied Visual Tracking
Jiahang Liu, Yunpeng Qi, Jiazhao Zhang et al. (+9 more)
Embodied Visual Tracking (EVT) is a fundamental ability that underpins practical applications, such as companion
robots, guidance
robots and service assistants, where continuously following moving targets is essential. Recent advances have enabled
language-guided tracking in complex and unstructured scenes. However, existing approaches lack explicit spatial
reasoning and effective temporal
memory, causing failures under severe occlusions or in the presence of similar-looking distractors. To address these challenges, we present TrackVLA++, a novel
Vision-
Language-Action (VLA) model that enhances
embodied visual tracking with two key modules, a spatial
reasoning mechanism and a Target Identification
Memory (TIM). The
reasoning module introduces a Chain-of-Thought paradigm, termed Polar-CoT, which infers the target's relative position and encodes it as a compact polar-coordinate
token for action prediction. Guided by these spatial priors, the TIM employs a gated update strategy to preserve long-horizon target
memory, ensuring spatiotemporal consistency and mitigating target loss during extended occlusions. Extensive experiments show that TrackVLA++ achieves state-of-the-art performance on public
benchmarks across both
egocentric and multi-camera settings. On the challenging EVT-Bench DT split, TrackVLA++ surpasses the previous leading approach by 5.1 and 12, respectively. Furthermore, TrackVLA++ exhibits strong zero-shot generalization, enabling robust real-world tracking in dynamic and occluded scenarios.
Read the full abstract ▾
All authors Jiahang Liu, Yunpeng Qi, Jiazhao Zhang, Minghan Li, Shaoan Wang, Kui Wu, Hanjing Ye, Hong Zhang, Zhibo Chen, Fangwei Zhong, Zhizheng Zhang, He Wang
Robotics
A Digital Twin Framework for Metamorphic Testing of Autonomous Driving Systems Using Generative Model
Tony Zhang, Burak Kantarci and Umair Siddique
Ensuring the safety of self-driving cars remains a major challenge due to the complexity and unpredictability of real-world driving environments. Traditional testing methods face significant limitations, such as the oracle problem, which makes it difficult to determine whether a system's behavior is correct, and the inability to cover the full range of scenarios an autonomous vehicle may encounter. In this paper, we introduce a digital twin-driven metamorphic testing framework that addresses these challenges by creating a virtual replica of the self-driving system and its operating environment. By combining digital twin technology with AI-based image generative models such as Stable
Diffusion, our approach enables the systematic generation of realistic and diverse driving scenes. This includes variations in weather, road
topology, and environmental features, all while maintaining the core semantics of the original scenario. The digital twin provides a synchronized simulation environment where changes can be tested in a controlled and repeatable manner. Within this environment, we define three metamorphic relations inspired by real-world traffic rules and vehicle behavior. We validate our framework in the Udacity self-driving simulator and demonstrate that it significantly enhances test coverage and effectiveness. Our method achieves the highest true positive rate (0.719), F1 score (0.689), and precision (0.662) compared to baseline approaches. This paper highlights the value of integrating digital twins with AI-powered scenario generation to create a scalable, automated, and high-fidelity testing solution for autonomous vehicle safety.
Read the full abstract ▾
Robotics
Sampling Strategies for Robust Universal Quadrupedal Locomotion Policies
David Rytz, Kim Tien Ly and Ioannis Havoutis
This work focuses on sampling strategies of configuration variations for generating robust universal locomotion policies for quadrupedal
robots. We investigate the effects of sampling physical
robot parameters and joint proportional-derivative gains to enable training a single reinforcement learning policy that generalizes to multiple parameter configurations. Three fundamental joint gain sampling strategies are compared: parameter sampling with (1) linear and
polynomial function mappings of mass-to-gains, (2) performance-based adaptive filtering, and (3) uniform random sampling. We improve the robustness of the policy by biasing the configurations using nominal priors and reference models. All training was conducted using the RaiSim simulation environment, tested in simulation on a range of diverse quadrupeds, and zero-shot deployed onto hardware using the ANYmal quadruped
robot. Compared to multiple baseline implementations, our results demonstrate the need for significant joint controller gains randomization for robust closing of the sim-to-real gap.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
Generative World Modelling for Humanoids: 1X World Model Challenge Technical Report
Riccardo Mereu, Aidan Scannell, Yuxin Hou et al. (+6 more)
World models are a powerful paradigm in AI and robotics, enabling
agents to reason about the future by predicting visual observations or compact latent states. The 1X World Model Challenge introduces an open-source
benchmark of real-world
humanoid interaction, with two complementary tracks: sampling, focused on
forecasting future image frames, and compression, focused on predicting future discrete latent codes. For the sampling track, we adapt the
video generation foundation model Wan-2.2 TI2V-5B to
video-state-conditioned future frame prediction. We condition the
video generation on
robot states using AdaLN-Zero, and further post-train the model using LoRA. For the compression track, we train a Spatio-Temporal
Transformer model from scratch. Our models achieve 23.0 dB PSNR in the sampling task and a Top-500 CE of 6.6386 in the compression task, securing 1st place in both challenges.
Read the full abstract ▾
All authors Riccardo Mereu, Aidan Scannell, Yuxin Hou, Yi Zhao, Aditya Jitta, Antonio Dominguez, Luigi Acerbi, Amos Storkey, Paul Chang
Robotics
Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
Kento Kawaharazuka, Jihoon Oh, Jun Yamada et al. (+2 more)
Amid growing efforts to leverage advances in large
language models (LLMs) and
vision-
language models (VLMs) for robotics,
Vision-
Language-Action (VLA) models have recently gained significant
attention. By unifying
vision,
language, and action data at scale, which have traditionally been studied separately, VLA models aim to learn policies that generalise across diverse tasks, objects, embodiments, and environments. This generalisation capability is expected to enable
robots to solve novel downstream tasks with minimal or no additional task-specific data, facilitating more flexible and scalable real-world deployment. Unlike previous surveys that focus narrowly on action representations or high-level model architectures, this work offers a comprehensive, full-stack review, integrating both software and hardware components of VLA systems. In particular, this paper provides a systematic review of VLAs, covering their strategy and architectural transition, architectures and building blocks, modality-specific processing techniques, and learning paradigms. In addition, to support the deployment of VLAs in real-world robotic applications, we also review commonly used
robot platforms, data collection strategies, publicly available
datasets, data augmentation methods, and evaluation
benchmarks. Throughout this comprehensive survey, this paper aims to offer practical guidance for the robotics community in applying VLAs to real-world robotic systems. All references categorized by training approach, evaluation method, modality, and
dataset are available in the table on our project website: https://vla-survey.github.io .
Read the full abstract ▾
All authors Kento Kawaharazuka, Jihoon Oh, Jun Yamada, Ingmar Posner, Yuke Zhu
Robotics
Bring the Apple, Not the Sofa: Impact of Irrelevant Context in Embodied AI Commands on VLA Models
Daria Pugacheva, Andrey Moskalenko, Denis Shepelev et al. (+3 more)
Vision Language Action (VLA) models are widely used in
Embodied AI, enabling
robots to interpret and execute
language instructions. However, their robustness to natural
language variability in real-world scenarios has not been thoroughly investigated. In this work, we present a novel systematic study of the robustness of state-of-the-art VLA models under linguistic perturbations. Specifically, we evaluate model performance under two types of instruction noise: (1) human-generated paraphrasing and (2) the addition of irrelevant context. We further categorize irrelevant contexts into two groups according to their length and their semantic and lexical proximity to
robot commands. In this study, we observe consistent performance degradation as context size expands. We also demonstrate that the model can exhibit relative robustness to random context, with a performance drop within 10%, while semantically and lexically similar context of the same length can trigger a quality decline of around 50%. Human paraphrases of instructions lead to a drop of nearly 20%. To mitigate this, we propose an LLM-based filtering framework that extracts core commands from noisy inputs. Incorporating our filtering step allows models to recover up to 98.5% of their original performance under noisy conditions.
Read the full abstract ▾
All authors Daria Pugacheva, Andrey Moskalenko, Denis Shepelev, Andrey Kuznetsov, Vlad Shakhuro, Elena Tutubalina
Robotics · Cross-listed from cs.HC
Artists' Views on Robotics Involvement in Painting Productions
Francesca Cocchella, Nilay Roy Choudhury, Eric Chen et al. (+1 more)
As robotic technologies evolve, their potential in artistic creation becomes an increasingly relevant topic of inquiry. This study explores how professional abstract artists perceive and experience co-creative interactions with an autonomous painting robotic arm. Eight artists engaged in six painting sessions – three with a human partner, followed by three with the
robot – and subsequently participated in semi-structured interviews analyzed through reflexive thematic analysis. Human-human interactions were described as intuitive, dialogic, and emotionally engaging, whereas human-
robot sessions felt more playful and reflective, offering greater autonomy and prompting for novel strategies to overcome the system's limitations. This work offers one of the first empirical investigations into artists' lived experiences with a
robot, highlighting the value of long-term engagement and a multidisciplinary approach to human-
robot co-creation.
Read the full abstract ▾
All authors Francesca Cocchella, Nilay Roy Choudhury, Eric Chen, Patrícia Alves-Oliveira
Robotics · Cross-listed from cs.LG
Introspection in Learned Semantic Scene Graph Localisation
Manshika Charvi Bissessur, Efimia Panagiotaki and Daniele De Martini
This work investigates how semantics influence localisation performance and robustness in a learned self-supervised, contrastive semantic localisation framework. After training a localisation network on both original and perturbed maps, we conduct a thorough post-hoc introspection analysis to probe whether the model filters environmental noise and prioritises distinctive landmarks over routine clutter. We validate various interpretability methods and present a comparative reliability analysis. Integrated gradients and
Attention Weights consistently emerge as the most reliable probes of learned behaviour. A semantic class ablation further reveals an implicit weighting in which frequent objects are often down-weighted. Overall, the results indicate that the model learns noise-robust, semantically salient relations about place definition, thereby enabling explainable registration under challenging visual and structural variations.
Read the full abstract ▾
Robotics
Diffusing Trajectory Optimization Problems for Recovery During Multi-Finger Manipulation
Abhinav Kumar, Fan Yang, Sergio Aguilera Marinovic et al. (+3 more)
Multi-fingered hands are emerging as powerful platforms for performing fine manipulation tasks, including tool use. However, environmental perturbations or execution errors can impede task performance, motivating the use of recovery behaviors that enable normal task execution to resume. In this work, we take advantage of recent advances in
diffusion models to construct a framework that autonomously identifies when recovery is necessary and optimizes contact-rich trajectories to recover. We use a
diffusion model trained on the task to estimate when states are not conducive to task execution, framed as an out-of-distribution detection problem. We then use
diffusion sampling to project these states in-distribution and use trajectory optimization to plan contact-rich recovery trajectories. We also propose a novel
diffusion-based approach that distills this process to efficiently diffuse the full parameterization, including constraints, goal state, and initialization, of the recovery trajectory optimization problem, saving time during online execution. We compare our method to a reinforcement learning baseline and other methods that do not explicitly plan contact interactions, including on a hardware screwdriver-turning task where we show that recovering using our method improves task performance by 96% and that ours is the only method evaluated that can attempt recovery without causing catastrophic task failure.
Videos can be found at https://dtourrecovery.github.io/.
Read the full abstract ▾
All authors Abhinav Kumar, Fan Yang, Sergio Aguilera Marinovic, Soshi Iba, Rana Soltani Zarrin, Dmitry Berenson
Robotics
Efficient View Planning Guided by Previous-Session Reconstruction for Repeated Plant Monitoring
Sicong Pan, Luca Lobefaro, Moein Taherkhani et al. (+4 more)
Repeated plant monitoring is essential for tracking crop growth, and 3D reconstruction enables consistent comparison across monitoring sessions. However, rebuilding a 3D model from scratch in every session is costly and overlooks informative geometry already observed previously. We propose efficient view planning guided by a previous-session reconstruction, which reuses a 3D model from the previous session to improve active perception in the current session. Based on this previous-session reconstruction, our method replaces iterative next-best-view planning with one-shot view planning that selects an informative set of views and computes the globally shortest execution path connecting them. Experiments on real multi-session
datasets, including public single-plant scans and a newly collected greenhouse crop-row
dataset, show that our method achieves comparable or higher surface coverage with fewer executed views and shorter
robot paths than iterative and one-shot baselines.
Read the full abstract ▾
All authors Sicong Pan, Luca Lobefaro, Moein Taherkhani, Xuying Huang, Rohit Menon, Cyrill Stachniss, Maren Bennewitz
Robotics
Tailoring materials into kirigami robots
Saravana Prashanth Murali Babu, Aida Parvaresh and Ahmad Rafsanjani
Kirigami, the traditional paper-cutting craft, holds immense potential for revolutionizing robotics by providing multifunctional, lightweight, and adaptable solutions. Kirigami structures, characterized by their bending-dominated deformation, offer resilience to tensile forces and facilitate shape morphing under small actuation forces. Kirigami components such as actuators, sensors, batteries, controllers, and body structures can be tailored to specific robotic applications by optimizing cut patterns. Actuators based on kirigami principles exhibit complex motions programmable through various energy sources, while kirigami sensors bridge the gap between electrical conductivity and compliance. Kirigami-integrated batteries enable energy storage directly within
robot structures, enhancing flexibility and compactness. Kirigami-controlled mechanisms mimic mechanical computations, enabling advanced functionalities such as shape morphing and
memory functions. Applications of kirigami-enabled
robots include grasping, locomotion, and wearables, showcasing their adaptability to diverse environments and tasks. Despite promising opportunities, challenges remain in the design of cut patterns for a given function and streamlining fabrication techniques.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
DecompGAIL: Learning Realistic Traffic Behaviors with Decomposed Multi-Agent Generative Adversarial Imitation Learning
Ke Guo, Haochen Liu, Xiaojun Wu et al. (+1 more)
Realistic traffic simulation is critical for the development of autonomous driving systems and urban mobility planning, yet existing imitation learning approaches often fail to model realistic traffic behaviors. Behavior cloning suffers from
covariate shift, while Generative
Adversarial Imitation Learning (GAIL) is notoriously unstable in multi-
agent settings. We identify a key source of this instability: irrelevant interaction misguidance, where a discriminator penalizes an ego vehicle's realistic behavior due to unrealistic interactions among its neighbors. To address this, we propose Decomposed Multi-
agent GAIL (DecompGAIL), which explicitly decomposes realism into ego-map and ego-neighbor components, filtering out misleading neighbor: neighbor and neighbor: map interactions. We further introduce a social PPO objective that augments ego rewards with distance-weighted neighborhood rewards, encouraging overall realism across
agents. Integrated into a lightweight SMART-based backbone, DecompGAIL achieves state-of-the-art performance on the WOMD Sim
Agents 2025
benchmark.
Read the full abstract ▾
All authors Ke Guo, Haochen Liu, Xiaojun Wu, Chen Lv
Robotics · Cross-listed from cs.CV
HARP-NeXt: High-Speed and Accurate Range-Point Fusion Network for 3D LiDAR Semantic Segmentation
Samir Abou Haidar, Alexandre Chariot, Mehdi Darouich et al. (+2 more)
LiDAR semantic
segmentation is crucial for autonomous vehicles and mobile
robots, requiring high accuracy and real-time processing, especially on resource-constrained embedded systems. Previous state-of-the-art methods often face a trade-off between accuracy and speed. Point-based and sparse convolution-based methods are accurate but slow due to the complexity of neighbor searching and 3D convolutions. Projection-based methods are faster but lose critical
geometric information during the 2D projection. Additionally, many recent methods rely on test-time augmentation (TTA) to improve performance, which further slows the inference. Moreover, the pre-processing phase across all methods increases execution time and is demanding on embedded platforms. Therefore, we introduce HARP-NeXt, a high-speed and accurate
LiDAR semantic
segmentation network. We first propose a novel pre-processing methodology that significantly reduces computational overhead. Then, we design the Conv-SE-NeXt feature extraction block to efficiently capture representations without deep layer stacking per network stage. We also employ a multi-scale range-point fusion backbone that leverages information at multiple abstraction levels to preserve essential
geometric details, thereby enhancing accuracy. Experiments on the nuScenes and SemanticKITTI
benchmarks show that HARP-NeXt achieves a superior speed-accuracy trade-off compared to all state-of-the-art methods, and, without relying on ensemble models or TTA, is comparable to the top-ranked PTv3, while running 24$\times$ faster. The code is available at https://github.com/SamirAbouHaidar/HARP-NeXt
Read the full abstract ▾
All authors Samir Abou Haidar, Alexandre Chariot, Mehdi Darouich, Cyril Joly, Jean-Emmanuel Deschaud
Robotics
Distributed 3D Source Seeking via SO(3) Geometric Control of Robot Swarms
Jesús Bautista and Héctor García de Marina
This paper presents a
geometric control framework on the Lie group SO(3) for 3D source-seeking by
robots with first-order attitude dynamics and constant translational speed. By working directly on SO(3), the approach avoids Euler-angle singularities and quaternion ambiguities, providing a unique, intrinsic representation of orientation. We design a proportional feed-forward controller that ensures exponential
alignment of each
agent to an estimated ascending direction toward a 3D scalar field source. The controller adapts to bounded unknown variations and preserves well-posed swarm formations. Numerical simulations demonstrate the effectiveness of the method, with all code provided open source for reproducibility.
Read the full abstract ▾