Robotics · Cross-listed from cs.CV
IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants
Vivek Chavan, Yasmina Imgrund, Tung Dao et al. (+5 more)
We introduce IndEgo, a
multimodal egocentric and exocentric
dataset addressing common industrial tasks, including assembly/disassembly, logistics and organisation, inspection and repair, woodworking, and others. The
dataset contains 3,460
egocentric recordings (approximately 197 hours), along with 1,092 exocentric recordings (approximately 97 hours). A key focus of the
dataset is collaborative work, where two workers jointly perform cognitively and physically intensive tasks. The
egocentric recordings include rich
multimodal data and added context via eye gaze, narration, sound, motion, and others. We provide detailed annotations (actions, summaries, mistake annotations, narrations), metadata, processed outputs (eye gaze, hand pose, semi-dense point cloud), and
benchmarks on procedural and non-procedural task understanding, Mistake Detection, and
reasoning-based Question Answering. Baseline evaluations for Mistake Detection, Question Answering and collaborative task understanding show that the
dataset presents a challenge for the state-of-the-art
multimodal models. Our
dataset is available at: https://huggingface.co/
datasets/FraunhoferIPK/IndEgo
Read the full abstract ▾
All authors Vivek Chavan, Yasmina Imgrund, Tung Dao, Sanwantri Bai, Bosong Wang, Ze Lu, Oliver Heimann, Jörg Krüger
Robotics · Cross-listed from math.OC
Anytime-Feasible First-Order Optimization via Safe Sequential QCQP
Jiarui Wang and Mahyar Fazlyab
This paper presents the Safe Sequential Quadratically Constrained Quadratic Programming (SS-QCQP)
algorithm, a first-order method for smooth inequality-constrained nonconvex optimization that guarantees feasibility at every iteration. The method is derived from a continuous-time dynamical system whose vector field is obtained by solving a convex QCQP that enforces monotonic descent of the objective and forward invariance of the feasible set. The resulting continuous-time dynamics achieve an $O(1/t)$ ergodic convergence rate for the stationarity measure under standard constraint qualification conditions. We then propose a safeguarded Euler discretization with adaptive step-size selection that preserves this convergence rate while maintaining both descent and feasibility in discrete time. To enhance scalability, we develop an active-set variant (SS-QCQP-AS) that selectively enforces constraints near the boundary, substantially reducing computational cost without compromising theoretical guarantees. Numerical experiments on a multi-
agent nonlinear optimal control problem demonstrate that SS-QCQP and SS-QCQP-AS maintain feasibility, exhibit the predicted convergence behavior, and deliver solution quality comparable to second-order solvers such as SQP and IPOPT.
Read the full abstract ▾
Robotics
Development of a Testbed for Autonomous Vehicles: Integrating MPC Control with Monocular Camera Lane Detection
Shantanu Rahman, Nayeb Hasin, Mainul Islam et al. (+1 more)
Autonomous vehicles are becoming popular day by day not only for autonomous road traversal but also for industrial automation, farming and military. Most of the standard vehicles follow the Ackermann style steering mechanism. This has become to de facto standard for large and long faring vehicles. The local planner of an autonomous vehicle controls the low-level vehicle movement upon which the vehicle will perform its motor actuation. In our work, we focus on autonomous vehicles in road and perform experiments to analyze the effect of low-level controllers in the simulation and a real environment. To increase the precision and stability of trajectory tracking in autonomous cars, a novel method that combines lane identification with Model Predictive Control (MPC) is presented. The research focuses on camera-equipped autonomous vehicles and uses methods like edge recognition, sliding window-based straight-line identification for lane line extraction, and dynamic region of interest (ROI) extraction. Next, to follow the identified lane line, an MPC built on a bicycle vehicle dynamics model is created. A single-lane road simulation model is built using ROS Gazebo and tested in order to verify the controller's performance. The root mean square error between the optimal tracking trajectory and the target trajectory was reduced by 27.65% in the simulation results, demonstrating the high robustness and flexibility of the developed controller.
Read the full abstract ▾
All authors Shantanu Rahman, Nayeb Hasin, Mainul Islam, Golam Sarowar
Robotics
Flow-Based Path Planning for Multiple Homogenous UAVs for Outdoor Formation-Flying
Mahmud Suhaimi Ibrahim, Shantanu Rahman, Muhammad Samin Hasan et al. (+2 more)
Collision-free path planning is the most crucial component in multi-
UAV formation-flying (MFF). We use unlabeled homogenous quadcopters (
UAVs) to demonstrate the use of a flow network to create complete (inter-
UAV) collision-free paths. This procedure has three main parts: 1) Creating a flow network graph from physical GPS coordinates, 2) Finding a path of minimum cost (least distance) using any graph-based path-finding
algorithm, and 3) Implementing the Ford-Fulkerson Method to find the paths with the maximum flow (no collision). Simulations of up to 64
UAVs were conducted for various formations, followed by a practical experiment with 3 quadcopters for testing physical plausibility and feasibility. The results of these tests show the efficacy of this method's ability to produce safe, collision-free paths.
Read the full abstract ▾
All authors Mahmud Suhaimi Ibrahim, Shantanu Rahman, Muhammad Samin Hasan, Minhaj Uddin Ahmad, Abdullah Abrar
Robotics
Online Learning-Enhanced High Order Adaptive Safety Control
Lishuo Pan, Mattia Catellani, Thales C. Silva et al. (+2 more)
Control barrier functions (CBFs) are an effective model-based tool to formally certify the safety of a system. With the growing complexity of modern control problems, CBFs have received increasing
attention in both optimization-based and learning-based control communities as a safety filter, owing to their provable guarantees. However, success in transferring these guarantees to real-world systems is critically tied to model accuracy. For example, payloads or wind disturbances can significantly influence the dynamics of an aerial vehicle and invalidate the safety guarantee. In this work, we propose an efficient yet flexible online learning-enhanced high-order adaptive control barrier function using Neural ODEs. Our approach improves the safety of a CBF controller on the fly, even under complex time-varying model perturbations. In particular, we deploy our hybrid adaptive CBF controller on a 38g nano quadrotor, keeping a safe distance from the obstacle, against 18km/h wind.
Read the full abstract ▾
All authors Lishuo Pan, Mattia Catellani, Thales C. Silva, Lorenzo Sabattini, Nora Ayanian
Robotics
Robot-Powered Data Flywheels: Deploying Robots in the Wild for Continual Data Collection and Foundation Model Adaptation
Jennifer Grannen, Michelle Pan, Kenneth Llontop et al. (+4 more)
Foundation models (FM) have unlocked powerful zero-shot capabilities in
vision and
language, yet their reliance on internet pretraining data leaves them brittle in unstructured, real-world settings. The messy, real-world data encountered during deployment (e.g. occluded or
multilingual text) remains massively underrepresented in existing corpora.
Robots, as
embodied agents, are uniquely positioned to close this gap: they can act in physical environments to collect large-scale, real-world data that enriches FM training with precisely the examples current models lack. We introduce the
Robot-Powered Data Flywheel, a framework that transforms
robots from FM consumers into data generators. By deploying
robots equipped with FMs in the wild, we enable a virtuous cycle:
robots perform useful tasks while collecting real-world data that improves both domain-specific adaptation and domain-adjacent generalization. We instantiate this framework with Scanford, a mobile manipulator deployed in the East Asia Library for 2 weeks. Scanford autonomously scans shelves, identifies books using a
vision-
language model (VLM), and leverages the library catalog to label images without human annotation. This deployment both aids librarians and produces a
dataset to finetune the underlying VLM, improving performance on the domain-specific in-the-wild library setting and on domain-adjacent
multilingual OCR
benchmarks. Using data collected from 2103 shelves, Scanford improves VLM performance on book identification from 32.0% to 71.8% and boosts domain-adjacent
multilingual OCR from 24.8% to 46.6% (English) and 30.8% to 38.0% (Chinese), while saving an ~18.7 hrs of human time. These results highlight how
robot-powered data flywheels can both reduce human effort in real deployments and unlock new pathways for continually adapting FMs to the messiness of reality. More details are at: https://scanford-
robot.github.io
Read the full abstract ▾
All authors Jennifer Grannen, Michelle Pan, Kenneth Llontop, Cherie Ho, Mark Zolotas, Jeannette Bohg, Dorsa Sadigh
Robotics · Cross-listed from cs.LG
Learning Massively Multitask World Models for Continuous Control
Nicklas Hansen, Hao Su and Xiaolong Wang
General-purpose control demands
agents that act across many tasks and embodiments, yet research on reinforcement learning (RL) for continuous control remains dominated by single-task or offline regimes, reinforcing a view that online RL does not scale. Inspired by the foundation model recipe (large-scale pretraining followed by light RL) we ask whether a single
agent can be trained on hundreds of tasks with online interaction. To accelerate research in this direction, we introduce a new
benchmark with 200 diverse tasks spanning many domains and embodiments, each with
language instructions, demonstrations, and optionally image observations. We then present Newt, a
language-conditioned multitask world model that is first pretrained on demonstrations to acquire task-aware representations and action priors, and then jointly optimized with online interaction across all tasks. Experiments show that Newt yields better multitask performance and data-efficiency than a set of strong baselines, exhibits strong open-loop control, and enables rapid adaptation to unseen tasks. We release our environments, demonstrations, code for training and evaluation, as well as 200+ checkpoints.
Read the full abstract ▾
Robotics
A Virtual Mechanical Interaction Layer Enables Resilient Human-to-Robot Object Handovers
Omar Faris, Sławomir Tadeja and Fulvio Forni
Object handover is a common form of interaction that is widely present in collaborative tasks. However, achieving it efficiently remains a challenge. We address the problem of ensuring resilient robotic actions that can adapt to complex changes in object pose during human-to-
robot object handovers. We propose the use of Virtual Model Control to create an interaction layer that controls the
robot and adapts to the dynamic changes in the handover process. Additionally, we propose the use of augmented reality to facilitate bidirectional communication between humans and
robots during handovers. We assess the performance of our controller in a set of experiments that demonstrate its resilience to various sources of uncertainties, including complex changes to the object's pose during the handover. Finally, we performed a user study with 16 participants to understand human preferences for different
robot control profiles and augmented reality visuals in object handovers. Our results showed a general preference for the proposed approach and revealed insights that can guide further development in adapting the interaction with the user.
Read the full abstract ▾
Robotics
Discover, Learn, and Reinforce: Scaling Vision-Language-Action Pretraining with Diverse RL-Generated Trajectories
Rushuai Yang, Zhiyuan Feng, Tianxiang Zhang et al. (+6 more)
Scaling
vision-
language-action (VLA) model pre-training requires large volumes of diverse, high-quality manipulation trajectories. Most current data is obtained via human teleoperation, which is expensive and difficult to scale. Reinforcement learning (RL) methods learn useful skills through autonomous exploration, making them a viable approach for generating data. However, standard RL training collapses to a narrow execution pattern, limiting its utility for large-scale pre-training. We propose Discover, Lea rn and Reinforce (DLR), an information-theoretic pattern discovery framework that generates multiple distinct, high-success behavioral patterns for VLA pretraining. Empirically, DLR generates a markedly more diverse trajectory corpus on LIBERO. Specifically, it learns multiple distinct, high-success strategies for the same task where standard RL discovers only one, and hence it covers substantially broader regions of the state-action space. When adapted to unseen downstream task suites, VLA models pretrained on our diverse RL data surpass counterparts trained on equal-sized standard RL
datasets. Moreover, DLR exhibits positive data-scaling behavior that single-pattern RL lacks. These results position multi-pattern RL as a practical, scalable data engine for
embodied foundation models.
Read the full abstract ▾
All authors Rushuai Yang, Zhiyuan Feng, Tianxiang Zhang, Kaixin Wang, Chuheng Zhang, Li Zhao, Xiu Su, Yi Chen, Jiang Bian
Robotics
Mixture of Horizons in Action Chunking
Dong Jing, Gang Wang, Jiaqi Liu et al. (+7 more)
Vision-
language-action (VLA) models have shown remarkable capabilities in robotic manipulation, but their performance is sensitive to the $\textbf{action chunk length}$ used during training, termed $\textbf{horizon}$. Our empirical study reveals an inherent trade-off: longer horizons provide stronger global foresight but degrade fine-grained accuracy, while shorter ones sharpen local control yet struggle on long-term tasks, implying fixed choice of single horizons being suboptimal. To mitigate the trade-off, we propose a $\textbf{mixture of horizons (MoH)}$ strategy. MoH rearranges the action chunk into several segments with different horizons, processes them in parallel with a shared action
transformer, and fuses outputs with a light linear gate. It has three appealing benefits. 1) MoH exploits long-term foresight and short-term precision jointly within a single model, improving both performance and generalizability to complex tasks. 2) MoH is plug-and-play for full-
attention action modules with minimal training or inference overhead. 3) MoH enables dynamic inference with adaptive horizons, which selects stable actions through cross-horizon consensus, achieving 2.5$\times$ higher throughput than baselines while preserving superior performance. Extensive experiments over flow-based policies $\pi_0$, $\pi_{0.5}$, and one-step regression policy $\pi_{\text{reg}}$ demonstrate that MoH yields consistent and significant gains on both simulations and real-world tasks. Notably, under mixed-task setting, $\pi_{0.5}$ with MoH reaches a new state-of-the-art with 99$\%$ average success rate on LIBERO after only $30k$ training iterations. Project page: https://timsty1.github.io/moh/
Read the full abstract ▾
All authors Dong Jing, Gang Wang, Jiaqi Liu, Weiliang Tang, Zelong Sun, Yunchao Yao, Zhenyu Wei, Yunhui Liu, Zhiwu Lu, Mingyu Ding
Robotics
Deployment Dynamics and Optimization of Novel Space Antenna Deployable Mechanism
Mamoon Aamir, Mariyam Sattar, Naveed Ur Rehman Junejo et al. (+1 more)
Given the increasing need for large aperture
antennas in space missions, the difficulty of fitting such structures into small launch vehicles has prompted the design of deployable
antenna systems. The thesis introduces a new Triple Scissors Deployable Truss Mechanism (TSDTM) for space
antenna missions. The new mechanism is to be stowed during launch and efficiently deploy in orbit, offering maximum aperture size while taking up minimal launch volume. The thesis covers the entire design process from
geometric modeling, kinematic analysis with screw theory and Newtonian approaches, dynamic analysis by
eigenvalue and simulation methods, and verification with SolidWorks. In addition, optimization routines were coded based on Support Vector Machines for material choice in LEO environments and machine learning method for
geometric setup. The TSDTM presented has enhanced structural dynamics with good comparison between simulation and analytical predictions. The structure optimized proved highly accurate, with a deviation of just 1.94% between machine learning-predicted and simulated natural frequencies, demonstrating the potential of incorporating AI-based methods in space structural design.
Read the full abstract ▾
All authors Mamoon Aamir, Mariyam Sattar, Naveed Ur Rehman Junejo, Aqsa Zafar Abbasi
Robotics · Cross-listed from cs.LG
Leveraging LLMs for reward function design in reinforcement learning control tasks
Franklin Cardenoso and Wouter Caarls
The challenge of designing effective reward functions in reinforcement learning (RL) represents a significant bottleneck, often requiring extensive human expertise and being time-consuming. Previous work and recent advancements in large
language models (LLMs) have demonstrated their potential for automating the generation of reward functions. However, existing methodologies often require preliminary evaluation metrics, human-engineered feedback for the refinement process, or the use of environmental source code as context. To address these limitations, this paper introduces LEARN-Opt (LLM-based Evaluator and Analyzer for Reward functioN Optimization). This LLM-based, fully autonomous, and model-agnostic framework eliminates the need for preliminary metrics and environmental source code as context to generate, execute, and evaluate reward function candidates from textual descriptions of systems and task objectives. LEARN-Opt's main contribution lies in its ability to autonomously derive performance metrics directly from the system description and the task objective, enabling unsupervised evaluation and selection of reward functions. Our experiments indicate that LEARN-Opt achieves performance comparable to or better to that of state-of-the-art methods, such as EUREKA, while requiring less prior knowledge. We find that automated reward design is a high-variance problem, where the average-case candidate fails, requiring a multi-run approach to find the best candidates. Finally, we show that LEARN-Opt can unlock the potential of low-cost LLMs to find high-performing candidates that are comparable to, or even better than, those of larger models. This demonstrated performance affirms its potential to generate high-quality reward functions without requiring any preliminary human-defined metrics, thereby reducing engineering overhead and enhancing generalizability.
Read the full abstract ▾
Robotics
Rethinking Intermediate Representation for VLM-based Robot Manipulation
Weiliang Tang, Jialin Gao, Jia-Hui Pan et al. (+6 more)
Vision-
Language Model (VLM) is an important component to enable robust
robot manipulation. Yet, using it to translate human instructions into an action-resolvable intermediate representation often needs a tradeoff between VLM-comprehensibility and generalizability. Inspired by context-free grammar, we design the Semantic Assembly representation named SEAM, by decomposing the intermediate representation into vocabulary and grammar. Doing so leads us to a concise vocabulary of semantically-rich operations and a VLM-friendly grammar for handling diverse unseen tasks. In addition, we design a new open-vocabulary
segmentation paradigm with a
retrieval-augmented few-shot learning strategy to localize fine-grained object parts for manipulation, effectively with the shortest inference time over all state-of-the-art parallel works. Also, we formulate new metrics for action-generalizability and VLM-comprehensibility, demonstrating the compelling performance of SEAM over mainstream representations on both aspects. Extensive real-world experiments further manifest its SOTA performance under varying settings and tasks.
Read the full abstract ▾
All authors Weiliang Tang, Jialin Gao, Jia-Hui Pan, Gang Wang, Li Erran Li, Yunhui Liu, Mingyu Ding, Pheng-Ann Heng, Chi-Wing Fu
Robotics
SENTINEL: A Fully End-to-End Language-Action Model for Humanoid Whole Body Control
Yuxuan Wang, Haobin Jiang, Shiqing Yao et al. (+2 more)
Existing
humanoid control systems often rely on teleoperation or modular generation pipelines that separate
language understanding from physical execution. However, the former is entirely human-driven, and the latter lacks tight
alignment between
language commands and physical behaviors. In this paper, we present SENTINEL, a fully end-to-end
language-action model for
humanoid whole-body control. We construct a large-scale
dataset by tracking human motions in simulation using a pretrained whole body controller, combined with their text annotations. The model directly maps
language commands and proprioceptive inputs to low-level actions without any intermediate representation. The model generates action chunks using flow matching, which can be subsequently refined by a residual action head for real-world deployment. Our method exhibits strong semantic understanding and stable execution on
humanoid robots in both simulation and real-world deployment, and also supports multi-modal extensions by converting inputs into texts.
Read the full abstract ▾
All authors Yuxuan Wang, Haobin Jiang, Shiqing Yao, Ziluo Ding, Zongqing Lu
Robotics · Cross-listed from cs.CV
Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving
Jianhua Han, Meng Tian, Jiangtong Zhu et al. (+16 more)
Autonomous driving heavily relies on accurate and robust spatial perception. Many failures arise from inaccuracies and instability, especially in long-tail scenarios and complex interactions. However, current
vision-
language models are weak at spatial grounding and understanding, and VLA systems built on them therefore show limited perception and localization ability. To address these challenges, we introduce Percept-WAM, a perception-enhanced World-Awareness-Action Model that is the first to implicitly integrate 2D/3D scene understanding abilities within a single
vision-
language model (VLM). Instead of relying on QA-style spatial
reasoning, Percept-WAM unifies 2D/3D perception tasks into World-PV and World-BEV
tokens, which encode both spatial coordinates and confidence. We propose a grid-conditioned prediction mechanism for dense object perception, incorporating IoU-aware scoring and parallel autoregressive decoding, improving stability in long-tail, far-range, and small-object scenarios. Additionally, Percept-WAM leverages pretrained VLM parameters to retain general intelligence (e.g., logical
reasoning) and can output perception results and trajectory control outputs directly. Experiments show that Percept-WAM matches or surpasses classical detectors and segmenters on downstream perception
benchmarks, achieving 51.7/58.9 mAP on COCO 2D detection and nuScenes BEV 3D detection. When integrated with trajectory decoders, it further improves planning performance on nuScenes and NAVSIM, e.g., surpassing DiffusionDrive by 2.1 in PMDS on NAVSIM. Qualitative results further highlight its strong open-vocabulary and long-tail generalization.
Read the full abstract ▾
All authors Jianhua Han, Meng Tian, Jiangtong Zhu, Fan He, Huixin Zhang, Sitong Guo, Dechang Zhu, Hao Tang, Pei Xu, Yuze Guo, Minzhe Niu, Haojie Zhu, Qichao Dong, Xuechao Yan, Siyuan Dong, Lu Hou, Qingqiu Huang, Xiaosong Jia, Hang Xu
Robotics
Soft Pneumatic Grippers: Topology optimization, 3D-printing and Experimental validation
Prabhat Kumar, Chandra Prakash, Josh Pinskier et al. (+2 more)
Typically, heuristic/trial-based approaches are used to design soft pneumatic grippers (SPGs). This paper presents a systematic
topology optimization framework for developing SPGs. The design-dependent nature of actuating load is modeled using Darcy's law with an added drainage term. A 2D soft arm unit is then optimized as a compliant mechanism under pneumatic loading. To ensure the design is robust and manufacturable, the problem is formulated as a min-max optimization, where output deformations of blueprint and eroded designs are considered. A volume constraint is imposed on the blueprint part, while a strain-energy constraint is enforced on the eroded part. The Method of Moving Asymptotes is employed to solve optimization problems. The optimized 2D part is extruded suitably to generate a 3D unit. Ten such 3D units are assembled to create a gripper arm. Both the optimized 2D unit and the corresponding gripper arm outperform their conventional rectangular designs under pneumatic loading, demonstrating the efficacy of the proposed approach. The arms are fabricated using the SLA printing technique. Numerical and experimental results are compared at different pneumatic loads. Four 3D-printed arms are integrated with a supporting structure to form the SPG. The gripping action of the SPG is demonstrated on objects with different weights, sizes, structures, stiffnesses, and shapes.
Read the full abstract ▾
All authors Prabhat Kumar, Chandra Prakash, Josh Pinskier, David Howard, Matthijs Langelaar
Robotics
Reference-Free Sampling-Based Model Predictive Control
Fabian Schramm, Pierre Fabre, Nicolas Perrin-Gilbert et al. (+1 more)
We present a sampling-based model predictive control (MPC) framework that enables emergent locomotion without relying on handcrafted gait patterns or predefined contact sequences. Our method discovers diverse motion patterns, ranging from trotting to galloping, robust standing policies, jumping, and handstand balancing, purely through the optimization of high-level objectives. Building on model predictive path integral (MPPI), we propose a cubic Hermite spline parameterization that operates on position and velocity control points. Our approach enables contact-making and contact-breaking strategies that adapt automatically to task requirements, requiring only a limited number of sampled trajectories. This sample efficiency enables real-time control on standard CPU hardware, eliminating the GPU acceleration typically required by other state-of-the-art MPPI methods. We validate our approach on the Go2 quadrupedal
robot, demonstrating a range of emergent gaits and basic jumping capabilities. In simulation, we further showcase more complex behaviors, such as backflips, dynamic handstand balancing and locomotion on a
Humanoid, all without requiring reference tracking or offline pre-training.
Read the full abstract ▾
All authors Fabian Schramm, Pierre Fabre, Nicolas Perrin-Gilbert, Justin Carpentier
Robotics
Efficient Optimization of a Permanent Magnet Array for a Stable 2D Trap
Ann-Sophia Müller, Moonkwang Jeong, Jiyuan Tian et al. (+2 more)
Untethered
magnetic manipulation of biomedical millirobots has a high potential for minimally invasive
surgical applications. However, it is still challenging to exert high actuation forces on the small
robots over a large distance. Permanent magnets offer stronger
magnetic torques and forces than electromagnetic coils, however, feedback control is more difficult. As proven by Earnshaw's
theorem, it is not possible to achieve a stable
magnetic trap in 3D by static permanent magnets. Here, we report a stable 2D
magnetic force trap by an array of permanent magnets to control a millirobot. The trap is located in an open space with a tunable distance to the magnet array in the range of 20 - 120mm, which is relevant to human anatomical scales. The design is achieved by a novel GPU-accelerated optimization
algorithm that uses mean squared error (MSE) and Adam optimizer to efficiently compute the optimal angles for any number of magnets in the array. The
algorithm is verified using numerical simulation and physical experiments with an array of two magnets. A millirobot is successfully trapped and controlled to follow a complex trajectory. The
algorithm demonstrates high scalability by optimizing the angles for 100 magnets in under three seconds. Moreover, the optimization workflow can be adapted to optimize a permanent magnet array to achieve the desired force vector fields.
Read the full abstract ▾
All authors Ann-Sophia Müller, Moonkwang Jeong, Jiyuan Tian, Meng Zhang, Tian Qiu
Robotics · Cross-listed from cs.CV
Three-Dimensional Anatomical Data Generation Based on Artificial Neural Networks
Ann-Sophia Müller, Moonkwang Jeong, Meng Zhang et al. (+4 more)
Surgical planning and training based on machine learning requires a large amount of 3D anatomical models reconstructed from medical imaging, which is currently one of the major bottlenecks. Obtaining these data from real patients and during surgery is very demanding, if even possible, due to legal, ethical, and technical challenges. It is especially difficult for soft tissue organs with poor imaging contrast, such as the prostate. To overcome these challenges, we present a novel workflow for automated 3D anatomical data generation using data obtained from physical organ models. We additionally use a 3D Generative
Adversarial Network (GAN) to obtain a
manifold of 3D models useful for other downstream machine learning tasks that rely on 3D data. We demonstrate our workflow using an artificial prostate model made of biomimetic hydrogels with imaging contrast in multiple zones. This is used to physically simulate endoscopic surgery. For evaluation and 3D data generation, we place it into a customized ultrasound scanner that records the prostate before and after the procedure. A neural network is trained to segment the recorded ultrasound images, which outperforms conventional, non-learning-based computer
vision techniques in terms of intersection over union (IoU). Based on the
segmentations, a 3D mesh model is reconstructed, and performance feedback is provided.
Read the full abstract ▾
All authors Ann-Sophia Müller, Moonkwang Jeong, Meng Zhang, Jiyuan Tian, Arkadiusz Miernik, Stefanie Speidel, Tian Qiu
Robotics · Cross-listed from cs.LG
First-order Sobolev Reinforcement Learning
Fabian Schramm, Nicolas Perrin-Gilbert and Justin Carpentier
We propose a refinement of temporal-difference learning that enforces first-order Bellman consistency: the learned value function is trained to match not only the Bellman targets in value but also their derivatives with respect to states and actions. By differentiating the Bellman backup through differentiable dynamics, we obtain analytically consistent gradient targets. Incorporating these into the critic objective using a Sobolev-type loss encourages the critic to align with both the value and local geometry of the target function. This first-order TD matching principle can be seamlessly integrated into existing
algorithms, such as Q-learning or actor-critic methods (e.g., DDPG, SAC), potentially leading to faster critic convergence and more stable policy gradients without altering their overall structure.
Read the full abstract ▾
Robotics
Autonomous Docking of Multi-Rotor UAVs on Blimps under the Influence of Wind Gusts
Pascal Goldschmid and Aamir Ahmad
Multi-rotor
UAVs face limited flight time due to battery constraints. Autonomous docking on blimps with onboard battery recharging and data offloading offers a promising solution for extended
UAV missions. However, the vulnerability of blimps to wind gusts causes trajectory deviations, requiring precise, obstacle-aware docking strategies. To this end, this work introduces two key novelties: (i) a temporal convolutional network that predicts blimp responses to wind gusts, enabling rapid gust detection and estimation of points where the wind gust effect has subsided; (ii) a model predictive controller (MPC) that leverages these predictions to compute collision-free trajectories for docking, enabled by a novel obstacle avoidance method for close-range manoeuvres near the blimp. Simulation results show our method outperforms a baseline constant-velocity model of the blimp significantly across different scenarios. We further validate the approach in real-world experiments, demonstrating the first autonomous multi-rotor docking control strategy on blimps shown outside simulation. Source code is available here https://github.com/
robot-perception-group/multi_rotor_airship_docking.
Read the full abstract ▾
Robotics
Analysis of Deep-Learning Methods in an ISO/TS 15066-Compliant Human-Robot Safety Framework
David Bricher and Andreas Mueller
Over the last years collaborative
robots have gained great success in manufacturing applications where human and
robot work together in close proximity. However, current ISO/TS-15066-compliant implementations often limit the efficiency of collaborative tasks due to conservative speed restrictions. For this reason, this paper introduces a deep-learning-based human-
robot-safety framework (HRSF) that aims at a dynamical adaptation of
robot velocities depending on the separation distance between human and
robot while respecting maximum biomechanical force and pressure limits. The applicability of the framework was investigated for four different deep learning approaches that can be used for human body extraction: human body recognition, human body
segmentation, human pose estimation, and human body part
segmentation. Unlike conventional industrial safety systems, the proposed HRSF differentiates individual human body parts from other objects, enabling optimized
robot process execution. Experiments demonstrated a quantitative reduction in cycle time of up to 15% compared to conventional safety technology.
Read the full abstract ▾
Robotics
Multi-Agent Monocular Dense SLAM With 3D Reconstruction Priors
Yuchen Zhou and Haihang Wu
Monocular Simultaneous Localization and Mapping (SLAM) aims to estimate a
robot's pose while simultaneously reconstructing an unknown 3D scene using a single camera. While existing
monocular SLAM systems generate detailed 3D geometry through dense scene representations, they are computationally expensive due to the need for iterative optimization. To address this challenge, MASt3R-SLAM utilizes learned 3D reconstruction priors, enabling more efficient and accurate estimation of both 3D structures and camera poses. However, MASt3R-SLAM is limited to single-
agent operation. In this paper, we extend MASt3R-SLAM to introduce the first multi-
agent monocular dense SLAM system. Each
agent performs local SLAM using a 3D reconstruction prior, and their individual maps are fused into a globally consistent map through a loop-closure-based map fusion mechanism. Our approach improves computational efficiency compared to state-of-the-art methods, while maintaining similar mapping accuracy when evaluated on real-world
datasets.
Read the full abstract ▾
Robotics
End-to-end Autonomous Vehicle Following System using Monocular Fisheye Camera
Jiale Zhang, Yeqiang Qian, Tong Qin et al. (+3 more)
The increase in vehicle ownership has led to increased traffic congestion, more accidents, and higher carbon emissions. Vehicle platooning is a promising solution to address these issues by improving road capacity and reducing fuel consumption. However, existing platooning systems face challenges such as reliance on lane markings and expensive high-precision sensors, which limits their general applicability. To address these issues, we propose a vehicle following framework that expands its capability from restricted scenarios to general scenario applications using only a camera. This is achieved through our newly proposed end-to-end method, which improves overall driving performance. The method incorporates a semantic mask to address causal confusion in multi-frame data fusion. Additionally, we introduce a dynamic sampling mechanism to precisely track the trajectories of preceding vehicles. Extensive closed-loop validation in real-world vehicle experiments demonstrates the system's ability to follow vehicles in various scenarios, outperforming traditional multi-stage
algorithms. This makes it a promising solution for cost-effective autonomous vehicle platooning. A complete real-world vehicle experiment is available at https://youtu.be/zL1bcVb9kqQ.
Read the full abstract ▾
All authors Jiale Zhang, Yeqiang Qian, Tong Qin, Mingyang Jiang, Siyuan Chen, Ming Yang
Robotics · Cross-listed from cs.LG
AVA-VLA: Improving Vision-Language-Action models with Active Visual Attention
Lei Xiao, Jifeng Li, Juntao Gao et al. (+6 more)
Vision-
Language-Action (VLA) models have shown remarkable progress in
embodied tasks recently, but most methods process visual observations independently at each timestep. This history-agnostic design treats
robot manipulation as a Markov Decision Process, even though real-world robotic control is inherently partially observable and requires
reasoning over past interactions. To address this mismatch, we reformulate VLA policy learning from a Partially Observable Markov Decision Process perspective and propose AVA-VLA, a framework that conditions action generation on a recurrent state that serves as a neural approximation to the
agent's belief over task history. Built on this recurrent state, we introduce Active Visual
Attention (AVA), which dynamically reweights visual
tokens in the current observation to focus on regions most relevant given both the instruction and execution history. Extensive experiments show that AVA-VLA achieves state-of-the-art performance on standard robotic
benchmarks, including LIBERO and CALVIN, and transfers effectively to real-world dual-arm manipulation tasks. These results demonstrate the effectiveness of temporally grounded active visual processing for improving VLA performance in robotic sequential decision-making. The project page is available at https://liauto-dsr.github.io/AVA-VLA-Page.
Read the full abstract ▾
All authors Lei Xiao, Jifeng Li, Juntao Gao, Feiyang Ye, Yan Jin, Jingjing Qian, Jing Zhang, Yong Wu, Xiaoyuan Yu
Robotics
Compressor-VLA: Instruction-Guided Visual Token Compression for Efficient Robotic Manipulation
Juntao Gao, Feiyang Ye, Jing Zhang et al. (+1 more)
Vision-
Language-Action (VLA) models have emerged as a powerful paradigm in
Embodied AI. However, the significant computational overhead of processing redundant visual
tokens remains a critical bottleneck for real-time robotic deployment. While standard
token pruning techniques can alleviate this, these task-agnostic methods struggle to preserve task-critical visual information. To address this challenge, simultaneously preserving both the holistic context and fine-grained details for precise action, we propose Compressor-VLA, a novel hybrid instruction-conditioned
token compression framework designed for efficient, task-oriented compression of visual information in VLA models. The proposed Compressor-VLA framework consists of two
token compression modules: a Semantic Task Compressor (STC) that distills holistic, task-relevant context, and a Spatial Refinement Compressor (SRC) that preserves fine-grained spatial details. This compression is dynamically modulated by the natural
language instruction, allowing for the adaptive condensation of task-relevant visual information. Experimentally, extensive evaluations demonstrate that Compressor-VLA achieves a competitive success rate on the LIBERO
benchmark while reducing FLOPs by 59% and the visual
token count by over 3x compared to its baseline. The real-
robot deployments on a dual-arm
robot platform validate the model's sim-to-real transferability and practical applicability. Moreover, qualitative analyses reveal that our instruction guidance dynamically steers the model's perceptual focus toward task-relevant objects, thereby validating the effectiveness of our approach.
Read the full abstract ▾
All authors Juntao Gao, Feiyang Ye, Jing Zhang, Wenjing Qian
Robotics
An Efficient Closed-Form Solution to Full Visual-Inertial State Initialization
Samuel Cerezo, Seong Hun Lee and Javier Civera
In this letter, we present a closed-form initialization method that recovers the full visual-inertial state without nonlinear optimization. Unlike previous approaches that rely on iterative solvers, our formulation yields analytical, easy-to-implement, and numerically stable solutions for reliable start-up. Our method builds on small-rotation and constant-velocity approximations, which keep the formulation compact while preserving the essential coupling between motion and inertial measurements. We further propose an observability-driven, two-stage initialization scheme that balances accuracy with initialization latency. Extensive experiments on the EuRoC
dataset validate our assumptions: our method achieves 10-20% lower initialization error than optimization-based approaches, while using 4x shorter initialization windows and reducing computational cost by 5x.
Read the full abstract ▾
Robotics
Accelerating Reinforcement Learning via Error-Related Human Brain Signals
Suzie Kim, Hye-Bin Shin and Hyo-Jeong Jang
In this work, we investigate how implicit neural feed back can accelerate reinforcement learning in complex robotic manipulation settings. While prior electroencephalogram (
EEG) guided reinforcement learning studies have primarily focused on
navigation or low-dimensional locomotion tasks, we aim to understand whether such neural evaluative signals can improve policy learning in high-dimensional manipulation tasks involving obstacles and precise end-effector control. We integrate error related potentials decoded from offline-trained
EEG classifiers into reward shaping and systematically evaluate the impact of human-feedback weighting. Experiments on a 7-DoF manipulator in an obstacle-rich reaching environment show that neural feedback accelerates reinforcement learning and, depending on the human-feedback weighting, can yield task success rates that at times exceed those of sparse-reward baselines. Moreover, when applying the best-performing feedback weighting across all sub jects, we observe consistent acceleration of reinforcement learning relative to the sparse-reward setting. Furthermore, leave-one subject-out evaluations confirm that the proposed framework remains robust despite the intrinsic inter-individual variability in
EEG decodability. Our findings demonstrate that
EEG-based reinforcement learning can scale beyond locomotion tasks and provide a viable pathway for human-aligned manipulation skill acquisition.
Read the full abstract ▾
Robotics · Cross-listed from cs.AI
GContextFormer: A global context-aware hybrid multi-head attention approach with scaled additive aggregation for multimodal trajectory prediction
Yuzhi Chen, Yuanchang Xie, Lei Zhao et al. (+3 more)
Multimodal trajectory prediction generates multiple plausible future trajectories to address vehicle motion uncertainty from intention ambiguity and execution variability. However, HD map-dependent models suffer from costly data acquisition, delayed updates, and vulnerability to corrupted inputs, causing prediction failures. Map-free approaches lack global context, with pairwise
attention over-amplifying straight patterns while suppressing transitional patterns, resulting in motion-intention misalignment. This paper proposes GContextFormer, a plug-and-play encoder-decoder architecture with global context-aware hybrid
attention and scaled additive aggregation achieving intention-aligned
multimodal prediction without map reliance. The Motion-Aware Encoder builds scene-level intention prior via bounded scaled additive aggregation over mode-embedded trajectory
tokens and refines per-mode representations under shared global context, mitigating inter-mode suppression and promoting intention
alignment. The Hierarchical Interaction Decoder decomposes social
reasoning into dual-pathway cross-
attention: a standard pathway ensures uniform
geometric coverage over
agent-mode pairs while a neighbor-context-enhanced pathway emphasizes salient interactions, with gating module mediating their contributions to maintain coverage-focus balance. Experiments on eight highway-ramp scenarios from TOD-VT
dataset show GContextFormer outperforms state-of-the-art baselines. Compared to existing
transformer models, GContextFormer achieves greater robustness and concentrated improvements in high-
curvature and transition zones via spatial distributions. Interpretability is achieved through motion mode distinctions and neighbor context modulation exposing
reasoning attribution. The modular architecture supports extensibility toward cross-domain
multimodal reasoning tasks. Source: https://fenghy-chen.github.io/sources/.
Read the full abstract ▾
All authors Yuzhi Chen, Yuanchang Xie, Lei Zhao, Pan Liu, Yajie Zou, Chen Wang
Robotics
AutoOdom: Learning Auto-regressive Proprioceptive Odometry for Legged Locomotion
Changsheng Luo, Yushi Wang, Wenhan Cai et al. (+1 more)
Accurate proprioceptive odometry is fundamental for legged
robot navigation in GPS-denied and visually degraded environments where conventional visual odometry systems fail. Current approaches face critical limitations: analytical filtering methods suffer from modeling uncertainties and cumulative drift, hybrid learning-filtering approaches remain constrained by their analytical components, while pure learning-based methods struggle with simulation-to-reality transfer and demand extensive real-world data collection. This paper introduces AutoOdom, a novel autoregressive proprioceptive odometry system that overcomes these challenges through an innovative two-stage training paradigm. Stage 1 employs large-scale simulation data to learn complex nonlinear dynamics and rapidly changing contact states inherent in legged locomotion, while Stage 2 introduces an autoregressive enhancement mechanism using limited real-world data to effectively bridge the sim-to-real gap. The key innovation lies in our autoregressive training approach, where the model learns from its own predictions to develop resilience against sensor noise and improve robustness in highly dynamic environments. Comprehensive experimental validation on the Booster T1
humanoid robot demonstrates that AutoOdom significantly outperforms state-of-the-art methods across all evaluation metrics, achieving 57.2% improvement in absolute trajectory error, 59.2% improvement in Umeyama-aligned error, and 36.2% improvement in relative pose error compared to the Legolas baseline. Extensive ablation studies provide critical insights into sensor modality selection and temporal modeling, revealing counterintuitive findings about IMU acceleration data and validating our systematic design choices for robust proprioceptive odometry in challenging locomotion scenarios.
Read the full abstract ▾
All authors Changsheng Luo, Yushi Wang, Wenhan Cai, Mingguo Zhao