Robotics
An offline approach to fNIRS-guided reinforcement learning for robot behavior
Julia Santaniello, Madelaine Brower, Benson Jiang et al. (+4 more)
Human-in-the-loop Reinforcement Learning has become a popular approach for training, finetuning, and aligning
robot behavior with user preferences. Our paper explores the feasibility of using
brain signals via functional near-infrared spectroscopy (fNIRS) to modulate
robot learning in simulation. We compare
agents trained on passive (observational) versus active (demonstrative) interaction tasks, and test multiple methods for enhancing the RL
algorithm with the neural signal, focusing on parameter augmentation in contrast to replacement. We further examine how model granularity and noise affect
agent learning. Our results show that this framework is effective. The neural signal improves learning when augmenting trajectory priorities and state-action q-targets. Additionally, the framework learns successfully from offline data, offering a practical alternative for settings where real-time BCI setups are impractical or only limited data is available.
Read the full abstract ▾
All authors Julia Santaniello, Madelaine Brower, Benson Jiang, Donatello Sassaroli, Chenyuan Zhang, Robert Jacob, Jivko Sinapov
Robotics · Cross-listed from cs.AI
Chat2Scenic: An Iterative RAG-Based Framework for Scenario Generation in Autonomous Driving
Yuan Gao, Wenting Miao, Mattia Piccinini et al. (+3 more)
Validating autonomous driving systems requires diverse, regulation-compliant test scenarios. In simulation-based testing, scenarios are defined as executable scripts. Yet automatically generating such scripts from regulatory descriptions remains an open challenge, and existing approaches face fundamental trade-offs.
Retrieval-assemble methods achieve reasonable compilation rates but lack scalability, whereas
retrieval-based full-script generation suffers from low compilation success rates. We present Chat2Scenic, the first iterative
retrieval-augmented framework to generate scenario scripts in Domain Specific
Language (DSL). Specifically, Chat2Scenic provides a chatbot interface that supports interactive scenario refinement and integrates
Retrieval-augmented Generation (RAG) to ground scenario generation in regulatory knowledge and DSL syntax. Furthermore, we propose an open
benchmark for scenario generation comprising 123 scenarios from various regulations, including NHTSA and United Nations Vehicle Regulations, as well as other sources. Extensive evaluation with State-of-the-Art (SOTA) Large
Language Models (LLMs) demonstrates that Chat2Scenic achieves 76.42% Compilation Success Rate (CSR) and 58.17% Framework Accuracy (FA), outperforming existing methods (
Retrieval Assemble with 30.08% CSR, 11.03% FA and
Retrieval full script generation with 16.26% CSR, 10.86% FA). To facilitate future research, we release our code as open source at https://github.com/TUM-AVS/chat2scenic.
Read the full abstract ▾
All authors Yuan Gao, Wenting Miao, Mattia Piccinini, Haoyu Wang, Qunying Song, Johannes Betz
Robotics
Beyond Visual Grasping: Benchmarking Complex Grasping from Detection to Execution
Hanyi Zhang, Khang Nguyen, Charith Munasinghe et al. (+10 more)
Robust robotic grasping remains a fundamental challenge for complex real-world applications. Recent advances in large-scale models demonstrate promising capabilities for
reasoning in robotic tasks. However, existing
benchmarks for grasping primarily focus on isolated, visual-based grasp pose detection, failing to capture the complexity of grasping tasks that require multi-step
reasoning and semantic understanding during execution. To address this gap, we propose GCA-Bench, a
benchmark featuring challenging grasping with complex action scenarios that involve both scene-level
reasoning and semantic constraints. GCA-Bench enables the evaluation of recent large foundation models under the same settings. To demonstrate the effectiveness of our new
benchmark, we implement a diverse set of baselines, ranging from traditional grasp detection pipelines to end-to-end learning methods. Empirical studies achieve success rates below 70% on complex grasping scenarios, underscoring critical limitations. In addition, we propose new evaluation metrics, analyze critical failure models, and provide insights to guide the development of more robust and generalizable grasping strategies.
Read the full abstract ▾
All authors Hanyi Zhang, Khang Nguyen, Charith Munasinghe, Basu Hela, Tianyu Li, Zihong Luo, Hoan Nguyen, Hans Wernher van de Venn, Yalin Zheng, Ravi Prakash, Tung D. Ta, Anh Nguyen, Baoru Huang
Robotics
DiMaS: Distribution Matching for Steering Vision-Language-Action Models
Pegah Khayatan, Sara Meziane, Jayneel Parekh et al. (+1 more)
Flow-matching-based
vision-
language-action (VLA) models have emerged as powerful policies for robotic manipulation, yet a critical capability remains underexplored: fine-grained behavioral control, the ability to govern how a
robot performs a task by intervening on its internal representations. Representation steering is a well-established interpretability tool for
language and
vision-
language models, where behavioral features are typically encoded as linear directions, but we show that these classic methods fall short in VLAs. We propose DiMaS, a Distribution-Matching Steering strategy tailored to flow-matching VLAs, which transports between representation distributions rather than shifting along a fixed direction, and show that it effectively controls behavior across two state-of-the-art VLAs. We further examine the generalizability of this strategy as the tasks it is learned from and evaluated on grow increasingly dissimilar, characterizing where behavioral control transfers and where it weakens. Finally, through an analysis of the representation structure of the action expert, we explain why classical linear steering falls short in the visuomotor setting: behavioral features are linearly decodable but not linearly steerable, which motivates the distribution-matching design of DiMaS. Our code is publicly available at https://github.com/pegah-kh/dimas, with additional results and
videos at https://pegah-kh.github.io/dimas/
Read the full abstract ▾
All authors Pegah Khayatan, Sara Meziane, Jayneel Parekh, Matthieu Cord
Robotics
Stochastic Filtering for Quorum Sensing in Robot Swarms under Anonymous Communication
Fabio Oddi, Andreagiovanni Reina and Vito Trianni
Quorum Sensing (QS) is a key capability for
robot swarms, useful for coordination of activities at the group level. Effective communication is instrumental for individuals to estimate the quorum level of the entire swarm. Anonymous communication protocols where individuals exchange local information without revealing unique identities are helpful to support quorum estimates by sampling information from neighbours and maintain scalability of the QS process. However, because anonymous protocols cannot distinguish message sources, repeated messages from the same sender may be double-counted, thereby biasing collective quorum estimates. In this study, we introduce a
stochastic filtering protocol inspired by $k$-priority sampling to improve estimate stability (\ANTk), and we compare it with a baseline anonymous protocols (\AN) and a randomised variant designed to improve accuracy (\ANT). We find that the baseline protocol \AN provides a parsimonious and fast solution, but remains highly inaccurate due to double-counting bias. The \ANT variant improves accuracy but suffers from information inertia, resulting in slower convergence. Finally, actively filtering the message buffer via the \ANTk protocol successfully decreases temporary errors and stabilises the estimate, at the cost of an increased time of recovery from errors.
Read the full abstract ▾
Robotics
Zihao Yu, Xiu Yuan and Chongjie Zhang
Embodied agents accumulate experience over time. We study how accumulated experience can be formed into persistent
memory for future
reasoning and action. We formulate
Embodied Action
Memory (EAM) as the capability to form and use
memory over
embodied experience, together with the persistent
memory state produced by that process. We introduce MEMORA, a framework that instantiates EAM through a formation-consolidation-
retrieval lifecycle and a multi-store world-
memory architecture. MEMORA organizes experience into participant-specific Environment, Entity, Activity, and Inferred Knowledge stores: online editing revises
memory as new evidence arrives, while offline consolidation abstracts repeated experience into reusable routines, habits, and preferences. We evaluate MEMORA with MEMORA-Bench, a 45-hour
egocentric-
video suite that measures both retrospective
memory faithfulness and prospective
memory-grounded planning. Across four open-weight answer models, MEMORA achieves the strongest aggregate planning performance among the evaluated
memory interfaces, with its largest gains on out-of-distribution planning. On these tasks, MEMORA improves
Robot-Grounded Plan score by up to 16.6 percent, suggesting that
memory formed and consolidated across experience can support planning for new goals beyond directly observed episodes. A physical-
robot demonstration further shows that
memory formed solely from human
egocentric video can ground high-level
robot plans in participant-specific objects and preferences. Project website: https://github.com/yuzihaowashu/MEMORA
Read the full abstract ▾
Robotics · Cross-listed from eess.SY
Information-Theoretic Adaptive Cooling for Deterministic MPPI via Entropy Feedback
Shuqi Wang, Wenrong Sun, Tao Han et al. (+2 more)
This paper investigates deterministic optimal control using Model Predictive Path Integral (MPPI) control, a sampling-based and derivative-free framework well suited for systems with complex dynamics and nonsmooth objectives. In deterministic MPPI, the temperature must be driven to zero to recover the true optimum, yet the design of an effective cooling schedule remains a fundamental challenge. Existing methods typically rely on predefined open-loop schedules, which limit the efficiency and robustness of the
algorithm. To overcome this limitation, we propose an Information-Theoretic Adaptive Cooling (ITAC) framework that uses the Shannon
entropy of the importance weights as an online feedback signal to regulate the temperature. The proposed mechanism adapts the cooling rate to the current sampling state, enabling fast progress when the weights are diffuse and cautious cooling when they become concentrated. We prove asymptotic convergence of the resulting scheme to the deterministic optimum, and further derive a critical
entropy threshold that leads to a smooth barrier against premature weight collapse. Experiments on nonsmooth signal temporal logic motion-planning tasks show that ITAC improves sampling efficiency and achieves substantially faster convergence than state-of-the-art baselines without sacrificing the derivative-free nature of MPPI.
Read the full abstract ▾
All authors Shuqi Wang, Wenrong Sun, Tao Han, Yue Gao, Xiang Yin
Robotics
Never Too Late for Force: Accelerating VLA Post-Training with Reactive Force Injection
Yi Wang, Wendi Chen, Zimo Wen et al. (+8 more)
Pretrained
vision-
language-action (VLA) policies provide strong
language-conditioned manipulation knowledge, but they remain largely
vision-driven and can struggle once manipulation enters contact states where the scene is occluded, depth is ambiguous, or small force errors push execution off the offline demonstration distribution. We present LIFT (Late Reactive Injection of Force for VLA Post-Training), a force-aware post-training framework that adds contact reactivity to a pretrained VLA policy while preserving its output at initialization. LIFT grafts a reactive action expert beside the original action expert, initializes it from pretrained action weights, and injects recent 6D end-effector force through causal force
memory and zero-initialized cross
attention, enabling actions to be refreshed during execution. To address the policy-dependent distribution shift of contact feedback, LIFT further couples reactive force injection with an online DAgger loop that trains on a mixture of offline task-
alignment data and human-corrected online rollouts. Across towel folding, book insertion, and Hanoi ring placement, LIFT learns faster and reaches higher peak and final performance than
vision-only DAgger, while ablations show that reactive force
memory and online corrective data are both important for robust contact-rich manipulation. Our code is publicly available at https://github.com/y-wng/lift
Read the full abstract ▾
All authors Yi Wang, Wendi Chen, Zimo Wen, Han Xue, Xueqi Li, Wenye Yu, Zhijie Chen, Hao Yang, Jun Lv, Chuan Wen, Cewu Lu
Robotics · Cross-listed from cs.AI
RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination
Haotian Liang, Mingkang Chen, Yufei Huang et al. (+27 more)
Embodied cognition requires
agents to connect high-level task
reasoning with the physical states to be achieved. We introduce Hy-
Embodied-RxBrain, an
embodied cognition foundation model with joint
language-visual
reasoning and imagination. Unlike
vision-
language models that emphasize scene understanding and textual decision making, or generative world models that mainly predict future visual states, RxBrain represents
embodied plans in a single planning sequence where
language and visual imagination play complementary roles.
Language provides the abstract structure of a plan, including task decomposition, planning primitives, constraints, temporal order, and decision logic, while visual imagination grounds this structure through world state prediction and joint subgoal planning, associating each planning step with intermediate and final physical states. RxBrain adopts a unified
multimodal Mixture-of-
Transformers architecture that supports
language, image, and
video understanding and generation within one model. To train this capability, we build an automatic pipeline that converts
embodied videos into joint text-visual planning supervision by decomposing
videos into planning steps and aligning them with visual state transitions. We further introduce RxBrain-Bench to evaluate whether models can represent
embodied plans through joint textual and visual components rather than separate understanding or generation. Experiments show that RxBrain maintains
embodied understanding and generation abilities, and produces plans with coupled textual
reasoning, world state prediction, and joint subgoal planning. We also extend RxBrain to continuous
robot action generation, where it shows promising real-
robot performance without large-scale action-data pretraining. These results provide an initial step toward foundation models for
embodied cognition.
Read the full abstract ▾
All authors Haotian Liang, Mingkang Chen, Yufei Huang, Yuchun Guo, Xiaomeng Zhu, Xiangli Shi, Kaixuan Wang, Yunxuan Mao, Weijie Zhou, Ling Chen, Shirong Zeng, Yueyu Long, Yuchen Si, Yajuan Zhu, Xingyu Zhou, Minghui Wang, Wanjia He, Xin Yang, Lingzhu Xiang, Zhiqing Liu, Bohan Ma, Xiran Huang, Tianshuo Yang, Zhiheng Liu, Xuantang Xiong, Zisheng Lu, Ping Luo, Yao Mu, Han Hu, Zhengyou Zhang
Robotics
Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learning
Zishuo Li, Bowen Yang, Changtao Miao et al. (+29 more)
Egocentric videos of human manipulation provide scalable supervision for
embodied intelligence, yet existing resources rarely combine low-cost continuous capture, manipulation-level structured annotations, and reusable tools for
robot learning. We present Open-AoE, an open, community-oriented
egocentric manipulation
dataset and toolchain spanning the full pipeline from smartphone capture to model training. Its first release contains approximately 2,000 hours of manipulation
video collected in natural environments by 500+ contributors using 400+ smartphones. The
dataset provides text annotations, MANO-based hand poses, camera trajectories, and temporally localized atomic actions. Open-AoE further includes a data processing pipeline that transforms raw recordings into structured samples through temporal action
segmentation, semantic annotation, hand reconstruction, and camera trajectory reconstruction. Meanwhile, we provide a separate downstream toolchain supports visualization, cross-embodiment retargeting, model-specific data conversion, and training recipes for VLA policies, WAMs, and World Models. By integrating scalable capture, structured processing, and downstream adaptation, Open-AoE reduces the barriers to both data contribution and reuse, providing practical open infrastructure for
embodied model training, human-to-
robot transfer, and world modeling.
Read the full abstract ▾
All authors Zishuo Li, Bowen Yang, Changtao Miao, Kai Zhu, Hao Chen, Qingze Guan, Zhengxing Wu, Wanke Zhan, Yang Sun, Zhiyi Huang, Zitong Shan, Zhenchao Jin, Jiadong Hong, Taowen Wang, Yushi Feng, You Liu, Yibo Wang, Yifan Yang, Zhaowen Zhou, Man Luo, Hao Cheng, Bo Zhang, Jianshu Li, Jiansheng Cai, Guocai Yao, Jize Zhang, Chenhao Lin, Renjing Xu, Lequan Yu, Chao Shen, Chunhua Shen, Zhe Li
Robotics
Semantic Audio-driven Understanding for Dynamic Humanoid Whole Body Control
J. M. A. Marcelo, M. Brienza, E. Bugli et al. (+4 more)
Recent advances in
humanoid robotics and reinforcement learning have enabled the acquisition of highly expressive whole-body motion policies. However, most robotic performances remain based on pre-scripted sequences or externally triggered behaviors, limiting autonomy and responsiveness to dynamic environments. In this work, we introduce a novel multi-modal orchestration framework for semantic audio-driven
humanoid control, enabling
robots to autonomously select and execute appropriate motion skills in real time. The system processes continuous audio streams and routes them into music or
speech branches. Music input is handled via audio fingerprinting and semantic
embeddings to retrieve track identity and temporal
alignment, allowing dynamic mapping between musical segments and motion policies.
Speech input is grounded into a discrete library of imitation-learned skills, enabling direct human-
robot interaction. Both modalities share a unified interface that schedules skill execution over a reinforcement learning control pipeline. We validate the approach in simulation and on a Unitree G1
humanoid, showing robust sim-to-real transfer and consistent audio-conditioned policy selection. Supplementary materials are available at the following site: https://lab-rococo-sapienza.github.io/semantic-WBC/
Read the full abstract ▾
All authors J. M. A. Marcelo, M. Brienza, E. Bugli, L. Comito, D. Nardi, D. D. Bloisi, V. Suriani
Robotics
Adaptive Control of Motor-Position-Controlled Flexible Joint Robots with Uncertain Joint Stiffness
Annika Kirner, Grazia Zambella, Igor Kovacevic et al. (+3 more)
Model-based control of flexible joint
robots with position-controlled actuators relies on accurate knowledge of the joint compliance. In practice, precise stiffness models are often unavailable as the properties of physical elastic elements vary with operating conditions and slowly change over time due to wear and aging. To improve model-based control of these systems, we propose an adaptive control approach in this work, which updates an estimate of the uncertain, nonlinear torque-deflection relation of each joint. As opposed to classical adaptive control approaches for non-elastic
robots, we rely on an implicit control law and a control-input-dependent regressor
matrix to account for the uncertain joint stiffness. We analyze robustness of the approach against errors induced by the motor position controller. Experimental results on a flexible joint with nonlinear stiffness characteristics demonstrate the effectiveness of the proposed approach.
Read the full abstract ▾
All authors Annika Kirner, Grazia Zambella, Igor Kovacevic, Hannes Höppner, Jee-Hwan Ryu, Christian Ott
Robotics
Zero2Skill: Bootstrapping Robot Skills through Autonomous Data Collection, Training, and Deployment
Boyuan Wang, Zhenyuan Zhang, Zhiqin Yang et al. (+16 more)
Autonomous data collection governs the volume and quality of real-world trajectories for manipulation policy learning. Existing pipelines reduce human effort via self-resetting, VLM verification, or
language-guided correction, yet episode-scoped fixes must be reissued whenever the same failure recurs, so oversight cost grows with session length rather than with the number of distinct problems. We present Zero2Skill, a human-
robot symbiotic agentic system in which corrections are retained and reused across rounds. The collection loop collects, verifies, and resets autonomously, pausing for a remote operator only when a phase exhausts an explicit retry budget. An LLM parser maps each natural-
language utterance to a structured adjustment stored in Corrective
Memory, so addressed failure modes typically need not be corrected again under the same conditions. On a real-
robot desktop-clearing testbed, Zero2Skill matches teleoperation episode success while reducing human working time to 16%.
Language corrections improve verifier-human agreement in all four evaluated settings and raise average single-attempt success from 12.5% to 47.5% (arm-selection: 20.0% to 50.0%). Policies fine-tuned on Zero2Skill data match teleoperation-trained policy success at a fraction of collection human cost.
Read the full abstract ▾
All authors Boyuan Wang, Zhenyuan Zhang, Zhiqin Yang, Peijun Gu, Shuya Wang, Xiaofeng Wang, Xianghui Ze, Yifan Chang, Guosheng Zhao, Jiangnan Shao, Guan Huang, Hengyu Liu, Yonggang Zhang, Wei Xue, Chunyuan Guan, Chenglin Pu, Yike Guo, Xingang Wang, Zheng Zhu
Robotics
Industrial Dexterity Benchmark: A Hardware-Software Benchmarking Platform for Industrial Dexterous Manipulation
Honglu He, Jacob Laufer, Zhiwu Zheng et al. (+8 more)
Dexterous manipulation remains a critical bottleneck in industrial automation; tasks such as cable routing, connector insertion, and precision assembly still rely heavily on manual labor despite decades of robotics research. This work presents a progression from classical, modular robotics pipelines toward an end-to-end
multimodal imitation-learning framework for industrial dexterous manipulation. As a part of this work, we introduce three key contributions: a set of Industrial Dexterity
Benchmark (IDB) boards aimed to mimic datacenter cable management, automotive cable harnesses, and gearbox assembly tasks; a scalable imitation learning framework (DAG-ROS); and a
multimodal diffusion-based policy framework (AG-iDP3) that creates models fusing RGB images, point clouds, joint positions, and wrist-frame wrench data. Focusing on the datacenter cable manipulation board, we evaluate the performance of a task involving cleaning a single cable over variations of an end-to-end AI policy using 48 trials per configuration. The best performing configuration, a
multimodal expansion
Diffusion Policy (DP), includes a multi-view RGB image source passed through an R3M encoder and reaches a 78% grasp and insert combined task success rate. This performance marks a significant improvement over the 36% observed from the single-camera RGB DP baseline. Each of the tested configurations requires only approximately 100 teleoperated demonstrations per task phase. These results indicate that the correct learned policy can outperform classical
vision and control robotic methods in robustness, generalization, and deployment efficiency, justifying a shift toward scalable robotic automation for high up-time industrial environments.
Read the full abstract ▾
All authors Honglu He, Jacob Laufer, Zhiwu Zheng, David Elkan-gonzalez, Raman Goyal, Xinyi Li, Su Lu, Mishek Musa, Berke Saat, Nicolas Tan, Colm Prendergast
Robotics
AeroMap3D: Anchoring Monocular UAV 6-DoF Localization to Visual-Geometric-Semantic Map Priors
Zhiyun Deng and Luis Sentis
We present AeroMap3D, a
monocular 6-DoF
UAV localization system that anchors onboard imagery to visual,
geometric, and semantic map priors for GNSS-denied
navigation. AeroMap3D addresses two fundamental challenges in map-referenced aerial localization: the cross-view discrepancy between
UAV imagery and satellite maps, and the structural inconsistency between bare-earth digital elevation models (DEMs) and urban scenes. First, we introduce a lightweight adapter that enables a dense matcher pretrained on internet-scale generic data to perform reliable
UAV-to-map registration without finetuning. By estimating the scale ratio and yaw offset between the
UAV image and map tile, the adapter removes the dominant
geometric misalignment induced by altitude, camera field of view, and heading before dense correspondence estimation. Second, AeroMap3D lifts 2D
UAV-map correspondences onto DEM terrain while using OpenStreetMap annotations to reject semantically unreliable matches before RANSAC-PnP pose estimation, thereby reducing errors caused by unmodeled building heights and off-nadir structures. Delayed map-based pose measurements are further fused with relative-motion priors using a delayed-state EKF for continuous trajectory estimation. Without
UAV-Terra3D retraining or tuning, AeroMap3D localizes all trajectories across eight Austin sites within 50 m and achieves 5.88 m mean 3D error over 55 km of flight.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
M$^\text{4}$World: A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming
Ke Cheng, Hanqiao Ye, Lei Shi et al. (+8 more)
Driving-world generation has emerged as a core capability for scalable autonomous-driving simulation, yet existing methods remain limited in object-level controllability and long-horizon stability. We present M$^\text{4}$World, a Multi-view and
Multimodal generative driving world model that synthesizes future surround-view
video streams and synchronized
LiDAR scans while supporting interactive object Manipulation and stable Minute-long streaming. Fine-grained object manipulation is realized through a flexible conditioning interface that supports explicit control over both the spatial layout and visual appearance of individual objects. Stable minute-long streaming, on the other hand, is achieved through a multi-stage training framework that enables online causal generation in only four denoising steps while maintaining coherent world dynamics throughout extended rollouts. Building on these components, we introduce an efficient few-clip post-training as well as a suite of visual reference-conditioned generation models, preserving general generation ability while allowing rare-case customization for long-tail controllability. To assess controllability beyond realism, we further introduce an automated VLM-based judging pipeline that evaluates scene-level condition adherence, view-wise object controllability, and cross-view object consistency. Comprehensive experiments show that M$^\text{4}$World consistently delivers high generation quality, precise controllability, and stable minute-long streaming. Together with downstream long-tail augmentation and scene editing, these results demonstrate the potential of M$^\text{4}$World for controllable, scalable driving simulation.
Read the full abstract ▾
All authors Ke Cheng, Hanqiao Ye, Lei Shi, Yahui Liu, Yunhan Shen, Jingtao Dong, Zhenke Wang, Wenxuan Ao, Weixiang Xu, Kaining Huang, Shuhan Shen
Robotics
GigaWorld-Policy-0.5: A Faster and Stronger WAM Empowered by AutoResearch
GigaWorld Team, Angen Ye, Angyuan Ma et al. (+26 more)
World Action Models (WAMs) improve
robot policy learning by jointly modeling actions and future visual observations, using future scene
evolution as dense supervision for physically grounded action generation. However, a common design in existing WAMs is to explicitly generate future
videos at inference time, incurring substantial computational overhead and hindering real-time closed-loop deployment. GigaWorld-Policy addresses this issue with an action-centered formulation, where future visual dynamics are used during training while action-only decoding is used at inference time. Building upon this framework, we present GigaWorld-Policy-0.5, an enhanced action-centered WAM designed for more efficient
robot control. During pretraining, GigaWorld-Policy-0.5 adopts a mixed Action-Conditioned World Modeling (AC-WM) and WAM training strategy. This strengthens the coupling between visual dynamics and
robot actions and improves the transferability of action representations for downstream policy learning. For efficient inference, GigaWorld-Policy-0.5 introduces a Mixture-of-
Transformers architecture that separates visual dynamics modeling and action generation into specialized experts, reducing active computation during action-only inference and achieving 85 ms inference latency on a local RTX 4090 setup. In addition, we employ an
agent-based AutoResearch pipeline to systematically search training configurations, enabling more efficient identification of optimal experimental setups while reducing the time and manual intervention required for hyperparameter tuning. Experiments and ablations show that GigaWorld-Policy-0.5 preserves the training benefits of future visual dynamics while improving inference efficiency for
robot control.
Read the full abstract ▾
All authors GigaWorld Team, Angen Ye, Angyuan Ma, Boyuan Wang, Chaojun Ni, Fangzheng Ye, Guan Huang, Guo Li, Guosheng Zhao, Haodong Yan, Hengtao Li, Jiwen Lu, Kai Wang, Mingming Yu, Qitang Hu, Qiuping Deng, Songling Liu, Xiaoyu Tian, Xiaofeng Wang, Xinyu Zhou, Xiuwei Xu, Xinze Chen, Yang Wang, Yejun Zeng, Yifan Chang, Yun Ye, Zhenyu Wu, Zhanqian Wu, Zheng Zhu
Robotics
Discriminative Barrier Functions for Safe Adversarial Imitation Learning from Observation
Anubhav Vishwakarma, Bhaumik Mehta, Caleb Hsu et al. (+3 more)
Inverse Reinforcement Learning (IRL)
algorithms are powerful tools for learning from and generalizing expert demonstrations, but they often rely on unconstrained exploration, rendering them unsafe for real-world deployment. Meanwhile, Control Barrier Functions (CBFs) can guarantee the safety of control systems, but the analytical design of CBFs can be time-consuming and esoteric. In this work, we address these limitations jointly by constraining reward function candidacy during IRL to the space of CBFs, yielding a formulation that exhibits safe online control with continuous experiential improvement. Crucially, this framework enables the data-driven recovery of barrier functions directly from unlabeled expert observations. We demonstrate that the recovered barrier function is robust to unsafe states entirely absent from the expert data. Furthermore, we
benchmark our method against standard IRL baselines in a simulated
navigation environment, demonstrating improved safety performance. Finally, we investigate the trade-offs of planning-based versus policy-based IRL methods across both simulation and a real world obstacle avoidance task.
Read the full abstract ▾
All authors Anubhav Vishwakarma, Bhaumik Mehta, Caleb Hsu, Byron Boots, Tyler Han, Karen Leung
Robotics
S-squared-VLA: Decoupling Semantic and Spatial Streams in Vision-Language-Action Models for Autonomous Driving
Jianguo Yu, Rukang Wang, Duanfeng Chu et al. (+3 more)
Vision-
Language Models (VLMs) have demonstrated remarkable potential for high-level
reasoning in autonomous driving, yet they fundamentally struggle to generate precise, low-level control actions. This limitation is rooted in a semantic-physical gap caused by the inherent mismatch between discrete
language tokens and continuous trajectory planning. While
Vision-
Language-Action (VLA) architectures attempt to bridge this gap by unifying perception and control into a single policy, this
entanglement creates a new bottleneck. Standard VLAs experience a severe spatial representation collapse, which irreversibly degrades the fine-grained spatial and
geometric priors essential for safe, boundary-aware
navigation. To address this limitation, we propose the S-squared-VLA, which explicitly decouples the semantic and spatial streams in
Vision-
Language-Action models. The semantic stream leverages hierarchical bridging to extract multi-scale VLM features for robust intent
reasoning. In parallel, an independent spatial stream bypasses the autoregressive
language bottleneck, directly preserving uncompressed spatial features from the visual encoder. By integrating auxiliary perception supervision, this stream explicitly equips the model with rich spatial and
geometric priors. Finally, a dual-stream planning adapter fuses high-level semantic intent with precise spatial constraints via cascaded
attention mechanisms. Evaluations on the NAVSIM closed-loop
benchmark show that S-squared-VLA achieves a Predictive Driver Model Score (PDMS) of 87.1, establishing a new state-of-the-art for VLA models under a purely supervised fine-tuning (SFT) setting. By mitigating the spatial representation collapse of traditional VLMs, our framework significantly outperforms baselines, achieving the highest No Collision (NC) rate of 98.4 among all evaluated methods.
Read the full abstract ▾
All authors Jianguo Yu, Rukang Wang, Duanfeng Chu, Chen Wang, Renju Feng, Liping Lu
Robotics
Learning Forward & Reverse Skills from a Single Unfinished Demonstration for Constrained Manipulation Tasks
Yexin Hu, Haoyi Zheng, Johannes Heidersberger et al. (+1 more)
Learning from demonstration (LfD) enables
robots to learn manipulation skills directly from expert demonstrations but remains challenging for contact-rich tasks involving
geometric constraints and force interaction. Existing approaches typically require multiple complete demonstrations and do not support reverse skill execution. In this paper, we present a unified one-shot framework for constrained manipulation that learns both forward and reverse execution from a single, possibly unfinished demonstration. Our method decomposes demonstrations into non-contact and contact phases, with non-contact motion encoded with dynamic movement primitives (DMP), and contact motion represented as a sequence of screw motion primitives segmented by our proposed geometry-driven twist-direction
segmentation algorithm. During execution, screw primitives are executed sequentially under admittance-guided pose correction and speed regulation, enabling task completion beyond the demonstrated trajectory length as well as reverse skill execution without additional learning data. Experiments on peg insertion, battery insertion, lock opening, and screw driving tasks demonstrate improved success rates and robustness over
segmentation and one-shot trajectory learning baselines. Details are available on the project website: https://tuwien-asl.github.io/LfD-Screw/.
Read the full abstract ▾
All authors Yexin Hu, Haoyi Zheng, Johannes Heidersberger, Dongheui Lee
Robotics
Merging Reaction to Cognition: A Hybrid Cognitive Strategy for Odour Source Localisation in Natural Environments
Hugo Magalhães, Rui Baptista and Lino Marques
Chemical pollutants released into the environment are transported by turbulent flows, generating complex, intermittent plume structures that threaten ecosystems and human health. Rapid localisation of emission sources is critical, and field
robots equipped with chemical sensors provide a viable means to perform this task. However, inferring source location from sensor readings remains difficult due to sparse detections and the absence of reliable concentration gradients. Existing approaches fall into two paradigms. Bio-inspired strategies rely on reactive behaviours triggered by detections, such as surge-casting, offering efficiency but requiring scenario-specific tuning. Cognitive strategies integrate observations into a probabilistic belief over source location. While more robust, they suffer from excessive exploration and strong dependence on belief accuracy. The Fast-Cognitive
algorithm reduced this computational burden but preserved the fundamental limitations. Previous Markov chain analysis revealed that source-directed motions occur roughly twice as often following odour detections, indicating that reactive behaviours naturally emerge within cognitive frameworks. This work proposes a hybrid strategy that explicitly incorporates bio-inspired reactivity into belief-dependent motion planning. It introduces a detection-triggered switching mechanism formalising transitions between crossflow exploration and source-directed motion, prioritising source proximity over information gain. Behavioural parameters are derived directly from belief metrics, enabling adaptive reactivity without manual tuning. The approach is validated through simulations under three
turbulence conditions and field experiments with an autonomous surface vehicle in the Mondego River, Portugal. Results show up to 50% reduction in travelled distance, 86% success rate, and 3.2m average localisation error.
Read the full abstract ▾
Robotics
Recursive ArUco Markers: A Scalable Fiducial Marker Design for Unmanned Aerial Vehicle Landing Pads
Rafael Munoz-Salinas, Francisco Jose Romero-Ramirez and Sergio Garrido-Jurado
Unmanned Aerial Vehicles (
UAVs) increasingly rely on visual fiducial markers for autonomous
navigation and precision landing. However, standard markers suffer from limited operational ranges, becoming undetectable when the camera is either too far or too close. While recursive and fractal markers have been proposed to address this issue, existing approaches either require the marker's center to remain visible, making them vulnerable to occlusion, or are limited in their recursion depth and placement. We propose a novel Recursive ArUco marker design. Our method allows any standard fiducial marker to be transformed into a recursive marker with an arbitrary depth. By employing a modified bit-sampling strategy during detection, we embed complete markers within both the black and white bits of the parent marker. This approach guarantees unlimited recursion depth and robust detection even with partial occlusion, as it does not rely on the marker's center being visible. Furthermore, by maintaining a single, unique identifier across all recursive scales, our proposal provides an extensive dictionary of multiple unique landing pads. This capability allows fleets of
UAVs to operate simultaneously, with each drone landing at its designated location – a feature not supported by existing Fractal and Harco markers due to their structural and dictionary constraints.
Read the full abstract ▾
Robotics
Learning Robust Execution in Robotic Manipulation with Agentic Reinforcement Learning
Xiaopeng Zhang, Yueyang Weng, Qi Liu et al. (+2 more)
Robotic manipulation poses fundamental challenges due to uncertainty, long-horizon execution, and compounding errors, which can easily destabilize execution and lead to task failure. Although recent
vision-
language-action (VLA) models exhibit strong generalization, they typically lack explicit mechanisms to assess execution stability and to recover when execution deviates from its nominal behavior. In this paper, we propose: (1) two complementary metrics to assess execution quality at runtime, and (2) an agentic reinforcement learning framework that learns to restore effective execution through high-level decision-making rather than directly learning low-level actions. In this framework, an agentic policy reasons over recent execution history and selects among a small set of execution modes to regulate the execution process. Under execution degradation, it triggers appropriate recovery mechanisms to restore the
robot to previously visited nominal states, enabling the task to continue. We evaluate the proposed method on the LIBERO
benchmark, achieving up to a 13.7% improvement in success rate under standard settings and up to a 39.2% improvement under disturbance settings, demonstrating substantially enhanced execution robustness.
Read the full abstract ▾
All authors Xiaopeng Zhang, Yueyang Weng, Qi Liu, Yongjin Mu, Yanjie Li
Robotics
A Deployed Hybrid Vehicle-in-the-Loop Platform for Validating Cooperative Perception
Anastasia Bolovinou, Giorgos Hadjipavlis, Markos Antonopoulos et al. (+6 more)
European safety regulation now permits a large share of automated-driving homologation evidence to be produced virtually, provided a validated physical-virtual facility generates it. We present a deployed hybrid Vehicle-in-the-Loop (ViL) platform that couples a real instrumented vehicle with a CARLA-based digital twin (DT) through a V2X message pipeline, and we report its first integrated operation on a public-road-representative test track. A real vehicle streams ETSI-compliant CAM/CPM messages into the DT, where a GPU-accelerated Cooperative Perception (CP) module fuses them into a probabilistic occupancy grid during scenario runtime. We demonstrate the platform on a multi-vehicle double T-intersection scenario, characterise the CP workload across nominal, rain and night conditions and five localization-noise levels, and discuss the platform's current architectural limits and the engineering targets they define. The results show that CP substantially widens field-of-view (FoV) coverage and improves occupied-
cell recall, and that beyond a moderate localization-noise threshold, positioning uncertainty, and not weather, becomes the dominant error source. We outline the platform's trajectory toward a Mediterranean operational design domain (ODD) testing service.
Read the full abstract ▾
All authors Anastasia Bolovinou, Giorgos Hadjipavlis, Markos Antonopoulos, Panagiotis Tachtalis, Konstantinos Petousakis, Konstantinos Lazaridis, Alexandros Siskos, Bill Roungas, Angelos Amditis
Robotics
Vision-Based Obstacle Separation for Strawberry Harvesting in Clusters Using Hierarchical Reinforcement Learning
Teng Li, Hanfei Shi, Chunjiang Zhao et al. (+1 more)
Selective harvesting in clustered strawberry environments is challenging because ripe fruits are often occluded by surrounding unripe fruits, making direct grasping unreliable. To address this problem, this paper proposes a hierarchical reinforcement learning framework, termed VGPA, which integrates a
vision-guided decision mechanism and a Progressive Adaptive Exploration Strategy (PAES) for
vision-based obstacle separation and harvesting. The task was decomposed into two sequential stages: obstacle separation and target grasping. At the high level, the
vision-guided mechanism improved option selection and accelerated policy convergence. At the low level, PAES improved exploration efficiency and training stability during continuous control learning. In simulation experiments, the learned policy achieved a success rate of 96.7%. In addition, sim-to-real transfer experiments on a self-developed parallel
robot showed that the proposed method achieved success rates ranging from 71.7% to 88.3%, outperforming direct picking while requiring only 1.22~s more average harvesting time. These results verified the effectiveness, generalization ability, and practical potential of the proposed method for robotic harvesting in complex clustered environments.
Read the full abstract ▾
All authors Teng Li, Hanfei Shi, Chunjiang Zhao, Ya Xiong
Robotics
Dynamical Vehicle Orienteering Problem for Multi-Rotor Unmanned Aerial Vehicles
František Nekovář, Matej Novosad, Martin Saska et al. (+1 more)
This paper introduces the Dynamical Vehicle Orienteering Problem (DVOP), a generalization of the Orienteering Problem (OP). The OP maximizes the reward collected from spatial targets under a limited travel budget; the DVOP extends it by accounting for both external and vehicle-actuated forces. We study the DVOP in the context of multi-rotor Unmanned Aerial Vehicle (
UAV) flight planning, using a three-dimensional Point-Mass Model (PMM) constrained by maximum velocity and acceleration magnitudes and subject to gravitational acceleration, with the travel budget expressed as a maximum flight time. Because the DVOP couples reward maximization with time-optimal trajectory planning, it cannot be formulated as a simple graph problem and solved exactly without relaxing or under-actuating the vehicle dynamics. We therefore propose two solution approaches: a Branch-and-Bound (BnB) procedure that combines Non-Linear Programming (NLP) and Mixed-Integer Linear Programming (MILP) to provide high-quality solutions, and a Large Neighborhood Search (LNS) metaheuristic that supplies an initial reward bound and scales to instances intractable for the BnB. The BnB relies on a novel MILP formulation of travel costs based on minimum-time trajectory primitives through target triplets, yielding a tight reward upper bound, while the LNS uses limited thrust decomposition to compute fast, high-quality PMM trajectories. Experiments on
benchmark instances show improvements of up to 37 % over state-of-the-art solutions for the Kinematic Orienteering Problem, and a real-world deployment on a multi-rotor
UAV verifies the proposed PMM solution trajectories.
Read the full abstract ▾
All authors František Nekovář, Matej Novosad, Martin Saska, Robert Pěnička
Robotics
The Nonsmooth Impact Direction (NSID) of Robotic Systems
Annika Kirner and Christian Ott
Collisions of rigid-link
robots and rigid environments are often modeled as instantaneous events. Under this idealization, the impact forces become impulsive and the system velocities nonsmooth. In this work, we systematically analyze pre- and post-impact velocities focusing on what we refer to as the nonsmooth impact direction (NSID). We show that it is a characteristic direction of a robotic impact and largely independent of contact properties. The results are directly applicable to large classes of backdrivable robotic systems with rigid links. We address particularities of systems with nonelastic and flexible joints, unconstrained as well as constrained systems. Further, we show that the approach direction w.r.t the NSID sets the direction of the impulsive force in frictional, inelastic impacts. The comprehensive theoretical analysis of this work supported by an experimental validation may serve as a foundation for future planning and control
algorithms for various robotic impact applications. These can include
humanoid locomotion on a slippery surface or repetitive hammering.
Read the full abstract ▾
Robotics
nuTruck: Benchmarking Autonomous Driving Planning for Distributed Electric-drive Trucks
Jinyu Miao, Pu Zhang, Yifei He et al. (+5 more)
The dominance of traditional rule-based methods in autonomous driving has gradually been replaced by learning-based approaches. While learning-based planners have achieved considerable success in passenger vehicles, their performance on heavy-duty trucks, particularly modern distributed electric-drive trucks (DETs), remains largely unexplored. To facilitate research and application of learning-based planners in DETs, this letter presents the first high-fidelity
benchmark, called nuTruck, designed to support large-scale neural network training and closed-loop evaluation. Given the complex dynamics and high rollover susceptibility of DETs, we first incorporate a highly accurate nonlinear truck dynamical model into the simulation, which enables independent driving and steering of all wheels and captures dynamic load transfer caused by acceleration, deceleration, and cornering, thereby allowing quantitative assessment of rollover risk in closed-loop simulation. Second, we adapt several rule-based and learning-based planners as baselines for DETs and evaluate their performance in closed-loop simulation. Finally, using real-world driving scenarios from the nuPlan
dataset, we conduct extensive closed-loop evaluations, analyzing not only conventional collision-free planning performance, but also the dynamical safety of the planned trajectories. The proposed nuTruck
benchmark is expected to serve as a new standard for fair and realistic evaluation of autonomous driving planners on DETs.
Read the full abstract ▾
All authors Jinyu Miao, Pu Zhang, Yifei He, Chengyao Zhang, Kun Jiang, Ke Wang, Mengmeng Yang, Diange Yang
Robotics
Towards a Modular Bin-picking Framework for Handling Object Pose Uncertainties
Frederik Hagelskjær
In recent years, there has been growing interest in robust robotic systems for precise bin-picking applications. To achieve reliable performance, such systems must address errors arising from both the object pose estimation and the grasping process. Although various approaches have been proposed, they typically target specific challenges and do not offer general solutions. In this paper, we present a modular framework that jointly handles both error types. The framework incorporates object pose distribution estimation to account for pose uncertainty, which frequently arises in situations with ambiguous observations where a single correct pose cannot be determined. To further reduce uncertainty, we introduce a second-viewpoint module that computes complementary pose distributions, which are subsequently fused. This fusion decreases overall uncertainty and improves system efficiency. Additionally, two independent modules are included to compensate for grasping errors. The modular design allows the components to be combined for optimal performance or used individually, depending on the physical setup. The proposed method is evaluated in a real-world setup with three different objects, with no errors, and all modules are shown to improve efficiency. These results suggest that incorporating pose distributions with grasping pose errors is a promising direction for developing more flexible and reliable robotic production systems. To the best of our knowledge, this is the first framework that jointly addresses both grasping and object pose uncertainties using interchangeable modules. We believe there is ample opportunity to integrate additional modules, resulting in improved performance and flexibility. The current framework is limited to pose uncertainties in SO(2), but it could be extended to SE(3), enabling additional modules to improve the system.
Read the full abstract ▾
Robotics
Anatomy of Uncertainty: Expressive Descriptors of Robot Motion for Nonverbal Human-Robot Communication
Ridhima Bector, Souravik Dutta, Poornima Ramachandran et al. (+4 more)
Robots working alongside humans must communicate their intended actions together with the uncertainty that arises from incomplete or ambiguous perception. This paper introduces a mathematical framework for expressing perceptual uncertainty through the motion of a robotic manipulator. Drawing on concepts of approach-avoidance and active perception,
robot behavior is organized in a Commitment-Vigilance state space whose dimensions are represented through Laban Effort factors, mapping five uncertainty-related states, namely confidence, curiosity, hesitance, fear and inactivity on the uncertainty continuum. A kinematic analysis decomposes goal-directed end-effector motion into a radial task-progress rate and a tangential target-bearing rate, which realize the two dimensions. From this decomposition, five motion primitives, namely approach, pause, retreat, probe and twitch, are derived and parameterized using eleven kinematic descriptors covering approach and retreat characteristics, pause behavior, gaze angles, end-effector tilt and shiver amplitude. A
video-based human-subject study evaluated the recognition of uncertainty-expressive trajectories and the influence of individual descriptors on perceived intensity. For every trajectory the intended behavioral state was the modal response and was selected significantly more often than chance. In single-descriptor comparisons, participants significantly preferred one variant as the more intense expression of the intended state. The results provide a perceptual basis for encoding
robot uncertainty in motion and for generating such trajectories autonomously from parametric movement representations. Expressive
robot motion
videos and questionnaire used in the user study are available at https://anonymous.4open.science/r/aou/.
Read the full abstract ▾
All authors Ridhima Bector, Souravik Dutta, Poornima Ramachandran, Ree Yan Yeoh, Jui Hien Tan, Domenico Campolo, Bernhard Johannes Schmitt