Robotics
Real-Time Service Robot Replanning via Simple Button Interaction for Improved Task Success and User Experience
Ryo Terashima, Yuga Yano, Koshun Arimura et al. (+1 more)
Service
robots must respond to unexpected instructions in real-world environments. However,
robots cannot detect all failures and exceptions during a task. To address these issues, we propose a real-time feedback function that enables
robots to modify their behavior based on human feedback. In this system, users can intuitively send feedback to the
robot by pressing a single button on a tablet when the
robot fails to act correctly.
Robots use this feedback to consider their failures and replan appropriate actions to complete the task. We conducted experiments with and without the feedback function to verify the following hypothesis: "Simple interactions do not cause a negative user experience." All questionnaire responses are evaluated on a five-point Likert scale. After adding the feedback function, the response score for the question "Did you feel that the
robot's behavior was unexpected?" improved by 0.5 points, and that for "Did you feel anxious about the
robot's behavior at times?" improved by 0.9 points. These results support the study's hypothesis and indicate that incorporating this real-time feedback function can simultaneously improve task success and the user experience.
Read the full abstract ▾
All authors Ryo Terashima, Yuga Yano, Koshun Arimura, Hakaru Tamukoh
Robotics
Human-Centric Grasp State Assessment: Toward Transferring Subjective Evaluation to Robots
Ryohei Kobayashi, Kosei Isomoto, Yuga Yano et al. (+2 more)
We propose a framework that transfers tacit human subjective criteria to robotic systems for the appropriate grasping of deformable objects. Achieving such behavior is challenging because a semantic gap exists between qualitative human expectations and quantitative robotic measurements. Conventional deep learning approaches for bridging this gap also require prohibitive amounts of manually annotated data for each newly encountered object. To address these challenges, our framework integrates a
Vision-
Language Model (VLM)-based semi-automated supervisor generator with a lightweight grasp state predictor, using a minimal set of human-annotated trials as contextual anchors to propagate subjective criteria to unannotated data. The prediction model then enables rapid online adaptation by sequentially estimating the grasp state from time-series tactile and grasping force measurements. Through experiments on three representative deformable objects and a human evaluation study with 25 participants, we demonstrate the feasibility of the proposed framework for adjusting grasping force according to human-perceived grasp appropriateness in the evaluated task setting.
Read the full abstract ▾
All authors Ryohei Kobayashi, Kosei Isomoto, Yuga Yano, Yuichiro Tanaka, Hakaru Tamukoh
Robotics
Risk-Aware Decision-Making for Autonomous Overtaking: A World Model-Based Mixture-of-Experts Framework
Yongzhi Liu, Sunan Zhang, Jinchang Xu et al. (+4 more)
Autonomous highway overtaking demands foresighted decision-making to handle complex interactions,
stochastic traffic
evolution, and temporal risk accumulation. However, standard safe reinforcement learning approaches typically rely on implicit value-based risk estimations rather than explicit dynamics modeling, thereby struggling to accurately capture complex risk propagation over multi-step horizons. This limitation frequently results in behaviors that are locally safe but induce substantial latent risks in the long term. To address this, a World Model-based Risk-aware Mixture-of-Experts (WM-RMoE) framework is proposed. First, a learned latent dynamics model facilitates parallel multi-step rollouts, elevating safety assessment from the action level to the trajectory level via cumulative risk evaluation. Second, to enhance robustness under varying interaction intensities, a hierarchical gating mechanism dynamically coordinates experts across long-horizon, short-horizon, and rule-based safety modules. Furthermore, a Gaussian Mixture Model is integrated to preserve
multimodal maneuvering branches, thereby mitigating the issue of behavioral mode averaging. Experimental results demonstrate that WM-RMoE significantly outperforms representative baselines in terms of safety compliance, decision stability, and generalization capability. Furthermore, benefiting from the risk-aware formulation, the proposed framework uniquely exhibits the ability to generate foresighted and semantically distinct overtaking maneuvers across diverse traffic densities.
Read the full abstract ▾
All authors Yongzhi Liu, Sunan Zhang, Jinchang Xu, Jiawei Wang, Yushu Qiu, Chen Lv, Weichao Zhuang
Robotics · Cross-listed from cs.AI
Generalist AI Control: Towards Multi-purpose Adaptive Algorithms
Klinsmann Agyei and Pouria Sarhadi
Traditional controllers are designed for specific systems and do not transfer across different system orders and dynamics. We present a Generalist Controller, a learning-based controller capable of controlling systems of varying orders and dynamics. The approach introduces a novel dynamic state-space representation using
attention mechanisms with masking, enabling a single neural network, trained in one shot, to handle systems with different dimensions without architectural modifications by assigning a system tag to each system. We generated 314,630 demonstrations from 25 diverse systems, including stable, unstable, minimum-phase, and non-minimum-phase dynamics, spanning linear and nonlinear systems from autonomous
underwater and aerospace vehicles to mechanical systems and chemical processes. The model learns cross-system control strategies through multi-scale temporal processing and a mixture-of-experts architecture. Simulation results demonstrate that the proposed generalist controller achieves comparable performance to system-specific LQI controllers across all tested systems, including challenging cases such as non-minimum-phase and unstable dynamics, whilst generalising to unseen operating conditions including actuator saturation, noise, disturbance, and reference trajectories not encountered during training. This work represents a significant step towards generalist control policies within a defined family of dynamical systems, demonstrating effective control across a range of single-input single-output (SISO) systems of varying order and dynamics using a single learned policy without system-specific tuning.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
xperception – Making Robotic Grasping Easier
Matteo Bortolon, Andrea Caraffa, Alice Fasoli et al. (+1 more)
The transition toward high-mix low-volume manufacturing demands flexibility in robotic manipulation. However, conventional
vision systems remain a bottleneck, requiring extensive data collection and model retraining whenever a new object is introduced to the production line. To overcome this
rigidity, we present xperception, a zero-shot 6D pose estimation technology that eliminates the need for object-specific fine-tuning and laborious data annotation. By directly utilizing typical CAD models and integrating the rich semantic features of foundation models (e.g. DINOv2, GeDi), xperception achieves millimeter-accurate 6D pose estimation. xperception showed robustness against severe occlusions in industrial tasks like bin picking and is engineered for deployment on industrial edge hardware, such as NVIDIA Jetson Thor. Validated at a TRL of 6, the core methodology behind xperception is based on the FreeZe
algorithm, which won the international BOP Challenge 2024, paving the way for scalable, plug-and-play robotic automation in unstructured high-mix low-volume manufacturing industries.
Read the full abstract ▾
All authors Matteo Bortolon, Andrea Caraffa, Alice Fasoli, Fabio Poiesi
Robotics
Real-Time sEMG-Based Telecontrol of an Assistive Robotic Arm Using a 1D Convolutional Neural Network
Edgar Manacorda, Mena Samir Kama Abouseffien, Olivier Lecompte et al. (+2 more)
Motor impairments affecting the upper limb significantly reduce autonomy in daily activities, particularly for tasks involving object manipulation. Assistive robotic arms offer a promising solution, provided they can be controlled in an intuitive, reliable, and responsive manner. Among human–machine interface approaches, surface electromyography (sEMG) enables non-invasive access to muscle activity and thus to the user's motor intentions. This work proposes a real-time sEMG-based interface for the teleoperation of an assistive robotic arm. The system relies on four-channel sEMG acquisition, signal preprocessing,
segmentation into sliding windows, and classification using a one-dimensional convolutional neural network (CNN). Several real-time strategies are investigated, including threshold-based onset detection, a two-stage classification approach (rest vs movement followed by gesture recognition), and a single classifier handling both rest and five gestures. The complete pipeline is implemented and evaluated both in simulation and on a real robotic platform. The CNN-based approach achieves high classification performance, with a test accuracy above 90% and strong generalization on experimentally acquired signals. The system exhibits stable real-time behavior, with an average latency of approximately 0.32 s consistent with the chosen windowing strategy, and the
robot can be controlled reliably using discrete gestures, producing coherent and smooth movements in both simulated and real environments. These findings demonstrate the feasibility of sEMG-based telecontrol for assistive robotics and highlight the importance of integrating signal processing, deep learning, and control strategies within a unified real-time framework. Future work may explore hybrid control approaches combining sEMG with additional sensing modalities to further improve robustness and usability.
Read the full abstract ▾
All authors Edgar Manacorda, Mena Samir Kama Abouseffien, Olivier Lecompte, Amandine Gesta, Abolfazl Mohebbi
Robotics
Control Design for a Rideable Animatronic Two-Wheeled Robot with Quadruped Form
Tadashi Sumioka, Kazushi Akimoto, Takuya Tsujimura et al. (+2 more)
In recent years, motorcycle popularity has been declining, particularly among the younger generations. To rekindle interest in motorcycles among this demographic, we developed a rideable two-wheeled
robot equipped with four limbs as a future partner mobility concept and intended for use in public events. The character reproduced by this
robot carries the protagonist on its back and exhibits a dynamic quadrupedal gait. To maximize the riding experience, we aimed to match the
robot's weight and size to the character's specifications while ensuring rider safety and enabling expressive movements of limbs, wrists, ankles, and facial features. However, achieving locomotion solely through limb movement would require excessive motor output and increased limb strength, resulting in higher weight and extremely slow gait, thereby reducing character fidelity. To overcome these challenges, the
robot performs its primary locomotion using a self-balancing two-wheeled base, while the limbs provide auxiliary support during mounting and dismounting and move in coordination with the wheeled locomotion speed. This approach enables an animatronic
robot capable of intuitive, weight-shift-based control for free and natural movement. In this paper, we focus on a robust selfbalancing control method that maintains stability even during rapid limb movements, as well as a motion control strategy that generates natural quadruped-like behavior.
Read the full abstract ▾
All authors Tadashi Sumioka, Kazushi Akimoto, Takuya Tsujimura, Sho Takayanagi, Yuya Horiuchi
Robotics
Design and Characterization of a Limb Encircling Actuator
Japmanjeet Singh Gill, Gray Cortright Thomas, Nikko Van Crey et al. (+2 more)
Lower-limb powered exoskeletons have demonstrated substantial improvements in mobility and function, but most designs place actuation components lateral to the legs or remotely at the waist or back. These configurations often extend beyond the body's natural envelope, making devices intrusive in everyday use and potentially limiting societal adoption. We posit that rethinking actuation geometry could enable exoskeletons that conform more closely to the body. Here, we explored an actuation layout in which the actuator encircles the limb in a plane orthogonal to the limb axis, potentially reducing its spatial footprint around the body. We developed the Limb-Encircling Actuator (LEA) and characterized its electromechanical properties using a custom-built testbed. The LEA also features a novel radial bearing layout with potential as a lightweight or lower-cost alternative to traditional large-diameter bearings. The actuator achieved a continuous torque density of 7.5 Nm/kg with a mass of 894 g. Despite this high torque density and innovative layout, the system remained difficult to contain close to the body. These results highlight opportunities and challenges in limb-encircling actuation and provide insights into torque-dense exoskeleton designs that could integrate more readily into everyday apparel if challenges in actuator sizing and geometry are overcome.
Read the full abstract ▾
All authors Japmanjeet Singh Gill, Gray Cortright Thomas, Nikko Van Crey, Roberto Leo Medrano, Elliott J. Rouse
Robotics
Adapting Generalist Vehicle Models for High-Speed MPC Across Terrains
Rwik Rana, Jesse Quattrociocchi, Christian Ellis et al. (+3 more)
High-speed off-road autonomy requires precise closed-loop control for a target vehicle while remaining robust across changing terrains. Recent forward kinodynamic (FKD) prediction foundation models suggest a promising path, starting from a generalist model and specializing it to the target platform. However, effective specialization remains challenging, as it often requires substantial real-world data, and models adapted to one setting can still overfit to specific terrains or driving regimes. We present OptCar (Optimized Car), a recipe for bridging the gap from generalist to specialist FKD models that preserves cross-terrain generalization while optimizing performance for a specific vehicle. OptCar introduces a
transformer FKD architecture that uses FiLM to condition multi-step predictions on a single dynamics context
token summarizing recent state-action history. It then specializes the generalist model using limited real-world data and targeted synthetic rollouts from environment-specific system identification. In closed-loop model predictive control (MPC) experiments across three terrains and an out-of-distribution cart-pulling task, the largest gains appear at 6 m/s, the highest speed evaluated and the regime in which slip dominates tracking error. On vegetation + dirt, the most slip-diverse terrain, OptCar reduces 6 m/s trajectory tracking error by roughly 55% relative to AnyCar fine-tuned on real data alone, and remains the most accurate even when an unseen cart payload changes the dynamics. With 5 minutes of real data per terrain, OptCar is competitive on road with a specialist trained on 30 minutes of road data and outperforms it when the terrain changes. Project Page : https://amrl.cs.utexas.edu/optcar/
Read the full abstract ▾
All authors Rwik Rana, Jesse Quattrociocchi, Christian Ellis, Nathan Tsoi, Garrett Warnell, Joydeep Biswas
Robotics
Parsimonious disturbance-aware minimum-time planning with parametric uncertainty
Martino Gulisano, Matteo Masoni and Marco Gabiccini
This study presents and validates a minimum-lap-time planning (MLTP) framework for motorsport applications that embeds robustness against both state disturbances and parameter uncertainty. The methodology builds upon a prior disturbance-aware framework that, at each track point, propagates
stochastic vehicle dynamics over a short horizon and tightens tyre-friction constraints based on the worst-case scenario at horizon end. We extend the formulation to account for uncertainty in key vehicle parameters: moment of inertia, centre-of-mass position, and aerodynamic drag coefficient. To keep the extended formulation computationally tractable, a spatially selective, parsimonious activation strategy confines the robust constraints to the
circuit segments where they are most critical. We demonstrate the improved driveability of the robust references by employing a model predictive controller (MPC) as a virtual test driver. For each reference, the same MPC drives a simulated FSAE (Formula SAE) car over 1000 runs on a representative Barcelona-Catalunya sector, with randomly realised impulsive disturbances and parameter scatter. We compare a nominal reference, planned without robustness, against its robust counterparts. The latter yield consistently fewer failed runs and, at a moderate sector-time cost, show tighter dispersion of key signals (vehicle inputs, axle saturations) around the reference values, evidence of better trackability.
Read the full abstract ▾
Robotics
Attitude Estimation Using Inertial and Barometric Measurements
Melone Nyoba Tchonkeu, Soulaimane Berkane and Tarek Hamel
Accurate and robust attitude estimation is a key challenge for autonomous vehicles, particularly in GNSS-denied conditions and during highly accelerated flight. In such conditions, Inertial Measurement Units (IMUs) alone are insufficient for reliable tilt estimation due to the ambiguity between gravitational and inertial accelerations. Although auxiliary velocity sensors such as GNSS, Pitot tubes, Doppler radar, or Visual Inertial Odometry are commonly used, they may be unavailable, intermittent, or costly. This paper introduces a barometer-aided attitude estimation architecture that exploits barometric altitude measurements to provide complementary information on the vehicle's vertical motion, thereby enhancing attitude estimation within nonlinear observers on SO(3). The contributions are twofold. First, we design a deterministic Riccati observer cascaded with a complementary filter, ensuring almost-global asymptotic stability (AGAS) under a uniform observability (UO) condition while preserving the
geometric structure of the attitude dynamics. Second, we propose a nonlinear observer evolving on SO(3)xR2, which integrates IMU measurements as inputs and barometer and magnetometer measurements as outputs within a unified framework, guaranteeing local exponential stability (LES) under relaxed uniform observability conditions. The proposed approaches are validated using both simulated and real flight data. The results demonstrate that barometer-aided estimation provides a lightweight, reliable, and effective complementary sensing modality for attitude estimation in minimal-sensing configurations, offering a practical alternative when conventional velocity measurements are unavailable or degraded.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Just-In-Time Scene Graph Growth: Combating Perceptual Saturation in Long-Horizon Robotics
Yue Chang, Rufeng Chen, Yifan Tian et al. (+7 more)
While 3D Scene Graphs (3DSGs) provide crucial structured representations for
embodied agents, conventional Ahead-of-Time, "build-everything-then-filter" pipelines conflict with the real-time, low-latency demands of edge platforms, inducing a perceptual saturation effect via severe observation redundancy. To resolve this, we present JITOMA (Just-In-Time On-demand
Memory Activation), a closed-loop framework that unifies task
reasoning, perception, and
memory into a just-in-time growth process. Instead of exhaustively mapping the entire environment, JITOMA leverages a top-down task heatmap at the frontend to filter continuous observations, routing minimal streams to maintain a global foundation of low-cost, dormant anchors. Upon a cognitive query, the backend Large
Language Model (LLM) parses the robotic intent to dynamically awaken task-relevant anchors, triggering expensive semantic operations such as dense node captioning exclusively within the activated local subgraph. To evaluate these dynamic capabilities and study perceptual saturation trade offs, we introduce JITOMA-Bench, a
benchmark for long-horizon task switching and complex intent grounding. Across JITOMA-Bench, JITOMA maintains only 1–6 active semantic objects and 0.25–0.28 s/frame across all tiers, showing that semantic computation remains bounded by current task demand rather than accumulated scene complexity.
Read the full abstract ▾
All authors Yue Chang, Rufeng Chen, Yifan Tian, Dazhi Huang, Zhaofan Zhang, Yi Chen, Wenze Zhang, Li Chen, Hui Xiong, Sihong Xie
Robotics · Cross-listed from physics.comp-ph
Towards end-to-end optimization in multimaterial 3D printing
Xue-Ling Luo, Steven Yang, Jingye Tan et al. (+3 more)
Multimaterial 3D printing enables the fabrication of functionally graded components, but optimizing their spatial material distribution alongside structural
topology remains a formidable challenge due to high-dimensional design spaces and complex constitutive modeling. This paper presents an end-to-end computational framework integrating sparsified physics-augmented neural networks with finite-element-based
topology optimization. By extracting closed-form, composition-aware hyperelastic constitutive laws from experimental data, this approach facilitates exact symbolic differentiation via the adjoint state method implemented with FEniCSx, efficiently circumventing the bottlenecks of applying neural network constitutive models. This pipeline is deployed on soft robotic gripper applications, demonstrating continuous composition optimization for highly anisotropic contact responses, and the concurrent optimization of macroscopic
topology and material distribution under non-failure stretch constraints. This methodology could replace laborious empirical prototyping, establishing interpretable machine-learning models as practical, robust design primitives for advanced multimaterial additive manufacturing.
Read the full abstract ▾
All authors Xue-Ling Luo, Steven Yang, Jingye Tan, Robert F. Shepherd, Noy Cohen, Nikolaos Bouklas
Robotics
Worlds in One Demo: A Synthetic Data Engine for Learning Open-World Mobile Manipulation
Lingxiao Guo, Huanyu Li and Guanya Shi
Learning open-world mobile manipulation policies requires vast data to achieve spatial generalization, long-horizon robustness, and scene generalization. Current prevailing data collection paradigms, teleoperation and UMI, demand prohibitive human effort and cost at scale. To scale beyond the limits of manual data collection, we seek to maximize the value of each human demonstration by scalable data generation. To this end, we introduce WANDA: learning open-World mobile mANipulation from one demonstration via a synthetic DAta engine. WANDA first reconstructs background Gaussian splats and
robot-object interaction trajectories from source RGBD observations, as a world substrate for later planning and rendering. It then rearranges contact-rich
robot-object interaction segments into extensive spatial configurations, utilizing whole-body motion planning to chain them into new trajectories. To enhance long-horizon robustness, it applies Corrective State Expansion to increase the
robot and object state diversity at different stages of mobile manipulation. To unlock cross-environment generalization, trajectories are synthesized on diverse generated 3D worlds from everyday photos. Furthermore, we synthesize photo-realistic observations by compositing rendered
robot and object meshes with Gaussian splatting backgrounds. We evaluate our approach on extensive simulation and real-world tasks in various scenes. Experiments show that policies trained with WANDA achieve long-horizon robustness, broad spatial generalization and cross-environment generalization from one real demonstration. Moreover, WANDA naturally supports cross-embodiment data generation, validated by zero-shot deployment on another mobile manipulator with a distinct morphology.
Read the full abstract ▾
Robotics
DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation
Yu Fang, Wanxi Dong, Jiaqi Liu et al. (+7 more)
Reinforcement learning holds great promise for improving
robot policies beyond the limits of imitation learning. However, its practical adoption remains bottlenecked by the lack of reliable
vision-
language reward models that provide dense and informative feedback. Two key challenges remain: acquiring diverse failure data at scale and obtaining fine-grained reward signals beyond sparse trajectory-level success labels. Collecting failure trajectories typically requires laborious human effort, while pseudo-failures constructed by relabeling successful demonstrations fail to capture the diverse physical failure modes that arise during
robot execution. Meanwhile, existing reward models often predict sparse binary or trajectory-level rewards, which provide limited guidance for efficient policy optimization. We introduce DenseReward, a dense robotic reward model that addresses both challenges. To train DenseReward, we develop an automated failure data generation pipeline that synthesizes physically realistic failure trajectories in simulation without human labeling, covering diverse failure modes such as collisions, missed grasps, object drops, and recovery behaviors. DenseReward predicts dense frame-level reward scores from visual observations and
language instructions, enabling fine-grained estimation of task progress throughout an episode. Experiments show that DenseReward outperforms general-purpose VLMs and existing robotic reward models in dense reward prediction across both simulated and real-world manipulation. We further demonstrate that DenseReward provides effective reward guidance for downstream model predictive control and reinforcement learning. We release the
dataset, trained reward models, and evaluation suite to support the development of failure-aware dense reward modeling for
robot learning.
Read the full abstract ▾
All authors Yu Fang, Wanxi Dong, Jiaqi Liu, Yue Yang, Mingxiao Huo, Yao Mu, Huaxiu Yao, Li Erran Li, Daniel Szafir, Mingyu Ding
Robotics · Cross-listed from cs.LG
TerraZero: Procedural Driving Simulation for Zero-Demonstration Self-Play at Scale
Zhouchonghao Wu, Akshay Rangesh, Weixin Li et al. (+5 more)
Training robust autonomous driving
agents requires a simulator fast enough for reinforcement learning at scale, realistic enough to ground behavior in real-world map structure, and diverse enough to cover the safety-critical long tail that logged data rarely contains. We present TerraZero, a procedural driving simulator and self-play training stack that meets these goals. A configurable C engine runs simulation on the CPU and policy inference on the GPU over a zero-copy path, sustaining 1.3M
agent-steps per second on a single server-grade GPU, far faster than existing object-level simulators, while keeping fidelity lighter single-
agent systems omit: heterogeneous
agents, multiple dynamics models, and full traffic-rule enforcement. TerraZero uses logged data only as a source of real-world map geometry, populating each map with randomized rule-based road users and signal controllers and randomizing
agent dynamics, rewards, and sizes per episode, so one map yields an effectively unbounded set of scenarios. Every reported policy trains from scratch by reinforcement learning alone, with zero human demonstrations, no imitation, no logged trajectories, and no fallback planner at inference, on a compute-efficient self-play recipe scaled across GPUs. The policies generalize zero-shot across cities and
datasets, including emergent left-hand-traffic driving without explicit supervision. As an ego policy, a single checkpoint is, to our knowledge, the first fully learned policy to top both val14 and the interactive long-tail InterPlan suite. On Waymo Open Sim
Agents realism the same recipe outperforms other demonstration-free methods and is competitive with the strongest reference-anchored self-play method. One stack serves both roles: state-of-the-art demonstration-free driving policies across dynamics for cars and trucks, and sim
agents that jointly control vehicles, pedestrians, and cyclists.
Read the full abstract ▾
All authors Zhouchonghao Wu, Akshay Rangesh, Weixin Li, Wei-Jer Chang, Zachary Lee, Saeed Bonab, Tim Wang, Wei Zhan
Robotics
FlowWAM: Optical Flow as a Unified Action Representation for World Action Models
Yixiang Chen, Peiyan Li, Yuan Xu et al. (+13 more)
World Action Models (WAMs) are able to leverage pretrained
video generators for both world modeling and action prediction. However, directly leveraging such
video generators for control raises a new challenge: how to represent actions in a suitable form that aligns with pretrained
video generators while carrying enough motion cues for accurate control. Existing numerical actions fail to satisfy the former, and prior visual action representations overlook the temporal motion structure across frames. We address this issue with FlowWAM, a dual-stream
diffusion framework that adopts optical flow as a unified,
video-native action representation. Flow
videos share the same format as RGB
videos and encode rich per-pixel displacement. By jointly modeling them within a shared pretrained
video generator, FlowWAM can naturally implement two modes of WAMs. In policy mode, FlowWAM generates flow for action prediction, while in world-model mode, it uses target flow sequences to guide future
video generation. Moreover, since flow can be easily extracted from raw
videos without action labels, FlowWAM can leverage large-scale action-unlabeled
video datasets for pretraining. We empirically find that our flow-based action representation delivers gains across both modes. On RoboTwin manipulation, FlowWAM raises the success rate to 92.94% on the Clean setting and 92.14% on Random, outperforming both VLA and WAM baselines. On WorldArena world modeling, it achieves the best overall EWMScore (63.71) with an 18.4% relative improvement in trajectory accuracy. More results can be found on our project website: https://flow-wam.github.io .
Read the full abstract ▾
All authors Yixiang Chen, Peiyan Li, Yuan Xu, Qisen Ma, Jiabing Yang, Kai Wang, Jianhua Yang, Dong An, He Guan, Gaoteng Liu, Jianlou Si, Jun Huang, Jing Liu, Nianfeng Liu, Yan Huang, Liang Wang
Robotics
ChunkFlow: Towards Continuity-Consistent Chunked Policy Learning
Zhao Yang, Yinan Shi, Mingyuan Yao et al. (+3 more)
Vision-
language action (VLA) models increasingly adopt chunked action heads to satisfy real-time constraints; however, this introduces boundary jitter: overlapping regions between consecutive chunks often yield inconsistent predictions, degrading temporal coherence and the task success rate. Existing methods, such as inference-time blending, merely reweight mismatched proposals without correcting underlying errors, leading to residual accumulation under biased or noisy histories. We propose ChunkFlow, a seam-aware training-and-execution framework for chunked policies that aligns chunk structure with boundary execution. It partitions each chunk into frozen, editable, and future zones, applies deterministic overlap blending at execution, and trains raw predictions with seam and first- and second-order continuity losses. History corruption and scheduled sampling improve robustness to executed-history errors, while an AWAC fine-tuning stage adapts the policy without removing these structural regularizers. Under mild smoothness assumptions, pre-blending seam discrepancies provably decay with increasing overlap. Experiments on CALVIN, LIBERO, and real
robots show an improved success-stability trade-off with low-latency inference. Project page: https://cytoderm-ai.github.io/chunkflow.
Read the full abstract ▾
All authors Zhao Yang, Yinan Shi, Mingyuan Yao, Wenyao Xue, Yawei Jueluo, Longjun Liu
Robotics
MAMMOTH: A Multi-Modal End-to-End Policy for Off-Road Mobility Robust to Missing Modality
Ahaan Kotian, Shivani Subramanyan and Suresh Sundaram
Reliable autonomous
navigation in unstructured off-road environments remains a critical unsolved challenge due to extreme terrain diversity, drastic illumination variations and acute sensor degradation. Recent developments have approached the problem as a traversability costmap estimation or visual
navigation task. However, many exhibit heavy reliance on RGB modality, leading to poor performance in varied illumination such as glares, shadows or low ambient light. Achieving robust generalization in such conditions requires integrating modalities that provide supplementary scene information. Such multi-modal methods suffer from a rigid dependency on the presence of near-perfect sensor inputs, leaving them unable to robustly handle sensor degradation or individual modality failure. To address these limitations, we introduce MAMMOTH (MAsking Multi-Modal inputs for Off-road Traversability Heuristic-informed
navigation), a unified end-to-end
navigation policy for robust off-road visual-goal-conditioned
navigation and undirected exploration. Specifically, MAMMOTH efficiently fuses multi-modal observations (RGB, Thermal, 3D Pointcloud and Ego Velocity) and is trained with a modality dropout scheme, enabling it to generalize to missing modalities at inference time. Furthermore, we employ a
diffusion policy to learn the joint conditional probability distribution of physically-grounded trajectories and a intrinsic traversability heuristic. MAMMOTH utilizes this heuristic to prefer safer, smoother trajectories. We validate MAMMOTH through extensive real-world
robot experiments in distinct off-road environments, including night-time operation. Our results demonstrate superior performance, with significant improvements in collision avoidance, terrain-aware planning and generalization to missing modalities. The code and
dataset used for this work will be made publicly available.
Read the full abstract ▾
Robotics
ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuning
Yilun Kong, Yunpeng Qing, Guozheng Ma et al. (+4 more)
Reinforcement Learning (RL) has demonstrated significant potential for improving
Vision-
Language-Action (VLA) models on complex manipulation tasks. However, its practical scalability remains severely limited by the substantial cost of environmental interactions. In this work, we first investigate the exploration stagnation bottleneck in current VLA-RL frameworks and reveal that trajectory diversity is fundamentally more important to sample efficiency than the sheer quantity of collected rollouts. Motivated by these insights, we introduce RL Exploration
Token (ExToken), a simple yet general framework that condition VLA policies on discrete behavioral priors derived from offline demonstrations for structured exploration. By conditioning the policy on different
tokens during rollout collection, ExToken encourages the
agent to explore diverse behavioral modes, substantially improving state-action coverage and exploration efficiency. To bridge exploration during training with deterministic inference at deployment, ExToken further incorporates a state-conditioned
token selector that adaptively predicts effective behavioral modes for unseen scenarios. Extensive experiments across simulated and real-world robotic manipulation tasks demonstrate that ExToken consistently accelerates convergence, improves task performance, and exhibits strong robustness under highly constrained interaction budgets.
Read the full abstract ▾
All authors Yilun Kong, Yunpeng Qing, Guozheng Ma, Haoyu Wang, Li Shen, Zhi Hou, Dacheng Tao
Robotics
UR-VC: Unsupervised Robotic Value Correction for Time-Derived Progress Proxies
Lirui Zhao, Modi Shi, Li Chen et al. (+3 more)
Modern
robot learning systems increasingly rely on dense progress or value signals to evaluate intermediate states, guide policy learning, and detect task completion, making the quality of these signals critical. Since such dense labels are rarely available at scale, normalized time within a demonstration is often used as a scalable substitute: later frames are treated as higher progress. However, this time-derived label is only a noisy proxy for physical task progress. In contact-rich manipulation, a
robot may make progress and then lose it through slips, failed grasps, or partial undoing, while the time-derived label continues to increase monotonically. We introduce Unsupervised Robotic Value Correction (UR-VC), an offline, training-free method for correcting time-derived progress labels. UR-VC exploits a simple regularity in demonstration data: similar states often recur across different episodes, but at different timestamps. Instead of trusting the timestamp from a single trajectory, UR-VC retrieves similar states from other episodes and aggregates their time-derived labels to obtain a corrected progress estimate. UR-VC requires no manual progress labels, reward annotations, or additional value model. We evaluate UR-VC on real bimanual cloth flatten-and-fold data, a long-horizon deformable-object manipulation task with visible intermediate progress. The corrected labels capture local regressions and non-uniform progress that normalized time cannot represent, while preserving the overall task trend. We further use the corrected signal to construct advantage labels for VLA training, following recent advantage-conditioned policy learning. UR-VC shows a positive trend in real-
robot task success under matched data, model, and training settings.
Read the full abstract ▾
All authors Lirui Zhao, Modi Shi, Li Chen, Qi Liu, Ping Luo, Hongyang Li
Robotics
Unveiling Complex Collective Behaviors from Simple Rewards
Yize Mi, Jianan Li, Liang Li et al. (+1 more)
Multi-
agent Reinforcement Learning (MARL) holds great potential for
robot swarms, but the black-box nature of neural policies complicates strategic analysis, limiting multi-
robot applications. Furthermore, complex swarm behaviors can surprisingly emerge from simple rewards without explicit aggregation incentives. Unveiling the mechanisms behind this emergence is critical, but the disconnection between simple rewards and collective behaviors exacerbates interpretability challenges. This paper aims to reveal the hidden mechanisms in this process. We propose a two-stage EEC (\LinkIII) explanatory framework. This includes a novel analytical tool called the
Agent Response Map (ARM), which reveals
agents' decision-making patterns across space and identifies regions of aggregation and avoidance. ARM reveals that the
robots implicitly learn the
geometric fields of the environment and utilize these structures as desired targets for coordinated movement. We validate this finding across two distinct tasks: a cooperative multi-
robot shape assembly and a competitive predator-prey pursuit-evasion. 1) In the cooperative task, ARM identifies the unoccupied target interior as the desired destination for
robot navigation. As the center becomes occupied, this target region automatically shifts toward the boundary, demonstrating the
robots' capacity to autonomously explore unoccupied areas. 2) In the competitive task, ARM surprisingly identifies the boundary of the predators' Voronoi diagram as the convergence destination for prey
agents. Together, these two tasks demonstrate the capability of ARM to discover the hidden
geometric structures underlying MARL policies in
robot swarms.
Read the full abstract ▾
All authors Yize Mi, Jianan Li, Liang Li, Shiyu Zhao
Robotics
PixelLoop: Shortcut Topological Navigation with Pixel-Level Loops
Sarthak Chittawar, Vansh Garg, Aditya Vadali et al. (+4 more)
Although topological mapping and
navigation have been studied extensively, the specific role and downstream effect of loop closures in purely topological representations has received relatively little
attention. Importantly, loop closure over topological maps is distinct from loop closure over globally referenced trajectories and metric maps. Building on recent denser
topologies grounded in pixel-level, relative 3D geometry, we propose PixelLoop which introduces loop closures directly in pixel space. Unlike sparse image-level edges or pose-graph corrections in SLAM, our pixel-level closures act as dense topological shortcuts that alter planning connectivity and cost propagation rather than merely aligning coordinates. This dense connectivity enables stable any-point-to-any-point
navigation and produces costmaps that align accurately with
geometric shortest paths. In particular, we showcase the distinct advantage of applying loop closures to fine-grained pixel
topologies rather than image-level
topologies. Across extensive simulated experiments, PixelLoop achieves over 35% absolute improvement in both Success Rate and SPL compared to image-relative baselines, with the largest gains in scenarios requiring shortcut exploitation. Results are further validated through real-world mobile
robot deployments, demonstrating that dense pixel-level loop closures provide a practical and robust foundation for topological visual
navigation. Project Page: https://pixelloop-nav.github.io/
Read the full abstract ▾
All authors Sarthak Chittawar, Vansh Garg, Aditya Vadali, Krish Pandya, Rohit Jayanti, Sourav Garg, Madhava Krishna
Robotics
Autonomous Tracking and Terminal Guidance of Moving Targets for Fixed-Wing UAVs
Wei-Hao Liou and Teng-Hu Cheng
This study introduces a unified control framework for fixed-wing unmanned aerial vehicles (
UAVs) fitted with a pan-tilt (PT) camera, intended to perform an end-to-end mission spanning from initial target detection to accurate terminal engagement. The proposed system employs a three-phase strategy: a
vision-based target acquisition phase, an NMPC-based tracking phase, and a terminal guidance phase. During tracking, the framework uses an Unscented Kalman Filter (UKF) to fuse YOLO-based visual detections with inertial measurements, enabling robust target state estimation under unknown dynamics. To ensure reliable visual contact, we introduce a constraint-aware Nonlinear Model Predictive Control (NMPC) strategy that incorporates Control Barrier Functions (CBFs) to explicitly prevent
UAV self-occlusion – a common limitation in fixed-wing tracking. Upon satisfying terminal engagement conditions, the system seamlessly transitions control to a quaternion-based Biased Proportional
Navigation Guidance (BPNG) law, enforcing precise impact angle constraints. High-fidelity simulations demonstrate that the framework achieves stable, robust tracking and accurate terminal interception while strictly respecting the vehicle's dynamic limits and camera field-of-view constraints.
Read the full abstract ▾
Robotics
Directional Constraints for Efficient Exploration in Safe Reinforcement Learning
Paolo Magliano, Puze Liu, Jan Peters et al. (+2 more)
Reinforcement Learning has revolutionized the landscape of robotic research, allowing robust learning of complex robotic skills in simulation. However, real-world deployment in open-ended environments requires strong safety guarantees to prevent dangerous or harmful behaviors. Safe Reinforcement Learning methods address this requirement by enforcing safety constraints. Nevertheless, learning under constraints often reduces learning speed and could lead to suboptimal task performance, as the
agent must solve a more complex constrained optimization problem compared to unconstrained settings. To tackle this issue, in this work, we propose an extension of the ATACOM framework, a state-of-the-art reliable safety layer that can be integrated with existing Reinforcement Learning
algorithms to enforce constraints derived from prior knowledge of the system or learned directly from data. Our proposed method, named ATACOM Directional Constraints (ATACOM-DC), significantly improves the safety-performance trade-off by introducing directional constraints that distinguish between actions approaching and moving away from constraint boundaries, activating constraint enforcement only when necessary. We evaluate our method across a range of challenging robotic control tasks in simulation, analyzing both constraint-violation costs and achieved task performance. Code and additional material at https://atacom-dc.
robot-learning.net.
Read the full abstract ▾
All authors Paolo Magliano, Puze Liu, Jan Peters, Davide Tateo, Raffaello Camoriano
Robotics · Cross-listed from cs.HC
Practical Judgment, Virtue, and Intuition in the Use of Opaque AI-Enabled Systems
Nathan G. Wood and Andrew P. Rebera
AI-enabled systems are seeing increasing deployment across numerous domains, with many being "black boxes" with respect to core functions and capabilities. I.e., many systems take inputs and give outputs, but without users having any ability to see how the former lead to the latter. AI-enabled systems are also being used to augment autonomy in systems, and autonomy coupled with opacity raises numerous concerns surrounding, e.g., the reliability of systems, their regularity in functioning, human ability to control them, or whether deploying opaque and potentially autonomous systems is in compliance with ethical and legal norms. In this article, we argue that many of these worries can be mitigated by leveraging practical judgment, virtue, and intuition in the deployment and use of opaque AI-enabled systems. We show that focusing on these distinctly human capabilities provides a means for bridging between the practical challenges created by opacity and the ethical, legal, and social norms underpinning particular domains. We argue that a core element in doing this is a recognition that many positive human traits are not quantifiable and we therefore must develop training regimen and guidelines on AI deployment anchored in humanistic but non-quantifiable values. Throughout the article, we focus on the military domain as an exemplar of the importance of practical judgment, virtue, and intuition as drivers for ethical and effective human decision-making surrounding AI deployments, but the underlying arguments apply to all domains where opaque and potentially autonomous systems are being deployed (subject to domain-specific alterations).
Read the full abstract ▾
Robotics
Globalized Constrained Stein Variational Inference for Diverse Feasible Robot Motion Planning
Jiayun Li and Georgia Chalvatzaki
Robot motion planning is inherently
multimodal, yet classical planners typically return only a single solution. Probabilistic formulations address this limitation by maintaining a distribution over motions, allowing the planner to reason over multiple low-cost alternatives. In robotics, however, motion samples must also satisfy strict constraints, including collision avoidance, joint limits, contact conditions, and dynamics consistency. These hard requirements make motion sampling substantially more challenging: within a limited planning budget, the ensemble must cover diverse low-cost motions while ensuring that every sample remains feasible under the relevant constraints. We propose SteinSQP (Stein Variational Sequential Quadratic Programming), a constrained Stein variational inference method for diverse feasible
robot motion sampling. SteinSQP evolves an interacting particle ensemble, as in Stein variational methods, while
embedding constraints directly into a kernel-space SQP subproblem. We solve the resulting constrained Stein-Newton subproblem with a GPU-friendly
matrix-free primal-dual
algorithm, enabling efficient batched ensemble updates. To globalize the method, we introduce an ensemble-level merit function that jointly balances objective value, constraint violation, and particle diversity. Across five constrained motion-planning tasks, SteinSQP returns fully feasible ensembles while preserving diverse motion alternatives. Compared with first-order constrained Stein baselines and serial multistart nonlinear programming, SteinSQP shows faster and more robust ensemble convergence in terms of iterations, improves particle-wise feasibility, and achieves faster batched time-to-solution on challenging
robot-scale tasks.
Read the full abstract ▾
Robotics
Vision-Based Dribbling for Humanoid Soccer via Privileged Representation Learning
Flavio Maiorana, Valerio Spagnoli, Eugenio Bugli et al. (+5 more)
Recent advances in
humanoid robotics have highlighted the importance of deployable loco-manipulation skills. Dribbling a soccer ball while evading active opponents requires simultaneous balance, precise ball control, and awareness of a dynamic adversary under onboard sensing and real-time constraints. Existing approaches typically separate perception and motion, which can be effective in controlled settings but may fail under occlusions, fast ball movements, and complex opponent interactions, since perception is not directly optimized for control. We propose an integrated approach in which a temporal depth encoder is embedded into a reinforcement learning policy through a task-specific projection layer. We apply this framework to a simulated Booster T1
humanoid robot and show that it is possible to learn
vision-based, opponent-aware dribbling directly from depth observations, without explicit state estimation or privileged scene information. The learned policy achieves 100% success in nominal target-driven dribbling and 96% success with a single static obstacle, while reaching 46% success against an actively moving ball-attacker opponent. These results demonstrate that the proposed framework supports robust
vision-based dribbling in nominal and moderately dynamic settings, and provides a strong foundation for handling more challenging moving-adversary scenarios.
Read the full abstract ▾
All authors Flavio Maiorana, Valerio Spagnoli, Eugenio Bugli, Flavio Volpi, Daniele Affinita, Vincenzo Suriani, Daniele Nardi, Luca Iocchi
Robotics
EagleVLA: Towards Onboard Real-Time Robot Control via Foresight-Aligned Asynchronous Inference
Zebin Yang, Qi Wang, Yunhe Wang et al. (+6 more)
Vision-
Language-Action (VLA) models have achieved impressive performance on diverse
embodied tasks. However, deploying VLA models on low-power onboard devices, such as the Jetson Orin, remains challenging due to their high computational complexity, which leads to substantial inference latency and low control frequency. Asynchronous inference can partially mask this latency by parallelizing action execution and subsequent inference, but it introduces two critical issues: perception-execution misalignment and long reaction time. In this paper, we propose EagleVLA, a method for efficient VLA deployment on onboard devices via Foresight-Aligned Asynchronous Correction. To address misalignment, we train a lightweight future correction module that predicts future environment representations, allowing the action expert to predict actions from the future time step. To reduce reaction time, we introduce confidence-based scheduling optimization that adaptively balances VLM and action expert invocations. We also build a llama.cpp-based inference engine tailored for onboard VLA deployment, with system-level optimizations including CUDA graph reuse, GPU-resident intermediate buffering, and flow unrolling. Extensive experiments demonstrate that EagleVLA achieves 9.85x and 6.19x improvements in control frequency compared with naive PyTorch and vla.cpp on Jetson Orin, while outperforming VLASH by 14.8% in success rate on LIBERO
benchmark. The code of our asynchronous
algorithm is available on https://github.com/PKU-SEC-Lab/EagleVLA, and our efficient llama.cpp-based inference engine is available on https://github.com/PKU-SEC-Lab/EagleVLA-Edge.
Read the full abstract ▾
All authors Zebin Yang, Qi Wang, Yunhe Wang, Xiurui Guo, Bo Yu, Shaoshan Liu, Jiafeng Xu, Hao Dong, Meng Li