Robotics
Probabilistic Plan Legibility with Off-the-shelf Planners
Michele Persiani and Thomas Hellström
Legible planning is the creation of plans that best disambiguate their goals from a set of other candidates from an observer's perspective. In this paper we propose a method for legible planning for arbitrary PDDL domains, by extending previous research on legibility to classical planning without requiring to construct ad-hoc planners. We also discuss how the observer perspective may be estimated through a second order theory of mind that connects the planner's and the observer's task spaces. Our solution can for example be deployed in human-
robot teaming scenarios, where an autonomous
robot in a team can implicitly communicate its goal by producing legible plans. We present
benchmark results on several PDDL planning domains. Our results generally show that plan legibility is a trade-off with plan efficiency, however, not all planning domains allows to increase legibility in the same way and a regularizing factor to balance legibility and efficiency was proved necessary.
Read the full abstract ▾
Robotics
Multi-Reference Path Tracking Control for an Agricultural Tractor with Nonlinear Model Predictive Control
Marcel Moll and Timo Oksanen
Guiding a tractor along a predefined reference path is a key component of precision agriculture. This study develops a path tracking controller based on Nonlinear Model Predictive Control, which incorporates multiple segments of a piecewise-linear reference path directly into the objective function. In addition, methods for selecting viable reference segments from the full path are presented. The control system is evaluated during a field test with a tractor controlled via the Tractor Implement Management steering interface. The NMPC solver converged on average after 3.45 ms and tracked the curved reference path with a mean absolute cross-track error of 6.1 cm.
Read the full abstract ▾
Robotics
Physical Kernel: Structured Visual Latents for Dark Manipulation
Jinting Hang, Hong Li, Zhenhui Cai et al. (+2 more)
We study dark manipulation: after a brief lit Write encodes z0 = Enc(rgb), a policy pi(z) and open-loop dynamics f(z,a) complete contact-rich skills without further pixels (dark_f). On ManiSkill StackCube (n=160; seed packs 0/1000), dark_f attains 68.1% stacked on the five-rung chain (near_A -> grasped -> lifted -> on_B -> stacked), compared with 35.6% for per-step lit_reenc and 0% for freeze/encode_black. On a shared Write->HOLD protocol (n=40), occlusion and camera-aligned GT contact-neighbor masks drive lit lift from 43% to 0%, while dark_f holds 82.5%; shuffling actions inflates dynamics MSE by ~9.4x; write-time appearance shifts break encoding (night: 0% stacked), yet the same shifts during HOLD leave dark_f lift unchanged; Write length Tw is flat once the stop phase is reached, while earlier stops and write-time blur/JPEG sharply cut stacked. A dedicated pi_write reaches 35%
vision-budget stacked (n=80); matched Dreamer-style/pixel nulls without privileged geom stay at 0%. Privileged state-RSSM MPC reaches ~35% stacked with 9D dark observations – a stronger-observation null, not a matched visual baseline.
Read the full abstract ▾
All authors Jinting Hang, Hong Li, Zhenhui Cai, Zhihao Zhao, Jian He
Robotics · Cross-listed from cs.CV
Time-Aware Assistive Navigation
Masaki Kuribayashi, Zhongkai Shangguan and Eshed Ohn-Bar
Can interactive
vision-and-
language agents learn not just what to say but also \textbf{when} to say it? Current
language models rarely plan over whether and when to realize a real-time response to a user. However, providing accurate and timely support for human decision-making, such as when guiding visually impaired individuals through urban environments, requires careful real-time responsiveness–poorly timed responses can distract users or add unnecessary cognitive load. As a machine intelligence challenge for
Multimodal Large
Language Model (MLLM)-based
agents, we introduce a large-scale
multimodal benchmark for an egocentric, assistive navigation task in complex outdoor environments. Using this
benchmark, we uncover a fundamental limitation of off-the-shelf MLLMs in delivering safe and time-sensitive navigation instructions, even with model fine-tuning on substantial amounts of data. We then demonstrate that a simple yet effective modification of the model, including direct supervision to predict the underlying reason for each instruction, yields significant performance gains across open-loop, closed-loop, and sim-to-real generalization settings. However, our analysis highlights persistent challenges in temporal
reasoning, safety-critical object awareness, and relational and distance understanding. To advance the development of scalable assistive
agents, we will release our simulation,
benchmark, and code (available at the project website: https://timeli-icra.github.io/).
Read the full abstract ▾
Robotics
Adaptive Cost-Sensitive Machine Learning for Autonomous Robot Navigation Failure Prediction: When Not All Errors Are Equal
Rifa Ferzana
Autonomous
robot navigation failures differ not only in categorical severity but also in the physical context in which they occur. A near-miss at low speed under reliable sensing is not equivalent to the same event during rapid motion, close obstacle approach or degraded perception. This paper reframes navigation failure prediction as consequence-sensitive forecasting. We first establish a fixed baseline in which training weights are modulated by categorical severity, then introduce an adaptive extension defining a state-dependent consequence function combining severity with normalised velocity, obstacle proximity and sensing uncertainty, together with a risk-sensitivity term that rises as conditions deteriorate. We evaluate on 2,000 simulated differential-drive episodes (~1,000,000 timesteps) using episode-level GroupKFold, with external validation on the UCI SCITOS G5
dataset. Fixed weighting raises Logistic Regression high-severity recall from 0.851 to 0.985 and reduces missed consequence cost from 1,940 to 313; the adaptive extension reaches 0.998 and 82. Under matched false-positive conditions, however, the discriminative advantage is modest (0.986 versus 0.984), so most of the gain reflects a more conservative operating point rather than better ranking. The effect is consistent across all five folds and stable across a threefold span of context coefficients. Because the primary simulation produced no collisions, we add a controlled extension in which 108 of 600 episodes terminate in contact: collision recall rises from 0.850 to 0.966 (fixed) and 0.984 (adaptive), with missed collision cost falling from 1,000 to 105, at false-positive rates of 0.413 and 0.799, respectively. Context-dependent consequence modelling thus provides a principled mechanism for allocating conservatism by physical risk.
Read the full abstract ▾
Robotics
GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation
Renhang Liu, Wenzhi Zhao, Zhuo Yang et al. (+42 more)
World-action models (WAM) predict future states to guide
robot actions, enabling learning from both action-free
video and action-labeled interaction. Most inherit pretrained
video generators, leaving WAM pretraining and scaling underexplored. We introduce Genie Envisioner Act 2.0 (GE-Act 2.0), a world-action model whose trainable generative and action components are all initialized from scratch on manipulation data. It combines a control-oriented autoencoder (CoAE), a single-step visual planner (SVP), and an inverse dynamics model (IDM). CoAE retains action- and instruction-relevant information under aggressive compression, while SVP produces a complete future state in one differentiable pass, so visual planning and inverse dynamics can be pretrained separately on complementary data. The components are then jointly trained with knowledge-aligned selective optimization (KASO), which reduces mismatched supervision by selecting only predicted futures judged behaviorally compatible with the recorded action. We evaluate pretrained checkpoints directly, without per-task fine-tuning, on 100 tasks across 20 manipulation skill groups with held-out scenes, backgrounds, lighting, and object instances. Scaling co-training data from 300 to 30,000 hours raises success from 17.1% to 44.1% on G1-OP and from 13.4% to 31.1% on G2-90D; despite comprising less than 2% of the co-training data, G2-90D improves by 17.7 points, suggesting cross-embodiment transfer. Gains span 19/20 and 18/20 skill groups, and skill-specific coverage strongly correlates with zero-shot out-of-distribution (OOD) success (Pearson r=0.80; Spearman rho=0.85). Under the same protocol, the model grounds object, color, shape, and position references in at least 90% of trials and follows explicit instructions even when they conflict with an already-committed behavior or a conventional scene association.
Read the full abstract ▾
All authors Renhang Liu, Wenzhi Zhao, Zhuo Yang, Liliang Chen, Pengfei Zhou, Shengcong Chen, Guanghui Ren, Youlun Peng, Rongjun Jin, Nan Wang, Sukai Wang, Xindong He, Jinyuan Feng, Ziyu Xiong, Linqing Zhong, Yifei Wei, Feng Han, Long Zhang, Da Huang, Nanshu Zhao, Chenghao Yin, Mo Wu, Zhaodong Yan, Kongtao Hu, Yuxiang Yan, Aogelijiang Niyazi, Yu Fang, Jia Zeng, Lizhu Meng, Daizhen Lv, Haoyu Cao, Zhiwen Hou, Lianjin Ye, Yuehan Niu, Zhikai Cai, Xuan Hu, Hui Min, Xiongfeng Cai, Yue Liao, Jing Wu, Soujanya Poria, Ye Li, Sanping Zhou, Maoqing Yao, AgiBot Research Team
Robotics
Benchmarking Dexterity of Multifingered Robot Hands: A Review and Perspective
Anthony Shilati, Anunth Ramaswami, Luke Batteas et al. (+16 more)
Robot hands are a key interface between AI and the physical world, making advances in robotic dexterity essential to realizing the
vision of physical AI. While impressive dexterity has been demonstrated with simple grippers, multifingered hands offer the potential for substantially greater versatility, precision, and adaptability in manipulation. In this review, we survey the state of the art in benchmarking the dexterity of multifingered
robot hands. Recognizing dexterity as a complex and multifaceted concept, we present the perspective of the U.S. National Science Foundation HAND Engineering Research Center, with a particular focus on fine in-hand manipulation. We introduce a framework consisting of three
benchmark levels that correspond to increasing system complexity, review representative
benchmarks at each level, and propose new
benchmarks and metrics to address limitations in the literature. More information can be found at https://hand-erc.github.io/benchmarking/.
Read the full abstract ▾
All authors Anthony Shilati, Anunth Ramaswami, Luke Batteas, Sylvia Tan, Anthony Barcio, Sairam Umakanth, Preksha Rao, David McDougall, Landry Graves, Ahmet A. Ozkan, Yunsoo Yoon, Arushi Pradhan, Michael G. Henry, Rohan Kota, Damian Gonzalez, Gray C. Thomas, Gary K. Fedder, J. Edward Colgate, Kevin M. Lynch
Robotics
Information-Guided Safe Reinforcement Learning for Autonomous Gas Source Localization using sUAS
Sachin Giri, Thomas Zhao, Matthew Huynh et al. (+1 more)
The autonomous localization of fugitive gas emissions using small Unmanned Aircraft Systems (sUAS) constitutes a fundamentally ill-posed inverse problem. In turbulent atmospheric boundary layers, highly intermittent scalar concentration fields violate the assumptions of classical gradient-based navigation, causing data-driven
estimators to suffer from severe noise and spurious local minima. To address these challenges, we introduce an Information-Guided Safe Reinforcement Learning framework evaluated within a custom, GPU-accelerated 3D simulation environment coupling an Eulerian wind solver with a Lagrangian puff dispersion model. We identify a critical vulnerability in deterministic information-seeking planners - a Gramian bias where
agents act greedily upon flawed early estimates, starving the
estimator of spatial diversity. To systematically break this degeneracy, our architecture integrates a classical empirical observability Gramian (EMGR) planner with a learned Soft Actor-Critic (SAC) exploratory policy. A deterministic meta-supervisor actively monitors
estimator reliability via Kullback-Leibler (KL) divergence, dynamically blending deterministic exploitation with learned exploration to steer the sUAS into high-information zones. Trained via a progressive curriculum and safeguarded by a strictly enforced Robust Control Barrier Function (RCBF), our RL framework achieves nearly 80% localization success on complex, mobile sources - drastically outperforming classical baselines (~30%) - while ensuring zero safety violations.
Read the full abstract ▾
All authors Sachin Giri, Thomas Zhao, Matthew Huynh, YangQuan Chen
Robotics
Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models
Wonje Jeung, Sangyeon Yoon, Hyesoo Hong et al. (+6 more)
Vision-
language models are increasingly used as reward functions for robotic learning, but this role requires paraphrase invariance: the same trajectory should receive the same reward under semantically equivalent goal descriptions. We show that current VLM reward models often violate this property. Paraphrasing the instruction alone can substantially change predicted progress scores, and can even flip identical
robot behavior between failure and success. To measure this failure mode, we introduce ROBORMBENCH, a
benchmark with 2,390 real-
robot trajectories, ground-truth progress labels, and 21,673 verified paraphrases spanning lexical, syntactic, and action-goal rewrites. Across proprietary and open-source VLMs, paraphrase-induced instability is widespread and severe, grows under more divergent rewrites, and is not reliably reduced by scale or explicit
reasoning. Dedicated reward models trained with trajectory-grounded supervision are substantially more stable. These results show that paraphrase robustness is a core requirement for reliable VLM-based reward modeling in robotics.
Read the full abstract ▾
All authors Wonje Jeung, Sangyeon Yoon, Hyesoo Hong, Yoonjun Cho, Dongjae Jeon, Bumjun Kim, Jean Oh, Youngjae Yu, Albert No
Robotics · Cross-listed from cs.CV
CrossDepth: Geometry-Constrained Attention for Generalizable Multi-View Surround Depth Estimation
Samer Abualhanud and Max Mehltretter
Reliable 3D understanding of the surrounding environment is a core requirement for autonomous driving. Multi-view surround camera rigs provide broad scene coverage, but the spatially adjacent images typically overlap only minimally. Consequently, the depth of most pixels must be inferred from monocular appearance cues. These cues can appear differently across images and may therefore be interpreted differently by the depth estimation model. We target two main sources of cross-image inconsistency: differences in camera intrinsics and the limited receptive field of each image. We address the former by conditioning the features on per-pixel camera-aware ray
embeddings, enabling the network to account for camera-dependent variations in monocular cues. We address the latter by extending each pixel's context beyond its own image through cross-image
attention constrained to geometrically plausible regions, derived from the calibrated rig setup. The model is trained in a fully self-supervised manner based on photometric consistency. Evaluations on DDAD and nuScenes show improved overall depth accuracy and cross-image depth consistency over state-of-the-art self-supervised methods under in-domain and cross-domain evaluation. Code is available at https://abualhanud.github.io/CrossDepthPage/.
Read the full abstract ▾
Robotics
What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies
Vivek Chavan, Pengtao Xie, Yahuan Shi et al. (+3 more)
Visuomotor imitation policies can achieve high performance under in-distribution visual conditions yet fail when visually similar objects or receptacles are introduced. We study this behavior as a problem of conditional visual grounding: the visual target required for successful control changes with the manipulation phase and, in more complex tasks, with the observed task state. Using Action Chunking with
Transformers (ACT), we systematically introduce distractor objects and receptacles with controlled color and shape similarity and localize failures to picking and placement. We find that distractor sensitivity is specific to both the type of visual similarity and the manipulation stage. Guided by this diagnosis, we evaluate distractor augmentation, phase-dependent
attention regularization, and appearance-based visual prompting as complementary interventions for improving target selection while preserving spatial information required for control. These interventions substantially improve robustness in simulation and on a physical UR3e. We further examine the same failure pattern in a pretrained
vision-
language-action policy on a state-conditioned instrument-handling task, where the observed state of a medical instrument determines the correct destination. Together, the results show that visual distractors can cause incorrect object or destination selection even when the underlying manipulation skill remains intact, and that explicitly improving target selection can substantially recover performance across distinct visuomotor policy-learning regimes.
Read the full abstract ▾
All authors Vivek Chavan, Pengtao Xie, Yahuan Shi, Oliver Heimann, Kevin Haninger, Jörg Krüger
Robotics
Towards Neuro-Symbolic Procedural Reasoning for Long-Horizon Vision-Language-Action Manipulation
Vivek Chavan, Yahuan Shi, Oliver Heimann et al. (+2 more)
Vision-
language-action (VLA) models can execute short manipulation skills, but remain brittle in long-horizon procedures requiring persistent task state, dependency-aware
reasoning, conditional decisions, and reliable grounding. We investigate a neuro-symbolic framework that combines learned VLA control with explicit task graphs and
multimodal procedural
memory. Task graphs encode action dependencies, valid transitions, and branch conditions, while
memory maintains the active step, completed actions, textual context, and task-relevant visual evidence. Together, these structures guide object selection, destination grounding, subgoal dispatch, and verification of expected state transitions. Human demonstrations provide additional spatial and temporal guidance through gaze or saliency cues. To isolate their effect on policy learning, our initial study bypasses cross-view gaze transfer and directly annotates pseudo-gaze in
robot-view teleoperation
videos. The resulting guidance is used during VLA fine-tuning and inference. We study two long-horizon manipulation domains, workspace clearing and surgical-instrument handling, which require ordered execution, visually grounded decisions, and conditional branching. We evaluate correct-object and destination selection, subtask completion, task progress, step-order consistency, complete-task success, and procedural or execution mistakes. This work positions structured symbolic
reasoning and demonstration-derived visual guidance as complementary mechanisms for reliable long-horizon VLA manipulation.
Read the full abstract ▾
All authors Vivek Chavan, Yahuan Shi, Oliver Heimann, Kevin Haninger, Jörg Krüger
Robotics
Development of a Humanoid Robot Prototype for Multimodal Human-Robot Interaction
Thang Tran Viet, Thanh Nguyen Canh, Huy Uong Gia et al. (+4 more)
Human-
robot interaction (HRI) enables intuitive and intelligent collaboration between humans and
robots in real-world environments. This paper introduces a humanoid
robot prototype designed as a flexible testbed for developing and integrating artificial intelligence (AI) modules in HRI tasks. The system features a 12 degree-of-freedom (DOFs) dual-arm mechanism and a 2 DOFs head with an expressive LCD screen to express facial emotions. All hardware components are controlled by a custom-designed controller board with real-time AI processing supported by an onboard Jetson module. The system incorporates three AI modules: (1) gesture recognition using MediaPipe Pose and an LSTM classifier, (2) object detection with YOLO and 3D localization, and (3) voice-command processing through
speech recognition and large
language model(LLM)-based semantic parsing. The platform is validated through experiments on positioning accuracy, with results showing average manipulation errors of approximately 1.83 cm. To demonstrate its versatility, experimental results show over 90% task accuracy, with gesture recognition reaching 96%,
speech recognition reaching 92%. The results confirm the effectiveness of the proposed system as a reproducible and accessible humanoid platform for research and prototyping in HRI.
Read the full abstract ▾
All authors Thang Tran Viet, Thanh Nguyen Canh, Huy Uong Gia, Phuc Dinh Van, Son Tran Duc, Ngoc Minh Do, Xiem HoangVan
Robotics
Adaptation Needs in Robotic Systems: Assessing Behavior Trees and Their Enhancement
Mehran Rostamnia, Gianluca Filippone, Ricardo Caldas et al. (+1 more)
Robotic systems increasingly operate in dynamic, uncertain, and open-ended environments, where design-time assumptions may no longer hold, and adaptation becomes necessary to maintain effective and safe operation. Behavior Trees (BTs) are widely used in robotic control architectures due to their modularity, readability, and reactivity. This raises a central question: are BTs sufficient to meet the adaptation needs of modern robotic systems? This paper investigates this question through a literature-driven study complemented by empirical validation. First, we derive a classification of robotic adaptation needs from the literature, organizing them into six categories: Knowledge, Perception, Actuation, System, Mission, and Environment. Then, we analyze the capabilities and limitations of classical BTs with respect to these needs. Then, we characterize BT-based approaches for adaptation from the existing literature and organize them into four primary families, i.e., generation, extension,
evolution, and refinement, including approaches that combine multiple families. Our analysis shows that the modularity, flexibility, and reactivity of classical BTs are insufficient for adaptation needs involving runtime restructuring,
reasoning under uncertainty, mission reinterpretation, learning, or integration with external knowledge and planning mechanisms. Enhanced BT approaches address several of these limitations, but to different extents and often with limitations of their own. Our findings relate adaptation needs to both the capabilities and limitations of classical and enhanced BTs, providing guidance on when classical BTs are sufficient, when enhanced mechanisms are needed, and which challenges remain or emerge for adaptive robotic control architectures.
Read the full abstract ▾
All authors Mehran Rostamnia, Gianluca Filippone, Ricardo Caldas, Patrizio Pelliccione
Robotics
FIRE-LIVWO: Robust LiDAR-Inertial-Visual-Wheel Odometry via Failure-Immune mmWave Radar Enhancement
Kun Hu, Menggang Li, Kaidi Wu et al. (+5 more)
Achieving robust SLAM in large-scale underground coal mines with complex structures and severe degeneracies remains highly challenging. Dense smoke and dust cause substantial loss of visual information and degrade
LiDAR point-cloud features, while long, self-similar corridors induce
geometric degeneration, leading to pronounced odometry drift. To address these issues, we propose FIRE-LIVWO: Failure-Immune mmWave Radar-Enhanced
LiDAR-Inertial-Visual-Wheel Odometry, a tightly coupled multi-modal odometry framework based on an iterated error-state Kalman filter (IESKF). The framework fuses 4D mmWave radar,
LiDAR, and visual features within a unified VoxelMap and jointly constructs
LiDAR-radar point-to-plane residuals and sparse visual photometric residuals. In smoke-filled environments, we exploit the strong penetration of 4D mmWave radar and introduce pointwise Doppler velocity constraints to preserve state observability. In geometrically degenerate corridors, we tightly couple wheel odometry using non-holonomic constraints (NHC) and online lever-arm compensation to reduce drift. Our central contribution is a degeneration detection and adaptive fusion model switching strategy grounded in
geometric and visual observability analysis, which quantifies observability online and dynamically adjusts modality weights. Real-world experiments in underground coal mines demonstrate that FIRE-LIVWO accurately identifies failure boundaries, enabling reliable modality switching under extreme conditions. Compared with baselines, it achieves superior accuracy and robustness (average localization error of 5.677m). We open source our code on Github to benefit the robotics community.
Read the full abstract ▾
All authors Kun Hu, Menggang Li, Kaidi Wu, Zhiwen Jin, Yingjie Zhao, Chaoquan Tang, Eryi Hu, Gongbo Zhou
Robotics
RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?
Zhenxuan Fan, Bo Zhang, Yutong Lin et al. (+9 more)
Vision-
Language-Action (VLA) models have shown promising progress in
language-conditioned robotic manipulation. However, existing
datasets and
benchmarks mainly evaluate task completion under predefined settings, offering limited insight into model
reasoning under increasing spatial and procedural complexity. We introduce RoboSPA (Robot Spatial-Procedural Assessment), a large-scale robotic manipulation
dataset and
benchmark for diagnosing embodied
reasoning in VLA models. RoboSPA focuses on two core dimensions, Fine-Grained Spatial
Reasoning and Long-Horizon Procedural Planning, covering 10 task categories and 56 base tasks. Each task is instantiated across five difficulty levels, yielding 280 variants with increasing spatial ambiguity and procedural complexity. We collect 527K trajectories across multiple embodiments and diverse scenes. Beyond binary success rate, RoboSPA introduces diagnostic metrics for more detailed evaluation. Experiments on representative VLA models show that current systems still struggle with complex spatial relations, precise low-level execution, and
memory-intensive planning. These results establish RoboSPA as a challenging diagnostic
benchmark for developing more capable, reliable, and generalizable embodied
agents. Our data and code are available at https://github.com/fanzhenxuan/RoboSPA.
Read the full abstract ▾
All authors Zhenxuan Fan, Bo Zhang, Yutong Lin, Yuqian Yuan, Juekai Lin, Liang Liang, Zhuoyi Huang, Wenqiao Zhang, Juncheng Li, Siliang Tang, Jun Xiao, Yueting Zhuang
Robotics
Human-Human & Human-Robot Interaction Transformer (H2INT) for Robot Navigation in Dense and Uncertain Crowds
Ao Shen, Kaixi Chen, Shiwei Liu et al. (+2 more)
Safe
robot navigation in dense crowds requires
reasoning about pedestrian motion and how it may change in response to a
robot. However, many learning-based approaches generate pedestrian motion independently of the
robot or assume uniform reciprocity, omitting an important source of interaction uncertainty. This paper presents a Human-Human & Human-
Robot Interaction
Transformer (H2INT), a reinforcement learning framework that retains
robot-conditioned changes in pedestrian motion during policy learning while allowing responsiveness to vary across pedestrians. Responsiveness affects the crowd dynamics when the
robot is visible but is not supplied as a policy input; the policy must instead infer its consequences from
robot-centered relative positions. A two-stage gated
Transformer progressively encodes human-human and human-
robot relations, while a recurrent policy captures their temporal
evolution. A curriculum gradually reduces pedestrian responsiveness to increase interaction difficulty. Simulation experiments demonstrate improved navigation safety and robustness over representative baselines across response conditions and crowd densities, and show transfer without retraining to structurally distinct crowd-flow layouts. Ablations support the hierarchical relational encoding and gated updates. Real-
robot deployment further verifies that the learned policy can operate with sparse observations in a physical environment.
Read the full abstract ▾
All authors Ao Shen, Kaixi Chen, Shiwei Liu, Fang Deng, Chen Chen
Robotics
Temporal Tactile Encoding and Compliance for Intent-Aware Robot-to-Human Bimanual Handover
Pasquale Marra, Stefano Berti, Gabriele Mario Caddeo et al. (+1 more)
Reliable
robot-to-human handover requires the
robot to infer when the person is ready to receive the object, and release it safely, comfortably, and at the right time. This is challenging because visual observations alone may not disambiguate clear taking intent from accidental contact, weak grasping, wrong-direction forces, or transient interactions. In this work we treat human-
robot handover as an intrinsically
multimodal problem. Our approach couples a VLA model with a compliance controller that reduces interaction forces during object transfer. We finetune the VLA model with human demonstrations using RGB observation, temporally encoded tactile feedback and proprioception. We evaluate the complete system in a human-subject study against two baselines: one without tactile feedback and one using tactile feedback without compliance control. We hypothesize that combining compliance and temporal tactile encoding yields the most reliable and comfortable handovers, as compliance facilitates physical interaction while tactile history captures sustained taking intent. Performance is measured through objective metrics and an ad-hoc questionnaire. The results show that the two components provide complementary benefits and substantially outperform the baselines. Code and data will be released upon acceptance.
Read the full abstract ▾
All authors Pasquale Marra, Stefano Berti, Gabriele Mario Caddeo, Lorenzo Natale
Robotics
TacPAC: Tactile Prediction and Real-Time Action Correction in World-Action Models for Contact-Rich Manipulation
Zipei Ma, Xiaofei Wei, Junzhe Jiang et al. (+2 more)
World-action models guide action generation with predicted future observations, but
vision-centric predictions miss the local contact cues that decide contact-rich manipulation. However, naively predicting future tactile observations as additional views recovers only a third of the achievable gain in our experiments. This gap reflects a timing mismatch: predictions precede execution, while tactile feedback arrives during it. We introduce TacPAC, which turns tactile prediction into real-time action correction. Once the base model has planned an action chunk, TacPAC caches the predicted contact that plan was conditioned on together with the plan's own representation, and a tactile expert reads each newly observed tactile image against that cache to correct the actions not yet executed. Feedback is thus interpreted against what the plan anticipated rather than in isolation, and one correction is a single pass over that cache, $20.7\times$ cheaper than regenerating the chunk. On five real-
robot tasks spanning precision insertion, fragile-object handling, object reorientation, and long-horizon manipulation, TacPAC leads every task and raises the average from 22% for its
vision-only base model to 64%. Code is available at https://github.com/LogosRoboticsGroup/TacPAC.
Read the full abstract ▾
All authors Zipei Ma, Xiaofei Wei, Junzhe Jiang, Shunlin Lu, Li Zhang
Robotics
One Word, Different Action: A Real-Robot Benchmark for Language-Conditioned Embodied Reasoning
Yiwei Liu, Luwei Yang and Shunbo Lei
Changes in natural-
language instructions can directly alter the behavior ultimately executed by a
robot, but such changes do not always imply that the task itself has changed. We propose One Word, Different Action, a
language-conditioned executable decision
benchmark based on real-
robot physical decision anchors. It evaluates
robot behavioral responses under task-preserving and task-changing conditions and examines joint
reasoning over multiple task constraints. The
benchmark uses decision invariance and decision sensitivity to measure action stability when task semantics are preserved and correct decision updates when task requirements change. Matched atomic constituents directly relate correct handling of individual constraints to the corresponding joint executable decision. Experiments show that atomic-task correctness does not sufficiently represent compositional
reasoning ability: even when every constituent constraint is handled correctly, a model may still produce an erroneous final action decision on the corresponding compositional task. These results indicate that the key challenge in
language-conditioned
robot reasoning lies not only in understanding individual task requirements, but also in integrating multiple correctly handled constraints into consistent executable behavior.
Read the full abstract ▾
Robotics
Morphology and actuation as inductive biases in robotic hand manipulation
Zalán Tari, Eszter Birtalan, Péter Polcz et al. (+1 more)
Robotic hands vary widely in anatomical fidelity and mechanical complexity, and these structural choices influence the coordination of joint motions and the difficulty of controlling the system. A unified framework is presented in which the kinematic and actuation stages are analysed separately and in composition, through the conditioning of the task Jacobian, the actuation
matrix, and their product. It is applied to two hands representing opposing design philosophies, the Shadow Dexterous Hand and the Anatomically Correct, Biomechatronic Hand, along four morphological aspects: joint axis geometry, actuator-to-DOF ratio, coupling architecture, and authority distribution. All parameters are derived from the hands' canonical digital representations. Anatomical fidelity carries no uniform advantage: oblique axes improve thumb conditioning but leave the long fingers worse conditioned than the orthogonal-axis design, while the branching tendon network improves the effective control mapping at every long finger and worsens it significantly at the thumb, where actuator authority is concentrated on thumb opposition. Predictions derived from these metrics are evaluated against reinforcement learning experiments using PPO, DDPG+HER, and TQC+HER, across three different tasks.
Read the full abstract ▾
All authors Zalán Tari, Eszter Birtalan, Péter Polcz, Miklós Koller
Robotics · Cross-listed from eess.SY
Risk-Aware Optimal Control with Rulebooks
Tichakorn Wongpiromsarn
We consider safety-critical control problems involving multiple requirements with different priorities and uncertainty in their evaluation. We represent these requirements using risk-aware rulebooks, where each requirement is assigned a risk measure and an acceptable threshold, and a priority relation is defined among the requirements. Each requirement induces a risk-evaluation function that maps a policy to the risk associated with its violation. We formulate risk-aware optimal control with rulebooks as a lexicographic optimization problem over excess risks and develop an anytime filtering and branch-and-bound
algorithm that progressively tightens the certified optimality gap while characterizing the corresponding set of policies at each priority level. The
algorithm returns a policy together with these gaps, which bound its suboptimality. We prove that these gaps are valid for any finite computational budget and, under additional assumptions, converge to zero as the computational budget increases. We evaluate the
algorithm on a synthetic
benchmark with a known optimum and a realistic highway-merging simulation with CVaR-based collision, rear-braking, headway, and comfort rules.
Read the full abstract ▾
Robotics
LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models
Lin Liu, Zhicheng Bao, Lu Zhang et al. (+7 more)
Vision-
Language-Action (VLA) or World Action (WAM) models have recently demonstrated remarkable performance in robotic manipulation. On LIBERO, SOTA method have achieved nearly 100% success rates, seemingly suggesting that the models are ready for deployment in real world. However, near perfect performance on existing
benchmarks can be misleading: success under ideal conditions does not imply real world robustness. Existing
benchmarks primarily evaluate task completion from predefined initial states, while real world interactions inevitably involve failures such as failed grasps, collisions, and unintended object movements. A
robot must therefore not only execute tasks successfully, but also recognize and recover from failures to continue the task. Yet this capability remains largely unmeasured, revealing a critical gap between
benchmark performance and real world reliability. To address this gap, we introduce LIBERO-Recover
Benchmark, a large scale
benchmark for failure recovery in robotic manipulation. Built upon LIBERO, we collect real execution failures from SOTA embodied models and construct 1,000+ scenarios across four recovery levels: (1) Action Retry, (2) Action Adaptation, (3) Object State Recovery, and (4) Environmental Recovery. We evaluate four core capabilities: spatial understanding, object structure
reasoning, interaction understanding, and topological
reasoning. As the first large-scale
benchmark for embodied failure recovery, LIBERO-Recover shifts evaluation from Can the
robot succeed?” to Can the
robot recover after failure?”, promoting robust and generalizable embodied
agents. The project will be avaible in \textcolor{blue}{https://liulin815.github.io/LIBERO-Recovery/}.
Read the full abstract ▾
All authors Lin Liu, Zhicheng Bao, Lu Zhang, Ziying Song, Wu Yang, Yuzheng Zhuang, Shuai Tao, Wulong Liu, Caiyan Jia, Huchuan Lu
Robotics · Cross-listed from cs.AR
APEX-RBD: Mixed-Precision Exploration Framework for Hardware-Efficient Robot Dynamics Accelerator Design
Xingyu Liu, Hanwei Fan, Chaofang Ma et al. (+4 more)
Rigid Body Dynamics (RBD) forms the computational core of real-time robotic control, but its immense computational complexity creates a performance bottleneck that necessitates dedicated hardware accelerators. However, the substantial hardware resource and power costs of these accelerators make their deployment on resource-constrained edge platforms highly challenging. While quantization offers a promising path to optimize RBD hardware for edge computing, existing uniform-precision approaches remain inefficient by ignoring the diverse quantization sensitivities of different variables. Although mixed-precision offers a superior alternative, its exploration is intractable due to a vast search space and the prohibitive cost of closed-loop simulation for motion accuracy evaluation. To address these challenges, we introduce APEX-RBD, an automated framework that makes mixed-precision exploration computationally tractable while effectively identifying hardware-efficient configurations. Specifically, it performs physics-driven search space
pruning via variable grouping and sensitivity analysis, and employs a data-efficient, prior-informed surrogate model to enable rapid trajectory error prediction. This formulation guides a hybrid optimizer to identify area- and power-efficient designs under user-defined accuracy and performance constraints. Experimental results demonstrate that APEX-RBD discovers designs achieving up to 1.9$\times$ area reduction and 1.8$\times$ power savings compared to uniform-precision baselines across diverse robotic platforms.
Read the full abstract ▾
All authors Xingyu Liu, Hanwei Fan, Chaofang Ma, Jiawei Liang, Guangyu Hu, Jiang Xu, Wei Zhang
Robotics
A Schema Bounded Language Model for Refining Robot Policies Without Destabilizing Local Learning
Chongwen Dong, Mithun Paul Saint-Germain, Pinjari Asif et al. (+1 more)
This paper addresses navigation by composite heterogeneous
robots in a decentralized system when policy
reasoning and local control operate at different update levels. In a NetLogo–Python implementation, three
robots share motion dynamics but use different LLM backends. Each
robot independently combines a large
language model (LLM) policy
agent, an Upper Confidence Bound (UCB) bandit, and a Double Deep Q-Network (Double DQN) controller; no central LLM generates team actions. LLM inference is confined to round-level policy generation and refinement rather than tick-level action selection. The
robots perform cross-LLM communication through a shared round summary containing policies, outcomes, and learning feedback. UCB performs refinement-mode selection, and the policy-conditioned Double DQN performs tick-level action selection from navigation variables, active policy parameters, and the LLM action prior. Each of the four configurations was evaluated over 30 rounds. In the fixed simulation, the complete configuration reached the goal in all 90 correlated
robot–round records and achieved the lowest median completion time (42 ticks) and P90 (73.2 ticks); its median was 25.0–39.1% lower than those of the other configurations. These observations provide descriptive, configuration-level evidence from the evaluated configurations.
Read the full abstract ▾
All authors Chongwen Dong, Mithun Paul Saint-Germain, Pinjari Asif, Carlo R. daCunha
Robotics
ToPos: Automated Optimal Positioning on Topographic Manifolds using Constrained Geodesic Voronoi Decomposition
Rajesh Raveendran, Akseli Vanhamaa, Jaakko Suutala et al. (+2 more)
Reliable autonomous mapping, environmental sampling, last-mile logistics, and infrastructure deployment depend on the optimal surface area-balanced distribution of Spatial Reference Sites (SRS). Conventional 2D Euclidean methods often fail in high-relief environments by neglecting topographic variations and physical obstructions. This leads to significant planimetric distortion, spatial clustering, and the placement of targets in inaccessible or shadowed regions, compromising both data integrity and operational safety. This paper introduces ToPos, an automated framework for TOPography-aware Optimal Sampling on topographic
manifolds. We treat the terrain as a discrete 2-dimensional
manifold embedded in 3D Euclidean space and replace standard flat-map distances with non-Euclidean geodesic distances that follow the actual surface geometry. The point distribution is formulated as an optimization problem using a Constrained Geodesic Voronoi Decomposition, solved via a Riemannian Nesterov Accelerated Gradient (NAG) engine. Our approach restricts target locations to a feasible "safe zone," accounting for non-traversable slopes, vegetation, environmental occlusions, etc. Through evaluations on non-convex sinusoidal
manifolds, we show that ToPos mitigates planimetric distortion by utilizing geodesic metrics. This approach results in a $\sim$74% improvement in optimal surface area-balanced distribution, as measured by the coefficient of variation (CV) of the Voronoi
cell areas. The framework is architected as a Geographic Information System (GIS)-ready micro-service to bolster the mentioned applications. Index Terms: Topographic
Manifolds, Geodesic Voronoi Decomposition, Infrastructure Deployment, 3D Mapping, Spatial Sampling, and Non-Euclidean Optimization.
Read the full abstract ▾
All authors Rajesh Raveendran, Akseli Vanhamaa, Jaakko Suutala, Antti Tikanmäki, Juha Röning
Robotics · Cross-listed from cs.CV
MINT: A Unified Model for World-Space Camera and Hand Motion Estimation from Scalable Egocentric Pipeline Supervision
Zijie Zhu, Weiren Cai, Yizhou Wang et al. (+4 more)
Recovering camera and hand motion in world coordinates from egocentric
video is a key capability for activity understanding,
robot learning, and augmented reality. Existing systems typically decompose this problem into separate stages for camera motion, depth estimation, hand reconstruction, and trajectory refinement, resulting in substantial computational overhead and preventing the joint modeling of camera and hand motion. We introduce MINT (Minting IN-the-Wild Trajectories), a foundation model for world-space hand motion reconstruction from ego-centric RGB
video. From a single shared spatiotemporal
video representation, MINT jointly predicts the camera trajectory, field of view (FoV), camera-frame hand states, and per-frame hand observability, and then produces world-space hand motion via explicit coordinate transformations. Training such a model at scale is challenging, since paired world-space camera and hand annotations are scarce. We therefore develop an open-source labeling EGOPIPELINE that converts large collections of public egocentric
videos into structured camera-and-hand trajectory supervision. MINT is first pretrained on these large-scale pseudo-labels and then fine-tuned on a small set of high-quality camera-and-hand annotations. Across public
benchmarks MINT approaches state-of-the-art accuracy without seeing either
benchmark in training, reaching 0.945 frame accuracy, 13.646 mm PA-MPJPE-p and 55.058 px EPE-p for camera-frame bimanual reconstruction on HOT3D, 4.690 mm RPE-T and 0.284 degrees RPE-R for camera trajectory, and a 3.67x end-to-end speedup over the labeling pipeline that supervises it. We release the model, training and inference code, labeling pipeline, and a curated 1,021-hour egocentric trajectory
dataset.
Read the full abstract ▾
All authors Zijie Zhu, Weiren Cai, Yizhou Wang, Zhenjie Yang, Yide Liu, Jiahao Chen, Guanqi He
Robotics · Cross-listed from cs.CV
One Diffusion Model, Two Roles: Guided Trajectory Planning and Safety-Critical Scenario Generation in Closed-Loop Simulation
Arka Pal, Rajesh Kumar, Hannes Eriksson et al. (+4 more)
Diffusion probabilistic models can capture the multi-modal, interaction-rich distribution of joint future trajectories in driving scenes. We show that a single pretrained
diffusion traffic model can serve two complementary roles in the autonomous driving development loop: as an ego motion planner, and as a controllable generator of safety-critical scenarios for stress-testing the planners. On the planning side, we introduce a Single-Stream Dual-Stream (SSDS)
diffusion-
transformer decoder that fuses scene context via joint
attention rather than late cross-
attention, improving closed-loop performance on nuPlan. We further propose Decoupled Annealing Posterior Sampling with Energy (DAPSE), a training-free guidance scheme that injects arbitrary energy functions at the clean-sample level, avoiding the first-order approximation errors while requiring no auxiliary networks. Beyond planning, we leverage the same
diffusion model as a controllable scenario generator to create realistic long-tail driving interactions for closed-loop evaluation. Through inference-time guidance, selected
agents are steered toward safety-critical behaviors, including aggressive cut-ins, lead-vehicle braking, and combined longitudinal-lateral interactions, while preserving realistic traffic behaviors. Evaluated in closed-loop nuPlan simulations with independent black-box planners, the generated scenarios expose failure modes that remain hidden under standard
benchmarks. Although the SSDS-based planner achieves stronger nominal performance, it experiences larger degradation under these challenging scenarios, demonstrating that
benchmark superiority does not necessarily translate to robustness. These results demonstrate that a single learned traffic prior can simultaneously improve motion planning and provide a realistic framework for systematic planner robustness evaluation.
Read the full abstract ▾
All authors Arka Pal, Rajesh Kumar, Hannes Eriksson, Rémi Lacombe, Arvid Laveno Ling, Ankit Gupta, Maciej Wozniak