Robotics
ICAT: Incident-Case-Grounded Adaptive Testing for Physical-Risk Prediction in Embodied World Models
Zhenglin Lai, Sirui Huang, Yuteng Li et al. (+3 more)
Reference-image-conditioned
video generators are increasingly used as neural simulators for
embodied planning and policy learning, yet their ability to anticipate physical hazards is rarely evaluated. We introduce ICAT, which grounds test construction in incident reports and safety guidance through structured risk
memories, constrained case generation, and human verification. The resulting
benchmark contains 909 evaluation units with explicit initial conditions, triggering actions, and documented consequences. Across six generation systems, severe outcomes are frequently missing or attenuated even when the initial state is preserved and the risky action is executed: the strongest system achieves 23.5% conditional strict success under best-of-eight selection. Expert
auditing, image-generator sensitivity analysis, matched safe-risky pairs, and alternative-outcome scoring provide complementary checks on this system-level finding. ICAT exposes a gap between reproducing risky actions and generating their documented consequences.
Read the full abstract ▾
All authors Zhenglin Lai, Sirui Huang, Yuteng Li, Changxin Huang, Jianqiang Li, Bingzhe Wu
Robotics
Hierarchical Motion Planning and Control under Unknown Nonlinear Dynamics via Predicted Reachability
Zhiquan Zhang and Melkior Ornik
Autonomous motion planning under unknown nonlinear dynamics requires learning system properties while navigating toward a target. In this work, we develop a hierarchical planning-control framework that enables online motion synthesis with limited prior system knowledge. The state space is partitioned into polytopes and approximates the unknown nonlinear system using a piecewise-affine (PWA) model. The local affine models are identified once the
agent enters the corresponding polytopes. To reduce computational complexity, we introduce a non-uniform adaptive state space partition strategy that refines the partition only in task-relevant regions. The resulting PWA system is abstracted into a directed weighted graph, whose edge existence is incrementally verified using reach control theory and predictive reachability conditions. Certified edges are weighted using provable time-to-reach bounds, while uncertain edges are assigned information-theoretic weights to guide exploration. The graph is updated online as new data becomes available, and high-level planning is performed by graph search, while low-level affine feedback controllers are synthesized to execute the plan. Furthermore, the conditions of classical reach control theory are often difficult to satisfy in underactuated settings. We therefore introduce relaxed reachability conditions to extend the framework to such systems. Simulations demonstrate effective exploration-exploitation trade-offs with formal reachability guarantees.
Read the full abstract ▾
Robotics
Play-Testing REMind: Evaluating an Educational Robot-Mediated Role-Play Game
Elaheh Sanoubari, Neil Fernandes, Keith Rebello et al. (+3 more)
This paper presents REMind, an innovative educational
robot-mediated role-play game designed to support anti-bullying bystander intervention among children. REMind invites players to observe a bullying scenario enacted by social
robots, reflect on the perspectives of the characters, and rehearse defending strategies by puppeteering a robotic avatar. We evaluated REMind through a mixed-methods play-testing study with 18 children aged 9–10. The findings suggest that the experience supported key learning goals related to self-efficacy, perspective-taking, understanding outcomes of defending, and intervention strategies. These results highlight the promise of
Robot-Mediated Applied Drama (RMAD) as a novel pedagogical framework to support Social-Emotional Learning.
Read the full abstract ▾
All authors Elaheh Sanoubari, Neil Fernandes, Keith Rebello, Alicia Pan, Andrew Houston, Kerstin Dautenhahn
Robotics
DreamControl-v2: Simpler and Scalable Autonomous Humanoid Skills via Trainable Guided Diffusion Priors
Sudarshan Harithas, Sangkyung Kwak, Pushkal Katara et al. (+6 more)
Developing robust autonomous loco-manipulation skills for
humanoids remains an open problem in robotics. While RL has been applied successfully to legged locomotion, applying it to complex, interaction-rich manipulation tasks is harder given long-horizon planning challenges for manipulation. A recent approach along these lines is DreamControl, which addresses these issues by leveraging off-the-shelf human motion
diffusion models as a generative prior to guide RL policies during training. In this paper, we investigate the impact of DreamControl's motion prior and propose an improved framework that trains a guided
diffusion model directly in the
humanoid robot's motion space, aggregating diverse human and
robot datasets into a unified embodiment space. We demonstrate that our approach captures a wider range of skills due to the larger training data mixture and establishes a more automated pipeline by removing the need for manual filtering interventions. Furthermore, we show that scaling the generation of reference trajectories is important for achieving robust downstream RL policies. We validate our approach through extensive experiments in simulation and on a real Unitree-G1.
Read the full abstract ▾
All authors Sudarshan Harithas, Sangkyung Kwak, Pushkal Katara, Srujan Deolasee, Dvij Kalaria, Srinath Sridhar, Sai Vemprala, Ashish Kapoor, Jonathan Chung-Kuan Huang
Robotics
Neural-Assisted in-Motion Self-Heading Alignment
Zeev Yampolsky, Felipe O. Silva, Adriano Frutuoso et al. (+1 more)
Autonomous platforms operating in the oceans require accurate
navigation to successfully complete their mission. In this regard, the initial heading estimation accuracy and the time required to achieve it play a critical role. The initial heading is traditionally estimated by model-based approaches employing orientation decomposition. However, methods such as the dual vector decomposition and optimized attitude decomposition achieve satisfactory heading accuracy only after long
alignment times. To allow rapid and accurate initial heading estimation, we propose an end-to-end, model-free, neural-assisted framework using the same inputs as the model-based approaches. Our proposed approach was trained and evaluated on real-world
dataset captured by an autonomous surface vehicle. Our approach shows a significant accuracy improvement over the model-based approaches achieving an average absolute error improvement of 53%. Additionally, our proposed approach was able to reduce the
alignment time by up to 67%. Thus, by employing our proposed approach, the reduction in
alignment time and improved accuracy allow for a shorter deployment time of an autonomous platform and increased
navigation accuracy during the mission.
Read the full abstract ▾
All authors Zeev Yampolsky, Felipe O. Silva, Adriano Frutuoso, Itzik Klein
Robotics
Long-Horizon Geometry-Aware Navigation among Polytopes via MILP-MPC and Minkowski-Based CBFs
Yi-Hsuan Chen, Salman Ghori, Ania Adil et al. (+2 more)
Autonomous
navigation in complex, non-convex environments remains challenging when
robot dynamics, control limits, and exact
robot geometry must all be taken into account. In this paper, we propose a hierarchical planning and control framework that bridges long-horizon guidance and geometry-aware safety guarantees for a polytopic
robot navigating among polytopic obstacles. At the high level, Mixed-Integer Linear Programming (MILP) is embedded within a Model Predictive Control (MPC) framework to generate a nominal trajectory around polytopic obstacles while modeling the
robot as a point mass for computational tractability. At the low level, we employ a control barrier function (CBF) based on the exact signed distance in the Minkowski-difference space as a safety filter to explicitly enforce the
geometric constraints of the
robot shape, and further extend its formulation to a high-order CBF (HOCBF). We demonstrate the proposed framework in U-shaped and maze-like environments under single- and double-integrator dynamics. The results show that the proposed architecture mitigates the
topology-induced local-minimum behavior of purely reactive CBF-based
navigation while enabling safe, real-time, geometry-aware
navigation.
Read the full abstract ▾
All authors Yi-Hsuan Chen, Salman Ghori, Ania Adil, Eric Feron, Calin Belta
Robotics · Cross-listed from cs.CY
Beyond Symbolic Control: Societal Consequences of AI-Driven Workforce Displacement and the Imperative for Genuine Human Oversight Architectures
Richard J. Mitchell
The accelerating displacement of human labor by artificial intelligence (AI) and robotic systems represents a structural transformation whose societal consequences extend far beyond conventional labor
market analysis. This paper presents a systematic multi-domain examination of the likely effects on economic structure, psychological well-being, political stability, education, healthcare, and geopolitical order. We identify a critical and underexamined dimension of this transition: the governance gap between nominal human oversight of AI systems – where humans occupy positions of formal authority over AI decisions – and genuine human oversight, where those humans possess the cognitive access, technical capability, and institutional authority to meaningfully understand, evaluate, and override AI outputs. We argue that this distinction, largely absent from current governance frameworks including the EU AI Act and NIST AI Risk Management Framework 1.0, represents the primary architectural failure mode in deployed AI governance. The societal consequences of labor displacement intensify this problem by concentrating consequential AI decision-making among an increasingly narrow class of technical and capital actors. We propose five architectural requirements for genuine human oversight systems and characterize the governance window – estimated at 10-15 years – before current deployment trajectories risk path-dependent social, economic, and institutional lock-in.
Read the full abstract ▾
Robotics
Advancing Multi-Robot Networks via MLLM-Driven Sensing, Communication, and Computation: A Comprehensive Survey
Hyun Jong Yang, Howon Lee, Kyuhong Shim et al. (+10 more)
Imagine advanced
humanoid robots, powered by
multimodal large
language models (MLLMs), coordinating missions across industries like warehouse logistics, manufacturing, and safety rescue. While individual
robots show local autonomy, realistic tasks demand coordination among multiple
agents sharing vast streams of sensor data. Communication is indispensable, yet transmitting comprehensive data can overwhelm networks, especially when a system-level orchestrator or cloud-based MLLM fuses
multimodal inputs for route planning or anomaly detection. These tasks are often initiated by high-level natural
language instructions. This intent serves as a filter for resource optimization: by understanding the goal via MLLMs, the system can selectively activate relevant sensing modalities, dynamically allocate bandwidth, and determine computation placement. Thus, R2X is fundamentally an intent-to-resource orchestration problem where sensing, communication, and computation are jointly optimized to maximize task-level success under resource constraints. This survey examines how integrated design paves the way for multi-
robot coordination under MLLM guidance. We review state-of-the-art sensing modalities, communication strategies, and computing approaches, highlighting how
reasoning is split between on-device models and powerful edge/cloud servers. We present four end-to-end demonstrations (sense -> communicate -> compute -> act): (i) digital-twin warehouse
navigation with predictive link context, (ii) mobility-driven proactive MCS control, (iii) a FollowMe
robot with a semantic-sensing switch, and (iv) real-hardware open-vocabulary trash sorting via edge-assisted MLLM grounding. We emphasize system-level metrics – payload, latency, and success – to show why R2X orchestration outperforms purely on-device baselines.
Read the full abstract ▾
All authors Hyun Jong Yang, Howon Lee, Kyuhong Shim, Jeongho Kwak, Hyunsoo Kim, Donghoon Kim, Khoa Anh Ngo, Sehyun Ryu, Jaehyun Choi, Youbin Kim, Chanjun Moon, Michael Ryoo, Byonghyo Shim
Robotics
MRReP: Mixed Reality-based Hand-drawn Reference Path Editing Interface for Mobile Robot Navigation
Takumi Taki, Masato Kobayashi and Yuki Uranishi
Autonomous mobile
robots operating in human-shared indoor environments often require paths that reflect human spatial intentions, such as avoiding interference with pedestrian flow or maintaining comfortable clearance. However, conventional path planners primarily optimize
geometric costs and provide limited support for explicit route specification by human operators. This paper presents MRReP, a Mixed Reality-based interface that enables users to draw a Hand-drawn Reference Path (HRP) directly on the physical floor using hand gestures. The drawn HRP is integrated into the
robot navigation stack through a custom Hand-drawn Reference Path Planner, which converts the user-specified point sequence into a global path for autonomous
navigation. We evaluated MRReP in a within-subject experiment against a conventional 2D baseline interface. The results demonstrated that MRReP enhanced path specification accuracy, usability, and perceived workload, while enabling more stable path specification in the physical environment. These findings suggest that direct path specification in MR is an effective approach for incorporating human spatial intention into mobile
robot navigation. Additional material is available at https://mertcookimg.github.io/mrrep
Read the full abstract ▾
Robotics
Generalizable Dense Reward for Long-Horizon Robotic Tasks
Silong Yong, Stephen Sheng, Carl Qi et al. (+6 more)
Existing robotic foundation policies are trained primarily via large-scale imitation learning. While such models demonstrate strong capabilities, they often struggle with long-horizon tasks due to distribution shift and error accumulation. While reinforcement learning (RL) can finetune these models, it cannot work well across diverse tasks without manual reward engineering. We propose VLLR, a dense reward framework combining (1) an extrinsic reward from Large
Language Models (LLMs) and
Vision-
Language Models (VLMs) for task progress recognition, and (2) an intrinsic reward based on policy self-certainty. VLLR uses LLMs to decompose tasks into verifiable subtasks and then VLMs to estimate progress to initialize the value function for a brief warm-up phase, avoiding prohibitive inference cost during full training; and self-certainty provides per-step intrinsic guidance throughout PPO finetuning. Ablation studies reveal complementary benefits: VLM-based value initialization primarily improves task completion efficiency, while self-certainty primarily enhances success rates, particularly on out-of-distribution tasks. On the CHORES
benchmark covering mobile manipulation and
navigation, VLLR achieves up to 56% absolute success rate gains over the pretrained policy, up to 5% gains over state-of-the-art RL finetuning methods on in-distribution tasks, and up to $10\%$ gains on out-of-distribution tasks, all without manual reward engineering. Additional visualizations can be found in https://silongyong.github.io/vllr_project_page/
Read the full abstract ▾
All authors Silong Yong, Stephen Sheng, Carl Qi, Xiaojie Wang, Evan Sheehan, Anurag Shivaprasad, Yaqi Xie, Katia Sycara, Yesh Dattatreya
Robotics
HapCompass: A Rotational Haptic Device for Contact-Rich Robotic Teleoperation
Xiangshan Tan, Jingtian Ji, Tianchong Jiang et al. (+2 more)
The contact-rich nature of manipulation makes it a significant challenge for robotic teleoperation. While haptic feedback is critical for contact-rich tasks, providing intuitive directional cues within wearable teleoperation interfaces remains a bottleneck. Existing solutions, such as non-directional vibrations from handheld controllers, provide limited information, while vibrotactile arrays are prone to perceptual interference. To address these limitations, we propose HapCompass, a novel, low-cost wearable haptic device that renders 2D directional cues by mechanically rotating a single linear resonant actuator (LRA). We evaluated HapCompass's ability to convey directional cues to human operators and showed that it increased the success rate, decreased the completion time and the maximum contact force for teleoperated manipulation tasks when compared to
vision-only and non-directional feedback baselines. Furthermore, we conducted a preliminary imitation-learning evaluation, suggesting that the directional feedback provided by HapCompass enhances the quality of demonstration data and, in turn, the trained policy. We release the design of the HapCompass device along with the code that implements our teleoperation interface: https://ripl.github.io/HapCompass/.
Read the full abstract ▾
All authors Xiangshan Tan, Jingtian Ji, Tianchong Jiang, Pedro Lopes, Matthew R. Walter
Robotics
Hybrid Framework for Robotic Manipulation: Integrating Reinforcement Learning and Large Language Models
Md Saad, Sajjad Hussain and Mohd Suhaib
This paper introduces a new hybrid framework that combines Reinforcement Learning (RL) and Large
Language Models (LLMs) to improve robotic manipulation tasks. By utilizing RL for accurate low-level control and LLMs for high level task planning and understanding of natural
language, the proposed framework effectively connects low-level execution with high-level
reasoning in robotic systems. This integration allows
robots to understand and carry out complex, human-like instructions while adapting to changing environments in real time. The framework is tested in a PyBullet-based simulation environment using the Franka Emika Panda robotic arm, with various manipulation scenarios as
benchmarks. The results show a 33.5% decrease in task completion time and enhancements of 18.1% and 36.4% in accuracy and adaptability, respectively, when compared to systems that use only RL. These results underscore the potential of LLM-enhanced robotic systems for practical applications, making them more efficient, adaptable, and capable of interacting with humans. Future research will aim to explore sim-to-real transfer, scalability, and multi-
robot systems to further broaden the framework's applicability.
Read the full abstract ▾
Robotics
Passive iFIR filters for data-driven velocity control in robotics
Yi Zhang, Zixing Wang and Fulvio Forni
We present a passive, data-driven velocity control method for nonlinear robotic manipulators that achieves better tracking performance than optimized PID with comparable design complexity. Using only three minutes of probing data, a VRFT-based design identifies passive iFIR controllers that (i) preserve closed-loop stability via passivity constraints and (ii) outperform a VRFT-tuned PID baseline on the Franka Research 3
robot in both joint-space and Cartesian-space velocity control, achieving up to a 74.5% reduction in tracking error for the Cartesian velocity tracking experiment with the most demanding reference model. When the
robot end-effector dynamics change, the controller can be re-learned from new data, regaining nominal performance. This study bridges learning-based control and stability-guaranteed design: passive iFIR learns from data while retaining passivity-based stability guarantees, unlike many learning-based approaches.
Read the full abstract ▾
Robotics
DIAL: Decoupling Intent and Action via Latent World Modeling for End-to-End VLA
Yi Chen, Yuying Ge, Hui Zhou et al. (+3 more)
The development of
Vision-
Language-Action (VLA) models has been significantly accelerated by pre-trained
Vision-
Language Models (VLMs). However, most existing end-to-end VLAs treat the VLM primarily as a
multimodal encoder, directly mapping
vision-
language features to low-level actions. This paradigm underutilizes the VLM's potential in high-level decision making and introduces training instability, frequently degrading its rich semantic representations. To address these limitations, we introduce DIAL, a framework bridging high-level decision making and low-level motor execution through a differentiable latent intent bottleneck. Specifically, a VLM-based System-2 performs latent world modeling by synthesizing latent visual foresight within the VLM's native feature space; this foresight explicitly encodes intent and serves as the structural bottleneck. A lightweight System-1 policy then decodes this predicted intent together with the current observation into precise
robot actions via latent inverse dynamics. To ensure optimization stability, we employ a two-stage training paradigm: a decoupled warmup phase where System-2 learns to predict latent futures while System-1 learns motor control under ground-truth future guidance within a unified feature space, followed by seamless end-to-end joint optimization. This enables action-aware gradients to refine the VLM backbone in a controlled manner, preserving pre-trained knowledge. Extensive experiments on the RoboCasa GR1 Tabletop
benchmark show that DIAL establishes a new state-of-the-art, achieving superior performance with 10x fewer demonstrations than prior methods. Furthermore, by leveraging heterogeneous human demonstrations, DIAL learns physically grounded manipulation priors and exhibits robust zero-shot generalization to unseen objects and novel configurations during real-world deployment on a
humanoid robot.
Read the full abstract ▾
All authors Yi Chen, Yuying Ge, Hui Zhou, Mingyu Ding, Yixiao Ge, Xihui Liu
Robotics
Reconfiguration of supernumerary robotic limbs for human augmentation
Mustafa Mete, Anastasia Bolotnikova, Alexander Schuessler et al. (+1 more)
Wearable
robots aim to seamlessly adapt to humans and their environment with personalized interactions. Existing supernumerary robotic limbs (SRLs), which enhance the physical capabilities of humans with additional extremities, have thus far been developed primarily for task-specific applications in structured industrial settings, limiting their adaptability to dynamic and unstructured environments. Here, we introduce a novel reconfigurable SRL framework grounded in a quantitative analysis of human augmentation to guide the development of more adaptable SRLs for diverse scenarios. This framework captures how SRL configuration shapes workspace extension and human-
robot collaboration. We define human augmentation ratios to evaluate collaborative, visible extended, and non-visible extended workspaces, enabling systematic selection of SRL placement, morphology, and autonomy for a given task. Using these metrics, we demonstrate how quantitative augmentation analysis can guide the reconfiguration and control of SRLs to better match task requirements. We validate the proposed approach through experiments with a reconfigurable SRL composed of origami-inspired modular elements. Our results suggest that reconfigurable SRLs, informed by quantitative human augmentation analysis, offer a new perspective for providing adaptable human augmentation and assistance in everyday environments.
Read the full abstract ▾
All authors Mustafa Mete, Anastasia Bolotnikova, Alexander Schuessler, Jamie Paik
Robotics
SafeDMPs: Integrating Formal Safety with DMPs for Adaptive HRI
Soumyodipta Nath, Pranav Tiwari and Ravi Prakash
Robots operating in human-centric environments must be both robust to disturbances and provably safe from collisions. Achieving these properties simultaneously and efficiently remains a central challenge. While Dynamic Movement Primitives (DMPs) offer inherent stability and generalization from single demonstrations, they lack formal safety guarantees. Conversely, formal methods like Control Barrier Functions (CBFs) provide provable safety but often rely on computationally expensive, real-time optimization, hindering their use in high-frequency control. This paper introduces SafeDMPs, a novel framework that resolves this trade-off. We integrate the closed-form efficiency and dynamic robustness of DMPs with a provably safe, non-optimization-based control law derived from Spatio-Temporal Tubes (STTs). This synergy allows us to generate motions that are not only robust to perturbations and adaptable to new goals, but also guaranteed to avoid static and dynamic obstacles. Our approach achieves a closed-form solution for a problem that traditionally requires online optimization. Experimental results on a 7-DOF
robot manipulator demonstrate that SafeDMPs is orders of magnitude faster and more accurate than optimization-based baselines, making it an ideal solution for real-time, safe, and collaborative robotics.
Read the full abstract ▾
Robotics
Design and Aerodynamic Modeling of MetaMorpher: A Hybrid Rotary andFixed-Wing Morphing UAV
Anja Bosak, Dorian Erić, Ana Milas et al. (+1 more)
In this paper, we present a generalized, comprehensive nonlinear mathematical model and conceptual design for the MetaMorpher, a metamorphic Unmanned Aerial Vehicle (
UAV) designed to bridge the gap between vertical takeoff and landing agility and fixed-wing cruising efficiency. Building on the successful design of the spincopter platform, this work introduces a simplified mechanical architecture using lightweight materials and a novel wing-folding strategy. Unlike traditional rigid-body approximations, we derive a nonlinear flight dynamics model that enables arbitrary force distributions across a segmented wing structure. This modularity allows for testing different airfoils, mass distributions, and chord lengths in a single environment. As part of this work, various flight modes were specifically tested and analyzed in the Simulink environment. The results show that the model behaves predictably under different structural configurations, demonstrating its reliability as a tool for rapid design evaluation.
Read the full abstract ▾
All authors Anja Bosak, Dorian Erić, Ana Milas, Stjepan Bogdan
Robotics
Semantic Zone-Based Map Management for Stable AI-Integrated Mobile Robots
Huichang Yun and Seungho Yoo
Recent advances in large AI models (VLMs and LLMs) and joint use of the 3D dense maps, enable mobile
robots to provide more powerful and interactive services grounded in rich spatial context. However, deploying both heavy AI models and dense maps on edge
robots is challenging under strict
memory budgets. When the
memory budget is exceeded, required keyframes may not be loaded in time, which can degrade the stability of position estimation and interfering model performance. We proposes a semantic zone-based map management approach to stabilize dense-map utilization under
memory constraints. We associate keyframes with semantic indoor regions (e.g., rooms and corridors) and keyframe management at the semantic zone level prioritizes spatially relevant map content while respecting
memory constraints. This reduces keyframe loading and unloading frequency and
memory usage. We evaluate the proposed approach in large-scale simulated indoor environments and on an NVIDIA Jetson Orin Nano under concurrent SLAM-VLM execution. With Qwen3.5:0.8b, the proposed method improves throughput by 3.3
tokens/s and reduces latency by 21.7% relative to a
geometric map-management strategy. Furthermore, while the
geometric strategy suffers from out-of-
memory failures and stalled execution under
memory pressure, the proposed method eliminates both issues, preserving localization stability and enabling robust VLM operation. These results demonstrate that the proposed approach enables efficient dense map utilization for
memory constrained, AI-integrated mobile
robots. Code is available at: https://github.com/huichangs/rtabmap/tree/segment
Read the full abstract ▾
Robotics · Cross-listed from eess.SY
Distributed Predictive Control Barrier Functions: Towards Scalable Safety Certification in Modular Multi-Agent Systems
Jonas Ohnemus, Alexandre Didier, Ahmed Aboudonia et al. (+2 more)
We consider safety-critical multi-
agent systems with distributed control architectures and potentially varying network
topologies. While learning-based distributed control enables scalability and high performance, a lack of formal safety guarantees in the face of unforeseen disturbances and unsafe network
topology changes may lead to system failure. To address this challenge, we introduce structured control barrier functions (s-CBFs) as a multi-
agent safety framework. The s-CBFs are augmented to a distributed predictive control barrier function (D-PCBF), a predictive, optimization-based safety layer that uses model predictions to guarantee recoverable safety at all times. The proposed approach enables a permissive yet formal plug-and-play protocol, allowing
agents to join or leave the network while ensuring safety recovery if a change in network
topology requires temporarily unsafe behavior. We validate the formulation through simulations and real-time experiments of a miniature race-car platoon.
Read the full abstract ▾
All authors Jonas Ohnemus, Alexandre Didier, Ahmed Aboudonia, Andrea Carron, Melanie N. Zeilinger
Robotics
GraSP-STL: A Graph-Based Framework for Zero-Shot Signal Temporal Logic Planning via Offline Goal-Conditioned Reinforcement Learning
Ancheng Hou, Ruijia Liu and Xiang Yin
This paper studies offline, zero-shot planning under Signal Temporal Logic (STL) specifications. We assume access only to an offline
dataset of state-action-state transitions collected by a task-agnostic behavior policy, with no analytical dynamics model, no further environment interaction, and no task-specific retraining. The objective is to synthesize a control strategy whose resulting trajectory satisfies an arbitrary unseen STL specification. To this end, we propose GraSP-STL, a graph-search-based framework for zero-shot STL planning from offline trajectories. The method learns a goal-conditioned value function from offline data and uses it to induce a finite-horizon reachability metric over the state space. Based on this metric, it constructs a directed graph abstraction whose nodes represent representative states and whose edges encode feasible short-horizon transitions. Planning is then formulated as a graph search over waypoint sequences, evaluated using arithmetic-
geometric mean robustness and its interval semantics, and executed by a learned goal-conditioned policy. The proposed framework separates reusable reachability learning from task-conditioned planning, enabling zero-shot generalization to unseen STL tasks and long-horizon planning through the composition of short-horizon behaviors from offline data. Experimental results demonstrate its effectiveness on a range of offline STL planning tasks.
Read the full abstract ▾
Robotics
Communication Outage-Resistant UUV State Estimation: A Variational History Distillation Approach
Shuyue Li, Miguel López-Benítez, Eng Gee Lim et al. (+5 more)
The reliable operation of Unmanned
Underwater Vehicle (UUV) clusters is highly dependent on continuous acoustic communication. However, this communication method is highly susceptible to intermittent interruptions. When communication outages occur, standard state
estimators such as the Unscented Kalman Filter (UKF) will be forced to make open-loop predictions. If the environment contains unmodeled dynamic factors, such as unknown ocean currents, this estimation error will grow rapidly, which may eventually lead to mission failure. To address this critical issue, this paper proposes a Variational History
Distillation (VHD) approach. VHD regards trajectory prediction as an approximate
Bayesian reasoning process, which links a standard motion model based on physics with a pattern extracted directly from the past trajectory of the UUV. This is achieved by synthesizing “virtual measurements” distilled from historical trajectories. Recognizing that the reliability of extrapolated historical trends degrades over extended prediction horizons, an adaptive confidence mechanism is introduced. This mechanism allows the filter to gradually reduce the trust of virtual measurements as the communication outage time is extended. Extensive Monte Carlo simulations in a high-fidelity environment demonstrate that the proposed method achieves a 91% reduction in prediction Root Mean Square Error (RMSE), reducing the error from approximately 170 m to 15 m during a 40-second communication outage. These results demonstrate that VHD can maintain robust state estimation performance even under complete communication loss.
Read the full abstract ▾
All authors Shuyue Li, Miguel López-Benítez, Eng Gee Lim, Fei Ma, Qian Dong, Mengze Cao, Limin Yu, Xiaohui Qin
Robotics · Cross-listed from eess.SY
Model Predictive Path Integral PID Control for Learning-Based Path Following
Teruki Kato, Koshi Oishi and Seigo Ito
Classical proportional–integral–derivative (PID) control remains widely used in industrial control systems, while model predictive control (MPC) is actively studied to achieve higher performance for systems with nonlinear dynamics. Model predictive path integral (MPPI) control is a sampling-based MPC method that optimizes control inputs without gradient calculations and can handle non-differentiable models and objective functions. However, conventional MPPI directly samples control-input sequences, which can produce large temporal input increments and causes the optimization dimension to grow with the prediction horizon. This study proposes MPPI–PID control, which uses MPPI to optimize PID gains online instead of directly optimizing the control-input sequences. By replacing high-dimensional input-sequence optimization with low-dimensional gain-space optimization while retaining the PID structure, the proposed formulation improves sampling efficiency and promotes smoother control inputs. Theoretical analyses are provided for a unified path-integral update, the relation between optimization dimension and effective sample size, and the temporal correlation of input perturbations induced by the PID structure. The method is evaluated on a learning-based path following of a mini forklift using a residual-learning dynamics model that combines a physical model and a neural network identified from real-machine driving data. Numerical results show that MPPI–PID improves tracking performance over fixed-gain PID, yields smaller input increments than conventional MPPI, and maintains favorable performance under reduced sampling budgets.
Read the full abstract ▾
Robotics
CReF: Cross-modal and Recurrent Fusion for Depth-conditioned Humanoid Locomotion
Yuan Hao, Ruiqi Yu, Shixin Luo et al. (+3 more)
Stable traversal over geometrically complex terrain increasingly requires exteroceptive perception, yet prior perceptive
humanoid locomotion methods often remain tied to explicit
geometric abstractions, either by mediating control through
robot-centric 2.5D terrain representations or by shaping depth learning with auxiliary geometry-related targets. While effective, these approaches introduce additional map-construction procedures or multi-stage skill-transfer processes beyond direct depth-to-control learning. We propose CReF (Cross-modal and Recurrent Fusion), a single-stage depth-conditioned
humanoid locomotion framework that learns locomotion-relevant features directly from raw forward-facing depth without explicit
geometric intermediates. CReF couples proprioception and depth
tokens through proprioception-queried cross-modal
attention, fuses the resulting representation with a gated residual fusion block, and performs temporal integration with a Gated Recurrent Unit (GRU) regulated by a highway-style output gate for state-dependent blending of recurrent and feedforward features. To further improve terrain interaction, we introduce a terrain-aware foothold placement reward that extracts supportable foothold candidates from foot-end point-cloud samples and rewards touchdown locations that lie close to the nearest supportable candidate. Experiments in simulation and on a physical
humanoid demonstrate robust traversal over diverse terrains and effective zero-shot transfer to real-world scenes containing handrails, hollow pallet assemblies, severe reflective interference, and visually cluttered outdoor surroundings.
Read the full abstract ▾
All authors Yuan Hao, Ruiqi Yu, Shixin Luo, Guoteng Zhang, Jun Wu, Qiuguo Zhu
Robotics
RAAP: Retrieval-Augmented Affordance Prediction with Cross-Image Action Alignment
Qiyuan Zhuang, He-Yang Xu, Yijun Wang et al. (+3 more)
Understanding object affordances is essential for enabling
robots to perform purposeful and fine-grained interactions in diverse and unstructured environments. However, existing approaches either rely on
retrieval, which is fragile due to sparsity and coverage gaps, or on large-scale models, which frequently mislocalize contact points and mispredict post-contact actions when applied to unseen categories, thereby hindering robust generalization. We introduce
Retrieval-Augmented Affordance Prediction (RAAP), a framework that unifies affordance
retrieval with
alignment-based learning. By decoupling static contact localization and dynamic action direction, RAAP transfers contact points via dense correspondence and predicts action directions through a
retrieval-augmented
alignment model that consolidates multiple references with dual-weighted
attention. Trained on compact subsets of DROID and HOI4D with as few as tens of samples per task, RAAP achieves consistent performance across unseen objects and categories, and enables zero-shot robotic manipulation in both simulation and the real world. Project website: https://github.com/SEU-VIPGroup/RAAP.
Read the full abstract ▾
All authors Qiyuan Zhuang, He-Yang Xu, Yijun Wang, Xin-Yang Zhao, Yang-Yang Li, Xiu-Shen Wei
Robotics · Cross-listed from cs.CV
Native-Domain Cross-Attention for Camera-LiDAR Extrinsic Calibration Under Large Initial Perturbations
Ni Ou, Zhuo Chen, Xinru Zhang et al. (+1 more)
Accurate camera-
LiDAR fusion relies on precise extrinsic calibration, which fundamentally depends on establishing reliable cross-modal correspondences under potentially large misalignments. Existing learning-based methods typically project
LiDAR points into depth maps for feature fusion, which distorts 3D geometry and degrades performance when the extrinsic initialization is far from the ground truth. To address this issue, we propose an extrinsic-aware cross-
attention framework that directly aligns image patches and
LiDAR point groups in their native domains. The proposed
attention mechanism explicitly injects extrinsic parameter hypotheses into the correspondence modeling process, enabling geometry-consistent cross-modal interaction without relying on projected 2D depth maps. Extensive experiments on the KITTI and nuScenes
benchmarks demonstrate that our method consistently outperforms state-of-the-art approaches in both accuracy and robustness. Under large extrinsic perturbations, our approach achieves accurate calibration in 88% of KITTI cases and 99% of nuScenes cases, substantially surpassing the second-best baseline. We have open sourced our code on https://github.com/gitouni/ProjFusion to benefit the community.
Read the full abstract ▾
All authors Ni Ou, Zhuo Chen, Xinru Zhang, Junzheng Wang
Robotics
CLaD: Planning with Grounded Foresight via Cross-Modal Latent Dynamics
Andrew Jeong, Jaemin Kim, Sebin Lee et al. (+1 more)
Robotic manipulation involves kinematic and semantic transitions that are inherently coupled via underlying actions. However, existing approaches plan within either semantic or latent space without explicitly aligning these cross-modal transitions. To address this, we propose CLaD, a framework that models how proprioceptive and semantic states jointly evolve under actions through asymmetric cross-
attention that allows kinematic transitions to query semantic ones. CLaD predicts grounded latent foresights via self-supervised objectives with EMA target encoders and auxiliary reconstruction losses, preventing representation collapse while anchoring predictions to observable states. Predicted foresights are modulated with observations to condition a
diffusion policy for action generation. On LIBERO-LONG
benchmark, CLaD achieves 94.7% success rate, competitive with large VLAs with significantly fewer parameters.
Read the full abstract ▾
All authors Andrew Jeong, Jaemin Kim, Sebin Lee, Sung-Eui Yoon
Robotics
Learning Semantic Priorities for Autonomous Target Search
Max Lodel, Nils Wilde, Robert Babuška et al. (+1 more)
The use of semantic features can improve the efficiency of target search in unknown environments for robotic search and rescue missions. Current target search methods rely on training with large
datasets of similar domains, which limits the adaptability to diverse environments. However, human experts possess high-level knowledge about semantic relationships necessary to effectively guide a
robot during target search missions in diverse and previously unseen environments. In this paper, we propose a target search method that leverages expert input to train a model of semantic priorities. By employing the learned priorities in a frontier exploration planner using combinatorial optimization, our approach achieves efficient target search driven by semantic features while ensuring robustness and complete coverage. The proposed semantic priority model is trained with several synthetic
datasets of simulated expert guidance for target search. Simulation tests in previously unseen environments show that our method consistently achieves faster target recovery than a coverage-driven exploration planner.
Read the full abstract ▾
All authors Max Lodel, Nils Wilde, Robert Babuška, Javier Alonso-Mora
Robotics
Interacting Multiple Model Proprioceptive Odometry for Legged Robots
Wanlei Li, Zichang Chen, Shilei Li et al. (+2 more)
State estimation for legged
robots remains challenging because legged odometry generally suffers from limited observability and therefore depends critically on measurement constraints to suppress drift. When exteroceptive sensors are unreliable or degraded, such constraints are mainly derived from proprioceptive measurements, particularly contact-related leg kinematics information. However, most existing proprioceptive odometry methods rely on an idealized point-contact assumption, which is often violated during real locomotion. Consequently, the effectiveness of proprioceptive constraints may be significantly reduced, resulting in degraded estimation accuracy. To address these limitations, we propose an interacting multiple model (IMM)-based proprioceptive odometry framework for legged
robots. By incorporating multiple contact hypotheses within a unified probabilistic framework, the proposed method enables online mode switching and probabilistic fusion under varying contact conditions. Extensive simulations and real-world experiments demonstrate that the proposed method achieves superior pose estimation accuracy over state-of-the-art methods while maintaining comparable computational efficiency.
Read the full abstract ▾
All authors Wanlei Li, Zichang Chen, Shilei Li, Xiaogang Xiong, Yunjiang Lou
Robotics
Industrial-Grade Robust Robot Vision for Screw Detection and Removal under Uneven Conditions
Tomoki Ishikura, Genichiro Matsuda, Takuya Kiyokawa et al. (+1 more)
As the amount of used home appliances is expected to increase despite the decreasing labor force in Japan, there is a need to automate disassembling processes at recycling plants. The automation of disassembling air conditioner outdoor units, however, remains a challenge due to unit size variations and exposure to dirt and rust. To address these challenges, this study proposes an automated system that integrates a task-specific two-stage detection method and a
lattice-based local calibration strategy. This approach achieved a screw detection recall of 99.8% despite severe degradation and ensured a manipulation accuracy of +/-0.75 mm without pre-programmed coordinates. In real-world validation with 120 units, the system attained a disassembly success rate of 78.3% and an average cycle time of 193 seconds, confirming its feasibility for industrial application.
Read the full abstract ▾
All authors Tomoki Ishikura, Genichiro Matsuda, Takuya Kiyokawa, Kensuke Harada
Robotics
Scaling Whole-Body Human Musculoskeletal Behavior Emulation for Specificity and Diversity
Yunyue Wei, Chenhui Zuo, Shanning Zhuang et al. (+3 more)
The
embodied learning of human motor control requires whole-body neuro-actuated musculoskeletal dynamics, while the internal muscle-driven processes underlying movement remain inaccessible to direct measurement. Computational modeling offers an alternative, but inverse dynamics methods struggled to resolve redundant control from observed kinematics in the high-dimensional, over-actuated system. Forward imitation approaches based on deep reinforcement learning exhibited inadequate tracking performance due to the curse of dimensionality in both control and reward design. Here we introduce a large-scale parallel musculoskeletal computation framework for biomechanically grounded whole-body motion reproduction. By integrating large-scale parallel GPU simulation with
adversarial reward aggregation and value-guided flow exploration, the MS-Emulator framework overcomes key optimization bottlenecks in high-dimensional reinforcement learning for musculoskeletal control, which accurately reproduces a broad repertoire of motions in a whole-body human musculoskeletal system actuated by approximately 700 muscles. It achieved high joint angle accuracy and body position
alignment for highly dynamic tasks such as dance, cartwheel, and backflip. The framework was also used to explore the musculoskeletal control solution space, identifying distinct musculoskeletal control policies that converge to nearly identical external kinematic and mechanical measurements. This work establishes a tractable computational route to analyzing the specificity and diversity underlying human
embodied control of movement. Project page: https://lnsgroup.cc/research/MS-Emulator.
Read the full abstract ▾
All authors Yunyue Wei, Chenhui Zuo, Shanning Zhuang, Haixin Gong, Yaming Liu, Yanan Sui