Robotics
Coverage Path Planning for Autonomous Sailboats in Inhomogeneous and Time-Varying Oceans: A Spatiotemporal Optimization Approach
Yang An, Zhikang Ge, Taiyu Zhang et al. (+3 more)
Autonomous sailboats are well suited for long-duration ocean observation due to their wind-driven endurance. However, their performance is highly anisotropic and strongly influenced by inhomogeneous and time-varying wind and current fields, limiting the effectiveness of existing coverage methods such as boustrophedon sweeping. Planning under these environmental and maneuvering constraints remains underexplored. This paper presents a spatiotemporal coverage path planning framework tailored to autonomous sailboats, combining (1)
topology-based morphological constraints in the spatial domain to promote compact and continuous coverage, and (2) forecast-aware look-ahead planning in the temporal domain to anticipate environmental
evolution and enable foresighted decision-making. Simulations conducted under
stochastic inhomogeneous and time-varying ocean environments, including scenarios with partial directional accessibility, demonstrate that the proposed method generates efficient and feasible coverage paths where traditional strategies often fail. To the best of our knowledge, this study provides the first dedicated solution to the coverage path planning problem for autonomous sailboats operating in inhomogeneous and time-varying ocean environments, establishing a foundation for future cooperative multi-sailboat coverage.
Read the full abstract ▾
All authors Yang An, Zhikang Ge, Taiyu Zhang, Jean-Baptiste R. G. Souppez, Gaofei Xu, Zhengru Ren
Robotics
Adaptive Illumination Control for Robot Perception
Yash Turkar, Shekoufeh Sadeghi and Karthik Dantu
Robot perception under low light or high dynamic range is usually improved downstream - via more robust feature extraction, image enhancement, or closed-loop exposure control. However, all of these approaches are limited by the image captured these conditions. An alternate approach is to utilize a programmable onboard light that adds to ambient illumination and improves captured images. However, it is not straightforward to predict its impact on image formation. Illumination interacts nonlinearly with depth, surface reflectance, and scene geometry. It can both reveal structure and induce failure modes such as specular highlights and saturation. We introduce Lightning, a closed-loop illumination-control framework for visual SLAM that combines relighting, offline optimization, and imitation learning. This is performed in three stages. First, we train a Co-Located Illumination Decomposition (CLID) relighting model that decomposes a
robot observation into an ambient component and a light-contribution field. CLID enables physically consistent synthesis of the same scene under alternative light intensities and thereby creates dense multi-intensity training data without requiring us to repeatedly re-run trajectories. Second, using these synthesized candidates, we formulate an offline Optimal Intensity Schedule (OIS) problem that selects illumination levels over a sequence trading off SLAM-relevant image utility against power consumption and temporal smoothness. Third, we distill this ideal solution into a real-time controller through behavior cloning, producing an Illumination Control Policy (ILC) that generalizes beyond the initial training distribution and runs online on a mobile
robot to command discrete light-intensity levels. Across our evaluation, Lightning substantially improves SLAM trajectory robustness while reducing unnecessary illumination power.
Read the full abstract ▾
Robotics
How Do We Research Human-Robot Interaction in the Age of Large Language Models? A Systematic Review
Yufeng Wang, Yuan Xu, Anastasia Nikolova et al. (+4 more)
Advances in large
language models (LLMs) are profoundly reshaping the field of human-
robot interaction (HRI). While prior work has highlighted the technical potential of LLMs, few studies have systematically examined their human-centered impact (e.g., human-oriented understanding, user modeling, and levels of autonomy), making it difficult to consolidate emerging challenges in LLM-driven HRI systems. Therefore, we conducted a systematic literature search following the PRISMA guideline, identifying 86 articles that met our inclusion criteria. Our findings reveal that: (1) LLMs are transforming the fundamentals of HRI by reshaping how
robots sense context, generate socially grounded interactions, and maintain continuous
alignment with human needs in
embodied settings; and (2) current research is largely exploratory, with different studies focusing on different facets of LLM-driven HRI, resulting in wide-ranging choices of experimental setups, study methods, and evaluation metrics. Finally, we identify key design considerations and challenges, offering a coherent overview and guidelines for future research at the intersection of LLMs and HRI.
Read the full abstract ▾
All authors Yufeng Wang, Yuan Xu, Anastasia Nikolova, Yuxuan Wang, Jianyu Wang, Chongyang Wang, Xin Tong
Robotics
Safe-SDL:Establishing Safety Boundaries and Control Mechanisms for AI-Driven Self-Driving Laboratories
Zihan Zhang, Haohui Que, Junhan Chang et al. (+3 more)
The emergence of Self-Driving Laboratories (SDLs) transforms scientific discovery methodology by integrating AI with robotic automation to create closed-loop experimental systems capable of autonomous hypothesis generation, experimentation, and analysis. While promising to compress research timelines from years to weeks, their deployment introduces unprecedented safety challenges differing from traditional laboratories or purely digital AI. This paper presents Safe-SDL, a comprehensive framework for establishing robust safety boundaries and control mechanisms in AI-driven autonomous laboratories. We identify and analyze the critical “Syntax-to-Safety Gap” – the disconnect between AI-generated syntactically correct commands and their physical safety implications – as the central challenge in SDL deployment. Our framework addresses this gap through three synergistic components: (1) formally defined Operational Design Domains (ODDs) that constrain system behavior within mathematically verified boundaries, (2) Control Barrier Functions (CBFs) that provide real-time safety guarantees through continuous state-space monitoring, and (3) a novel Transactional Safety Protocol (CRUTD) that ensures atomic consistency between digital planning and physical execution. We ground our theoretical contributions through analysis of existing implementations including UniLabOS and the Osprey architecture, demonstrating how these systems instantiate key safety principles. Evaluation against the LabSafety Bench reveals that current foundation models exhibit significant safety failures, demonstrating that architectural safety mechanisms are essential rather than optional. Our framework provides both theoretical foundations and practical implementation guidance for safe deployment of autonomous scientific systems, establishing the groundwork for responsible acceleration of AI-driven discovery.
Read the full abstract ▾
All authors Zihan Zhang, Haohui Que, Junhan Chang, Xin Zhang, Hao Wei, Tong Zhu
Robotics
CLOT: Closed-Loop Global Motion Tracking for Whole-Body Humanoid Teleoperation
Tengjie Zhu, Guanyu Cai, Yang Zhaohui et al. (+8 more)
Long-horizon whole-body
humanoid teleoperation remains challenging due to accumulated global pose drift, particularly on full-sized
humanoids. Although recent learning-based tracking methods enable agile and coordinated motions, they typically operate in the
robot's local frame and neglect global pose feedback, leading to drift and instability during extended execution. In this work, we present CLOT, a real-time whole-body
humanoid teleoperation system that achieves closed-loop global motion tracking via high-frequency localization feedback. CLOT synchronizes operator and
robot poses in a closed loop, enabling drift-free human-to-
humanoid mimicry over long timehorizons. However, directly imposing global tracking rewards in reinforcement learning, often results in aggressive and brittle corrections. To address this, we propose a data-driven randomization strategy that decouples observation trajectories from reward evaluation, enabling smooth and stable global corrections. We further regularize the policy with an
adversarial motion prior to suppress unnatural behaviors. To support CLOT, we collect 20 hours of carefully curated human motion data for training the
humanoid teleoperation policy. We design a
transformer-based policy and train it for over 1300 GPU hours. The policy is deployed on a full-sized
humanoid with 31 DoF (excluding hands). Both simulation and real-world experiments verify high-dynamic motion, high-precision tracking, and strong robustness in sim-to-real
humanoid teleoperation. Motion data, demos and code can be found in our website.
Read the full abstract ▾
All authors Tengjie Zhu, Guanyu Cai, Yang Zhaohui, Guanzhu Ren, Haohui Xie, ZiRui Wang, Junsong Wu, Jingbo Wang, Xiaokang Yang, Yao Mu, Yichao Yan
Robotics
RynnBrain: Open Embodied Foundation Models
Ronghao Dang, Jiayan Guo, Bohan Hou et al. (+23 more)
Despite rapid progress in
multimodal foundation models,
embodied intelligence community still lacks a unified, physically grounded foundation model that integrates perception,
reasoning, and planning within real-world spatial-temporal dynamics. We introduce RynnBrain, an open-source spatiotemporal foundation model for
embodied intelligence. RynnBrain strengthens four core capabilities in a unified framework: comprehensive
egocentric understanding, diverse spatiotemporal localization, physically grounded
reasoning, and physics-aware planning. The RynnBrain family comprises three foundation model scales (2B, 8B, and 30B-A3B MoE) and four post-trained variants tailored for downstream
embodied tasks (i.e., RynnBrain-Nav, RynnBrain-Plan, and RynnBrain-VLA) or complex spatial
reasoning tasks (i.e., RynnBrain-CoP). In terms of extensive evaluations on 20
embodied benchmarks and 8 general
vision understanding
benchmarks, our RynnBrain foundation models largely outperform existing
embodied foundation models by a significant margin. The post-trained model suite further substantiates two key potentials of the RynnBrain foundation model: (i) enabling physically grounded
reasoning and planning, and (ii) serving as a strong pretrained backbone that can be efficiently adapted to diverse
embodied tasks.
Read the full abstract ▾
All authors Ronghao Dang, Jiayan Guo, Bohan Hou, Sicong Leng, Kehan Li, Xin Li, Jiangpin Liu, Yunxuan Mao, Zhikai Wang, Yuqian Yuan, Minghao Zhu, Xiao Lin, Yang Bai, Qian Jiang, Yaxi Zhao, Minghua Zeng, Junlong Gao, Yuming Jiang, Jun Cen, Siteng Huang, Liuyi Wang, Wenqiao Zhang, Chengju Liu, Jianfei Yang, Shijian Lu, Deli Zhao
Robotics
AsyncVLA: An Asynchronous VLA for Fast and Robust Navigation on the Edge
Noriaki Hirose, Catherine Glossop, Dhruv Shah et al. (+1 more)
Robotic foundation models achieve strong generalization by leveraging internet-scale
vision-
language representations, but their massive computational cost creates a fundamental bottleneck: high inference latency. In dynamic environments, this latency breaks the control loop, rendering powerful models unsafe for real-time deployment. We propose AsyncVLA, an asynchronous control framework that decouples semantic
reasoning from reactive execution. Inspired by hierarchical control, AsyncVLA runs a large foundation model on a remote workstation to provide high-level guidance, while a lightweight, onboard Edge Adapter continuously refines actions at high frequency. To bridge the domain gap between these asynchronous streams, we introduce an end-to-end finetuning protocol and a trajectory re-weighting strategy that prioritizes dynamic interactions. We evaluate our approach on real-world
vision-based
navigation tasks with communication delays up to 6 seconds. AsyncVLA achieves a 40% higher success rate than state-of-the-art baselines, effectively bridging the gap between the semantic intelligence of large models and the reactivity required for edge robotics.
Read the full abstract ▾
All authors Noriaki Hirose, Catherine Glossop, Dhruv Shah, Sergey Levine
Robotics
Inferring Turn-Rate-Limited Engagement Zones with Sacrificial Agents for Safe Trajectory Planning
Grant Stagg and Cameron K. Peterson
This paper presents a learning-based framework for estimating pursuer parameters in turn-rate-limited pursuit-evasion scenarios using sacrificial
agents. Each sacrificial
agent follows a straight-line trajectory toward an adversary and reports whether it was intercepted or survived. These binary outcomes are related to the pursuer's parameters through a
geometric reachable-region (RR) model. Two formulations are introduced: a boundary-interception case, where capture occurs at the RR boundary, and an interior-interception case, which allows capture anywhere within it. The pursuer's parameters are inferred using a gradient-based multi-start optimization with custom loss functions tailored to each case. Two trajectory-selection strategies are proposed for the sacrificial
agents: a
geometric heuristic that maximizes the spread of expected interception points, and a
Bayesian experimental-design method that maximizes the D-score of the expected Gauss-Newton information
matrix, thereby selecting trajectories that yield maximal information gain. Monte Carlo experiments demonstrate accurate parameter recovery with five to twelve sacrificial
agents. The learned engagement models are then used to generate safe, time-optimal paths for high-value
agents that avoid all feasible pursuer engagement regions.
Read the full abstract ▾
Robotics
FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation
Huajian Zeng, Lingyun Chen, Jiaqi Yang et al. (+4 more)
Recent
vision-
language-action (VLA) models can generate plausible end-effector motions, yet they often fail in long-horizon, contact-rich tasks because the underlying hand-object interaction (HOI) structure is not explicitly represented. An embodiment-agnostic interaction representation that captures this structure would make manipulation behaviors easier to validate and transfer across
robots. We propose FlowHOI, a two-stage flow-matching framework that generates semantically grounded, temporally coherent HOI sequences, comprising hand poses, object poses, and hand-object contact states, conditioned on an
egocentric observation, a
language instruction, and a 3D Gaussian splatting (3DGS) scene reconstruction. We decouple geometry-centric grasping from semantics-centric manipulation, conditioning the latter on compact 3D scene
tokens and employing a motion-text
alignment loss to semantically ground the generated interactions in both the physical scene layout and the
language instruction. To address the scarcity of high-fidelity HOI supervision, we introduce a reconstruction pipeline that recovers aligned hand-object trajectories and meshes from large-scale
egocentric videos, yielding an HOI prior for robust generation. Across the GRAB and HOT3D
benchmarks, FlowHOI achieves the highest action recognition accuracy and a 1.7$\times$ higher physics simulation success rate than the strongest
diffusion-based baseline, while delivering a 40$\times$ inference speedup. We further demonstrate real-
robot execution on four dexterous manipulation tasks, illustrating the feasibility of retargeting generated HOI representations to real-
robot execution pipelines.
Read the full abstract ▾
All authors Huajian Zeng, Lingyun Chen, Jiaqi Yang, Yuantai Zhang, Fan Shi, Peidong Liu, Xingxing Zuo
Robotics · Cross-listed from cs.CV
Learning on the Fly: Replay-Based Continual Object Perception for Indoor Drones
Sebastian-Ion Nae, Mihai-Eugen Barbu, Sebastian Mocanu et al. (+1 more)
Autonomous
agents such as indoor drones must learn new object classes in real-time while limiting catastrophic forgetting, motivating Class-Incremental Learning (CIL). However, most unmanned aerial vehicle (
UAV)
datasets focus on outdoor scenes and offer limited temporally coherent indoor
videos. We introduce an indoor
dataset of $14,400$ frames capturing inter-drone and ground vehicle footage, annotated via a semi-automatic workflow with a $98.6\%$ first-pass labeling agreement before final manual verification. Using this
dataset, we
benchmark 3 replay-based CIL strategies: Experience Replay (ER), Maximally Interfered
Retrieval (MIR), and Forgetting-Aware Replay (FAR), using YOLOv11-nano as a resource-efficient detector for deployment-constrained
UAV platforms. Under tight
memory budgets ($5-10\%$ replay), FAR performs better than the rest, achieving an average accuracy (ACC, $mAP_{50-95}$ across increments) of $82.96\%$ with $5\%$ replay. Gradient-weighted class activation mapping (Grad-CAM) analysis shows
attention shifts across classes in mixed scenes, which is associated with reduced localization quality for drones. The experiments further demonstrate that replay-based continual learning can be effectively applied to edge aerial systems. Overall, this work contributes an indoor
UAV video dataset with preserved temporal coherence and an evaluation of replay-based CIL under limited replay budgets. Project page: https://spacetime-
vision-robotics-laboratory.github.io/learning-on-the-fly-cl
Read the full abstract ▾
All authors Sebastian-Ion Nae, Mihai-Eugen Barbu, Sebastian Mocanu, Marius Leordeanu
Robotics
Force Sensing for Wearable Human-Robot Interfaces via Fluidic Innervation
Noah Rubin, Ava Schraeder, Hrishikesh Sahu et al. (+2 more)
Mechanically characterizing the human-machine interface is essential to understanding user behavior and optimizing wearable
robot performance. This interface has been challenging to sensorize due to manufacturing complexity and non-linear sensor responses. Here, we measure human limb-device interaction via fluidic innervation, creating a 3D-printed silicone pad with embedded air channels to measure forces. As forces are applied to the pad, the air channels compress, resulting in a pressure change measurable by off-the-shelf pressure transducers. We demonstrate in benchtop testing that pad pressure is highly linearly related to applied force ($R^2 = 0.998$) and confirmed strong linear relationships to isometric knee torque in a clinical dynamometer with strategic pad placement. We built on these idealized settings to test pad performance in more unconstrained settings, including during cyclic dynamic and stepwise isometric bicep curls. Finally, we integrated the sensor into a lower-extremity robotic exoskeleton and recorded pad pressure during repeated squats with the device unpowered. Pad pressure tracked squat phase and overall task dynamics consistently. Collectively, our preliminary results suggest fluidic innervation is a readily customizable sensing modality with high signal-to-noise ratio and temporal resolution for capturing human-machine interaction. In the long-term, this modality may provide an alternative real-time sensing input to control / optimize wearable robotic systems and to capture user function during device use.
Read the full abstract ▾
All authors Noah Rubin, Ava Schraeder, Hrishikesh Sahu, Thomas C. Bulea, Lillian Chin
Robotics · Cross-listed from cs.NI
Location as a service with a MEC architecture
Christopher Schahn, Jorin Kouril, Bernd Schaeufele et al. (+1 more)
In recent years, automated driving has become viable, and advanced driver assistance systems (ADAS) are now part of modern cars. These systems require highly precise positioning. In this paper, a cooperative approach to localization is presented. The GPS information from several road users is collected in a Mobile Edge Computing cloud, and the characteristics of GNSS positioning are used to provide lane-precise positioning for all participants by applying probabilistic filters and HD maps.
Read the full abstract ▾
All authors Christopher Schahn, Jorin Kouril, Bernd Schaeufele, Ilja Radusch
Robotics
Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videos
Albert J. Zhai, Kuo-Hao Zeng, Jiasen Lu et al. (+3 more)
The ability to learn manipulation skills by watching
videos of humans has the potential to unlock a new source of highly scalable data for
robot learning. Here, we tackle prehensile manipulation, in which tasks involve grasping an object before performing various post-grasp motions. Human
videos offer strong signals for learning the post-grasp motions, but they are less useful for learning the prerequisite grasping behaviors, especially for
robots without human-like hands. A promising way forward is to use a modular policy design, leveraging a dedicated grasp generator to produce stable grasps. However, arbitrary stable grasps are often not task-compatible, hindering the
robot's ability to perform the desired downstream motion. To address this challenge, we present Perceive-Simulate-Imitate (PSI), a framework for training a modular manipulation policy using human
video motion data processed by paired grasp-trajectory filtering in simulation. This simulation step extends the trajectory data with grasp suitability labels, which allows for supervised learning of task-oriented grasping capabilities. We show through real-world experiments that our framework can be used to learn precise manipulation skills efficiently without any
robot data, resulting in significantly more robust performance than using a grasp generator naively.
Read the full abstract ▾
All authors Albert J. Zhai, Kuo-Hao Zeng, Jiasen Lu, Ali Farhadi, Shenlong Wang, Wei-Chiu Ma
Robotics
William Chen, Jagdeep Singh Bhatia, Catherine Glossop et al. (+6 more)
Pretrained
vision-
language models (VLMs) can make semantic and visual inferences across diverse settings, providing valuable common-sense priors for robotic control. However, effectively grounding this knowledge in
robot behaviors remains an open challenge. Prior methods often employ a hierarchical approach where VLMs reason over high-level commands to be executed by separate low-level policies, e.g.,
vision-
language-action models (VLAs). The interface between VLMs and VLAs is usually natural
language task instructions, which fundamentally limits how much VLM
reasoning can steer low-level behavior. We thus introduce Steerable Policies: VLAs trained on rich synthetic commands at various levels of abstraction, like subtasks, motions, and grounded pixel coordinates. By improving low-level controllability, Steerable Policies can unlock pretrained knowledge in VLMs, enabling improved task generalization. We demonstrate this benefit by controlling our Steerable Policies with both a learned high-level
embodied reasoner and an off-the-shelf VLM prompted to reason over command abstractions via in-context learning. Across extensive real-world manipulation experiments, these two novel methods outperform prior
embodied reasoning VLAs and VLM-based hierarchical baselines, including on challenging generalization and long-horizon tasks. Website: steerable-policies.github.io
Read the full abstract ▾
All authors William Chen, Jagdeep Singh Bhatia, Catherine Glossop, Nikhil Mathihalli, Ria Doshi, Andy Tang, Danny Driess, Karl Pertsch, Sergey Levine
Robotics
Human Emotion-Mediated Soft Robotic Arts: Exploring the Intersection of Human Emotions, Soft Robotics and Arts
Saitarun Nadipineni, Chenhao Hong, Tanishtha Ramlall et al. (+4 more)
Soft robotics has emerged as a versatile field with applications across various domains, from healthcare to industrial automation, and more recently, art and interactive installations. The inherent flexibility, adaptability, and safety of soft
robots make them ideal for applications that require delicate, organic, and lifelike movement, allowing for immersive and responsive interactions. This study explores the intersection of human
emotions, soft robotics, and art to establish and create new forms of human
emotion-mediated soft robotic art. In this paper, we introduce two soft embodiments: a soft character and a soft flower as an art display that dynamically responds to
brain signals based on alpha waves, reflecting different
emotion levels. We present how human
emotions can be measured as alpha waves based on
brain/
EEG signals, how we map the alpha waves to the dynamic movements of the two soft embodiments, and demonstrate our proposed concept using experiments. The findings of this study highlight how soft robotics can embody human emotional states, offering a new medium for insightful artistic expression and interaction, and demonstrating how art displays can be
embodied.
Read the full abstract ▾
All authors Saitarun Nadipineni, Chenhao Hong, Tanishtha Ramlall, Chapa Sirithunge, Kaspar Althoefer, Fumiya Iida, Thilina Dulantha Lalitharatne
Robotics
Temporally-Sampled Efficiently Adaptive State Lattices for Autonomous Ground Robot Navigation in Partially Observed Environments
Ashwin Satish Menon, Eric R. Damm, Eli S. Lancaster et al. (+3 more)
Due to sensor limitations, environments that off-road mobile
robots operate in are often only partially observable. As the
robots move throughout the environment and towards their goal, the optimal route is continuously revised as the sensors perceive new information. In traditional autonomous
navigation architectures, a regional motion planner will consume the environment map and output a trajectory for the local motion planner to use as a reference. Due to the continuous revision of the regional plan guidance as a result of changing map information, the reference trajectories which are passed down to the local planner can differ significantly across sequential planning cycles. This rapidly changing guidance can result in unsafe
navigation behavior, often requiring manual safety interventions during autonomous traversals in off-road environments. To remedy this problem, we propose Temporally-Sampled Efficiently Adaptive State
Lattices (TSEASL), which is a regional planner arbitration architecture that considers updated and optimized versions of previously generated trajectories against the currently generated trajectory. When tested on a Clearpath Robotics Warthog Unmanned Ground Vehicle as well as real map data collected from the Warthog, results indicate that when running TSEASL, the
robot did not require manual interventions in the same locations where the
robot was running the baseline planner. Additionally, higher levels of planner stability were recorded with TSEASL over the baseline. The paper concludes with a discussion of further improvements to TSEASL in order to make it more generalizable to various off-road autonomy scenarios.
Read the full abstract ▾
All authors Ashwin Satish Menon, Eric R. Damm, Eli S. Lancaster, Felix A. Sanchez, Jason M. Gregory, Thomas M. Howard
Robotics · Cross-listed from math.OC
A Data-Driven Algorithm for Model-Free Control Synthesis
Sean Bowerfind, Matthew R. Kirchner and Gary Hewer
Presented is an
algorithm to synthesize the optimal infinite-horizon LQR feedback controller for continuous-time systems. The
algorithm does not require knowledge of the system dynamics but instead uses only a finite-length sampling of input-output data. A necessary condition that relates the optimal LQR gain to any arbitrary solution trajectory of the system is presented. An
algorithm using this necessary condition, based on constrained optimization, is developed that estimates the LQR gain
matrix. In addition to calculating the standard feedback gain
matrix, a feedforward gain can be found to implement a reference tracking controller. This paper presents a theoretical justification for the method and shows several examples, including a validation test flight on a real scale aircraft with unknown dynamics.
Read the full abstract ▾
Robotics
UniManip: Open-World Robotic Manipulation via Agentic Operational Graphs
Haichao Liu, Yuanjiang Xue, Yuheng Zhou et al. (+5 more)
Achieving general-purpose robotic manipulation requires
robots to bridge high-level semantic intent with low-level physical interaction in unstructured environments. However, zero-shot generalization remains challenging: end-to-end
Vision-
Language-Action (VLA) models often lack the precision required for long-horizon tasks, while traditional hierarchical planners suffer from semantic
rigidity under open-world variations. We present UniManip, a unified framework centered on an Agentic Operational Graph (AOG), an explicit cognitive and operational backbone for task orchestration, state tracking, execution verification, and closed-loop failure recovery. High-level semantic intents are grounded into actionable primitives via a physically grounded scene graph that provides
geometric affordances, contact-aware interaction parameters, and collision-free trajectory synthesis. By coupling a high-level Agentic Layer for task orchestration with a low-level Scene Layer for dynamic state representation, UniManip continuously aligns abstract planning with
geometric constraints, enabling robust execution. Unlike static pipelines, UniManip operates as a dynamic agentic loop: it instantiates object-centric scene graphs from unstructured perception, parameterizes these representations into collision-free trajectories via a safety-aware local planner, and exploits structured
memory to autonomously diagnose and recover from execution failures. UniManip achieves state-of-the-art performance on LIBERO-Pro, while extensive real-world experiments demonstrate robust zero-shot generalization across diverse tasks, embodiments, and dexterous hands, outperforming leading VLA models and hierarchical baselines by 22.5% and 25.0% in task success rate, respectively. Open-source project page: https://henryhcliu.github.io/unimanip.
Read the full abstract ▾
All authors Haichao Liu, Yuanjiang Xue, Yuheng Zhou, Li Yang, Haoyuan Deng, Yinan Liang, Lihua Xie, Ziwei Wang
Robotics
Agentic AI for Robot Control: Flexible but still Fragile
Oscar Lima, Marc Vinci, Martin Günther et al. (+10 more)
Recent work leverages the capabilities and commonsense priors of generative models for
robot control. In this paper, we present an agentic control system in which a
reasoning-capable
language model plans and executes tasks by selecting and invoking
robot skills within an iterative planner and executor loop. We deploy the system on two physical
robot platforms in two settings: (i) tabletop grasping, placement, and box insertion in indoor mobile manipulation (Mobipick) and (ii) autonomous agricultural
navigation and sensing (Valdemar). Both settings involve uncertainty, partial observability, sensor noise, and ambiguous natural-
language commands. The system exposes structured introspection of its planning and decision process, reacts to exogenous events via explicit event checks, and supports operator interventions that modify or redirect ongoing execution. Across both platforms, our proof-of-concept experiments reveal substantial fragility, including non-deterministic suboptimal behavior, instruction-following errors, and high sensitivity to
prompt specification. At the same time, the architecture is flexible: transfer to a different
robot and task domain largely required updating the system
prompt (domain model, affordances, and action catalogue) and re-binding the same tool interface to the platform-specific skill API.
Read the full abstract ▾
All authors Oscar Lima, Marc Vinci, Martin Günther, Marian Renz, Alexander Sung, Sebastian Stock, Johannes Brust, Lennart Niecksch, Zongyao Yi, Felix Igelbrink, Benjamin Kisliuk, Martin Atzmueller, Joachim Hertzberg
Robotics
SENSE-STEP: Learning Sim-to-Real Locomotion for a Sensory-Enabled Soft Quadruped Robot
Storm de Kam, Ebrahim Shahabi and Cosimo Della Santina
Robust closed-loop locomotion remains challenging for soft quadruped
robots due to high-dimensional dynamics, actuator hysteresis, and difficult-to-model contact interactions, while conventional proprioception provides limited information about ground contact. In this paper, we present a learning-based control framework for a pneumatically actuated soft quadruped equipped with tactile suction-cup feet, and we validate the approach experimentally on physical hardware. The control policy is trained in simulation through a staged learning process that starts from a reference gait and is progressively refined under randomized environmental conditions. The resulting controller maps proprioceptive and tactile feedback to coordinated pneumatic actuation and suction-cup commands, enabling closed-loop locomotion on flat and inclined surfaces. When deployed on the real
robot, the closed-loop policy outperforms an open-loop baseline, increasing forward speed by 41% on a flat surface and by 91% on a 5-degree incline. Ablation studies further demonstrate the role of tactile force estimates and inertial feedback in stabilizing locomotion, with performance improvements of up to 56% compared to configurations without sensory feedback.
Read the full abstract ▾
Robotics
How Swarms Differ: Challenges in Collective Behaviour Comparison
André Fialho Jesus and Jonas Kuckling
Collective behaviours often need to be expressed through numerical features, e.g., for classification or imitation learning. This problem is often addressed by proposing an ad-hoc feature set for a particular swarm behaviour context, usually without further consideration of the solution's resilience outside of the conceived context. Yet, the development of automatic methods to design swarm behaviours is dependent on the ability to measure quantitatively the similarity of swarm behaviours. Hence, we investigate the impact of feature sets for collective behaviours. We select swarm feature sets and similarity measures from prior swarm robotics works, which mainly considered a narrow behavioural context and assess their robustness. We demonstrate that the interplay of feature set and similarity measure makes some combinations more suitable to distinguish groups of similar behaviours. We also propose a self-organised map-based approach to identify regions of the feature space where behaviours cannot be easily distinguished.
Read the full abstract ▾
Robotics
Learning Native Continuation for Action Chunking Flow Policies
Yufeng Liu, Hang Yu, Juntu Zhao et al. (+9 more)
Action chunking enables
Vision Language Action (VLA) models to run in real time, but naive chunked execution often exhibits discontinuities at chunk boundaries. Real-Time Chunking (RTC) alleviates this issue but is external to the policy, leading to spurious
multimodal switching and trajectories that are not intrinsically smooth. We propose Legato, a training-time continuation method for action-chunked flow-based VLA policies. Specifically, Legato initializes denoising from a schedule-shaped mixture of known actions and noise, exposing the model to partial action information. Moreover, Legato reshapes the learned flow dynamics to ensure that the denoising process remains consistent between training and inference under per-step guidance. Legato further uses randomized schedule condition during training to support varying inference delays and achieve controllable smoothness. Empirically, Legato produces smoother trajectories and reduces spurious
multimodal switching during execution, leading to less hesitation and shorter task completion time. Extensive real-world experiments show that Legato consistently outperforms RTC across five manipulation tasks, achieving approximately 10% improvements in both trajectory smoothness and task completion time.
Read the full abstract ▾
All authors Yufeng Liu, Hang Yu, Juntu Zhao, Bocheng Li, Di Zhang, Mingzhu Li, Wenxuan Wu, Yingdong Hu, Junyuan Xie, Junliang Guo, Dequan Wang, Yang Gao
Robotics
INHerit-SG: Incremental Hierarchical Semantic Scene Graphs for Complex Embodied Retrieval
YukTungSamuel Fang, Zhikang Shi, Jiabin Qiu et al. (+5 more)
Object
retrieval in 3D scenes remains difficult when queries combine hierarchical context, attributes, negation, and chained spatial relations. Existing semantic maps, scene graphs, and agentic
memory support open-vocabulary
retrieval, but jointly resolving these constraints remains challenging. In this work, we propose INHerit-SG, an incremental hierarchical semantic scene graph with an accompanying
retrieval method for complex
embodied retrieval. Asynchronous
geometric and semantic processing and structural event-driven maintenance organize observations into a hierarchy whose node semantics and spatial relations are recorded as natural
language text. The same
memory supports direct reading and
reasoning by more capable
language models or query parsing followed by structured
retrieval. Controlled comparisons show that direct reading is more accurate with powerful hosted models, while structured
retrieval improves the accuracy of smaller local models and lowers per-query API cost for hosted models. We also construct HM3DSem-SQR, containing 6,084 queries from 36 HM3D-Sem scenes, and evaluate on Main-864, a class-balanced subset spanning all 36 scenes, and in three real-world environments. Experiments show that INHerit-SG outperforms the strongest baselines in both simulator and real-world evaluations. The largest gains come on attribute and compositional queries (7.6 and 6.3 strict Top-1 points on Main-864), with a 9.8-point overall success margin on real
robots. Project Page: https://fangyuktung.github.io/INHeritSG.github.io/
Read the full abstract ▾
All authors YukTungSamuel Fang, Zhikang Shi, Jiabin Qiu, Zixuan Chen, Jieqi Shi, Hao Xu, Jing Huo, Yang Gao
Robotics
Adding internal audio sensing to internal vision enables human-like in-hand fabric recognition with soft robotic fingertips
Iris Andrussow, Jans Solano, Benjamin A. Richardson et al. (+2 more)
Distinguishing the feel of smooth silk from coarse cotton is a trivial everyday task for humans. When exploring such fabrics, fingertip skin senses both spatio-temporal force patterns and texture-induced vibrations that are integrated to form a haptic representation of the explored material. It is challenging to reproduce this rich, dynamic perceptual capability in
robots because tactile sensors typically cannot achieve both high spatial resolution and high temporal sampling rate. In this work, we present a system that can sense both types of haptic information, and we investigate how each type influences robotic tactile perception of fabrics. Our robotic hand's middle finger and thumb each feature a soft tactile sensor: one is the open-source Minsight sensor that uses an internal camera to measure fingertip deformation and force at 50 Hz, and the other is our new sensor Minsound that captures vibrations through an internal MEMS microphone with a bandwidth from 50 Hz to 15 kHz. Inspired by the movements humans make to evaluate fabrics, our
robot actively encloses and rubs folded fabric samples between its two sensitive fingers. Our results test the influence of each sensing modality on overall classification performance, showing high utility for the audio-based sensor. Our
transformer-based method achieves a maximum fabric classification accuracy of 97 % on a
dataset of 20 common fabrics. Incorporating an external microphone away from Minsound increases our method's robustness in loud ambient noise conditions. To show that this audio-visual tactile sensing approach generalizes beyond the training data, we learn general representations of fabric stretchiness, thickness, and roughness.
Read the full abstract ▾
All authors Iris Andrussow, Jans Solano, Benjamin A. Richardson, Georg Martius, Katherine J. Kuchenbecker
Robotics
SKYSURF: A Self-learning Framework for Persistent Surveillance using Cooperative Aerial Gliders
Houssem Eddine Mohamadi and Nadjia Kara
The success of surveillance applications involving small unmanned aerial vehicles (
UAVs) depends on how long the limited on-board power would persist. To cope with this challenge, alternative renewable sources of lift are sought. One promising solution is to extract energy from rising masses of buoyant air. This paper proposes a local-global behavioral management and decision-making approach for the autonomous deployment of soaring-capable
UAVs. The cooperative
UAVs are modeled as non-deterministic finite state-based rational
agents. In addition to a mission planning module for assigning tasks and issuing dynamic
navigation waypoints for a new path planning scheme, in which the concepts of visibility and prediction are applied to avoid the collisions. Moreover, a delayed learning and tuning strategy is employed optimize the gains of the path tracking controller. Rigorous comparative analyses carried out with three benchmarking baselines and 15 evolutionary
algorithms highlight the adequacy of the proposed approach for maintaining the surveillance persistency (staying aloft for longer periods without landing) and maximizing the detection of targets (two times better than non-cooperative and semi-cooperative approaches) with less power consumption (almost 6% of battery consumed in six hours).
Read the full abstract ▾
Robotics
SafeFlowMPC: Predictive and Safe Trajectory Planning for Robot Manipulators with Learning-based Policies
Thies Oelerich, Gerald Ebmer, Christian Hartl-Nesic et al. (+1 more)
The emerging integration of
robots into everyday life brings several major challenges. Compared to classical industrial applications, more flexibility is needed in combination with real-time reactivity. Learning-based methods can train powerful policies based on demonstrated trajectories, such that the
robot generalizes a task to similar situations. However, these black-box models lack interpretability and rigorous safety guarantees. Optimization-based methods provide these guarantees but lack the required flexibility and generalization capabilities. This work proposes SafeFlowMPC, a combination of flow matching and online optimization to combine the strengths of learning and optimization. This method guarantees safety at all times and is designed to meet the demands of real-time execution by using a suboptimal model-predictive control formulation. SafeFlowMPC achieves strong performance in three real-world experiments on a KUKA 7-DoF manipulator, namely two grasping experiment and a dynamic human-
robot object handover experiment. A
video of the experiments is available at http://www.acin.tuwien.ac.at/42d6. The code is available at https://github.com/TU-Wien-ACIN-CDS/SafeFlowMPC.
Read the full abstract ▾
All authors Thies Oelerich, Gerald Ebmer, Christian Hartl-Nesic, Andreas Kugi
Robotics · Cross-listed from cs.HC
Media Framing Moderates Risk-Benefit Perceptions and Value Tradeoffs in Human-Robot Collaboration
Philipp Brauner, Felix Glawe, Luisa Vervier et al. (+1 more)
Public acceptance of industrial human-
robot collaboration (HRC) is shaped by how risks and benefits are perceived by affected employees. Positive or negative media framing may shape and shift how individuals evaluate HRC. This study examines how message framing moderates the effects of perceived risks and perceived benefits on overall attributed value. In a pre-registered study, participants (N = 1150) were randomly assigned to read either a positively or negatively framed newspaper article in one of three industrial contexts (autonomy, employment, safety) about HRC in production. Subsequently, perceived risks, benefits, and value were measured using reliable and publicly available psychometric scales. Two multiple regressions (one per framing condition) tested for main and interaction effects. Framing influenced absolute evaluations of risk, benefits, and value. In both frames, risks and benefits significantly predicted attributed value. Under positive framing, only main effects were observed (risks: beta = -0.52; benefits: beta = 0.45). Under negative framing, both predictors had stronger main effects (risks: beta = -0.69; benefits: beta = 0.63) along with a significant negative interaction (beta = -0.32), indicating that higher perceived risk diminishes the positive effect of perceived benefits. Model fit was higher for the positive frame (R^2 = 0.715) than for the negative frame (R^2 = 0.583), indicating greater explained variance in value attributions. Framing shapes the absolute evaluation of HRC and how risks and benefits are cognitively integrated in trade-offs. Negative framing produces stronger but interdependent effects, whereas positive framing supports additive evaluations. These findings highlight the role of strategic communication in fostering acceptance of HRC and underscore the need to consider framing in future HRC research.
Read the full abstract ▾
All authors Philipp Brauner, Felix Glawe, Luisa Vervier, Martina Ziefle
Robotics
Scaling Single Human Demonstrations for Imitation Learning using Generative Foundational Models
Nick Heppert, Minh Quang Nguyen and Abhinav Valada
Imitation learning is a popular paradigm to teach
robots new tasks, but collecting
robot demonstrations through teleoperation or kinesthetic teaching is tedious and time-consuming. In contrast, directly demonstrating a task using our human embodiment is much easier and data is available in abundance, yet transfer to the
robot can be non-trivial. In this work, we propose Real2Gen to train a manipulation policy from a single human demonstration. Real2Gen extracts required information from the demonstration and transfers it to a simulation environment, where a programmable expert
agent can demonstrate the task arbitrarily many times, generating an unlimited amount of data to train a flow matching policy. We evaluate Real2Gen on human demonstrations from three different real-world tasks and compare it to a recent baseline. Real2Gen shows an average increase in the success rate of 26.6% and better generalization of the trained policy due to the abundance and diversity of training data. We further deploy our purely simulation-trained policy zero-shot in the real world. We make the data, code, and trained models publicly available at real2gen.cs.uni-freiburg.de.
Read the full abstract ▾