Robotics
Reinforcement Learning for Optimal Experiment Design in Parameter Identification of Mechatronic Systems
Julian Langschwert, Georg Schaefer, Jakob Rehrl et al. (+2 more)
Informative excitation signals are critical for accurate system identification of mechatronic systems, yet classical system identification (SI) approaches require expert knowledge and hand-crafted signal design to respect hardware safety constraints, limiting their generalizability. We propose a reinforcement learning (RL)
agent that learns optimal excitation signals for a Quanser Aero 2 testbed while autonomously enforcing safety constraints through reward shaping. Evaluated across 10 independent training seeds, our comprehensive
agent achieves competitive estimation accuracy across all three identified parameters, outperforming classical baselines while incurring only 0.75% safety violations.
Read the full abstract ▾
All authors Julian Langschwert, Georg Schaefer, Jakob Rehrl, Stefan Huber, Simon Hirlaender
Robotics · Cross-listed from cs.MA
Differentiable Model Predictive Safety for Heterogeneous Mobility at Urban Intersections
Wenzhe Song and Hao Zhang
The imminent integration of autonomous vehicles and mobile
robots in urban settings presents a critical safety challenge for future intelligent transportation systems. This paper addresses the complex problem of coordinating heterogeneous
agents with disparate dynamics at unregulated intersections. We introduce a novel framework, differentiable model predictive safety (DMPS), which embeds the foresight of model-predictive control into a data-driven, end-to-end reinforcement learning architecture. DMPS
agents learn a latent dynamics model to predict future trajectories contingent on their actions. A learned, differentiable safety critic then evaluates the risk of these trajectories. Crucially, by leveraging backpropagation through the entire unrolled predictive model,
agents can efficiently compute the gradient of future safety with respect to their current action, enabling a minimal and precise online safety correction. Integrated into a multi-
agent training scheme, DMPS virtually eliminates collisions to less than 5.6% in high-density, mixed vehicle-
robot traffic simulations, demonstrating state-of-the-art safety without compromising energy and traffic efficiency.
Read the full abstract ▾
Robotics · Cross-listed from cs.AI
Reason–Imagine–Act: Closed-Loop LLM Decision Making with World Models for Autonomous Driving
Zhengqi Sun, Yiwen Sun, Boxuan Liu et al. (+3 more)
Large
language models (LLMs) are promising for autonomous driving, but semantics-only decision policies can yield physically unsafe behavior in dynamic traffic. Existing methods either perform online
language reasoning without explicit dynamics verification or use world models mainly in offline pipelines, leaving a gap between semantic intent and physical feasibility at decision time. We propose Reason–Imagine–Act (RIA), a closed-loop framework that couples an LLM reasoner with an action-conditioned world model for online safety verification. At each step, the LLM proposes an action template and candidate sub-actions, the world model performs short-horizon rollouts, and a safety scorer selects the safest executable action with feedback to the next
reasoning step. Under a unified CARLA point-goal protocol (1000 episodes), RIA achieves 80.05% route completion, 51.10% arrival rate, and 0.20% collision rate. Under the same closed-loop interface, RIA consistently outperforms training-free baselines, including CARLA TM and MADA, on core closed-loop metrics. For reproducibility, code is available at https://github.com/pku-smart-city/source_code/tree/main/RIA.
Read the full abstract ▾
All authors Zhengqi Sun, Yiwen Sun, Boxuan Liu, Tailai Chen, Tianxu Guo, Jiabin Liu
Robotics
Conflict-Aware Active Perception and Control in 3D Gaussian Splatting Fields via Control Barrier Functions
Amirhossein Mollaei Khass, Athanasios Cosse, Vivek Pandey et al. (+1 more)
Active perception in uncertain environments requires
robots to navigate safely while acquiring informative observations to reduce map uncertainty. These objectives inherently conflict, as informative viewpoints often lie near uncertain regions with higher collision risk. To address this challenge, we develop a conflict-aware active perception and control framework for robotic systems operating in environments represented by 3D Gaussian Splatting (3DGS). Safety is enforced using a Control Barrier Function (CBF) derived from an Average Value-at-Risk AV@R collision-risk metric that accounts for
geometric uncertainty and guarantees forward invariance of a safe set. To improve perception, we propose a risk-aware Expected Information Gain (EIG) formulation for selecting the next-best-view and introduce perception barrier functions that align the camera orientation with the local information-ascent direction. To obtain a tractable formulation for these conflicting safety and perception objectives, we propose a unified safety-critical, perception-aware quadratic program that enforces safety as a hard constraint while relaxing perception constraints through slack variables. Simulation results demonstrate that the proposed method improves both safety and information acquisition compared to existing 3DGS-based approaches.
Read the full abstract ▾
All authors Amirhossein Mollaei Khass, Athanasios Cosse, Vivek Pandey, Nader Motee
Robotics
Fault-Tolerant, Rigidity-Preserving Control of Inflatable Truss Robots
James Wade, Isaac Weaver, Mihai Stanciu et al. (+1 more)
Isoperimetric robotic trusses can adapt to different tasks and environments due to their high strength-to-weight ratio and the ability to reconfigure dramatically into new shapes. However, motor failures in operational environments can severely limit capability if not properly addressed. This paper presents a fault-tolerant control framework for an inflatable robotic truss that maintains functionality despite motor failures, demonstrated through three key contributions. First, we extend the kinematic optimization to handle arbitrary combinations of motor failures by imposing equality constraints to ensure failed actuators are not used. Second, we introduce discrete-time control barrier function (DTCBF) constraints that, under the assumed kinematic model, mathematically guarantee structural
rigidity while maximizing workspace utilization. Third, we implement closed-loop position control using onboard encoder feedback and a forward-kinematics-based state
estimator, improving positional accuracy in the presence of disturbances. We validate our approach through simulation and hardware experiments on a 2D isoperimetric truss testbed. For a 2D configuration with 6 actuators, we demonstrate >62% workspace preservation under single-motor failures, a >22% reduction in tracking RMSE using failure-aware closed-loop control relative to the fully functional open-loop baseline, and a 39% closed-loop RMSE reduction over failure-unaware open-loop control under unmodeled partial actuator degradation. These results establish a foundation for more robust and resilient isoperimetric truss
robots operating under degraded actuation.
Read the full abstract ▾
All authors James Wade, Isaac Weaver, Mihai Stanciu, Nathan Usevitch
Robotics · Cross-listed from cs.CV
Faster or Stronger: Towards Flexible Visual Place Recognition via Weighted Aggregation and Token Pruning
Zichao Zeng, June Moh Goo, Junwei Zheng et al. (+4 more)
Visual Place Recognition (VPR) aims to match a query image to reference images of the same place in a large-scale database. Recent state-of-the-art methods employ
Vision Transformers (ViTs) as backbone foundation models to extract patch-level features that are robust to viewpoint, illumination, and seasonal variations, which are then aggregated into a compact global descriptor for
retrieval. Most existing aggregation methods uniformly pool patch
tokens into learned clusters, despite the fact that different clusters often encode distinct spatial or semantic patterns and contribute unequally to VPR performance. To address this limitation, we propose Weighted Aggregated Descriptor (WeiAD), which assigns weights to clusters during aggregation, producing more discriminative global representations. Beyond accuracy,
retrieval latency is a critical concern for large-scale deployments and resource-constrained edge devices. Prior work mainly reduces latency by compressing global descriptors, while overlooking the cost of feature extraction, an issue exacerbated by ViT-based backbones. We therefore introduce WeiToP, a VPR-oriented
token pruning framework that reduces feature extraction cost via self-
distillation, where aggregation-induced
token importance supervises a lightweight
pruning module attached to an early
transformer layer, enabling inference-time
token pruning. After a single joint training phase, WeiToP enables plug-and-play
token pruning at inference time, allowing flexible and on-demand control over the accuracy-efficiency trade-off without additional training. Moreover, WeiToP outperforms existing
token pruning methods adapted from general
vision tasks.
Read the full abstract ▾
All authors Zichao Zeng, June Moh Goo, Junwei Zheng, Weijia Fan, Jiaming Zhang, Rainer Stiefelhagen, Jan Boehm
Robotics
The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agents
Doguhan Yeke, Elif Su Temirel, Ananth Shreekumar et al. (+3 more)
Vision-
language models (VLMs) are used as high-level planners for
embodied agents, translating natural
language instructions and visual observations into action plans. While prior work has studied abstention in LLMs, existing
benchmarks are largely text-only and do not capture the perceptual grounding and physical constraints inherent to
embodied robotics environments. In such settings, abstention requires recognizing when instructions are ambiguous, physically infeasible, based on false premises, or otherwise unresolvable given the available sensory modalities and context. To address this gap, we introduce a taxonomy to categorize abstention in the context of
embodied robotics and present RoboAbstention, a scalable and auditable framework for generating abstention instructions grounded in images gathered from five robotics
datasets. RoboAbstention instantiates the taxonomy through a three-phase pipeline: (1) structured visual grounding, (2) deterministic constraint derivation, and (3) controlled instruction generation via category-specific templates. This enables the construction of a diverse
dataset with verifiable abstention conditions. We evaluate several frontier VLMs and find that all models exhibit significant weaknesses in abstention, including those with advanced
reasoning capabilities. The best-performing model, Gemini 2.5 Flash, abstains on only 39.0% of our 6,069
benchmark instructions, while the
embodied planner Gemini Robotics ER 1.6 Preview abstains on just 16.5%. We further explore methods for improving abstention in VLM planners, such as defensive prompting and in-context learning, and find that these interventions substantially improve performance, reaching 93.6% abstention rate for Gemini Robotics ER 1.6 Preview and 88.6% for GPT 5.4 Mini, yet no approach fully solves the problem. We open-source RoboAbstention at https://purseclab.github.io/RoboAbstention/.
Read the full abstract ▾
All authors Doguhan Yeke, Elif Su Temirel, Ananth Shreekumar, Brandon Lee, Dongyan Xu, Z Berkay Celik
Robotics
Enhancing Graph-Based SLAM in GNSS-Denied environments by leveraging leg odometry
Léon Perruchot-Triboulet, Luc Jaulin and Kai Xiao
Autonomous
navigation in GNSS-denied environments remains a core challenge for legged
robots, where exteroceptive sensors such as
LiDAR are prone to elevation drift in geometrically sparse or repetitive scenes. We present a factor graph architecture that augments the LIO-SAM framework with a parallel kinematic lane driven by proprioceptive leg odometry, coupled to the main
LiDAR-inertial lane via an identity relative pose constraint with a selective noise model. Applied to a Linxai D50 quadruped platform across two outdoor loops totaling over one kilometer, our approach reduces elevation drift from over 30m to under 30cm and enables convergence in a scene where the baseline pipeline fails entirely. These results suggest that proprioceptive data, already computed onboard for gait control, constitutes a lightweight and effective vertical anchor for SLAM in GNSS-denied settings.
Read the full abstract ▾
Robotics
Spacetime Optimal-Transport Attention for Visuo-Haptic Imitation Learning of Contact-Rich Manipulation
Yue Feng, Weicheng Huang and I-Ming Chen
Contact-rich manipulation tasks such as tight-clearance insertion, connector mating, polishing, and surface-conforming wiping remain difficult for data-driven controllers because they couple discontinuous contact dynamics, partial observability, and strict safety constraints. No single sensing modality suffices:
vision supplies global context before contact, force/torque (F/T) feedback governs interaction after contact, and proprioceptive pose provides a consistent kinematic backbone. Most prior imitation-learning policies for contact-rich tasks operate on uni- or bi-modal signals, and the few that fuse three modalities typically adopt off-the-shelf
attention modules with no explicit prior on how
attention mass should be distributed across task-relevant regions. We present Spacetime Optimal-Transport
Attention (SO-TA), a tri-modal fusion backbone that replaces softmax-normalized patch
attention by an
entropy-regularized Optimal Transport (OT)
alignment between force-pose-derived sub-queries and visual patches. Explicit marginal constraints act as a structured inductive bias for contact-rich tasks, encouraging conditioning-aware spatial selection that is stable across illumination, distractors, and partial occlusion. SO-TA is paired with a
diffusion-based sequence policy mapping observation windows to pose-action chunks. We evaluate SO-TA on three real-
robot tasks: tight peg-in-hole assembly, BCM wiring-connector insertion, and curved-surface mark erasing. With ~200 rollouts per condition, SO-TA reaches 100% success on tight peg-in-hole versus 93% for cross-
attention at matched capacity, and retains 82.5% success under illumination, distractor, and partial-occlusion perturbations where a concatenation baseline drops to 43.5%. OT-derived patch heatmaps and leave-one-out modality-influence ratios provide interpretable, phase-dependent diagnostics.
Read the full abstract ▾
Robotics · Cross-listed from cs.HC
Multi-Week, In-Class Deployments of Telepresence Robots With Four Homebound K-12 Students: Benefits, Challenges, and Recommendations
Matthew Rueben, Rhianna Lee, Thomas R. Groechel et al. (+4 more)
Missing significant amounts of school during K-12 education is known to put students' cognitive and social development at risk. Alternatives such as home instruction and online learning are common, but lack sufficient interaction with peers and teachers in the classroom. Mobile remote presence systems, or telepresence
robots, are promising for homebound students because they provide embodiment and mobility in addition to the real-time participation offered by
video conferencing technologies. Research is needed, however, for telepresence
robots to meet the complex needs of homebound students participating remotely in the K-12 classroom context. We present findings from four multi-week deployments with homebound K-12 students attending classes via telepresence
robots. The homebound students' experiences were documented in a total of 15 interviews and analyzed qualitatively as case studies. The homebound student participants and their deployment contexts differed from one another along multiple dimensions, and while some benefits of mobile remote attendance were enjoyed by all participants, each participant also experienced unique benefits. Some challenges with hearing, seeing, and moving the
robot around the classroom warranted improvements to the design of the telepresence system. Other challenges suggested priorities for managing a classroom deployment, such as ensuring that the remote student is included in classroom activities, accountable to the teacher, and treated with respect by classmates. Based on insights from the study, we make recommendations for real-world deployment procedures in similar contexts.
Read the full abstract ▾
All authors Matthew Rueben, Rhianna Lee, Thomas R. Groechel, Hengzhi Chen, Haemi Lee, Gisele Ragusa, Maja J. Matarić
Robotics
Scalable Multi-robot Motion Planning via Hierarchical Subproblem Expansion and Workspace Decomposition Refinement
Isaac Ngui, Courtney McBeth, James D. Motes et al. (+2 more)
A fundamental challenge in multi-
robot motion planning is achieving sufficient coordination to avoid inter-
robot conflicts without incurring the large computational expense of searching the joint configuration space of the
robot group. In this work, we present a method for multiple mobile
robot motion planning that achieves an improvement in planning time up to an order of magnitude by leveraging the insight that we can use discrete search over a workspace decomposition to provide coordination between
robots during planning. While prior work uses workspace
topology to inform when coordination between
robots is needed and then composes
robots into their joint configuration space, we take a step further by iteratively refining our workspace representation to allow our planner to search smaller, decoupled configuration spaces.
Read the full abstract ▾
All authors Isaac Ngui, Courtney McBeth, James D. Motes, Marco Morales, Nancy M. Amato
Robotics
VBT-MPC: Vision-Based Tactile MPC for Contour Following
Edison Velasco-Sanchez, Luis F. Recalde, Guanrui Li et al. (+1 more)
Tactile sensing plays a key role in robotic manipulation, particularly in tasks like surface inspection. Successful execution requires maintaining contact while accurately tracking object contours. In this work, we propose a
Vision-Based Tactile Model Predictive Control (VBT-MPC) framework for robotic contour following using a
Vision-Based Tactile Sensor (VBTS) mounted in an eye-in-hand configuration. The proposed controller operates directly in contour features space, thereby avoiding the need for separate pose-estimation modules or complex force-control architectures. We further compare our VBT-MPC with visual-servoing strategies adapted to tactile features, and evaluate contour tracking on objects with diverse geometries and materials in both simulation and real-world experiments.
Read the full abstract ▾
All authors Edison Velasco-Sanchez, Luis F. Recalde, Guanrui Li, Pablo Gil
Robotics · Cross-listed from cs.CV
STELLAR: Scaling 3D Perception Large Models for Autonomous Driving
Yingwei Li, Xin Huang, Yang Liu et al. (+13 more)
Model scaling has demonstrated remarkable success through large-scale training on diverse
datasets. It remains an open question whether the same paradigm would apply to autonomous driving perception systems due to unique challenges, such as fusing heterogeneous sensor data and the need for sophisticated 3D spatial understanding. To bridge this gap, we present a comprehensive study on systematically analyzing the impact of scale on these systems. We develop our
STELLAR model based on Sparse Window
Transformer, by extending the input modalities to include
LiDAR, radar, camera, and map prior. We train the model on a large-scale
dataset of 50 million driving examples with up to 500 million parameters. Our large-scale experiments reveal empirical scaling trends that connect model performance to model size, data, and compute. The resulting model establishes a new state-of-the-art on the Waymo Open
Dataset challenge, outperforming prior arts by a large margin. Our work demonstrates that large-scale training is a highly promising path for advancing the capabilities of perception models for autonomous driving.
Read the full abstract ▾
All authors Yingwei Li, Xin Huang, Yang Liu, Yang Fu, Alex Zihao Zhu, Chen Song, Junwen Yao, Anant Subramanian, Hao Xiang, Weijing Shi, Yuliang Zou, Tom Hoddes, Zhaoqi Leng, Govind Thattai, Dragomir Anguelov, Mingxing Tan
Robotics
SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Framework
Tianshu Wu, Xiangqi Kong, Yue Chen et al. (+5 more)
Building
humanoid robots capable of generalizable whole-body loco-manipulation in the real world remains a fundamental challenge. Existing methods either rely on laborious task-specific reward engineering, rigidly replay reference motions that fail to generalize, or depend on costly teleoperation that limits scalability. While human
videos capture diverse human behaviors, motion priors inferred from them are inherently imperfect, suffering from occlusion, contact artifacts, and retargeting errors that render them unsuitable for direct policy learning. To address this, we present SUGAR, a scalable data-driven framework that converts diverse human
videos into deployable
humanoid loco-manipulation skills, without any task-specific reward engineering or reference-motion conditioning at inference. SUGAR proceeds in three stages. First, a fully automated pipeline extracts kinematic interaction priors including human-object motion trajectories and contact labels from unstructured human
videos. Second, a privileged physics-based refiner uses a unified mimic reward and progressive state pool to transform imperfect priors into physically feasible, high-fidelity skills. Third, refined skills are distilled into a hierarchical autonomous policy consisting of a command generator and a command tracker. We evaluate SUGAR on six representative loco-manipulation tasks in simulation and real-world
humanoid hardware. Our method substantially outperforms reference-tracking baselines, and performance scales clearly with the amount of human
video data. It also achieves zero-shot real-world transfer with reliable closed-loop execution, autonomous failure recovery, and stable long-horizon performance under external perturbations. Project Page: https://tianshuwu.github.io/sugar-
humanoid/
Read the full abstract ▾
All authors Tianshu Wu, Xiangqi Kong, Yue Chen, Qize Yu, Hang Ye, Jia Li, Yizhou Wang, Hao Dong
Robotics
Proximal State Nudging: Reducing Skill Atrophy from AI Assistance
Megha Srivastava, Jonathan Ouyang, Eric Zhou et al. (+6 more)
Skill atrophy, the gradual decline of human capability under AI assistance, poses a safety risk in shared-control of semi-autonomous systems, where operators may be unable to distinguish their own inputs from autonomous corrections. We propose Proximal State Nudging (PSN), a shared autonomy
algorithm that jointly optimizes for skill development and task performance by nudging users toward states estimated to be most learnable. We first show that PSN outperforms existing shared autonomy baselines in balancing student improvement in unassisted reward with overall shared performance, using simulated students in the classic LunarLander environment. We then present, to the best of our knowledge, the first human subject studies of a planner incorporating learning-compatible shared autonomy: across two driving tasks in the CARLA simulator (High Performance Racing and Parallel Parking, n = 60), PSN produces up to 7x larger gains in unassisted skill than standard blended shared autonomy, while incurring 50% fewer collisions than unassisted self-practice.
Read the full abstract ▾
All authors Megha Srivastava, Jonathan Ouyang, Eric Zhou, Andrew Silva, Emily Sumner, Dorsa Sadigh, Yuchen Cui, Deepak Gopinath, Guy Rosman
Robotics
Terrestrial Soft Mobile Robots: A Review
Dimuthu D. K. Arachchige
Soft mobile
robots have emerged as a promising area of research with potential applications in various disciplines including but not limited to search-and-rescue, service, surveillance, explorations, and manufacturing. In this article, we provide a comprehensive review of the current state of soft mobile
robot research, focusing on wheelless terrestrial locomotive systems. We include past and present developments in locomotion strategies, actuation methods, modeling approaches, and control systems. Further, we identify key research challenges that must be overcome to enable the widespread adoption of soft mobile
robots in various applications. Overall, this article provides a valuable resource for researchers and practitioners interested in the field of soft mobile
robots and soft robotics.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
Mechanisms of Misgeneralization in Physical Sequence Modeling
Kento Nishi, Raphael Tang, Karun Kumar et al. (+2 more)
Generative sequence models are often trained to plan motion in physical domains, from robotics to mechanical simulations. When constructing a
dataset to train such a model, engineers may curate demonstrations to specify how trajectories should be distributed over a physical quantity like travel distance or mechanical energy. For example, a roboticist building a maze
navigation agent might choose demonstrations whose travel distances cover a fixed range uniformly, hoping to constrain the
agent's expected power usage. We find that standard deep learning can violate this intent: each generated trajectory can seem plausible on its own, but the aggregate distribution over the physical quantity is wrong. We call this failure physical misgeneralization, and develop an account of its mechanism. Using controlled synthetic tasks, we show that physical misgeneralization arises when local errors typical of the model class propagate through the physical measurement to shift the recovered distribution. We estimate these errors with a data deviation kernel, and we use it to predict which physical quantities gain or lose mass in both our synthetic and more applied maze
navigation and double-pendulum motion tasks. Finally, our mechanistic interpretation helps identify which mitigation strategies are structurally promising, and we use it to propose a kernel-informed intervention.
Read the full abstract ▾
All authors Kento Nishi, Raphael Tang, Karun Kumar, Core Francisco Park, Hidenori Tanaka
Robotics
Hamilton–Jacobi Reachability for Spacecraft Collision Avoidance
Larry Hui, Jordan Kam, William Su et al. (+1 more)
This article presents a Hamilton–Jacobi (HJ) reachability framework for a two–satellite collision avoidance problem operating in the same circular orbit, where relative motion is modeled in the radial–tangential–normal (RTN) frame using planar Hill–Clohessy–Wiltshire (HCW) dynamics. We define the target state space as unsafe relative configurations in the orbit plane corresponding to minimum separation requirements consistent with Federal Communications Commission (FCC) orbital standards. The interaction between spacecraft is formulated as a zero–sum differential game, where Player 1 is the controlled satellite and Player 2 is modeled as a bounded
adversarial disturbance with unknown intent. We present the HJ formulation and compute backward reachable sets that characterize relative states from which collision cannot be avoided under worst-case disturbances, while states outside this set admit provably collision-free trajectories. These reachable sets are integrated with supervisory hybrid control logic to determine when evasive maneuvers must be initiated, enabling mathematically grounded safety guarantees for scalability.
Read the full abstract ▾
All authors Larry Hui, Jordan Kam, William Su, Jianshu Zhou
Robotics
Topology-Optimized Pneumatic Soft Actuator: Design and Experimental Validation
Sumit Mehta and Konstantinos Poulios
This paper demonstrates the computational design of soft elastomeric pneumatic actuators using nonlinear
topology optimization. An existing density- and porohyperelasticity-based
topology optimization framework was extended from 2D to 3D and used to generate two manufacturable actuator designs, which were then studied numerically and experimentally. For both designs, the objective was to maximize the bending response for a prescribed actuation pressure under two different allowable strain limits. A key advantage of the employed
topology optimization framework is that it can consistently, during the optimization, account for the very large deformations induced upon pressurization. The two optimized 3D designs were fabricated using stereolithography and experimentally tested to validate their performance.
Read the full abstract ▾
Robotics · Cross-listed from cs.AI
Probing an Embodied LLM: When Higher Observation Fidelity Hurts Problem Solving
Oussama Zenkri and Oliver Brock
Large
Language Models (LLMs) are increasingly proposed as cognitive components for robotic systems, yet their opaque decision processes make it difficult to explain success or failure in closed-loop
embodied tasks. Following an empirical AI methodology, we study an
embodied LLM
agent behaviorally by varying the available information and measuring the resulting changes in behavior. Using the Lockbox, a sequential mechanical puzzle with hidden interdependencies, we evaluate LLMs across RGB, RGB-D, and ground-truth symbolic observations in a physical robotic setup and use simulation to probe the resulting behavior. Counterintuitively,
agents perform best under raw RGB input and worst under perfect ground-truth observations. In simulation, we probe this effect by randomly flipping perceived action outcomes and find that moderate noise improves performance, peaking at a 40% flip probability with a 2.85-fold success rate increase over the noise-free baseline. Further analysis links this gain to a reduction in repetitive action loops. These findings suggest that success rates alone are insufficient for evaluating LLMs, as measured performance may reflect the interaction between perceptual errors and
reasoning failures rather than robust problem solving.
Read the full abstract ▾
Robotics · Cross-listed from cs.SE
Towards LLM-Assisted Architecture Recovery for Real-World ROS~2 Systems: An Agent-Based Multi-Level Approach to Hierarchical Structural Architecture Reconstruction
Dominique Briechle, Raj Chanchad, Tobias Geger et al. (+5 more)
Explicit software architecture models are essential artifacts for communicating, analyzing, and evolving complex software-intensive systems. In ROS~2-based robotic systems, however, structural (de-)composition and integration semantics are often only implicitly encoded across distributed artifacts such as source code and launch files, making recovery of hierarchical architecture particularly difficult. Existing approaches mainly focus on node-level entities and communication wiring, while providing limited support for recovering hierarchical structural (de-)composition across multiple abstraction levels. In this paper, we extend our previously proposed blueprint-guided LLM-assisted architecture recovery pipeline for ROS~2 systems through two major enhancements: (1) refined prompting to improve the consistency and controllability of architecture synthesis, and (2) a staged recovery strategy based on multi-level intermediate architectural representations that incorporate the atomic ROS node list and launch file dependencies, thereby enabling structurally constrained reconstruction across multiple abstraction levels. The approach is evaluated on a real-world automated product disassembly system based on cooperative robotic arms and heterogeneous ROS~2 artifacts. Compared to our previous work, the considered case study exhibits substantially higher integration complexity and richer functionality. The results demonstrate improved structural consistency, scalability, and robustness of architecture recovery, while also revealing remaining challenges related to dynamic integration semantics in large-scale ROS~2 systems.
Read the full abstract ▾
All authors Dominique Briechle, Raj Chanchad, Tobias Geger, Ruidi He, Dhruv Jajadiya, Dhruv Kapadiya, Andreas Rausch, Meng Zhang
Robotics
Minimalist Visual Inertial Odometry
Francesco Pasti, Jeremy Klotz, Nicola Bellotto et al. (+1 more)
Visual-Inertial Odometry (VIO), which is critical to mobile
robot navigation, uses cameras with a large number of pixels. Capturing and processing camera images requires significant resources. This work presents a minimalist approach to planar odometry, showing that just four visual sensors (pixels) and an IMU provide robust motion estimation for differential-drive
robots. Our key insight is that four downward-facing photodiodes that sense the world through optical Gabor masks produce signals that encode speed. Based on this, we jointly optimize the mask parameters alongside a Temporal Convolutional Network (TCN) using a physically-grounded simulator. The resulting model decodes speed from the four photodiode measurements. Pairing these estimates with an IMU's angular speed yields a continuous planar trajectory. We validate our approach with a prototype sensor mounted on a differential drive
robot. Across diverse indoor and outdoor terrains, our system closely tracks the reference trajectories without any real-world fine-tuning. Our work shows that minimalist sensing enables efficient and accurate planar odometry.
Read the full abstract ▾
All authors Francesco Pasti, Jeremy Klotz, Nicola Bellotto, Shree K. Nayar
Robotics
Beyond Binary Success: A Diagnostic Meta-Evaluation Framework for Fine-Grained Manipulation
He-Yang Xu, Pengyuan Zhang, Zongyuan Ge et al. (+5 more)
Fine-grained manipulation marks a regime where global scene context no longer suffices, and success hinges on the tight coupling of local attribute grounding, high-fidelity spatial perception, and constraint-respecting motor execution. However, current
embodied AI
benchmarks collapse these capacities into binary success rates, systematically inflating reported capabilities by up to 70% and masking the architectural bottlenecks that impede real-world deployment. We introduce MetaFine, a diagnostic meta-evaluation framework that disentangles manipulation competency along three axes: understanding, perception, and controlled behavior. Built on a compositional task graph, MetaFine absorbs heterogeneous external
benchmarks and reconstructs them into diagnostic scenarios of varying complexity under a unified protocol. Evaluating state-of-the-art
vision-
language-action (VLA) models through this lens exposes severe dimension-specific failures invisible to conventional metrics. Through targeted causal intervention, we identify the visual encoder's ability to preserve local spatial structure as a key bottleneck for fine-grained precision: improving it directly unlocks previously inaccessible manipulation capabilities without modifying downstream policies. MetaFine further supports hybrid real-sim validation, using limited paired real-world rollouts to calibrate scalable simulation-based estimates for more stable physical benchmarking. By shifting evaluation from ranking to diagnosis, MetaFine turns benchmarking into an actionable compass for repairing the layered capacities underlying genuine physical dexterity. The MetaFine framework,
benchmarks, and supporting resources will be publicly released at our project page: https://metafine.github.io/.
Read the full abstract ▾
All authors He-Yang Xu, Pengyuan Zhang, Zongyuan Ge, Xiaoshuai Hao, Serge Belongie, Xin Geng, Yuxin Peng, Xiu-Shen Wei
Robotics
CEER: Compliant End-Effector and Root Control as a Unified Interface for Hierarchical Humanoid Loco-Manipulation
Xinyuan Luo, Xingrui Chen, Xunjian Yin et al. (+6 more)
Humanoid robots have achieved impressive locomotion performance, yet contact-rich and long-horizon manipulation remains a major bottleneck. Manipulation is inherently contact-rich and demands compliant whole-body control for stable interaction, while its diversity and long-horizon nature favor modular, planner-compatible interfaces over joint-space tracking. We propose CEER, a compliant end-effector-root (EE-root) control abstraction for modular
humanoid loco-manipulation within a hierarchical planning framework. CEER enables compliance-aware whole-body control in an interpretable task space defined by root motion commands and end-effector pose targets, and supports plug-and-play integration with heterogeneous high-level planners. A teacher-student framework is adopted to distill a general motion-tracking controller into a low-level policy that consumes only EE-root commands. We further construct a hierarchical system that integrates heterogeneous planners and task modules through the EE-root interface, enabling diverse manipulation tasks without retraining the underlying whole-body policy. Experiments in simulation and on hardware demonstrate 3.3 cm end-effector tracking accuracy with substantially reduced jerk compared to baselines, stable contact-rich manipulation under teleoperation, and up to 70% success in simulated single-object loco-manipulation tasks within a room-scale environment. These results indicate that compliant EE-root control provides a practical abstraction for
humanoid loco-manipulation, enabling modular and scalable integration of diverse skills.
Read the full abstract ▾
All authors Xinyuan Luo, Xingrui Chen, Xunjian Yin, Hongxuan Wu, Boxi Xia, Zhuoqun Chen, Jinzhou Li, Boyuan Chen, Xianyi Cheng
Robotics
TravExplorer: Cross-Floor Embodied Exploration via Traversability-Aware 3-D Planning
Han Zheng, Zhe Chen, Yudong Huang et al. (+4 more)
Zero-shot Object
Navigation (ZSON) has shown promise for open-vocabulary target search in unseen environments, yet most existing systems remain tied to planar representations and single-floor assumptions. These assumptions become inadequate in real buildings, where
navigation involves floors, stairs, landings, and vertically overlapping spaces. This article presents TravExplorer, a cross-floor
embodied exploration framework that couples zero-shot semantic guidance with traversability-aware 3-D planning. TravExplorer maintains a unified volumetric map that distinguishes occupied structures from
robot-reachable support surfaces and extracts traversable frontiers from connected support surfaces, including floors, stairs, and landings. A FOV-aware active perception strategy further resolves incomplete observations during cross-floor traversal. To reduce semantic-
reasoning latency, a lightweight guidance module aligns a probabilistic instance map from online open-vocabulary
segmentation with a spatial value map from fast image-to-text matching. Based on these
geometric and semantic
memories, a hierarchical planner performs target-aware frontier touring over object hypotheses, traversable frontiers, and stair landmarks, and generates executable cross-floor motions through foothold-guided 3-D search and vertically constrained local trajectory optimization. Experiments over 4,195 simulated episodes on HM3D and MP3D demonstrate consistent advantages over representative ObjectNav baselines. Fifty real-world trials on a Unitree Go2 further validate open-vocabulary target search across single-floor and cross-floor indoor environments without prior maps or human intervention. The code will be released at https://github.com/wuyi2121/TravExplorer.
Read the full abstract ▾
All authors Han Zheng, Zhe Chen, Yudong Huang, Haoran Liu, Jinghao Wang, Ming Yang, Tong Qin
Robotics · Cross-listed from cs.CV
World-Ego Modeling for Embodied Video Generation in Long-Horizon Navigation-Manipulation Tasks
Zuyao Lin, Jianhui Zhang, Peidong Jia et al. (+4 more)
Embodied video world models typically capture both scene
evolution and the
robot's behavior, which we refer to as the world and the ego, respectively. The world and the ego exhibit different underlying dynamics: world prediction relies primarily on visual history and emphasizes scene stability, whereas ego prediction relies more strongly on the current instruction and emphasizes accurate instruction following. Modeling both components within a single generation stream can entangle these different dependencies, making it difficult to specialize the prediction of either component. Consequently, it becomes difficult to simultaneously maintain scene consistency and accurate instruction following, particularly in long-horizon
navigation-manipulation tasks. In this paper, we propose to decompose an
embodied video into the world and the ego and disentangle their generation processes. Specifically, we define the world as the background and currently unmanipulated objects, and the ego as the
robot and currently manipulated objects. Based on this definition, we develop the World-Ego Model (WEM), which combines a
vision-
language state predictor using role-conditioned
attention (RCA) and asymmetric query budgets with a semantic-routed mixture-of-experts (SR-MoE)
diffusion generator. To enable rigorous evaluation, we further construct HTEWorld, a
dataset and
benchmark for long-horizon
embodied video generation with hybrid
navigation-manipulation tasks, providing 125K training
video clips comprising over 4.5M frames with fine-grained instructions, together with 300 multi-turn evaluation trajectories covering over 2K instructions. Extensive experiments show that WEM achieves state-of-the-art performance on HTEWorld while remaining competitive on existing manipulation-oriented evaluations.
Read the full abstract ▾
All authors Zuyao Lin, Jianhui Zhang, Peidong Jia, Xiaoguang Zhao, Shanghang Zhang, Jingdong Wang, Xingyu Chen
Robotics · Cross-listed from cs.AI
Robotics-Inspired Guardrails for Foundation Models in Socially Sensitive Domains
Rebecca Ramnauth, Drazen Brscic and Brian Scassellati
Foundation models are increasingly deployed in socially sensitive domains such as education, mental health, and caregiving, where failures are often cumulative and context-dependent. Existing guardrail approaches – ranging from training-time
alignment to prompting, decoding constraints, and post-hoc moderation – primarily provide empirical risk reduction rather than enforceable behavioral guarantees, and largely treat safety as a property of individual outputs rather than interaction trajectories. We reframe guardrails as a problem of runtime behavioral control over interaction trajectories, drawing on robotics to introduce formal constructs for constraint enforcement in uncertain, closed-loop systems. We instantiate these ideas in the Grounded Observer framework and apply it across three real-world deployments: small talk, in-home autism therapy, and behavioral de-escalation in schools. Across settings, the framework enables runtime interventions that mitigate drift into undesirable interaction regimes while adapting to diverse social contexts. We discuss extensions to the framework and propose research directions toward stronger guarantees.
Read the full abstract ▾
Robotics
RoHIL: Robust Human-in-the-Loop Robotic Reinforcement Learning Against Illumination Variations
Shuoqin Zhang, Yixin Xiong, Xiru Gao et al. (+4 more)
Human-in-the-loop reinforcement learning systems achieve near-perfect success on the workstation where they are trained, but collapse when the same
robot is moved to a workstation a few meters away due to shifts in the visual input distribution caused by new lamp positions and window light. Re-collecting demonstrations and re-running HIL on every workstation is incompatible with deployment, and naively fine-tuning on shifted-light data triggers catastrophic forgetting of the source workstation. To close this cross-domain gap, we present RoHIL, an offline fine-tuning framework that uses no extra real-
robot interaction. RoHIL combines (i) a world-model-based image relighter that re-synthesises the visual stream of source-workstation trajectories under multiple virtual HDRI environments, leaving actions and rewards real; (ii) Illumination-Retention Replay (IRR), a data-level anti-forgetting mechanism that interleaves relit adaptation transitions with original-light retention transitions to preserve source-workstation Bellman coverage; and (iii) an anchored Bellman-actor regulariser that constrains representation and policy drift from the original source-workstation policy. Across four real-
robot manipulation tasks under significant cross-workstation illumination variations, RoHIL substantially improves shifted-light performance where standard HIL-RL collapses, while preserving source-workstation performance, eliminating the need to re-collect data and retrain for every new workstation and environment. Project page: https://anonymous4365.github.io/RoHIL/
Read the full abstract ▾
All authors Shuoqin Zhang, Yixin Xiong, Xiru Gao, Kai Liu, Ke Wang, Xichuan Zhou, Zhe Hu
Robotics
Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learning
Dongjie Yu, Kun Lei, Zhennan Jiang et al. (+2 more)
Pretrained imitation policies have become a strong foundation for
robot manipulation, but they often require online improvement to overcome execution errors, limited
dataset coverage, and deployment mismatch. A central question is therefore how reinforcement learning (RL) should adapt policies after offline pretraining. Existing lightweight methods commonly apply residual corrections directly in action space, but this often leads to noisy and poorly structured exploration. In this work, we propose Z-Perturbation Reinforcement Learning (ZPRL), an approach that steers pretrained policies through a compact bottleneck latent rather than through policy weights or output actions. During offline training, we augment the policy with a plug-and-play variational information bottleneck (VIB) module to extract a task-relevant latent interface from observation
embeddings. During online finetuning, the base policy is frozen and RL learns only a residual perturbation on this latent, whose decoded representation conditions the frozen action generator. We instantiate ZPRL on flow-matching policies and evaluate it on eight simulation tasks and four real-world tasks. Across diverse manipulation settings, ZPRL improves both sample efficiency and final performance over strong post-training baselines. In the real world, ZPRL improves the average success rate on four tasks by 33.7% over imitation base policies while producing smoother exploration behaviors than an action residual counterpart. These results suggest that a compact, task-aligned bottleneck latent provides an effective interface for online RL adaptation. More
videos can be found at https://manutdmoon.github.io/ZPRL/.
Read the full abstract ▾
All authors Dongjie Yu, Kun Lei, Zhennan Jiang, Jia Pan, Huazhe Xu
Robotics · Cross-listed from cs.DC
DAG-Based QoS-Aware Dynamic Task Placement for Networked Multi-Stage Control Pipelines
Thien Tran, Jonathan Kua, Thuong Hoang et al. (+3 more)
Current Physical AI (PAI) relies heavily on closed-loop visual-servoing pipelines, whose perception and planning stages may become computationally intensive onboard due to complex models embedded on
robots. In practice, offloading the perception task to on-site edges statically is inappropriate for latency-sensitive, precise industrial settings over a standardized industrial network. This emphasizes the importance of Control-Communication-Computing (3C) co-design in industrial automation: monolithic local execution saturates AI-accelerated machine and
robot hardware, while static edge offloading exposes the control loop to network jitter. Existing adaptive task placement (ATP) controllers can partially address the gap by relocating a single pipeline stage on binary threshold rules, without a multi-stage model and an explicit cost on placement switching. In this paper, we propose a directed acyclic graph (DAG) based quality-of-service (QoS)-aware dynamic task placement (DTP) framework for sensing-perception-planning-control pipelines in networked robotics. This pipeline is formalized as a DAG with task-level and node-level attributes for compute cost, communication delay, and feasible placement sets; over a small interpretable candidate set (fully local, static offload, hybrid), a window-based cost function combines tail end-to-end latency, deadline violation rate, hardware utilization, and a Hamming-distance switching penalty, and a DTP
algorithm with hysteresis and a minimum dwell-time bounds placement chatter. Our work presents the theoretical framework, a structured qualitative analysis, and a two-phase simulation plus hardware-in-the-loop validation roadmap.
Read the full abstract ▾
All authors Thien Tran, Jonathan Kua, Thuong Hoang, Minh Tran, Yuemin Ding, Jiong Jin