Robotics
Self-supervised Geometry Reasoning for LiDAR Simultaneous Localization and Mapping
Jiwoo Kim, Jinwoo Lee, Woojae Shin et al. (+2 more)
LiDAR simultaneous localization and mapping (SLAM) relies on local
geometric quantities such as covariances, correspondences, and surface structures. However, most existing pipelines rely on hand-crafted estimates of local geometry and use them as fixed inputs to
LiDAR SLAM, which can make the estimated local geometry noisy and unstable in sparse regions of a point cloud or when using low-resolution
LiDAR. To address this issue, this paper introduces a self-supervised framework that learns an explicit symbolic representation of local geometry and uses it to improve
LiDAR SLAM recursively. Specifically, each point is represented as a Gaussian distribution, allowing local geometry to be described by a covariance. Without dense geometry labels or ground-truth poses, the framework learns by maximizing the likelihood of local geometry, with self-supervision derived from consistency relations over symbolic
geometric representations, including predicted covariances, correspondences, and trajectory from SLAM. The learned geometry is then fed back into
LiDAR SLAM, forming a reciprocal loop in which improved geometry enhances localization and mapping, and improved localization provides cleaner supervision for subsequent geometry
reasoning. This framework is backend-agnostic and can be plugged into existing
LiDAR SLAM pipelines without architectural changes. Experiments on KITTI under varying
LiDAR resolutions show that the proposed method improves both odometry and global registration.
Read the full abstract ▾
All authors Jiwoo Kim, Jinwoo Lee, Woojae Shin, Giseop Kim, Hyondong Oh
Robotics
AERIS: Aerial-Edge Role-Driven Intelligence at Runtime via Orchestrated Language-Model Swarm
Jiabin Lou, Haopeng Wang, Xinyu Liu et al. (+3 more)
Integrating large
language models into robotic systems holds promise for enhancing autonomy, yet practical deployment remains constrained by strict heartbeat-constrained scheduling and limited computational power. We propose AERIS: an edge deployment framework for aerial platforms. It organizes dedicated small
language models combined with lightweight perception and control modules into roles that can be instantiated at runtime, and dynamically rebinds them across different executors as resources change, thereby pushing intelligent capabilities to the edge. AERIS achieves long-horizon instruction decomposition through an
attention-subgoal
alignment mechanism, which involves annotating the currently active instruction step in messages, thereby progressively approaching long-term objectives. We evaluate AERIS on a high-fidelity
UAV Vision-and-
Language Navigation benchmark. Under a heartbeat-timed execution mechanism, AERIS maintains a stable perception-decision-control loop between a low-frequency planner and a high-frequency controller, supporting real-time closed-loop operation. We further validate its deployability through two real-world experiments focused on planning and fast response. A demonstration
video is provided in the supplementary materials.
Read the full abstract ▾
All authors Jiabin Lou, Haopeng Wang, Xinyu Liu, Yu Zhang, Rongye Shi, Wenjun Wu
Robotics
SA-VLA: State-aware tokenizer for improving Vision-Language-Action Models' performance
Tengyue Jiang, Chunpu Xu, Jiayue Kang et al. (+1 more)
Discrete action tokenization provides a compact interface for autoregressive VLA policies, but accurately recovering continuous
robot actions from discrete codes remains challenging. Existing tokenizers typically map each discrete code to a fixed continuous action prototype, ignoring the
robot's current proprioceptive state. This limitation is particularly pronounced in manipulation, where the same action
token may require different continuous controls under different joint configurations, object poses, and contact conditions. We therefore propose SA-VLA, a state-aware action tokenizer that conditions action decoding on
robot state. We study two state-injection mechanisms for VQ-based action tokenization: cross-
attention between state and action features, and a lightweight state adapter that predicts action-wise modulation factors for state-conditioned action modulation and reconstruction. The adapter formulation expands the effective support of a finite codebook by allowing each discrete
token to represent a family of state-dependent continuous actions, while preserving the efficiency and compatibility of discrete action modeling. Integrated into an LLM-based VLA policy, SA-VLA supports both autoregressive and parallel action-
token decoding with minimal changes to the model interface. On 12 RoboTwin manipulation tasks, SA-VLA improves the average success rate from 0.29 to 0.56 over the strongest tokenizer baseline. In zero-shot sim-to-real experiments on three real-world tasks, it further improves average success from 0.15 to 0.33 over the strongest tokenizer baseline. These results demonstrate that state-conditioned action decoding is a simple and effective mechanism for reducing the compression gap in discrete VLA policies.
Read the full abstract ▾
All authors Tengyue Jiang, Chunpu Xu, Jiayue Kang, Yao Mu
Robotics
Automating the Design of Embodied Agent Architectures
Jian Zhou, Sihao Lin, Jin Li et al. (+3 more)
Embodied agents are typically built as hand-designed compositions of perception,
memory, planning, and action modules. This modularity exposes a large architectural design space, but current systems still rely on researcher intuition to choose where information is stored, how observations are processed, and how model calls are connected.
Agent Architecture Search (AAS) automates such design for text-domain
agents, but has not been systematically evaluated on perceptual
embodied agents. We study this transfer. To make
embodied architectures searchable, we introduce AgentCanvas on the executor side, representing
embodied agents as editable typed graphs with simulator execution and episode-level logs. On the optimizer side, we lift AAS from LLM-call loops to tool-using coding-
agent sessions, adapting ADAS and AFlow and introducing Knowledge Distill Loop (KDLoop). Across four
embodied executors (MapGPT, SmartWay, ExploreEQA, and VoxPoser), search discovers improvements that persist on episodes never queried during search. KDLoop and AFlow independently discover elevation-tolerant observation processing. KDLoop's implementation also improves success rates over the baseline when transferred to real
robots. These improvements are not uniform: some gains remain uncertain or reverse on held-out episodes, and an apparent gain exploits evaluator-side information. The experiments further show how rollout noise, local edit basins, and incomplete use of episode-level evidence constrain search. Together, these results establish the potential of automated architecture search for
embodied agents and characterize the challenges of applying it in simulated and real-world environments. Code is available at https://github.com/jianzhou0420/AgentCanvas.
Read the full abstract ▾
All authors Jian Zhou, Sihao Lin, Jin Li, Shuai Fu, Gengze Zhou, Qi Wu
Robotics
TacEvo: Self-Evolving Architecture Discovery for Robotic Tactile Perception via LLM-Driven Quality-Diversity Search
Mohammed AbuSadeh, Lan Wei and Dandan Zhang
Vision-based tactile sensing converts contact-induced surface deformation into images, enabling
robots to infer contact forces and fine surface textures that are not accessible through conventional
vision alone. However, tactile images are sensor- and physics-specific, so effective architectures often require expert intuition and extensive manual iteration. Existing neural architecture search (NAS) pipelines can reduce this burden, but they are often computationally expensive and restricted to hand-designed search spaces, which limits architectural novelty and diversity. We introduce TacEvo, a self-evolving architecture discovery framework that improves network designs from downstream feedback. TacEvo uses an LLM to generate code-level mutations and crossovers, and a MAP-Elites quality-diversity loop that preserves diverse elite architectures while preferentially reusing
prompts that consistently yield improvements. Exploration is guided by two behavioural descriptors, Architectural Diversity and Efficiency Ratio, which encourage coverage across structural variations and compute-size trade-offs. On ViTacTip force regression and grating classification, TacEvo achieves high autonomous generation reliability (96.0%/94.5% trainable) and improves best validation fitness over 20 generations by 56.1%/96.1%. In a 20-seed post-search high-fidelity evaluation, TacEvo matches the expert baseline on force prediction and outperforms it on fine-grained grating classification. These results suggest that LLM-driven self-evolving search constitutes a practical paradigm for AI-assisted scientific discovery in specialised robotic sensing.
Read the full abstract ▾
Robotics
SIR: Structured Image Representations for Explainable Robot Learning
Paul Mattes, Jan Schwab, Jens Bosch et al. (+5 more)
Existing
robot policies based on learned visual
embeddings lack explicit structure and are sensitive to visual distractions. Thus, the representations that drive their behaviour are often opaque, making their decision-making process difficult to interpret. To address this, we introduce Structured Image Representations (SIR), a method that leverages Scene Graphs (SGs) as an intermediate representation for
robot policy learning. Our approach first constructs a fully connected graph, using image-derived features as initial node representations. Then, a module learns to sparsify this graph end-to-end, creating a task-relevant sub-graph that is passed to the action generation model. This process makes our model intrinsically explainable. Evaluations on RoboCasa show that our sparse graph policies outperform image-based baselines on average with 19.5% vs 14.81% success rate. Most importantly, we show that the learned sparse graphs are a powerful tool for model analysis. By analysing when the model's sub-graph deviates from human expectation, such as by including distractor nodes or omitting key objects, we successfully uncover
dataset biases, including spurious correlations and positional biases. https://github.com/intuitive-
robots/SIR_Model
Read the full abstract ▾
All authors Paul Mattes, Jan Schwab, Jens Bosch, Nils Blank, Maximilian Xiling Li, Minh-Trung Tang, Moritz Haberland, Rudolf Lioutikov
Robotics · Cross-listed from cs.CV
CylindTrack: Depth-Aware Cylindrical Motion Modeling for Panoramic Multi-Object Tracking
Buyin Deng, Kai Luo, Lingxin Huang et al. (+5 more)
Multi-Object Tracking (MOT) is essential for
embodied perception, where panoramic cameras provide a 360{\deg} field of view to reduce blind spots and keep surrounding targets observable for longer durations. However, panoramic MOT requires motion estimates and association cues consistent with its imaging geometry. In
videos represented in equirectangular projection (ERP), horizontal coordinates wrap at the image seam, while frame-wise relative-depth estimates may fluctuate over time. These inconsistencies can disrupt trajectory prediction and identity association. To address these challenges, we propose CylindTrack, a depth-aware cylindrical tracking-by-detection framework that integrates continuous angular motion modeling with temporally filtered relative-depth cues. Specifically, Depth-Temporal Trajectory Modeling (DTM) promotes instance depth from an isolated frame-wise cue into temporally filtered trajectory-level states. Spherical Spatio-Temporal Consistency Learning (SSTC) combines a Temporal Mixer and Spherical Geometry-aware
Attention to enhance temporal coherence and panoramic
geometric alignment in depth-aware representations. Furthermore, a
Topology-Aware Cylindrical Motion Model (TCMM) lifts horizontal motion into a continuous angular state space, enabling seam-consistent motion prediction and association. By jointly modeling trajectory-level depth consistency and panoramic
topology, CylindTrack improves identity preservation and trajectory continuity in challenging panoramic scenes. Experiments on QuadTrack and JRDB demonstrate improvements in identity-oriented metrics with practical efficiency, achieving HOTA scores of 33.67 and 31.12, IDF1 scores of 40.45 and 34.33, and tracking speeds of 28.56 and 21.34 FPS, respectively. The source code will be released at https://github.com/warriordby/CylindTrack.
Read the full abstract ▾
All authors Buyin Deng, Kai Luo, Lingxin Huang, Xinqi Liu, Fei Cheng, Hang Zheng, Liming Yin, Kailun Yang
Robotics
Jianxin Bi, Qiang Wang, Jayaram Reddy et al. (+4 more)
Tactile sensors are inherently heterogeneous: a model trained on one sensor cannot be directly used on another, which limits learning contact-rich manipulation policies from diverse tactile data at scale. To bridge this gap, we propose the Heterogeneous Tactile
Transformer (HTT), a framework that learns shared tactile representations across heterogeneous sensors. HTT consists of sensor-specific encoders and a shared
transformer trunk, and is pretrained with per-modality masked reconstruction together with cross-modal
alignment between paired sensors. Pretraining uses our novel Heterogeneous Paired Tactile (HPT)
dataset, containing 1.6M synchronized paired frames across four
vision- and array-based tactile sensors. Across distinct tactile perception and real-world manipulation tasks, HTT is shown to learn transferable representations that adapt to new tasks and previously unseen sensors.
Dataset, code, and model checkpoints are released at https://jxbi1010.github.io/htt-gh-page/.
Read the full abstract ▾
All authors Jianxin Bi, Qiang Wang, Jayaram Reddy, Kelvin Lin, Soibkhon Khajikhanov, Ruihan Gao, Harold Soh
Robotics
Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation
Shengqi Xu, Guojin Zhong, Yang Liu et al. (+7 more)
Visuo-Tactile policies leveraging optical tactile sensors have shown great promise in contact-rich manipulation. These sensors achieve high spatial resolution and multi-dimensional force sensing by utilizing an internal camera to monitor the deformation of their elastic gel surface, thereby indirectly inferring tactile cues. Despite their advantages, extracting fine-grained contact states necessary for contact-rich manipulation remains an open challenge. Existing methods typically use either raw images or cumulative motion fields to represent tactile cues. However, both are prone to perception ambiguity. Raw tactile images mainly capture appearance changes, while cumulative motion fields only reflect the aggregate gel deformation. Consequently, distinct fine-grained contact states can exhibit highly similar patterns, making it difficult to explicitly distinguish subtle contact variations. To address this issue, we explore the dynamic priors of tactile motion and discover that the correlation between transient and cumulative motion can explicitly distinguish fine-grained contact states. Based on this insight, we propose a motion-aware tactile representation to facilitate contact-rich manipulation. Beyond tactile representation, effective fusion of tactile and visual modalities is also critical. Most existing fusion methods either directly concatenate features from each modality or train modality-specific networks separately and fuse their outputs. However, these strategies struggle to simultaneously model cross-modal interactions and preserve modality-specific characteristics. In this work, we take advantage of the Mixture-of-
Transformers architecture and propose a unified modality-aware visuo-tactile policy that captures cross-modal complementarity while maintaining modality-specific properties.
Read the full abstract ▾
All authors Shengqi Xu, Guojin Zhong, Yang Liu, Fanjie Wang, Hu Luo, Hanyu Zhou, Weiyao Zhang, Ziyi Ye, Zuxuan Wu, Yu-Gang Jiang
Robotics
WARP: Whole-Body Retargeting for Learning from Offline Human Demonstrations
Zhenyang Chen, Chuizheng Kong, Chuye Zhang et al. (+4 more)
Direct transfer from human demonstration to learnable
robot action is a crucial step towards scalable whole-body mobile manipulation. While human data scales better than mobile teleoperation, it requires overcoming significant embodiment gaps. Existing retargeting methods yield imprecise or inconsistent solutions, causing action multi-modality that prevents supervised policies from reliably converging. We present Whole-body-Aware Retargeting from human Pose (WARP), an offline pipeline that explicitly models embodiment differences to extract precise, unique whole-body actions. WARP leverages a closed-form Shoulder-Elbow-Wrist (SEW)
geometric solver for exact end-effector tracking while preserving whole-body structural intent. Paired with lazy mobile-base control, it extracts accurate, consistent
robot trajectories. Evaluations show WARP provides highly reliable data for open-loop real-world replay. To our knowledge, WARP is the first framework to achieve zero-shot whole-body mobile manipulation directly from offline human demonstrations, eliminating the need for human-in-the-loop teleoperation action data. More details on https://warp-retargeting.github.io/
Read the full abstract ▾
All authors Zhenyang Chen, Chuizheng Kong, Chuye Zhang, Yuanshao Yang, Lawrence Y. Zhu, Shreyas Kousik, Danfei Xu
Robotics
REPAIR-Bench: A Benchmark for Robot Error Perception And Interaction Recovery
Giuliano Pioldi, Yashika Batra, Arman Ibrayeva et al. (+4 more)
Understanding how users perceive and respond to
robot failures is essential for building robust and trustworthy
robot systems. Prior work, however, (i) often treats failures as independent events, (ii) emphasizes binary failure detection, (iii) with rule-based recovery modeling. We present REPAIR-Bench, built on 214 interaction trials from 41 participants, the
benchmark spans four induced failure types and provides synchronized facial action units, head pose,
speech transcripts, and post-interaction affect and recovery reports. The
benchmark spans three novel evaluation tasks that jointly capture the lifecycle of failure in human-
robot interaction (HRI): (i) failure detection over inter-dependent interaction sessions, modeling longitudinal user adaptation across repeated failures; (ii) visual failure-type classification beyond binary success/failure formulations; and (iii) user-centered recovery prediction, inferring users' preferred recovery strategies from interaction context rather than relying on manually designed or rule-based strategies. In baseline experiments, hierarchical recurrent modeling improved failure detection over a single-session model (strict F1: 0.80 vs. 0.68), achieved a failure localization mean signed error of -0.51 s, median absolute error of 2.97 s and, for recovery prediction, a QLoRA-tuned Mistral-7B reached Hit@5=0.76 and F1@5=0.32. REPAIR-Bench provides both the HRI and Medical HRI communities with a standardized framework for (1) evaluating
robot failures and (2) building transparent, adaptive, and trustworthy recovery systems.
Read the full abstract ▾
All authors Giuliano Pioldi, Yashika Batra, Arman Ibrayeva, Yuanchen Bai, Purnjay Maruur, Promise Ekpo, Angelique Taylor
Robotics
OpenSPM: An Environment-Transferable Robotic Key Spatial Pose Memory and Closed-Loop High-Frequency Flow-Matching Action Generation Model
Iok Tong Lei, Qingchen Xie, Yifan Wang et al. (+2 more)
Open-environment tabletop robotic manipulation requires systems to possess semantic understanding, precise
geometric pose estimation, and high-frequency action generation. While end-to-end
vision-
language-action (VLA) models excel at semantic generalization, they often lack explicit
geometric constraints for fine-grained tasks and require costly training. To bridge the gap between high-level semantics and low-level physical execution, we propose OpenSPM, an open environment spatial persistent
memory framework consisting of spatial pose
memory and flow-matching action generation model. OpenSPM first leverages semantically conditioned 3D perception and Kalman filtering to track continuous 6D poses. It then extracts key spatial poses from human demonstrations, keeping them as transferable, object-centric spatial persistent
memory entries. During inference, OpenSPM retrieves relevant
memory entries in terms of natural
language instructions, transfers the spatial poses to new scenes using SE(3) transformations, and generates high-frequency action chunks via a lightweight conditional flow-matching model. Combined with real-time proprioceptive state feedback and terminal residual correction, the system effectively suppresses trajectory error accumulation. Evaluated on ten LIBERO-GOAL tasks, OpenSPM achieves an 85.6% success rate and an equivalent control frequency of 1033.3 Hz, while requiring minimal inference AI computing power. Extensive ablations illustrate that structured spatial persistent
memory and closed-loop residual correction play a crucial role in reliable, high-frequency robotic manipulation.
Read the full abstract ▾
All authors Iok Tong Lei, Qingchen Xie, Yifan Wang, Yap Ying Jie, Zhidong Deng
Robotics
RoamFlow: Reinforcement-Aligned One-Step Action MeanFlow Policy for Image-Goal Navigation
Zixuan Zhang, Yuqi Chen, Junjie Gao et al. (+4 more)
Image-goal
navigation is a key challenge in
embodied robotics, where an
agent must reach a target specified solely by a goal image. While existing reinforcement learning approaches map perceptual observations directly to actions, they struggle to model long-horizon dependencies, often leading to suboptimal trajectories. To address this limitation, we propose RoamFlow, a generative
navigation framework that leverages MeanFlow to predict the average velocity field for trajectory synthesis, enabling efficient few-step generation and reducing inference latency. We further adopt a two-stage training strategy that combines expert imitation for stable initialization with reinforcement learning for task-specific policy refinement. Extensive experiments in both Habitat simulation and real-world robotic platforms demonstrate that RoamFlow achieves efficient inference while maintaining strong
navigation performance under real-time constraints.
Read the full abstract ▾
All authors Zixuan Zhang, Yuqi Chen, Junjie Gao, Siyuan Song, Yongzhou Pan, Beichen Wang, Mir Feroskhan
Robotics
Flying to Image-Specified Objects: 3D Quadrotor Navigation via Cross-Graph Memory and Viewpoint Planning
Junjie Gao, Yuqi Chen, Yongzhou Pan et al. (+3 more)
Instance-Specific Image-Goal
Navigation (InstanceImageNav) requires a
robot to navigate toward the exact object instance depicted in a query image. Extending this task to quadrotors is challenging due to continuous 3D control, limited field of view (FOV), and safety constraints, which make successful
navigation highly dependent on selecting informative viewpoints. We propose a hierarchical
navigation framework for quadrotor InstanceImageNav that separates high-level decision making from low-level motion execution. Instead of navigating directly to spatial locations, the system generates viewpoint-aware action nodes around frontier regions and potential target objects, enabling the
robot to explore while maintaining informative viewpoints for detecting the target instance. A lightweight semantic
memory maintains object-level and observation-level context, allowing semantic cues to propagate to candidate action nodes for decision making. A learning-based policy selects the most promising action node, and a trajectory planner generates dynamically feasible 3D flight paths for safe execution. Experiments in simulation demonstrate consistent improvements over strong baselines, and real-world quadrotor flights validate the practicality and robustness of the proposed framework.
Read the full abstract ▾
All authors Junjie Gao, Yuqi Chen, Yongzhou Pan, Yaosheng Deng, Jiaping Xiao, Mir Feroskhan
Robotics
Sphere-VIO: Fast and Robust Visual-Inertial Odometry via Unified Spherical Representation for Heterogeneous Multi-Camera Systems
Yueteng Yang, Yusen Xie, Hao Wei et al. (+5 more)
Multi-camera visual-inertial odometry (VIO) overcomes the inherent limitations of pure visual systems by expanding the field of view. However, existing
algorithms are typically tailored for fixed camera setups and lack unified compatibility with heterogeneous multi-camera systems. Meanwhile, due to the absence of a unified cross-camera representation and association mechanism, current methods struggle to achieve a balance among robust cross-camera feature tracking, stable depth estimation, and reliable real-time performance. To address these issues, we present Sphere-VIO, a lightweight filter-based VIO framework with unified spherical representation for heterogeneous multi-camera systems. Specifically, we first propose a Unified Spherical Panorama Model (USPM) that supports all standard camera models and enables bidirectional fast mapping between multi-camera images and a shared spherical space without sequential stitching, simplifying cross-camera feature management and improving triangulation efficiency. Second, we design a parallel-accelerated depth-guided semi-direct tracking pipeline, namely Hierarchical Omnidirectional Feature
Alignment (HOFA), with global spherical constraints for robust cross-camera matching, and fuse multi-camera depth observations into a standard depth filter for stable initialization. Finally, we develop a multi-camera-adapted ESKF backend that employs spherical bearing residuals and Schur complement marginalization to minimize computational overhead, enabling accurate real-time state estimation on resource-constrained devices. Extensive experiments on public
benchmarks and a custom omnidirectional
dataset show that Sphere-VIO achieves superior trade-offs between accuracy, robustness, efficiency, and cross-camera generality.
Read the full abstract ▾
All authors Yueteng Yang, Yusen Xie, Hao Wei, Qianhao Wang, Boyu Zhou, Fei Gao, Jun Ma, Jinni Zhou
Robotics
Pondering the Way: Spatial-perceiving World Action Model for Embodied Navigation
Hong Chen, Daqi Liu, Zehan Zhang et al. (+10 more)
Existing world model-based planners for visual
navigation typically follow a verification-centric paradigm, decoupling goal intent from trajectory synthesis. This approach suffers from candidate dependence, heavy computational overhead, and inconsistencies between sampled actions and predicted visuals. To address these issues, we propose SWAM (Spatial-perceiving World Action Model), a task-centric joint observation-action generation framework. Given start and goal RGB observations, SWAM performs single-pass inference to simultaneously generate intermediate RGB-D sequences and corresponding action trajectories, promoting goal-consistent trajectory generation and improved spatial feasibility. While SWAM leverages depth pseudo-labels during training to internalize spatial priors, it requires only
monocular RGB input at inference time. We further introduce a visual-guided action refinement module and a trajectory-scale regularization loss to enforce fine-grained
alignment between motion and visual cues while stabilizing predictions across varying distances. Extensive experiments show that SWAM significantly outperforms state-of-the-art two-stage planners in success rate, trajectory accuracy, and inference efficiency, while demonstrating robust zero-shot generalization to unseen environments.
Read the full abstract ▾
All authors Hong Chen, Daqi Liu, Zehan Zhang, Haiguang Wang, Tianhao Lu, Longfei Yan, Haiyang Sun, Fangzhen Li, Hongwei Xie, Bing Wang, Guang Chen, Hangjun Ye, Yihua Tan
Robotics
Critical Interval MSE: Toward Reliable Offline Validation for Robot Manipulation Policies
Haoxu Huang, Tongsam Zheng, Yifan Chen et al. (+2 more)
Real-world evaluation is the gold standard for
robot manipulation policies because it tests them against the physical conditions and deployment challenges they are ultimately designed to handle. However, real-world evaluation is also the bottleneck for iterating on
robot imitation learning policies: it is costly, difficult to reproduce, and often too sparse to reliably compare nearby model variants. A straightforward proxy for performance is validation loss on expert demonstrations, but this proxy is often poorly correlated with real-world performance. In this paper, we introduce Critical Interval MSE (CI-MSE), an intuitively simple yet effective offline validation metric. CI-MSE uses a zero-shot
agent to localize task-critical segments, restricts error computation to those segments, and applies simple action-
alignment procedures that better match rollout-time behavior. Across simulation and real-world experiments, CI-MSE generally yields a stronger correlation between validation error and rollout performance than raw MSE. Across 27 distinct policy checkpoints, CI-MSE achieves a Spearman rank correlation of $-0.87$, closer to the ideal value of $-1$ than raw MSE's $-0.61$, demonstrating a significant improvement. Sensitivity analysis shows that our metric is robust across a wide range of hyperparameter settings. We further study the effectiveness of CI-MSE under evaluation distribution shifts and identify practical boundaries for its use. In summary, this paper provides a simple and reliable offline validation tool for accelerating policy iteration. Project webpage: https://ci-mse.github.io/
Read the full abstract ▾
All authors Haoxu Huang, Tongsam Zheng, Yifan Chen, Jiacheng You, Yang Gao
Robotics
Trust Your Instincts: Confidence-Driven Test-Time RL for Vision-Language-Action Models
Siyao Chen, Jiakang Yuan, Jiaxin Wang et al. (+1 more)
Reinforcement learning (RL) has become indispensable for pushing
Vision-
Language-Action Models (VLAs) beyond static imitation learning. However, existing RL methods typically require external environmental feedback, relying on predefined success signals to guide policy updates. In this work, we show that VLA models possess useful internal evaluative capabilities: in discrete-action VLAs, trajectories with higher generation confidence are significantly more likely to succeed. Based on this observation, we introduce T^2VLA (Test-time VLA), an architecture-agnostic test-time RL framework that enables VLA models to achieve self-bootstrapping policy improvement. Instead of relying on external rewards, T^2VLA leverages trajectory-level similarity to high-confidence expert demonstrations as an intrinsic reward signal. In addition, we propose a Confidence-Driven Dual Expert Bootstrapping mechanism, which dynamically balances a Local Pseudo-Expert for exploration and a Global Expert Pool for training stability. Extensive experiments on the LIBERO and RoboTwin
benchmarks show that T^2VLA consistently outperforms supervised baselines and approaches oracle RL performance with ground-truth rewards, achieving effective improvement without external reward feedback. Furthermore, T^2VLA adapts to distinct VLA paradigms, including both OpenVLA-OFT and the pi series.
Read the full abstract ▾
All authors Siyao Chen, Jiakang Yuan, Jiaxin Wang, Tao Chen
Robotics
AUSLUN: A Fixed-Hover UAV–USV System for GNSS-Denied Maritime Search and Navigation
Siyuan Yang, Zikai Jia, Hailiang Kuang et al. (+4 more)
Global
navigation satellite system (GNSS) denial can prevent an unmanned surface vehicle (USV) from both finding a distant vessel and maintaining a globally referenced approach. This paper presents AUSLUN (Automatic
UAV Search, Localization, and USV
Navigation), a fixed-hover aerial-surface system that uses a coastal unmanned aerial vehicle (
UAV), which estimates its own pose through visual-inertial odometry (VIO), as a long-range sensing and
navigation anchor. The central design shifts sensing motion from
UAV translation to a zoom pod and closes the loop through three coupled elements: polygon-aware annular pod scanning, modality-aware bearing-range localization, and target-relative USV guidance with visual-loss recovery. The same gated recursive
estimator uses
laser range for the non-cooperative target and datalink range for the cooperative USV. Search-planning simulations show that the adaptive yaw bounds reduce scan time and redundant coverage relative to a matched fixed-sector scan, and GPS-referenced field data show that the gated recursive
estimator outperforms non-recursive baselines in localization accuracy. An integrated maritime mission further demonstrates the complete search-to-
navigation sequence, including a deliberately triggered visual-loss recovery. These results establish the feasibility and operating boundary of fixed-hover
UAV assistance for stationary-target approach in coastal GNSS-denied environments. The source code and a
video demonstration are publicly available at https://github.com/xirhxq/pod_search and https://youtu.be/S-5RkJs35JI.
Read the full abstract ▾
All authors Siyuan Yang, Zikai Jia, Hailiang Kuang, Xiaoyu He, Qizhi Guo, Yihao Dong, Shaoming He
Robotics
Normalizing Flow-Enhanced Message Passing for Multirobot Collaborative Localization
Han Shen, Guanghui Wen, Liangming Chen et al. (+1 more)
Accurate, robust, and adaptive localization is essential for various robotic operations. This paper proposes a new message passing (MP)
algorithm for realizing collaborative localization in a distributed manner. The
algorithm unifies Gaussian belief propagation (GBP) and mean-field (MF) approximation, where GBP preserves dependencies among
robot states, and MF enables estimation of noise statistics. To effectively handle non-conjugate terms from nonlinear measurement models, the
algorithm adopts a parametric formulation in which these terms are treated by gradient
estimators. Beyond linearization and sampling, we further design a normalizing flow (NF)-based gradient
estimator, enabling learnable sampling. End-to-end training tunes NF parameters according to the behavior of MP, improving the overall estimation performance. To support estimation of practical robotic states that involve rotations, the method is then extended to Lie group state spaces. Finally, the method is applied to multirobot localization task fusing odometry, global
navigation satellite system (GNSS) measurements, and inter-
robot ultra wideband (UWB) ranging. Simulations and experiments on autonomous surface vehicles (ASVs) demonstrate its improved accuracy, robustness, and adaptability.
Read the full abstract ▾
All authors Han Shen, Guanghui Wen, Liangming Chen, Ming Cao
Robotics
TACO: A Test and Check Framework for Robust Pose Graph Optimization
Emilio Olivastri, Alberto Pretto and Tobias Fischer
Pose Graph Optimization (PGO) is one of the most widely adopted approaches for solving Simultaneous Localization and Mapping (SLAM) problems. However, PGO approaches are particularly sensitive to outliers, which can substantially degrade the quality of the estimated trajectories. These outliers arise from incorrect place recognition associations caused by perceptual aliasing in the environment. In this paper, we present TACO (short for Test And Check Optimization), a robust optimization framework designed to filter out outliers from PGO systems. Rather than explicitly modeling measurements as inliers or outliers, TACO finds an approximation to the maximally consistent set of measurements incrementally through two complementary components: (i) The test component, namely the Incremental Probabilistic Consensus (IPC)
algorithm, evaluates the consistency of each incoming loop closure online. (ii) The check component dubbed Switchable Outlier Sanitization leverages the existing Switchable Constraints to periodically sanitize any inconsistent measurements from the consistent set that IPC may have mistakenly included. We evaluate TACO on 2D SLAM and 3D Visual SLAM
datasets against several state-of-the-art methods. The results show robustness comparable to state-of-the-art offline methods while preserving the computational efficiency required for online deployment, achieving a success rate above 90% in 2D and 83% in 3D across outlier rates up to 50%, with mean convergence times of approximately 45 ms and 100 ms, respectively. We release an open-source implementation of our method with this paper.
Read the full abstract ▾
Robotics
Legible Shared Autonomy: Implicit Communication of Robot Belief through Motion
Jinwei Liu, Pengfei Li, Shaofeng Chen et al. (+2 more)
Shared autonomy systems combine user input with autonomous assistance to help users with motor impairments control
robot arms to perform everyday manipulation tasks, by inferring user goals and providing appropriate guidance. However, the
robot's internal beliefs about user goals cannot be observed by users. Traditional shared autonomy systems provide assistance along efficient shortest paths toward inferred goals, but when multiple objects lie in similar directions, such assistive motion remains ambiguous and fails to reveal the specific goal identified by the
robot. This creates two critical problems. First, when the
robot correctly infers the goal, users continue controlling because they cannot perceive understanding from ambiguous assistive motion, wasting effort when autonomous completion would suffice. Second, when the
robot misunderstands intent, users cannot quickly detect errors until assistive motion diverges significantly, requiring substantial corrective input. We address this by introducing legible motion into shared autonomy, where
robot actions must both advance toward the goal and clearly reveal which goal has been inferred, enabling users to understand the
robot's beliefs and adjust control accordingly. The
robot modulates communication strength through confidence-aware adaptive authority allocation by providing assertive legible assistive actions when confident while increasing user authority when uncertain, transforming shared autonomy into transparent bidirectional collaboration. User studies including simulation, and physical experiments with a six-degree-of-freedom
robot arm demonstrate that legible shared autonomy significantly improves users' understanding of
robot beliefs and reduces user control effort compared to standard shared autonomy.
Read the full abstract ▾
All authors Jinwei Liu, Pengfei Li, Shaofeng Chen, Tao Wang, Yun-Bo Zhao
Robotics
STEAM: Self-Supervised Temporal Ensemble Advantage Modeling for Real-World Robot Learning
Zhihao Liu, Qiuyi Gu, Yitao Wang et al. (+16 more)
Real-world
robot learning increasingly relies on heterogeneous data, but demonstrations and rollouts often mix useful progress with stalls, corrections, and suboptimal behavior. Effective policy learning therefore requires frame-level advantages that distinguish reliable local progress from failures and regressions. We propose Self-supervised Temporal Ensemble Advantage Modeling (STEAM), a label-free method that learns such advantages from expert demonstrations. STEAM trains an ensemble of temporal-offset predictors on frame pairs within expert trajectories, using the normalized temporal offset between two frames as a self-supervised signal. Each predictor maps a frame pair to a distribution over temporal offsets, which is converted into a scalar advantage. STEAM then takes the minimum advantage across the ensemble to score mixed-quality rollout data conservatively. Across real-world bimanual towel folding, chip checkout, table clearing, and cola restocking tasks, STEAM identifies stalls, failures, and recoveries. When combined with CFGRL, STEAM achieves the highest policy success rate on all four tasks, averaging $83.5\%$ against $58.8\%$ for the best baseline. Project page: https://rlinf.github.io/steam/.
Read the full abstract ▾
All authors Zhihao Liu, Qiuyi Gu, Yitao Wang, Dongming Qiao, Yixian Zhang, Shuaihang Chen, Liangzhi Shi, Tianxing Zhou, Zefang Huang, Kang Chen, Zhen Guo, Quanlu Zhang, Jincheng Yu, Xiaodan Liang, Guoliang Fan, Yu Wang, Feng Gao, Xinlei Chen, Chao Yu
Robotics
Data-Driven Modeling and Control for Tethered Space Systems with Koopman-Informed Graphs
Ao Jin, Yifeng Ma, Panfeng Huang et al. (+1 more)
Modeling tethered space systems is critical for advanced orbital operations. Flexible components such as tethers and space nets are integral to these systems but present significant control challenges due to their high dimensional, strongly coupled, and nonlinear dynamics. While data driven methods offer alternative modeling approaches, they frequently struggle with long term predictive stability and spatial generalization. To address this, we propose the Koopman Graph Dynamics (KGD) framework to learn the structural dynamics by integrating the global linear
evolution of the Koopman operator with the local topological priors of Graph Neural Networks. Building upon this representation, we develop a KGD based Model Predictive Control strategy for tethered space systems. Subsequently, the ground experiments on flexible tether and space net demonstrate the high precision modeling capabilities of the proposed method. Crucially, the framework exhibits exceptional capacity for spatial transfer without retraining. Models trained exclusively on small configurations successfully predict and control significantly larger, unseen physical scales. Furthermore, the orbit simulations within a physics engine verify the effectiveness of the proposed approach for tethered space systems.
Read the full abstract ▾
All authors Ao Jin, Yifeng Ma, Panfeng Huang, Fan Zhang
Robotics · Cross-listed from cs.CV
OP3DSG: Open-Vocabulary Part-Aware 3D Scene Graph Generation for Real-World Environments
Yirum Kim and Ue-Hwan Kim
3D scene graphs (3DSGs) provide a compact and structured abstraction of 3D environments. Although advances in foundation models have enabled open-vocabulary 3DSG generation, existing approaches remain object-centric and encode limited relational information – restricting their applicability in real-world scenarios that require fine-grained understanding. We propose OP3DSG, an open-vocabulary part-aware 3DSG generation framework that constructs unified graphs that jointly model objects, interactive parts, spatial relations, functional relations, and affordances. OP3DSG integrates object-part knowledge-guided detection with part-aware 3D fusion to preserve small and interaction-relevant components, and employs a geometry-initialized prior graph with LLM-based refinement to reduce spurious relational predictions while enabling efficient graph construction. To systematically evaluate unified 3D scene graph construction, we introduce UniGraph3D, a
benchmark designed for part-aware perception and multi-level relational
reasoning. Experimental results show that OP3DSG achieves state-of-the-art performance and demonstrates its effectiveness as a perception backbone in diverse real-world robotics tasks.
Read the full abstract ▾
Robotics
FalconTrack: Photorealistic Auto-Labeled Perception and Physics-Aware Vision-Based Aerial Tracking
Yan Miao, Karteek Gandiboyina, Noah Giles et al. (+4 more)
Vision-based aerial tracking is critical in GPS-denied environments. Reliable perception for tracking depends on large-scale labeled data, yet most photorealistic
datasets rely on heavy manual annotation and are time-consuming to produce. We present FalconTrack, a unified perception-and-tracking framework that (i) leverages a photorealistic editable simulator for automated label generation and (ii) combines multi-head perception with physics-aware tracking for zero-shot sim-to-real transfer. FalconTrack provides an automated labeling pipeline in a Gaussian Splatting simulator that isolates target Gaussians from short object
videos and composites them with randomized backgrounds to generate RGB, mask, class, and 6-DoF pose labels, producing about 10k labeled images in under 20 minutes. Using this
dataset, we train a multi-head perception module with staged learning and reprojection consistency, and fuse its outputs with class-conditioned dynamics priors in an EKF for tracking. Our perception model outperforms two baselines and reaches 96-100% class accuracy in zero-shot sim-to-real transfer on three geometrically diverse objects and two environments, while maintaining consistent performance in unseen simulated and real scenes. In real hardware closed-loop visual tracking, the onboard system runs at about 25 Hz and achieves 100% success in sim-to-real F1-tenth and gate tracking in five trajectories across two environments, while a mask-centered
vision baseline drops to 60% success on F1-tenth during fast out-of-view scenarios.
Read the full abstract ▾
All authors Yan Miao, Karteek Gandiboyina, Noah Giles, Hideki Okamoto, Bardh Hoxha, Georgios Fainekos, Sayan Mitra
Robotics
Analytic Concept-Centric Memory for Agentic Embodied Manipulation
Mingyang Sun, Xiujian Liang, Jiude Wei et al. (+4 more)
Long-horizon
embodied manipulation requires
agents to remember persistent objects, track changing scene states, and reuse prior interaction knowledge. However, existing
agent memories are often stored as unstructured histories or
embedding-based records, making it difficult to retrieve manipulation-relevant object parts, physical states, action effects, and executable skills. We propose an analytic concept-centric
memory framework for agentic
embodied manipulation. Our
memory organizes experience around structured analytic concepts, where objects are represented by semantic parts, parametric templates, grounded poses, affordances, and manipulation states. It further connects object and scene
memories with transition
memory for action-induced state changes and skill
memory for template-grounded and policy-grounded execution. At runtime, the
agent performs structured coarse-to-fine
retrieval to identify relevant objects, states, transitions, and skills, supporting state-consistent
reasoning and skill reuse. Experiments on
memory-dependent manipulation, articulated-object generalization, real-world
memory evaluation, and ablations show that our approach improves task completion,
retrieval accuracy, object re-identification, and cross-object skill generalization over unstructured and
embedding-based
memory baselines.
Read the full abstract ▾
All authors Mingyang Sun, Xiujian Liang, Jiude Wei, Qichen He, Donglin Wang, Cewu Lu, Jianhua Sun
Robotics
Trajectory Optimization for Collision-Aware Redundant Robotic Multi-Axis Additive Manufacturing by Constrained Gradient Projection
Zhikai Shen, Jiasheng Qu, Chenyu Xu et al. (+4 more)
Redundant robotic multi-axis additive manufacturing (MAAM) enables support-free and conformal fabrication, but trajectory optimization for long-horizon paths remains challenging under strict deposition-position constraints and time-varying collision constraints. This work proposes a computational framework for collision-aware trajectory optimization in redundant robotic MAAM. We first formulate nozzle-workpiece relative kinematics using a relative Jacobian, and develop a differentiable SDF-based collision model that captures fabrication-induced geometry
evolution and provides optimization gradients. The deposition position is then enforced as a hard waypoint-wise equality constraint through iterative projection onto the self-motion
manifold, with the loss gradient restricted to the corresponding tangent space. Experiments on an 8-DOF robotic MAAM platform with diverse long-horizon support-free and conformal toolpaths show that our method maintains a mean nozzle-position error below 10{\mu}m, reduces maximum joint jerk by up to $77.6\%$, and eliminates all sampled collision and orientation violations. Compared with the SQP-based baseline, it achieves up to a 10.2x speedup and improved convergence. Physical fabrication experiments further verify that the resulting smooth, collision-free trajectories enable successful printing of complex geometries with fewer visible deposition artifacts.
Read the full abstract ▾
All authors Zhikai Shen, Jiasheng Qu, Chenyu Xu, Zhuo Huang, Chengkai Dai, Yongzhe Li, Guoxin Fang
Robotics
Cross-Spectral Stereo Inertial Odometry
Seungsang Yun, Hyunsoo Jang, Tai Hyoung Rhee et al. (+3 more)
Standard stereo VIO focuses exclusively on the benefit of metric scale via single-spectrum baselines, often overlooking the risks of spectral redundancy. This structural limitation leads to correlated failures, where both sensors simultaneously fail in degraded environments that affect their shared spectrum. Leveraging a cross-spectral system presents a complementary solution to this issue, yet the significant appearance gap between modalities renders standard matching ineffective. Existing deep learning-based matchers, while effective, introduce inference latencies that violate real-time constraints. To bridge this gap, we present an asynchronous real-time cross-spectral visual-thermal-inertial (VTI) system that temporally decouples high-latency deep matching from high-rate state estimation. Our architecture incorporates a spectral-aware weighting scheme that dynamically balances modality reliance based on photometric
entropy and thermal noise, ensuring robustness against both abrupt lighting changes and thermal artifacts. Furthermore, we introduce a seamless handling mechanism for thermal Non-uniformity Correction (NUC) to maintain tracking continuity. Extensive experiments across diverse scenarios confirm that our system overcomes spectral redundancy, yielding superior accuracy in nominal daylight while ensuring robustness in visually degraded environments. We will open source our code and data: https://github.com/seungsang07/cross-spectral-stereo-inertial-odometry
Read the full abstract ▾
All authors Seungsang Yun, Hyunsoo Jang, Tai Hyoung Rhee, Hyunho Song, Hyeonjae Gil, Ayoung Kim
Robotics
Multi-UAV Formation Cooperative Obstacle Avoidance and Adaptive Shape Deformation Control in Complex Environments Based on BI-APF-RRT and Affine Transformation
Yiliang Wu, Weican Chen and Yendo Hu
Aiming at the problem that obstacle avoidance flexibility and formation integrity are difficult to coexist in multi-
UAV formation motion in complex obstacle environments , and that the traditional artificial potential field (APF) method easily falls into local optima, a cooperative obstacle avoidance
algorithm for multi-
UAV formations integrating BI-APF-RRT and affine transformation is proposed. First, abandoning the traditional APF centroid path planning method , a goal-biased Bidirectional Artificial Potential Field method RRT (BI-APF-RRT)
algorithm is adopted to conduct global collision-free path planning for the centroid of the leader formation. By introducing an improved artificial potential field and cubic B-spline interpolation, the smoothness and rapid convergence of the global path are ensured. Secondly, using the generated global path as the guiding trajectory for the formation's centroid , combined with an affine transformation
matrix (including non-uniform scaling and rotation) , the formation can adaptively deform based on the distance to obstacles while moving along the optimal path. Finally, the followers track the leaders through a distributed control law , enabling the entire formation to safely cross complex obstacle areas without disassembling.
Read the full abstract ▾