Robotics · Cross-listed from eess.SY
Hierarchical JEPA Meets Predictive Remote Control in Beyond 5G Networks
Abanoub M. Girgis, Ibtissam Labriji and Mehdi Bennis
In wireless networked control systems, ensuring timely and reliable state updates from distributed devices to remote controllers is essential for robust control performance. However, when multiple devices transmit high-dimensional states (e.g., images or
video frames) over bandwidth-limited wireless networks, a critical trade-off emerges between communication efficiency and control performance. To address this challenge, we propose a Hierarchical Joint-
Embedding Predictive Architecture (H-JEPA) for scalable predictive control. Instead of transmitting states, device observations are encoded into low-dimensional
embeddings that preserve essential dynamics. The proposed architecture employs a three-level hierarchical prediction, with high-level, medium-level, and low-level predictors operating across different temporal resolutions, to achieve long-term prediction stability, intermediate interpolation, and fine-grained refinement, respectively. Control actions are derived within the
embedding space, removing the need for state reconstruction. Simulation results on inverted cart-pole systems demonstrate that H-JEPA enables up to 42.83 % more devices to be supported under limited wireless capacity without compromising control performance.
Read the full abstract ▾
Robotics
When Simultaneous Localization and Mapping Meets Wireless Communications: A Survey
Konstantinos Gounis, Sotiris A. Tegos, Dimitrios Tyrovolas et al. (+2 more)
This paper surveys the state-of-the-art in the nexus of SLAM and Wireless Communications, attributing the bidirectional impact of each with a focus on visual SLAM (V-SLAM) integration. We provide an overview of key concepts related to wireless signal propagation,
geometric channel modeling, and
radio frequency (RF)-based localization and sensing. In addition to this, we show image processing techniques that can detect landmarks, proactively predicting optimal paths for wireless channels. Several dimensions are considered, including the prerequisites, techniques, background, and future directions and challenges of the intersection between SLAM and wireless communications. We analyze estimation and control approaches such as
Bayesian filters, feature-based pose estimation, perception-aware motion control, spatial methods for signal processing such as vector fields, and key technological aspects. We expose techniques and items towards enabling a highly effective
retrieval of the autonomous
robot state. Among other interesting findings, we observe that
monocular V-SLAM would benefit from RF relevant information, as the latter can serve as a proxy for the scale ambiguity resolution. Conversely, we find that wireless communications in the context of 5G and beyond can potentially benefit from visual odometry that is central in SLAM. Moreover, we examine other sources besides the camera for SLAM and describe the twofold relation with wireless communications. Finally, integrated solutions performing joint communications and SLAM appear to be in their infancy: theoretical and practical advancements are required to add higher-level localization and semantic perception capabilities to RF and multi-
antenna technologies.
Read the full abstract ▾
All authors Konstantinos Gounis, Sotiris A. Tegos, Dimitrios Tyrovolas, Panagiotis D. Diamantoulakis, George K. Karagiannidis
Robotics
LangGS-SLAM: Real-Time Language-Feature Gaussian Splatting SLAM
Seongbo Ha, Sibaek Lee, Kyungsu Kang et al. (+3 more)
In this paper, we propose a RGB-D SLAM system that reconstructs a
language-aligned dense feature field while sustaining low-latency tracking and mapping. First, we introduce a Top-K Rendering pipeline, a high-throughput and semantic-distortion-free method for efficiently rendering high-dimensional feature maps. To address the resulting semantic-
geometric discrepancy and mitigate the
memory consumption, we further design a multi-criteria map management strategy that prunes redundant or inconsistent Gaussians while preserving scene integrity. Finally, a hybrid field optimization framework jointly refines the
geometric and semantic fields under real-time constraints by decoupling their optimization frequencies according to field characteristics. The proposed system achieves superior
geometric fidelity compared to
geometric-only baselines and comparable semantic fidelity to offline approaches while operating at 15 FPS. Our results demonstrate that online SLAM with dense, uncompressed
language-aligned feature fields is both feasible and effective, bridging the gap between 3D perception and
language-based
reasoning.
Read the full abstract ▾
All authors Seongbo Ha, Sibaek Lee, Kyungsu Kang, Joonyeol Choi, Seungjun Tak, Hyeonwoo Yu
Robotics
Kun Wang, Xiao Feng, Mingcheng Qu et al. (+1 more)
Vision Language Action (VLA) models have recently shown great potential in bridging
multimodal perception with robotic control. However, existing methods often rely on direct fine-tuning of pre-trained
Vision-
Language Models (VLMs), feeding semantic and visual features directly into a policy network without fully addressing the unique semantic
alignment challenges in the VLA domain. In this paper, we propose HMVLA, a novel VLA framework that exploits the inherent hierarchical structures in
vision and
language for comprehensive semantic
alignment. Unlike traditional methods that perform
alignment in Euclidean space, our HMVLA embeds
multimodal features in
hyperbolic space, enabling more effective modeling of the hierarchical relationships present in image text data. Furthermore, we introduce a sparsely gated Mixture of Experts (MoE) mechanism tailored for semantic
alignment, which enhances
multimodal comprehension between images and text while improving efficiency. Extensive experiments demonstrate that HMVLA surpasses baseline methods in both accuracy and generalization. In addition, we validate its robustness by reconstructing
datasets to further test cross domain adaptability.
Read the full abstract ▾
All authors Kun Wang, Xiao Feng, Mingcheng Qu, Tonghua Su
Robotics · Cross-listed from cs.CV
PovNet+: A Deep Learning Architecture for Socially Assistive Robots to Learn and Assist with Multiple Activities of Daily Living
Fraser Robinson, Souren Pashangpour, Matthew Lisondra et al. (+1 more)
A significant barrier to the long-term deployment of autonomous socially assistive
robots is their inability to both perceive and assist with multiple activities of daily living (ADLs). In this paper, we present the first
multimodal deep learning architecture, POVNet+, for multi-activity recognition for socially assistive
robots to proactively initiate assistive behaviors. Our novel architecture introduces the use of both ADL and motion
embedding spaces to uniquely distinguish between a known ADL being performed, a new unseen ADL, or a known ADL being performed atypically in order to assist people in real scenarios. Furthermore, we apply a novel user state estimation method to the motion
embedding space to recognize new ADLs while monitoring user performance. This ADL perception information is used to proactively initiate
robot assistive interactions. Comparison experiments with state-of-the-art human activity recognition methods show our POVNet+ method has higher ADL classification accuracy. Human-
robot interaction experiments in a cluttered living environment with multiple users and the socially assistive
robot Leia using POVNet+ demonstrate the ability of our multi-modal ADL architecture in successfully identifying different seen and unseen ADLs, and ADLs being performed atypically, while initiating appropriate assistive human-
robot interactions.
Read the full abstract ▾
All authors Fraser Robinson, Souren Pashangpour, Matthew Lisondra, Goldie Nejat
Robotics · Cross-listed from cs.MA
AI-Augmented Density-Driven Optimal Control (D2OC) for Decentralized Environmental Mapping
Kooktae Lee and Julian Martinez
This paper presents an AI-augmented decentralized framework for multi-
agent (multi-
robot) environmental mapping under limited sensing and communication. While conventional coverage formulations achieve effective spatial allocation when an accurate reference map is available, their performance deteriorates under uncertain or biased priors. The proposed method introduces an adaptive and self-correcting mechanism that enables
agents to iteratively refine local density estimates within an optimal transport-based framework, ensuring theoretical consistency and scalability. A dual multilayer perceptron (MLP) module enhances adaptivity by inferring local mean-variance statistics and regulating virtual uncertainty for long-unvisited regions, mitigating stagnation around local minima. Theoretical analysis rigorously proves convergence under the Wasserstein metric, while simulation results demonstrate that the proposed AI-augmented Density-Driven Optimal Control consistently achieves robust and precise
alignment with the ground-truth density, yielding substantially higher-fidelity reconstruction of complex multi-modal spatial distributions compared with conventional decentralized baselines.
Read the full abstract ▾
Robotics
Multi-Robot Decentralized Collaborative SLAM in Planetary Analogue Environments: Dataset, Challenges, and Lessons Learned
Pierre-Yves Lajoie, Karthik Soma, Haechan Mark Bong et al. (+4 more)
Decentralized collaborative simultaneous localization and mapping (C-SLAM) is essential to enable multirobot missions in unknown environments without relying on preexisting localization and communication infrastructure. This technology is anticipated to play a key role in the exploration of the Moon, Mars, and other
planets. In this article, we share insights and lessons learned from C-SLAM experiments involving three
robots operating on a Mars analogue terrain and communicating over an ad hoc network. We examine the impact of limited and intermittent communication on C-SLAM performance, as well as the unique localization challenges posed by planetary-like environments. Additionally, we introduce a novel
dataset collected during our experiments, which includes real-time peer-to-peer inter-
robot throughput and latency measurements. This
dataset aims to support future research on communication-constrained, decentralized multirobot operations.
Read the full abstract ▾
All authors Pierre-Yves Lajoie, Karthik Soma, Haechan Mark Bong, Alice Lemieux-Bourque, Rongge Zhang, Vivek Shankar Varadharajan, Giovanni Beltrame
Robotics
Track-centric Iterative Learning for Global Trajectory Optimization in Autonomous Racing
Youngim Nam, Jungbin Kim, Kyungtae Kang et al. (+1 more)
This paper presents a global trajectory optimization framework for minimizing lap time in autonomous racing under uncertain vehicle dynamics. Optimizing the trajectory over the full racing horizon is computationally expensive, and tracking such a trajectory in the real world hardly assures global optimality due to uncertain dynamics. Yet, existing work mostly focuses on dynamics learning at the tracking level, without updating the trajectory itself to account for the learned dynamics. To address these challenges, we propose a track-centric approach that directly learns and optimizes the full-horizon trajectory. We first represent trajectories through a track-agnostic parametric space in light of the wavelet transform. This space is then efficiently explored using
Bayesian optimization, where the lap time of each candidate is evaluated by running simulations with the learned dynamics. This optimization is embedded in an iterative learning framework, where the optimized trajectory is deployed to collect real-world data for updating the dynamics, progressively refining the trajectory over the iterations. The effectiveness of the proposed framework is validated through simulations and real-world experiments, demonstrating lap time improvement of up to 20.7% over a nominal baseline and consistently outperforming state-of-the-art methods.
Read the full abstract ▾
All authors Youngim Nam, Jungbin Kim, Kyungtae Kang, Cheolhyeon Kwon
Robotics
Meta-ROS: A Next-Generation Middleware Architecture for Adaptive and Scalable Robotic Systems
Anshul Ranjan, Anoosh Damodar, Neha Chougule et al. (+3 more)
The field of robotics faces significant challenges related to the complexity and interoperability of existing middleware frameworks, like ROS2, which can be difficult for new developers to adopt. To address these issues, we propose Meta-ROS, a novel middleware solution designed to streamline robotics development by simplifying integration, enhancing performance, and ensuring cross-platform compatibility. Meta-ROS leverages modern communication protocols, such as Zenoh and ZeroMQ, to enable efficient and low-latency communication across diverse hardware platforms, while also supporting various data types like audio, images, and
video. We evaluated Meta-ROS's performance through comprehensive testing, comparing it with existing middleware frameworks like ROS1 and ROS2. The results demonstrated that Meta-ROS outperforms ROS2, achieving up to 30% higher throughput, significantly reducing message latency, and optimizing resource usage. Additionally, its robust hardware support and developer-centric design facilitate seamless integration and ease of use, positioning Meta-ROS as an ideal solution for modern, real-time robotics AI applications.
Read the full abstract ▾
All authors Anshul Ranjan, Anoosh Damodar, Neha Chougule, Dhruva S Nayak, Anantharaman P.N, Shylaja S S
Robotics
Quick Heuristic Validation of Edges in Dynamic Roadmap Graphs
Yulie Arad, Stav Ashur and Nancy M. Amato
In this paper we tackle the problem of adjusting roadmap graphs for
robot motion planning to non-static environments. We introduce the "Red-Green-Gray" paradigm, a modification of the SPITE method, capable of classifying the validity status of nodes and edges using cheap heuristic checks, allowing fast semi-lazy roadmap updates. Given a roadmap, we use simple computational geometry methods to approximate the swept volumes of
robots and perform lazy collision checks, and label a subset of the edges as invalid (red), valid (green), or unknown (gray). We present preliminary experimental results comparing our method to the well-established technique of Leven and Hutchinson, and showing increased accuracy as well as the ability to correctly label edges as invalid while maintaining comparable update runtimes.
Read the full abstract ▾
Robotics
End-to-end example-based sim-to-real RL policy transfer based on neural stylisation with application to robotic cutting
Jamie Hathaway, Alireza Rastegarpanah and Rustam Stolkin
Whereas reinforcement learning has been applied with success to a range of robotic control problems in complex, uncertain environments, reliance on extensive data - typically sourced from simulation environments - limits real-world deployment due to the domain gap between simulated and physical systems, coupled with limited real-world sample availability. We propose a novel method for sim-to-real transfer of reinforcement learning policies, based on a reinterpretation of neural style transfer from image processing to synthesise novel training data from unpaired unlabelled real world
datasets. We employ a variational autoencoder to jointly learn self-supervised feature representations for style transfer and generate weakly paired source-target trajectories to improve physical realism of synthesised trajectories. We demonstrate the application of our approach based on the case study of
robot cutting of unknown materials. Compared to baseline methods, including our previous work, CycleGAN, and conditional variational autoencoder-based time series translation, our approach achieves improved task completion time and behavioural stability with minimal real-world data. Our framework demonstrates robustness to
geometric and material variation, and highlights the feasibility of policy adaptation in challenging contact-rich tasks where real-world reward information is unavailable.
Read the full abstract ▾
Robotics · Cross-listed from cs.AI
MemCtrl: Using MLLMs as Active Memory Controllers on Embodied Agents
Vishnu Sashank Dorbala and Dinesh Manocha
Foundation models rely on in-context learning for personalized decision making. The limited size of this context window necessitates
memory compression and
retrieval systems like RAG. These systems however often treat
memory as large offline storage spaces, which is unfavorable for
embodied agents that are expected to operate under strict
memory and compute constraints, online. In this work, we propose MemCtrl, a novel framework that uses
Multimodal Large
Language Models (MLLMs) for
pruning memory online. MemCtrl augments MLLMs with a trainable
memory head \mu that acts as a gate to determine which observations or reflections to retain, update, or discard during exploration. We evaluate with training two types of \mu, 1) via an offline expert, and 2) via online RL, and observe significant improvement in overall
embodied task completion ability on \mu-augmented MLLMs. In particular, on augmenting two low performing MLLMs with MemCtrl on multiple subsets of the EmbodiedBench
benchmark, we observe that \mu-augmented MLLMs show an improvement of around 16% on average, with over 20% on specific instruction subsets. Finally, we present a qualitative analysis on the
memory fragments collected by \mu, noting the superior performance of \mu augmented MLLMs on long and complex instruction types.
Read the full abstract ▾
Robotics
A Methodology for Designing Knowledge-Driven Missions for Robots
Guillermo GP-Lenza, Carmen DR.Pita-Romero, Miguel Fernandez-Cortizas et al. (+1 more)
This paper presents a comprehensive methodology for implementing knowledge graphs in ROS 2 systems, aiming to enhance the efficiency and intelligence of autonomous robotic missions. The methodology encompasses several key steps: defining initial and target conditions, structuring tasks and subtasks, planning their sequence, representing task-related data in a knowledge graph, and designing the mission using a high-level
language. Each step builds on the previous one to ensure a cohesive process from initial setup to final execution. A practical implementation within the Aerostack2 framework is demonstrated through a simulated search and rescue mission in a Gazebo environment, where drones autonomously locate a target. This implementation highlights the effectiveness of the methodology in improving decision-making and mission performance by leveraging knowledge graphs.
Read the full abstract ▾
All authors Guillermo GP-Lenza, Carmen DR.Pita-Romero, Miguel Fernandez-Cortizas, Pascual Campoy
Robotics
Learning From a Steady Hand: A Weakly Supervised Agent for Robot Assistance under Microscopy
Huanyu Tian, Martin Huber, Lingyun Zeng et al. (+7 more)
This paper rethinks steady-hand robotic manipulation by using a weakly supervised framework that fuses calibration-aware perception with admittance control. Unlike conventional automation that relies on labor-intensive 2D labeling, our framework leverages reusable warm-up trajectories to extract implicit spatial information, thereby achieving calibration-aware, depth-resolved perception without the need for external fiducials or frame-by-frame manual depth annotation. By explicitly characterizing residuals from observation and calibration models, the system establishes a task-space error budget from recorded warm-ups. Under the evaluated laboratory conditions, the recorded trajectories yielded an empirical lateral error level of approximately 49{\mu}m and a conservative axial error estimate of approximately 291 {\mu}m during large in-plane moves. In a within-subject user study (N=8), the learned
agent reduces overall NASA-TLX workload by 77.1% relative to the simple steady-hand assistance baseline. Together, these results demonstrate successful closed-loop micromanipulation and reduced subjective workload under the evaluated laboratory conditions, without external fiducials or frame-by-frame manual depth annotation, supporting the proposed approach as a framework for microscope-guided robotic assistance.
Read the full abstract ▾
All authors Huanyu Tian, Martin Huber, Lingyun Zeng, Zhe Han, Wayne Bennett, Giuseppe Silvestri, Gerardo Mendizabal-Ruiz, Tom Vercauteren, Alejandro Chavez-Badiola, Christos Bergeles
Robotics · Cross-listed from cs.CV
Li-ViP3D++: Query-Gated Deformable Camera-LiDAR Fusion for End-to-End Perception and Trajectory Prediction
Matej Halinkovic, Nina Masarykova, Alexey Vinel et al. (+1 more)
End-to-end perception and trajectory prediction from raw sensor data is one of the key capabilities for autonomous driving. Modular pipelines restrict information flow and can amplify upstream errors. Recent query-based, fully differentiable perception-and-prediction (PnP) models mitigate these issues, yet the complementarity of cameras and
LiDAR in the query-space has not been sufficiently explored. Models often rely on fusion schemes that introduce heuristic
alignment and discrete selection steps which prevent full utilization of available information and can introduce unwanted bias. We propose Li-ViP3D++, a query-based
multimodal PnP framework that introduces Query-Gated Deformable Fusion (QGDF) to integrate multi-view RGB and
LiDAR in query space. QGDF (i) aggregates image evidence via masked
attention across cameras and feature levels, (ii) extracts
LiDAR context through fully differentiable BEV sampling with learned per-query offsets, and (iii) applies query-conditioned gating to adaptively weight visual and
geometric cues per
agent. The resulting architecture jointly optimizes detection, tracking, and multi-hypothesis trajectory
forecasting in a single end-to-end model. On nuScenes, Li-ViP3D++ improves end-to-end behavior and detection quality, achieving higher EPA (0.335) and mAP (0.502) while substantially reducing false positives (FP ratio 0.147), and it is faster than the prior Li-ViP3D variant (139.82 ms vs. 145.91 ms). These results indicate that query-space, fully differentiable camera-
LiDAR fusion can increase robustness of end-to-end PnP without sacrificing deployability.
Read the full abstract ▾
All authors Matej Halinkovic, Nina Masarykova, Alexey Vinel, Marek Galinski
Robotics
OGPO: One-Step Generative Policy Optimization for Real-Time Robot Control
Guowei Zou, Haitao Wang, Hejun Wu et al. (+3 more)
Real-time
robot control demands fast action generation.
Diffusion and flow matching policies for
robot control require multi-step sampling, limiting their deployment in real-time scenarios. Natively reducing the sampling steps to one sacrifices representation quality and task performance, creating a trilemma among speed, fidelity, and performance. We present One-Step Generative Policy Optimization (OGPO), a systematic framework to resolve this trilemma. OGPO first pairs a lightweight architecture with the interval velocity principle for
distillation-free one-step inference, while representation spreading prevents representation quality degradation. It then performs on-policy reinforcement learning (RL) fine-tuning on this fast, stable policy to break the imitation learning ceiling. Experiments on RoboMimic and OpenAI Gym
benchmarks show that OGPO matches or exceeds multi-step baselines while achieving a 5-20 times inference speedup and over 120Hz control frequency. Physical deployment on a Franka-Emika-Panda
robot validates real-world applicability. Project page: https://ogpo-project.github.io/
Read the full abstract ▾
All authors Guowei Zou, Haitao Wang, Hejun Wu, Yukun Qian, Yuhang Wang, Weibing Li
Robotics
Tendon-based modelling, estimation and control for a simulated high-DoF anthropomorphic hand model
Péter Polcz, Katalin Schäffer and Miklós Koller
Tendon-driven anthropomorphic robotic hands often lack direct joint angle sensing, as the integration of joint encoders can compromise mechanical compactness and dexterity. This paper presents a computational method for estimating joint positions from measured tendon displacements and tensions. An efficient kinematic modeling framework for anthropomorphic hands is first introduced based on the Denavit-Hartenberg convention. Using a simplified tendon model, a system of nonlinear equations relating tendon states to joint positions is derived and solved via a nonlinear optimization approach. The estimated joint angles are then employed for closed-loop control through a Jacobian-based proportional-integral (PI) controller augmented with a feedforward term, enabling gesture tracking without direct joint sensing. The effectiveness and limitations of the proposed estimation and control framework are demonstrated in the MuJoCo simulation environment using the Anatomically Correct Biomechatronic Hand, featuring five degrees of freedom for each long finger and six degrees of freedom for the thumb.
Read the full abstract ▾
Robotics
GPO: Growing Policy Optimization for Legged Robot Locomotion and Whole-Body Control
Shuhao Liao, Peizhuo Li, Xinrong Yang et al. (+7 more)
Training reinforcement learning (RL) policies for legged
robots remains challenging due to high-dimensional continuous actions, hardware constraints, and limited exploration. Existing methods for locomotion and whole-body control work well for position-based control with environment-specific heuristics (e.g., reward shaping, curriculum design, and manual initialization), but are less effective for torque-based control, where sufficiently exploring the action space and obtaining informative gradient signals for training is significantly more difficult. We introduce Growing Policy Optimization (GPO), a training framework that applies a time-varying action transformation to restrict the effective action space in the early stage, thereby encouraging more effective data collection and policy learning, and then progressively expands it to enhance exploration and achieve higher expected return. We prove that this transformation preserves the PPO update rule and introduces only bounded, vanishing gradient distortion, thereby ensuring stable training. We evaluate GPO on both quadruped and hexapod
robots, including zero-shot deployment of simulation-trained policies on hardware. Policies trained with GPO consistently achieve better performance. These results suggest that GPO provides a general, environment-agnostic optimization framework for learning legged locomotion.
Read the full abstract ▾
All authors Shuhao Liao, Peizhuo Li, Xinrong Yang, Linnan Chang, Zhaoxin Fan, Qing Wang, Lei Shi, Yuhong Cao, Wenjun Wu, Guillaume Sartoretti
Robotics
MeCo: Enhancing LLM-Empowered Multi-Robot Collaboration via Similar Task Memoization
Baiqing Wang, Helei Cui, Bo Zhang et al. (+3 more)
Multi-
robot systems have been widely deployed in real-world applications, providing significant improvements in efficiency and reductions in labor costs. However, most existing multi-
robot collaboration methods rely on extensive task-specific training, which limits their adaptability to new or diverse scenarios. Recent research leverages the
language understanding and
reasoning capabilities of large
language models (LLMs) to enable more flexible collaboration without specialized training. Yet, current LLM-empowered approaches remain inefficient: when confronted with identical or similar tasks, they must replan from scratch because they omit task-level similarities. To address this limitation, we propose MeCo, a similarity-aware multi-
robot collaboration framework that applies the principle of “cache and reuse” (a.k.a., memoization) to reduce redundant computation. Unlike simple task repetition, identifying and reusing solutions for similar but not identical tasks is far more challenging, particularly in multi-
robot settings. To this end, MeCo introduces a new similarity testing method that retrieves previously solved tasks with high relevance, enabling effective plan reuse without re-invoking LLMs. Furthermore, we present MeCoBench, the first
benchmark designed to evaluate performance on similar-task collaboration scenarios. Experimental results show that MeCo substantially reduces planning costs and improves success rates compared with state-of-the-art approaches.
Read the full abstract ▾
All authors Baiqing Wang, Helei Cui, Bo Zhang, Xiaolong Zheng, Bin Guo, Zhiwen Yu
Robotics
Vibro-Sense: Robust Vibration-based Impulse Response Localization and Trajectory Tracking for Robotic Hands
Wadhah Zai El Amri and Nicolás Navarro-Guerrero
Rich contact perception is crucial for robotic manipulation, yet traditional tactile skins remain expensive and complex to integrate. This paper presents a scalable alternative: high-accuracy whole-body touch localization via vibro-acoustic sensing. By equipping a robotic hand with seven low-cost piezoelectric microphones and leveraging an Audio Spectrogram
Transformer, we decode the vibrational signatures generated during physical interaction. Extensive evaluation across stationary and dynamic tasks reveals a localization error of under 5 mm in static conditions. Furthermore, our analysis highlights the distinct influence of material properties: stiff materials (e.g., metal) excel in impulse response localization due to sharp, high-bandwidth responses, whereas textured materials (e.g., wood) provide superior friction-based features for trajectory tracking. The system demonstrates robustness to the
robot's own motion, maintaining effective tracking even during active operation. Our primary contribution is demonstrating that complex physical contact dynamics can be effectively decoded from simple vibrational signals, offering a viable pathway to widespread, affordable contact perception in robotics. To accelerate research, we provide our full
datasets, models, and experimental setups as open-source resources.
Read the full abstract ▾
Robotics
A Practical Framework of Key Performance Indicators for Multi-Robot Lunar and Planetary Field Tests
Julia Richter, David Oberacker, Gabriela Ligeza et al. (+10 more)
Robotic prospecting for critical resources on the Moon, such as ilmenite, rare earth elements, and water ice, requires robust exploration methods given the diverse terrain and harsh environmental conditions. Although numerous analog field trials address these goals, comparing their results remains challenging because of differences in
robot platforms and experimental setups. These missions typically assess performance using selected, scenario-specific engineering metrics that fail to establish a clear link between field performance and science-driven objectives. In this paper, we address this gap by deriving a structured framework of KPI from three realistic multi-
robot lunar scenarios reflecting scientific objectives and operational constraints. Our framework emphasizes scenario-dependent priorities in efficiency, robustness, and precision, and is explicitly designed for practical applicability in field deployments. We validated the framework in a multi-
robot field test and found it practical and easy to apply for efficiency- and robustness-related KPI, whereas precision-oriented KPI require reliable ground-truth data that is not always feasible to obtain in outdoor analog environments. Overall, we propose this framework as a common evaluation standard enabling consistent, goal-oriented comparison of multi-
robot field trials and supporting systematic development of robotic systems for future planetary exploration.
Read the full abstract ▾
All authors Julia Richter, David Oberacker, Gabriela Ligeza, Valentin T. Bickel, Philip Arm, William Talbot, Marvin Grosse Besselmann, Florian Kehl, Tristan Schnell, Hendrik Kolvenbach, Rüdiger Dillmann, Arne Roennau, Marco Hutter
Robotics
STORM: Slot-based Task-aware Object-centric Representation for robotic Manipulation
Alexandre Chapin, Emmanuel Dellandréa and Liming Chen
Visual foundation models provide strong perceptual features for robotics, but their dense representations lack explicit object-level structure, limiting robustness and controllability in manipulation tasks. We propose STORM (Slot-based Task-aware Object-centric Representation for robotic Manipulation), a lightweight object-centric adaptation module that augments frozen visual foundation models with a small set of task-aware slots for robotic manipulation. Rather than fully tuning large backbones on the task, STORM employs an efficient two-stage training strategy: few layers of object-centric representation are first trained on top of the frozen backbone through visual–semantic pretraining using
language embeddings, then jointly adapted with a downstream manipulation policy for task alignement. This staged learning prevents degenerate slot formation and preserves semantic consistency while aligning perception with task objectives. Experiments on object discovery
benchmarks and robotic manipulation tasks show that STORM improves control performance and generalization to visual shifts (distractors, textures, lighting) compared to directly using frozen or fine-tuned foundation model features, or existing object-centric representations. STORM serves not only as an efficient mechanism for refining generic foundation model features, but also as a novel way of injecting beneficial structural and semantic bias into policy learning.
Read the full abstract ▾
Robotics
RF-MatID: Dataset and Benchmark for Radio Frequency Material Identification
Xinyan Chen, Qinchun Li, Ruiqin Ma et al. (+3 more)
Accurate material identification plays a crucial role in
embodied AI systems, enabling a wide range of applications. However, current
vision-based solutions are limited by the inherent constraints of optical sensors, while
radio-frequency (RF) approaches, which can reveal intrinsic material properties, have received growing
attention. Despite this progress, RF-based material identification remains hindered by the lack of large-scale public
datasets and the limited benchmarking of learning-based approaches. In this work, we present RF-MatID, the first open-source, large-scale, wide-band, and geometry-diverse RF
dataset for fine-grained material identification. RF-MatID includes 16 fine-grained categories grouped into 5 superclasses, spanning a broad frequency range from 4 to 43.5 GHz, and comprises 142k samples in both frequency- and time-domain representations. The
dataset systematically incorporates controlled geometry perturbations, including variations in incidence angle and stand-off distance. We further establish a multi-setting, multi-protocol
benchmark by evaluating state-of-the-art deep learning models, assessing both in-distribution performance and out-of-distribution robustness under cross-angle and cross-distance shifts. The 5 frequency-allocation protocols enable systematic frequency- and region-level analysis, thereby facilitating real-world deployment. RF-MatID aims to enable reproducible research, accelerate algorithmic advancement, foster cross-domain robustness, and support the development of real-world application in RF-based material identification.
Read the full abstract ▾
All authors Xinyan Chen, Qinchun Li, Ruiqin Ma, Jiaqi Bai, Li Yi, Jianfei Yang
Robotics
Demonstration-Free Robotic Control via LLM Agents
Brian Y. Tsui, Alan Y. Fang and Tiffany J. Hwu
Robotic manipulation has increasingly adopted
vision-
language-action (VLA) models, which achieve strong performance but typically require task-specific demonstrations and fine-tuning, and often generalize poorly under domain shift. We investigate whether general-purpose large
language model (LLM)
agent frameworks, originally developed for software engineering, can serve as an alternative control paradigm for
embodied manipulation. We introduce FAEA (Frontier
Agent as
Embodied Agent), which applies an LLM
agent framework directly to
embodied manipulation without modification. Using the same iterative
reasoning that enables software
agents to debug code, FAEA enables
embodied agents to reason through manipulation strategies. We evaluate an unmodified frontier
agent, Claude
Agent SDK, across the LIBERO, ManiSkill3, and MetaWorld
benchmarks. With privileged environment state access, FAEA achieves success rates of 84.9%, 85.7%, and 96%, respectively. This level of task success approaches that of VLA models trained with less than 100 demonstrations per task, without requiring demonstrations or fine-tuning. With one round of human feedback as an optional optimization, performance increases to 88.2% on LIBERO. This demonstration-free capability has immediate practical value: FAEA can autonomously explore novel scenarios in simulation and generate successful trajectories for training data augmentation in
embodied learning. Our results indicate that general-purpose
agents are sufficient for a class of manipulation tasks dominated by deliberative, task-level planning. This opens a path for robotics systems to leverage actively maintained
agent infrastructure and benefit directly from ongoing advances in frontier models. Code is available at https://github.com/robiemusketeer/faea-sim
Read the full abstract ▾
Robotics
TaF-VLA: Tactile-Force Alignment in Vision-Language-Action Models for Force-aware Manipulation
Yuzhe Huang, Pei Lin, Wanlin Li et al. (+5 more)
Vision-
Language-Action (VLA) models have recently emerged as powerful generalists for robotic manipulation. However, due to their predominant reliance on visual modalities, they fundamentally lack the physical intuition required for contact-rich tasks that require precise force regulation and physical
reasoning. Existing attempts to incorporate
vision-based tactile sensing into VLA models typically treat tactile inputs as auxiliary visual textures, thereby overlooking the underlying correlation between surface deformation and interaction dynamics. To bridge this gap, we propose a paradigm shift from tactile-
vision alignment to tactile-force
alignment. Here, we introduce TaF-VLA, a framework that explicitly grounds high-dimensional tactile observations in physical interaction forces. To facilitate this, we develop an automated tactile-force data acquisition device and curate the TaF-
Dataset, comprising over 10 million synchronized tactile observations, 6-axis force/torque, and
matrix force map. To align sequential tactile observations with interaction forces, the central component of our approach is the Tactile-Force Adapter (TaF-Adapter), a tactile sensor encoder that extracts discretized latent information for encoding tactile observations. This mechanism ensures that the learned representations capture history-dependent, noise-insensitive physical dynamics rather than static visual textures. Finally, we integrate this force-aligned encoder into a VLA backbone. Extensive real-world experiments demonstrate that TaF-VLA policy significantly outperforms state-of-the-art tactile-
vision-aligned and
vision-only baselines on contact-rich tasks, verifying its ability to achieve robust, force-aware manipulation through cross-modal physical
reasoning.
Read the full abstract ▾
All authors Yuzhe Huang, Pei Lin, Wanlin Li, Daohan Li, Jiajun Li, Jiaming Jiang, Chenxi Xiao, Ziyuan Jiao
Robotics
Shallow-{\pi}: Knowledge Distillation for Flow-based VLAs
Boseong Jeon, Yunho Choi and Taehan Kim
The growing demand for real-time robotic deployment necessitates fast and on-device inference for
vision-
language-action (VLA) models. Within the VLA literature, efficiency has been extensively studied at the
token level, such as visual
token pruning. In contrast, systematic
transformer layer reduction has received limited
attention and, to the best of our knowledge, has not been explored for flow-based VLA models under knowledge
distillation. In this work, we propose Shallow-pi, a principled knowledge
distillation framework that aggressively reduces the
transformer depth of both the VLM backbone and the flow-based action head, compressing the model from 18 to 6 layers. Shallow-pi achieves over two times faster inference with less than one percent absolute drop in success rate on standard manipulation
benchmarks, establishing state-of-the-art performance among reduced VLA models. Crucially, we validate our approach through industrial-scale real-world experiments on Jetson Orin and Jetson Thor across multiple
robot platforms, including
humanoid systems, in complex and dynamic manipulation scenarios.
Read the full abstract ▾
Robotics
TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance
Zhemeng Zhang, Jiahua Ma, Xincheng Yang et al. (+13 more)
Fine-grained and contact-rich manipulation remain challenging for
robots, largely due to the underutilization of tactile feedback. To address this, we introduce TouchGuide, a novel cross-policy visuo-tactile fusion paradigm that fuses modalities within a low-dimensional action space. Specifically, TouchGuide operates in two stages to guide a pre-trained
diffusion or flow-matching visuomotor policy at inference time. First, the policy produces a coarse, visually-plausible action using only visual inputs during early sampling. Second, a task-specific Contact Physical Model (CPM) provides tactile guidance to steer and refine the action, ensuring it aligns with realistic physical contact conditions. Trained through contrastive learning on limited expert demonstrations, the CPM provides a tactile-informed feasibility score to steer the sampling process toward refined actions that satisfy physical contact constraints. Furthermore, to facilitate TouchGuide training with high-quality and cost-effective data, we introduce TacUMI, a data collection system. TacUMI achieves a favorable trade-off between precision and affordability; by leveraging rigid fingertips, it obtains direct tactile feedback, thereby enabling the collection of reliable tactile data. Extensive experiments on five challenging contact-rich tasks, such as shoe lacing and chip handover, show that TouchGuide consistently and significantly outperforms state-of-the-art visuo-tactile policies.
Read the full abstract ▾
All authors Zhemeng Zhang, Jiahua Ma, Xincheng Yang, Xin Wen, Yuzhi Zhang, Boyan Li, Yiran Qin, Jin Liu, Can Zhao, Li Kang, Haoqin Hong, Zhenfei Yin, Philip Torr, Hao Su, Ruimao Zhang, Daolin Ma
Robotics
TRACER: Texture-Robust Affordance Chain-of-Thought for Deformable-Object Region Grounding
Wanjun Jia, Kang Li, Fan Yang et al. (+7 more)
The central challenge in robotic manipulation of deformable objects lies in aligning high-level semantic instructions with physical interaction points under complex appearance and texture variations. Existing
vision-based affordance prediction methods often suffer from boundary overflow and fragmented functional regions. To address these issues, we propose TRACER, a Texture-Robust Affordance Chain-of-Thought for Deformable-Object Region Grounding framework that maps hierarchical semantic
reasoning to appearance-robust and physically consistent interaction regions. Specifically, a Tree-structured Affordance Chain-of-Thought (TA-CoT) decomposes high-level task intentions into hierarchical affordance-semantic instructions. A Spatially-Constrained Boundary Refinement (SCBR) mechanism suppresses prediction spillover and guides responses toward valid object regions. Furthermore, an Interactive Convergent Refinement Flow (ICRF) refines dispersed affordance responses into coherent regions, improving spatial continuity and physical plausibility. Experiments on the Fine-AGDDO15
dataset and a real-world robotic platform demonstrate that TRACER improves affordance grounding precision across diverse textures and patterns. It also enhances long-horizon manipulation success, bridging high-level semantic
reasoning and low-level physical execution. The source code and
dataset will be made publicly available at https://github.com/Dikay1/TRACER.
Read the full abstract ▾
All authors Wanjun Jia, Kang Li, Fan Yang, Mengfei Duan, Wenrui Chen, Yiming Jiang, Hui Zhang, Kailun Yang, Zhiyong Li, Yaonan Wang
Robotics
A Taylor Series Approach to Correct Localization Errors in Robotic Field Mapping using Gaussian Processes
Muzaffar Qureshi, Tochukwu Elijah Ogri, Kyle Volle et al. (+1 more)
Gaussian Processes (GPs) are powerful non-parametric
Bayesian models for regression of scalar fields, formulated under the assumption that measurement locations are perfectly known and the corresponding field measurements have Gaussian noise. However, many real-world scalar field mapping applications rely on sensor-equipped mobile
robots to collect field measurements, where imperfect localization introduces state uncertainty. Such discrepancies between the estimated and true measurement locations degrade GP mean and covariance estimates. To address this challenge, we propose a method for updating the GP models when improved estimates become available. Leveraging the differentiability of the kernel function, a second-order correction
algorithm is developed using the precomputed Jacobians and Hessians of the GP mean and covariance functions for real-time refinement based on measurement location discrepancy data. Simulation results demonstrate improved prediction accuracy and computational efficiency compared to full model retraining.
Read the full abstract ▾
All authors Muzaffar Qureshi, Tochukwu Elijah Ogri, Kyle Volle, Rushikesh Kamalapurkar