Robotics
FoldQuantVLA: Native Low-Bit Quantization of Vision-Language-Action Models via Consistent Folding
Hung T. Ho, Khanh D. Nguyen, Quang D. Nguyen et al. (+5 more)
Low-bit
vision-
language-action inference must reduce observation-to-action latency while preserving
robot behavior. We present FoldQuantVLA, a post-training quantization framework that carries a consistent activation representation through calibration, weight rounding, and native integer execution. It combines channel scaling and block Hadamard transforms with dynamic per-
token quantization, without policy retraining. Custom TensorRT plugins execute projections in both the
language backbone and iterative action expert with four-bit weights and activations (W4A4) on Ada GPUs and Jetson AGX Orin. Evaluation spans LIBERO, SimplerEnv, and two
robot platforms. Across three GR00T checkpoints and $\pi_{0.5}$, W4A4 achieves $1.20$ to $1.33\times$ speedups over floating-point TensorRT on Orin and $1.25$ to $1.52\times$ on desktop. Retaining
language attention-output and feed-forward down projections at eight bits (W8A8) improves held-out action fidelity on all four checkpoints. Across four real-
robot tasks, this configuration raises observed GR00T N1.7 success from $80.0\%$ with uniform W4A4 to $92.5\%$ over 80 trials per configuration, with a measured additional Orin latency of 1 ms.
Read the full abstract ▾
All authors Hung T. Ho, Khanh D. Nguyen, Quang D. Nguyen, Thanh Q. Duong, Ngan Le, Meng Guo, Vien A. Ngo, An T. Le
Robotics
Robotic Valve Turning: Axial Misalignment Correction Using Reaction Torque Feedback
Amit Kumar, Sri Harsha Turlapati, Gautami Golani et al. (+3 more)
In this work, we propose a haptic update control law that uses reaction torques to correct axial misalignment during robotic valve manipulation. Unlike
vision-based estimates, which can be affected by calibration errors, occlusion, and uncertainty in the contact geometry, reaction torques arise directly from the physical interaction between the gripper and valve. A
geometric relationship exists between the error (misalignment) vector and these torques. The primary aim of this work is to propose a stable controller exploiting this
geometric property. Our control law is proven to be uniformly asymptotically stable. Simulations are performed for verification. Furthermore, we experimentally test the robustness of our method using a Kinova Gen3 robotic arm for initial misalignments ranging from $-15^\circ$ to $15^\circ$ at 3 different valve positions and report the resulting data distribution. The absolute value of the median misalignment across all 18 test cases is found to be within $2.46^\circ$ and that of reaction torques within $0.23\mathrm{Nm}$.
Read the full abstract ▾
All authors Amit Kumar, Sri Harsha Turlapati, Gautami Golani, Yang Lin, Ravi N. Banavar, Domenico Campolo
Robotics
Zeva-Ego: Egocentric Mid-Training with In-Context Causal Learning for Robot Manipulation
Bingjia Huang, Xin Ding, Fu Chen et al. (+5 more)
Egocentric
video offers a scalable source of physical interaction experience, yet translating it into
robot-executable knowledge and enabling continual adaptation remain challenging. We introduce Zeva-Ego, a unified framework that learns physical priors from human experience and evolves through
robot interaction. An Action-Centric Encoder (ACE) converts egocentric visual transitions into action-centered supervision for VLA mid-training, while In-Context Causal Learning (ICCL) enables parameter-free adaptation from action-effect feedback at deployment. Scaling Ego data to 10K hours improves RoboTwin success from 63.8% to 75.3%, matching 2K hours of
robot demonstrations (74.7%), corresponding to an empirical data ratio of roughly 4-5:1. With accumulated interaction experience, ICCL further improves success from 58% to 89% within four attempts without parameter updates. These results demonstrate a scalable path toward embodied intelligence that learns from human experience and continuously improves through its own interaction.
Read the full abstract ▾
All authors Bingjia Huang, Xin Ding, Fu Chen, Kun Li, Wei Sun, Hao Wu, Yunxin Liu, Ting Cao
Robotics
Tactile-JEPA: Topology-Aware Self-Supervised Representation Learning for Distributed Tactile Sensors
Elizaveta Kovtun, Matvey Konovalov, Andrey Sakhovskiy et al. (+1 more)
Tactile sensing is an essential modality for
robots performing contact-rich, dexterous manipulation, particularly under visual occlusion. While pre-trained image encoders are standard in
robot learning pipelines, tactile encoders are still commonly trained from scratch from raw, noisy signals, which might limit their expressivity. Existing self-supervised learning (SSL) approaches focus predominantly on
vision-based tactile sensors, leaving distributed electronic skins largely unaddressed. These sensors, however, have a distinctive property: their sensing elements are sparse and irregularly arranged over the surface they cover, which makes direct reuse of visual SSL methods suboptimal. We present Tactile-JEPA, an efficient self-supervised pre-training method that uses the spatial arrangement of tactile sensors to learn
topology-aware representations. Specifically, it is trained to predict the
embeddings of masked sensing elements from the unmasked remainder, using the sensor connectivity graph to guide spatial masking. Our analysis shows that effective tactile representations require capturing both local contact details and the global state of the tactile surface, which we achieve through dual-scale masking. Across three diverse
datasets spanning
magnetic and piezoresistive sensors, different
robot embodiments, and single- and paired-sensor configurations, Tactile-JEPA reduces force estimation error by 6.3% and in-hand orientation error by 20.8% over the prior state-of-the-art, with consistent gains in other downstream applications, including policy learning. Overall, our results demonstrate that the benefit of tactile sensing depends critically on the quality of encoder pre-training, a problem which Tactile-JEPA addresses directly. Code is available at https://github.com/E-Kovtun/tactile.
Read the full abstract ▾
All authors Elizaveta Kovtun, Matvey Konovalov, Andrey Sakhovskiy, Semen Budennyy
Robotics
Multi-Agent Transportation of Free-Flyers in Microgravity Via Pushing Interaction Under Human-in-the-Loop Control
Gregorio Marchesini, Nicola De Carli, Sihyun Cho et al. (+5 more)
We propose a safety-critical framework for the cooperative transportation of passive targets in microgravity, where a team of chaser
robots acts through unilateral pushing contacts to track a human-provided desired twist while ensuring safe target motion. The pushing-only nature of the interaction introduces sparse, configuration-dependent actuation constraints requiring chasers to physically relocate on the target body when the desired pushing allocation changes. To address these challenges, we formulate a delay-aware feedback control architecture leveraging Control Lyapunov Function (CLF) and Control Barrier Function (CBF) constraints within a mixed-integer thrust allocation program to enforce stability and safety of the target, respectively. The proposed framework enables reference tracking while guaranteeing obstacle avoidance with a circular obstacle despite intermittent control authority, providing a foundation for human-supervised cooperative transportation of free-flyers in space environments. The proposed framework is validated through Gazebo simulations.
Read the full abstract ▾
All authors Gregorio Marchesini, Nicola De Carli, Sihyun Cho, Youngkyoung Kong, Elias Krantz, Mani Hemanth Dhullipalla, Dimos V. Dimarogonas, H. Jin Kim
Robotics
LIBERO-VPro: Benchmarking Closed-Loop Visual Robustness of Robotic Foundation Models
Huiqiong Li, Zhiting Mei, Anirudha Majumdar et al. (+3 more)
Robotic foundation models achieve impressive performance on standard manipulation
benchmarks, yet these evaluations typically assume clean, timely, and consistent visual observations throughout execution. We introduce LIBERO-VPro, a
benchmark for systematically evaluating the closed-loop visual robustness of robotic foundation models by perturbing the visual evidence available during execution. LIBERO-VPro covers four complementary dimensions, including Visual Evidence Degradation, Camera Staleness, Visual Source Consistency, and Task-Relevant Scene Variation, spanning 12 challenge categories, 96 experimental settings, and 3,296 task-condition cases. We evaluate three
vision-
language-action models and three world-action models over approximately 196,000 simulated episodes, complemented by 200 real-world rollouts on a Franka Research 3. Our results reveal that strong nominal performance can mask substantial weaknesses in visual grounding and adaptation. Models often remain successful despite severe object-level occlusion, yet degrade sharply when local interaction cues are disrupted or familiar spatial priors are violated. They are also highly sensitive to stale or missing observations and struggle when changed task preconditions require behavioral adaptation. Finally, VLAs and WAMs exhibit distinct robustness profiles, showing that visual robustness is multi-dimensional and architecture-dependent. LIBERO-VPro provides a systematic diagnostic framework for developing robotic foundation models that can more reliably ground and adapt their actions under challenging visual conditions.
Read the full abstract ▾
All authors Huiqiong Li, Zhiting Mei, Anirudha Majumdar, Jingjing Chen, Yu-Gang Jiang, Bin Zhu
Robotics
Performance-Preserving Online Adaptation in Social Navigation via Diffusion Steering
Haruto Nagahisa, Kohei Matsumoto, Yuki Hyodo et al. (+1 more)
In social navigation, modeling the complex interactions between humans and
robots is difficult, and deep reinforcement learning has therefore been actively studied. However, because simulation alone cannot fully reproduce diverse scenarios,
robot dynamics, and the social conventions that vary across deployment environments, fine-tuning in the deployment environment is promising. In doing so, learning that preserves the base model's performance is required, so as not to compromise the primary objective of navigation, namely avoiding pedestrians and reaching the destination. In this study, we propose a method that applies
diffusion steering via reinforcement learning (DSRL), which trains only the noise policy while keeping the
diffusion policy fixed, thereby achieving learning that preserves performance. Furthermore, we integrate
diffusion-based RL policies trained with multiple seeds to construct the base policy, improving learning performance. Our evaluation shows that, compared with other methods, the proposed method enables efficient learning while preserving performance, and we confirm flexible behavior control through adaptation to social conventions, as well as its effectiveness on a physical
robot through hardware-in-the-loop simulation.
Read the full abstract ▾
All authors Haruto Nagahisa, Kohei Matsumoto, Yuki Hyodo, Ryo Kurazume
Robotics · Cross-listed from cs.CV
AnalogDepth: Multi-view Geometry from FPV drones under Analog Video Transmission
André Amorim and Pedro F. Proença
Analog
video transmission (VTX) remains widespread in FPV drones due to low latency, weight and low cost. However analog VTX suffers from complex spatially structured image degradation which differ fundamentally from digital image corruption (e.g. AWGN) used in standard training augmentation. This work shows that this type of noise severely degrades the accuracy of Depth Anything 3 (DA3), a state-of-the-art feed forward visual geometry foundation model. To address this gap, we present AnalogDepth, a parameter-efficient training pipeline that adapts DA3 to analog FPV imagery using student-teacher knowledge
distillation with Low-Rank Adaptation (LoRA) injected into the DINOv2 backbone. Rather than synthesizing noise analytically, we build a noise bank from static FPV recordings under diverse conditions and compare real-noise injection against PSD-matched Gaussian synthesis and AWGN as baselines. Experiments on six real FPV flight sequences across three indoor scenes show that training with our noise bank consistently reduces per-frame depth RMSE and 3D reconstruction Chamfer distance compared to the pretrained DA3 baseline and both Gaussian noise variants. These results demonstrate that replicating the spatial structure of real analog transmission noise is critical for effective adaptation.
Read the full abstract ▾
Robotics
vla.simd: Efficient CPU Inference for Language-Conditioned Manipulation
Khanh D. Nguyen, Hoang M. Truong and An T. Le
Deploying
language-conditioned manipulation without a dedicated GPU requires efficient inference and action chunks that cover the delay between policy queries. We present vla.simd, a CPU inference engine that combines shared SIMD micro-kernels, reusable computation, and target-specific optimization. We relate query latency and execution horizon to action availability under lagged and time-aligned execution, distinguishing action supply from feedback frequency. Across six policies and four CPUs, vla.simd achieves approximately $1.4\times$ median speedup over compiled PyTorch references while preserving fp32 numerical fidelity. We also introduce IMPACT, an ACT-based policy with cached text representations and
language-modulated visual features. IMPACT is the only
language-conditioned policy in our evaluated set that supplies at least 30 actions/s on the Raspberry Pi 5: after a 90 s thermal soak, it supplies 33.5 actions/s in fp32 and 81.2 with int8. Separate GPU evaluations yield $76.4\%$ mean success across four LIBERO suites without
robot pretraining; instruction-shuffling tests demonstrate selection among familiar goals. Trials with IMPACT on an SO-101 arm and SmolVLA on a UR10e with a Robotiq gripper demonstrate CPU deployment on two
robot embodiments.
Read the full abstract ▾
Robotics
ME-Brain-1.0: Memory, Cognition and Action for Evolving Embodied Intelligence
Wei He, Hengtao Li, Chenfeng Wang et al. (+21 more)
Current embodied systems largely rely on pretrained capabilities that remain fixed after deployment, limiting their ability to learn from physical interaction. We introduce MachEmbodied-
Brain (ME-
Brain), a self-evolving embodied system organized around a closed loop of action execution, experience acquisition, experience
evolution, and improved execution. Evolvable
Memory consolidates
multimodal trajectories into hierarchical, reusable experience; Cognitive Core transforms physical experience into transferable skills; and the Action Model combines event-driven keyframes, EventCell local-world prediction, and action-conditioned
memory modulation to focus computation on decision-critical moments, regions, and historical evidence. Together, these modules shift embodied intelligence from train-and-freeze to deploy-and-evolve without model retraining. Cognitive Core outperforms the strongest comparison models by 8.2 and 9.6 points on embodied and
agent benchmarks. The Action Model achieves 47.88% mean success on RoboMME, a 3.26-point improvement over the strongest baseline. On RoboDojo, it reaches a 21.51 mean Score and 16.03% success rate, exceeding $\pi_{0.5}$ by 10.10 and 9.12 points. On the six-task ME-RealBench, ME-
Brain achieves a 69.5 mean Score and 66.7% success rate, outperforming DM0.5 by 12.8 and 11.7 points, respectively.
Read the full abstract ▾
All authors Wei He, Hengtao Li, Chenfeng Wang, Zhongrui Yu, Xuhan Zhu, Maokui He, Zide Liu, Xiyue Zhang, Xianwei Mao, Chunpeng Zhou, Jia Shi, Yanze Xin, Jingwen Li, Jingxie Zheng, Sijie Zeng, Fan Lu, Zeyu Zhang, Shuai Guo, Hengxuan Zhang, Pengfei Yu, Jia Shi, Yu Liu, Kun Zhan, Yan Xie
Robotics
OpenFlyScan: A Quality-Guided Aerial Reconstruction System for Consumer Drones
Zhongrui You, Zhen Li, Junli Liu et al. (+2 more)
3D Gaussian Splatting (3DGS) provides high-fidelity scenes for large-scale embodied simulation, but constructing large-scale urban assets remains constrained by expensive equipment and delayed quality feedback. Preset surveys can leave complex surfaces insufficiently observed, with defects discovered only after reconstruction, requiring return visits and repeated processing. We present OpenFlyScan, a quality-guided aerial reconstruction system for consumer drones that integrates a GS quality model, a reacquisition planner, and a custom-designed mobile app. The model learns from GS rendering errors to predict regional reconstruction quality. Based on these predictions, the planner then generates complementary reacquisition strips to be executed through the app, which also supports automated oblique surveys and data transfer without additional hardware on board. Across real aerial scenes, the model effectively identifies regions that are likely to be poorly reconstructed. In the Expo West field experiment, targeted reacquisition improves PSNR at additional views by 10.95 dB. With consumer drones, OpenFlyScan integrates capture, targeted reacquisition, and reconstruction to support rapid, low-cost urban asset creation. Code and models will be made publicly available at https://openflyscan.github.io/.
Read the full abstract ▾
All authors Zhongrui You, Zhen Li, Junli Liu, Zhigang Wang, Bin Zhao
Robotics
Audio-based UAV Localization with Adaptive Temporal Correspondence via Reinforcement Learning
Haoxiang Lei, Mingzheng Feng, Daotong Wang et al. (+1 more)
Audio-based localization provides a low-cost and illumination-independent sensing solution for anti-
UAV early warning. However, existing methods typically rely on a predefined fixed audio segment length, which limits temporal correspondence and creates a trade-off between sufficient acoustic evidence and timely localization. To address this issue, we propose an audio-based localization framework with adaptive temporal correspondence. A probe segment is first used to extract a compact acoustic state that characterizes the reliability and consistency of the observation. Guided by the state, a reinforcement learning controller dynamically determines the required audio window size for each localization decision. The selected audio segment is then processed by a Mamba-based localization network with adaptive temporal feature modulation for 3D position estimation. Extensive experiments demonstrate that our method achieves competitive 3D localization accuracy with substantially reduced temporal correspondence latency compared to SOTA methods and exhibits strong generalization across scenarios.
Read the full abstract ▾
All authors Haoxiang Lei, Mingzheng Feng, Daotong Wang, Shenghai Yuan
Robotics
Odometry-Aided Real-Time Mapping for Underwater Robots Using Forward-Looking Sonar
Siyuan Du, Kanzhong Yao, Youdong Wang et al. (+5 more)
Reliable perception is essential for underwater vehicles operating in complex environments, where light attenuation and scattering often degrade visibility and compromise optical sensing. Forward-looking sonar (FLS) offers an alternative by providing high-frame-rate acoustic imaging under poor optical conditions. However, real-time FLS mapping remains challenging due to unresolved target elevation, spatially non-uniform noise, and fragmented target boundaries, which hinder feature extraction and introduce
geometric ambiguity during projection. To address these challenges, we propose a cascaded feature reconstruction pipeline combining fast
Fourier transform (FFT)-based denoising, fast multiscale constant false alarm rate (MCFAR) detection, and gradient-adaptive boundary connection to extract
geometric features from degraded sonar images with low latency. We integrate attitude-aware
geometric projection with incremental occupancy accumulation to construct a depth-referenced 2.5D map for local mapping in confined underwater environments. The sonar's vertical position is referenced to an external sensor, while target elevation is assigned under an explicit
geometric assumption rather than measured directly by FLS. Experiments in a 3 m X 5 m pool demonstrate centimeter-scale planar mapping accuracy, with a root-mean-square error (RMSE) below 3 cm across three sequences and an average processing time of 42.4 ms per frame.
Read the full abstract ▾
All authors Siyuan Du, Kanzhong Yao, Youdong Wang, Yingqi Liu, Qingwen Liu, Qunhui Yang, Zhe Sun, Xuelong Li
Robotics
A Topological Representation with Object-Path Graphs for Open-Vocabulary Instance Navigation
Linwei Zheng, Daojie Peng, Bingtao Wang et al. (+2 more)
Vision-
language navigation requires embodied
agents to navigate environments using natural
language instructions and visual observations. Existing approaches typically decompose navigation into sequential
language-guided decisions or rely on online exploration without prior environmental knowledge. Scene graph representations offer compact semantic
memory but remain decoupled from downstream navigation, which still depends on dense metric maps. To close this gap, we propose an object–path graph that unifies open-vocabulary semantic
reasoning with topological navigation. The proposed representation jointly supports semantic grounding, graph-based localization, and navigation within a single lightweight topological framework. Building on this graph, we introduce a navigation strategy that combines global path planning with local inter-node execution through lightweight node localization and semantic visual servoing, enabling navigation directly over the graph without dense metric reconstruction. Experiments on HM3D and Replica demonstrate competitive performance in open-vocabulary object grounding through the proposed hierarchical graph structure, while achieving effective navigation performance. Real-world
robot experiments further validate the practicality of the proposed framework.
Read the full abstract ▾
All authors Linwei Zheng, Daojie Peng, Bingtao Wang, Haoang Li, Jun Ma
Robotics
StenoVLA-3D: 3D-Aware Reasoning VLA for Navigation Through Gastrointestinal Stenoses
Tamima Tabassum, Yiming Huang, Tianchun Wu et al. (+7 more)
Autonomous endoscopic navigation requires the policy model to predict actions from texture-poor monocular observations, make safe control decisions, and retain evidence of lesions after they leave the field of view. Existing
vision-
language-action (VLA) models primarily rely on visual appearance and short-term context, limiting
geometric grounding and episode-level reporting. We introduce StenoVLA-3D, a 3D-aware VLA framework for navigating through stenotic regions. We integrate point-maps into the Cosmos-Reason 2 backbone through learned geometry-gated fusion, and also propose a temporal state branch to model traversal progress. Our
reasoning-and-action backbone predicts grounded
reasoning with actions, while dedicated heads estimate stenosis shape and generate the final lesion report. We further introduce EndoCausal, an episode-level
dataset with lesion annotations, actions, and temporally grounded
reasoning. On 40 held-out recorded test episodes, StenoVLA-3D reaches 95.2% semantic accuracy and 83.4% action accuracy. On the physical 3-DoF endoscope, it attains 88.9% and 77.8% task success in esophageal and colonic phantoms (36 trials each), substantially outperforming the evaluated baselines.
Read the full abstract ▾
All authors Tamima Tabassum, Yiming Huang, Tianchun Wu, Changjing Liu, Zhiqing Tang, Chikit Ng, Beilei Cui, Liangjing Shao, Jiewen Lai, Hongliang Ren
Robotics
GraspTune: Tactile-Driven Execution Refinement for Robust Grasping
Juntao Li, Xingke Xia, Sichao Liu et al. (+1 more)
Visual grasp proposal generation has advanced rapidly, yet converting a selected proposal into a stable physical grasp remains a central execution-stage challenge. This paper introduces GraspTune, a tactile-driven execution-stage refinement framework that starts from a nominal proposal and applies bounded residual TCP motions during approach, contact formation, and final grasp execution. GraspTune learns control-facing contact semantics from local depth, tactile signals, state, and history using state-conditioned expert contact queries and multi-task supervision for contact change, contact risk, and post-close readiness. The representation conditions a
diffusion-pretrained residual policy and is aligned with PPO for closed-loop execution. Across more than 60,000 simulated executions over 20 object categories, GraspTune establishes an execution-layer benefit across four proposal generators, raising stable grasp success by +19.22, +9.55, +12.45, and +20.70 percentage points for GraspNet, Contact-GraspNet, AnyGrasp, and VGN. A four-fold held-out category study raises unseen-object execution from 54.58% to 70.33%, showing category-disjoint generalization of contact correction. Across more than 1,000 real-
robot trials on a UR5e setup with Xense fingertip sensors, GraspTune raises GraspNet execution from 71.0% to 84.3%, validating direct transfer without realworld policy fine-tuning. Together, these results turn visually plausible proposals into stable physical grasps for downstream contact-rich manipulation. A supplementary
video is available at https://youtu.be/kcq7fSLNtzU.
Read the full abstract ▾
All authors Juntao Li, Xingke Xia, Sichao Liu, Daqiang Guo
Robotics
Object-Centric Conditioning for Visuomotor Flow Matching
Jijie Li, Xu Yang, Junhong Zou et al. (+4 more)
Robot visuomotor policies are commonly formulated as autoregressive,
diffusion-based, or more recently, flow matching models. Among them, Action-to-Action (A2A) flow matching improves inference efficiency by initializing generation from historical action priors rather than
stochastic noise. However, stale historical motion patterns and entangled global visual representations can jointly reduce robustness under spatial out-of-distribution (OOD) shifts and visual distractors. In this work, we propose SlotFlow, an object-centric flow matching policy for robust visuomotor manipulation. SlotFlow decouples scene observations into semantic ("what") features and lightweight image-plane spatial ("where") cues to provide object-aware policy conditioning and current-state grounding. The semantic representation suppresses irrelevant background correlations, while the spatial cue improves adaptation to shifted object configurations. Extensive simulation and real-world experiments demonstrate improved robustness under visual distractors and severe spatial perturbations while preserving the low-step inference efficiency of A2A. Controlled initialization and perception ablations further identify object-centric grounding as a major source of the gains and show that it complements, rather than replaces, useful historical motion priors.
Read the full abstract ▾
All authors Jijie Li, Xu Yang, Junhong Zou, Chunhai Zhao, Chaoyang Zhao, Zhen Lei, Xiangyu Zhu
Robotics
MimicAgent: Quadruped Skills via Prompt-to-Trajectory Generation
Lucky Kant Nayak, Narayanan Palghat Parameswaran, Neehar Peri et al. (+1 more)
We present MimicAgent, a
prompt-to-trajectory generation framework for learning dynamic quadruped skills. Although reward shaping is extensively used when training quadruped policies, navigating the resulting reward landscape is notoriously difficult, requiring hours of "graduate student descent". Eureka attempts to automate reward design with LLMs, but we find that it struggles to generalize across diverse skills and morphologies. Our key observation is that it is far easier for a human - and by association, an LLM - to generate reference motions than to shape reward functions. Our hypothesis is motivated by the success of example-guided RL for humanoids, which exploits large-scale motion capture
datasets as references for training locomotion policies. Unlike humanoids, quadrupeds lack such reference motion data. Towards this end, we propose MimicAgent, an agentic harness that, given a skill
prompt, generates quadruped reference trajectories with coding
agents. These coarse reference trajectories are then used to train example-guided RL policies that are deployable in simulation and in the real-world. Notably, we find that when prompting Claude Fable 5.1 within our agentic harness, 87% of
prompts yield semantically aligned reference trajectories.
Read the full abstract ▾
All authors Lucky Kant Nayak, Narayanan Palghat Parameswaran, Neehar Peri, Deva Ramanan
Robotics
BayesianGS-SLAM: Uncertainty-Aware Neural Rendering SLAM via Probabilistic Formulation
Kyeongsu Kang, Seongbo Ha, Sibaek Lee et al. (+1 more)
Neural-rendering-based SLAM relies on rendered RGB-D residuals for camera tracking and map optimization, but the reliability of these predictions can vary substantially because of sensor noise, limited observation coverage, and incomplete map representations. Without an explicit reliability estimate, unreliable residuals may adversely affect pose optimization, while frames already well explained by the current map may trigger redundant mapping updates. In this paper, we present BayesianGS-SLAM, an uncertainty-aware 3D Gaussian Splatting SLAM framework that estimates predictive color and depth uncertainty during mapping and consistently reuses it across the SLAM pipeline. Our tractable probabilistic formulation combines a sensor-noise uncertainty component with an opacity-induced map-representation component propagated through the rendering process. The resulting predictive uncertainty is used to augment mapping, normalize tracking residuals through a robust pose objective, and evaluate incoming frames using a predictive-surprise-based keyframe criterion. Unlike prior uncertainty-aware neural-rendering SLAM methods that primarily consider color uncertainty or use uncertainty only during mapping, our framework estimates predictive uncertainty for both color and depth and integrates it into mapping, tracking, and keyframe selection. Evaluations on real-world RGB-D
datasets demonstrate substantially improved depth uncertainty-error ranking compared with existing uncertainty-aware SLAM methods. Moreover, the proposed keyframe-selection strategy reduces the number of selected keyframes and mapping calls while maintaining competitive tracking and rendering performance.
Read the full abstract ▾
All authors Kyeongsu Kang, Seongbo Ha, Sibaek Lee, Hyeonwoo Yu
Robotics
Phrase-Level Robotic Guqin Performance: Bimanual Motion Planning and Audio-Tactile Interaction Monitoring
Zhen Wang, Zhiheng Chen, Tianyuan Bao et al. (+1 more)
Recent advances in humanoid robotics and embodied intelligence have enabled
robots to perform increasingly complex manipulation tasks. However, musical instrument performance remains a formidable
benchmark, demanding not only collision-free trajectory execution but also precise contact timing, asymmetric bimanual coordination, and target acoustic outcomes on physical instruments. The guqin, a seven-string fretless zither, presents unique manipulation challenges due to its millimetric string spacing, transient right-hand plucking, and sustained left-hand harmonic contacts. In this work, we present a physical heterogeneous dual-arm robotic system for phrase-level autonomous guqin performance. We formulate guqin playing as a hybrid discrete–continuous execution problem and develop a hierarchical planning framework that coordinates working finger assignment, configuration continuity, obstacle avoidance, and tight bimanual contact schedules across consecutive musical events. The system integrates
vision-guided instrument localization, tactile-based harmonic contact monitoring, and auditory feedback-informed plucking parameter calibration. Real-world experiments on a 25-event phrase demonstrate that the system reliably executes coordinated open-string and seventh-hui harmonic sequences on a physical guqin, achieving 93.6% and 96.8% event correctness across repeated trials.
Read the full abstract ▾
All authors Zhen Wang, Zhiheng Chen, Tianyuan Bao, Tianwei Zhang
Robotics
ActiveArena: Benchmarking and Understanding Active Perception in Robotic Manipulation
Yibo Li, Enshen Zhou, Rui Chen et al. (+7 more)
Active perception and manipulation are crucial for
robots to interact with complex scenes. Existing
benchmarks struggle to evaluate how
robots effectively acquire and maintain information in
memory in an active manner. To this end, we introduce ActiveArena-Sim, an active-perception simulator with controllable viewpoints and large-scale workspaces as the foundation. Built on this, we propose ActiveArena-Bench, which comprises 35 tasks across 5 fine-grained categories, covering visual exploration and interactive information acquisition. Each task is difficult to solve from passive observations alone, requiring multi-round evidence acquisition and
memory-based
reasoning. The
benchmark provides rich
memory annotations, standardized training data, and ID/OOD protocols featuring disjoint scenes, unseen distractor configurations, and novel backgrounds. Moreover, we present ActiveArena-VLA, a modular suite of 13
vision-
language-action configurations for controlled studies of
memory writing,
memory capacity, proprioceptive state, subtask supervision, and high-level planning in active perception.
Benchmark results reveal a substantial ID-OOD gap: uniform
memory sampling, increased
memory capacity under reliable write policies, proprioceptive inputs, and subtask supervision improve OOD generalization, while planner-guided
memory management and decision-making achieve performance close to the best-performing configuration using only sparse
memory. ActiveArena thus provides a unified testbed to develop and diagnose models for active perception and manipulation.
Read the full abstract ▾
All authors Yibo Li, Enshen Zhou, Rui Chen, Yanjun Ding, Mengzhen Liu, Yi Han, Jiabo Zhan, Lipeng Wang, Shanghang Zhang, Lu Sheng
Robotics
CARE: Experience-Guided Atomic Corrective Execution for Vision-Language-Action Policies
Junlan Xiao, Junwei Jiang, Zaibin Zhang et al. (+4 more)
Vision-
Language-Action (VLA) policies achieve strong performance in robotic manipulation but remain brittle once execution deviates from nominal trajectories. We propose CARE (Corrective Atomic Robotic Execution), a framework that improves recovery by learning from failures encountered during execution. Instead of generating corrective data from manually designed or random perturbations, CARE collects failed rollouts, models stage-conditioned post-failure deviations, and uses the resulting empirical distributions to synthesize representative failure states and corrective demonstrations. At inference time, CARE combines stage-wise planning with physically grounded 3D monitoring to trigger atomic adjustments or re-operations while preserving task progress. We further introduce the Failure State Recovery
Benchmark (FSR-Bench), which evaluates recovery from intermediate failure states under local deviations and structural anomalies. Experiments across multiple VLA backbones, simulation
benchmarks, and real-world dual-arm tasks show consistent improvements, with average task-success gains of 14.5 points in simulation and 15.9 points in the real world. Code, models, and data are available at https://github.com/xiaojunlan/care
Read the full abstract ▾
All authors Junlan Xiao, Junwei Jiang, Zaibin Zhang, Yifan Wang, Zhongbo Zhang, Huchuan Lu, Lijun Wang
Robotics
Ask Before It Tells: Benchmark-to-Robot Body-Cue Transfer for a Question-First Bedside Robot
Dongsik Yoon
Body-cue recognition can support assistive
robots, but
benchmark accuracy does not guarantee reliable behavior under a
robot-camera viewpoint. We present Nuni, a bedside
robot prototype that treats a detected distress cue as a reason to ask rather than a reason to alert. We compare two X3D-UGT RGB appearance classifiers, which reach 97.7% and 94.8% six-way accuracy on NTU RGB+D, with a pose-centric hybrid pipeline on 28 single-actor scripted clips recorded from the
robot camera. The hybrid path achieved 0.71 six-way macro recall, versus 0.25 and 0.29 for the fine-tuned and from-scratch RGB variants. More importantly for interaction, it produced a question-triggering distress cue in 12/16 distress clips and would have prompted unnecessarily in 2/8 normal clips; the RGB variants yielded a question-triggering cue in only 2/16 and 3/16 distress clips. We separately tested the question-first controller through event injection. All 13 state-transition trials passed: valid responses caused stand-down, two unanswered
prompts produced one alert, and three boundary conditions were handled correctly. These results are a preliminary technical evaluation, not a user study or medical validation, but they show how interaction policy can limit the consequences of uncertain perception.
Read the full abstract ▾
Robotics
Dexterous Robot Manipulation from Human Demonstrations via Contact-Anchored Retargeting and Residual Policy Learning
Zihao Yang, Chengyuan Liu, Yu Zhou et al. (+6 more)
Learning dexterous manipulation from demonstrations is bottlenecked by data: the contact forces that determine whether a grasp succeeds are absent from every scalable source of human demonstrations. This paper builds on two observations. First, what survives the change from a human hand to a
robot hand is the contact structure of a demonstration - which finger regions touch which object locations, and in what order - rather than its joint motion. Second, physical consistency need not be engineered per task: a single residual reinforcement learning (RL) policy, trained once across diverse demonstrations, can repair kinematic recordings into physically consistent, contact-annotated trajectories, and the same residual formulation restores dynamic feasibility after retargeting. These observations yield a three-stage pipeline that converts human motion-capture recordings into dexterous
robot policies with no real-
robot training data: physics refinement with a simulated MANO hand recovers contacts and forces, contact-anchored retargeting transfers the demonstrated contact structure through an objective independent of hand morphology, and residual policy learning adapts the result to
robot actuation. The pipeline reconstructs 25,454 single-hand trajectories (success 7.3% -> 59.3%) and 25 dual-hand tasks (16.0% -> 62.4%) with one shared policy per setting, transfers one human
dataset to four morphologically distinct
robot hands (+62.4 pp), and executes four contact-rich bimanual tasks on physical hardware with zero real-
robot training data.
Read the full abstract ▾
All authors Zihao Yang, Chengyuan Liu, Yu Zhou, Runze Lv, Tianyu Cui, Sheng Yi, Haohua Zhu, Irvine Lu, JieQ Sun
Robotics
When Does Touch Matter? Charting the Vision-Interaction Gap in Cluttered Dexterous Grasping
Hao Jiang, Luis Dominguez and Daniel Seita
Dexterous grasping in clutter poses a basic sensing question: when do tactile measurements and external wrench estimates improve on visual geometry? Occlusion and contact can obscure grasp quality, motivating a controlled evaluation of these interaction signals. We present a controlled real-world study over five tabletop scene conditions on a dexterous hand (LEAP Hand) with
vision, per-finger and wrist wrench estimates, and distributed fingertip taxels. With demonstrations, visual observations, action space, and compliant control fixed, we compare
diffusion policies trained by imitation learning on
vision only, wrench, taxels, or all combined, plus representation and fusion baselines. The combined policy succeeds in 24/25 trials versus 14/25 for
vision only, and 15/15 versus 6/15 across the three confined conditions. Ablations show that wrench and taxel feedback are complementary. Behavioral comparisons show that interaction feedback enables earlier rejection of inadequate contacts, regrasping before lift, and more stable grasps. To our knowledge, this is the first real-world study to combine and separately evaluate these interaction modalities for target-oriented dexterous grasping in clutter. These results chart a widening
vision-interaction gap and position cluttered dexterous grasping as a
benchmark for determining when the learned policy needs interaction sensing. Project website: https://interaction-dex-grasp.github.io/
Read the full abstract ▾
Robotics
Safety Control of a Hyper-redundant Robot via Adaptive Weighted Control Barrier Functions
Zijian Cai, Kiwan Wong, Wenci Xin et al. (+3 more)
Hyper-redundant
robots are well suited for confined-space manipulation due to their high dexterity, but safe operation in cluttered environments remains challenging. In addition, their slender structures often lead to uneven load distributions and nonuniform tracking errors along the body. To address these issues, this work proposes a weighted control barrier functions (W-CBFs) framework that enforces safety constraints while reducing tracking errors caused by uneven loading. The proposed controller was first evaluated on a circular path-following task under different obstacle configurations. With fixed weights, compared to the non-weighted method, the maximum reduction in root-mean-square (RMS) tracking error was 59.6% in simulation and 87.7% in physical experiments. An adaptive weighting strategy was then investigated based on the discrepancy between simulated and experimental performance under different mapping functions. The RMS errors were further reduced by 21.9% and 8.5%, respectively, although the error increases when obstacles were located close to the
robot body. Finally, the
robot was evaluated in a cleaning task requiring coverage of a rectangular area and compared with manual teleoperation. Although the controller was not explicitly optimized for area coverage, the autonomous strategy achieved comparable or better coverage performance while avoiding collisions with the surrounding frame, whereas collisions occurred during manual operation.
Read the full abstract ▾
All authors Zijian Cai, Kiwan Wong, Wenci Xin, Wei Xiao, Daniela Rus, Cecilia Laschi
Robotics
Automatic Labelling for Bimanual Mobile Manipulation
Yupu Lu and Jia Pan
Semantically meaningful subtask labels can provide useful contexts for long-horizon policies, but automatically identifying both reliable temporal boundaries and broad semantic descriptions for annotations remains difficult. We present an automatic labelling pipeline that assigns temporal localisation to deterministic trajectory analysis and semantic interpretation to
vision-
language (VL)
reasoning. The pipeline segments synchronised kinematic signals into phases, performs phase-localised VL
reasoning to describe the contents, and aggregates the outputs for the base, left arm, and right arm actions. We evaluate this pipeline primarily on 29 real Galaxea bimanual mobile-manipulation tasks. Repeating the VL
reasoning three times first produces the same output value for 87.4% on selected tasks. A review by nine participants across all 29 tasks then judgements on the labelled phases and shows positive acceptance of temporal divisions (90.5%), body labels (90.7%), and arm labels (78.7%). The results indicate that the segmentation-VL design can produce structured annotations while preserving asynchronous bimanual behaviour, providing a basis for richer semantic subtask identification and state-based verification.
Read the full abstract ▾
Robotics
Toward Human-in-the-Loop Robot Failure Recovery: Bridging Communication Gaps in Human-Robot Collaboration
Promise Ekpo, Teju Vijay, Dhruv Mandalik et al. (+5 more)
Robots can recover from failures by asking bystanders for help, but effective human-in-the-loop recovery requires communication that accounts for differences in people's knowledge. Prior inverse-semantics work generates requests using a single listener model, leaving differences in listener knowledge untested. We introduce Listener Differences in Human-
Robot Interaction (LD-HRI), a game,
dataset, and
benchmark that evaluates speakers through human listener performance. Our evaluation examines request properties, large
language model (LLM) speakers, and inverse-semantics request-selection
algorithms under controlled differences in listener information. The corpus contains 446 human-written requests and 1{,}302 listener trials. We additionally evaluated 24 frozen LLM-written requests with 70 human listeners across 560 trials. Novice success is descriptively higher with model-written requests across all four tasks, yet both request sources leave substantial expert–novice gaps, including 16 percentage points for LLM requests. LD-HRI makes these gaps measurable, providing a foundation for designing more robust communication in human-
robot and human-
agent interaction.
Read the full abstract ▾
All authors Promise Ekpo, Teju Vijay, Dhruv Mandalik, Tisha Jain, Arman Ibrayeva, Sunishka Sil, Stefanie A. Tellex, Angelique Taylor
Robotics
AquaOrbit: Sim-to-Real Reinforcement Learning for Underwater Target Orbiting under Intermittent Visual Feedback
Kanzhong Yao, Jinyi Leng, Hao Zhang et al. (+2 more)
Intermittent visual loss disrupts target-relative feedback during underwater orbiting, making it difficult to maintain coordinated motion and reacquire a moving target. We present AquaOrbit, a reinforcement-learning controller with a recovery module for underwater target orbiting under interrupted visual feedback. During detection loss, the recovery module uses latched line-of-sight, roll, and depth references to support stabilization and target reacquisition. We train the controller in Isaac Sim with dynamics, observation, and
vision-loss randomization. Evaluated without retraining in Gazebo/ROS2 under a different physics engine and perception perturbations, AquaOrbit completes 20/20 orbiting trials in each of the static- and moving-target conditions on an unseen variable-depth 3-D trajectory. In the moving-target condition, it reduces mean line-of-sight error by approximately 46% relative to a PID-based visual servoing controller with recovery while maintaining comparable path-tracking accuracy; removing the recovery module reduces completion to 9/20. Zero-shot physical deployment with fully onboard perception and control demonstrates elliptical, figure-eight, and variable-depth circular trajectories, including the latter two trajectory types absent from training. The
robot maintains attitude stability during manual occlusions lasting up to 8s and reacquires the target within 2.5s in the reported attitude-induced field-of-view loss events.
Read the full abstract ▾
All authors Kanzhong Yao, Jinyi Leng, Hao Zhang, Zhe Sun, Xuelong Li
Robotics
What Matters in Designing World Action Models: An Empirical Study
Chao Tang, Haoqing Wang, Zilang Cen et al. (+9 more)
World Action Models (WAMs) have emerged as a promising paradigm for generalizable
robot control. Despite the growing number of WAM systems, existing works often introduce unified systems that bundle together multiple design choices, such as architecture and training strategy, making it difficult to isolate individual contributions and systematically compare alternative designs. In this work, we present a controlled study that disentangles these design choices and analyzes not only their empirical effects, but also how and why they shape WAMs. More specifically, we focus on three fundamental questions in building WAMs: (1) what causal structure should govern the interaction between world modeling and action generation? (2) in which latent space should world modeling be performed? and (3) how do different world-action modeling objectives affect model behavior and performance? Through structurally controlled experiments on three representative
benchmarks, RoboCasa-GR1, LIBERO, and LIBERO-Plus, we systematically compare six causal structures, eight latent representations, and four training objectives, covering popular design choices in existing WAMs. We further validate our key findings on real-
robot data from the DROID
dataset. We hope to provide a systematic understanding of how core design choices affect world-action modeling and what principles can guide the development of future WAM systems.
Read the full abstract ▾
All authors Chao Tang, Haoqing Wang, Zilang Cen, Weishi Mi, Wei Xia, Fangcheng Liu, Anda Cheng, Yeqing Shen, Xiaohui Cui, Xiaoyuan Zhang, Yehui Tang, Tingguang Li