Robotics · Cross-listed from cs.CV
Sign Language Recognition using Parallel Bidirectional Reservoir Computing
Nitin Kumar Singh, Arie Rachmad Syulistyo, Yuichiro Tanaka et al. (+1 more)
Sign
language recognition (SLR) facilitates communication between deaf and hearing communities. Deep learning based SLR models are commonly used but require extensive computational resources, making them unsuitable for deployment on edge devices. To address these limitations, we propose a lightweight SLR system that combines parallel bidirectional reservoir computing (PBRC) with MediaPipe. MediaPipe enables real-time hand tracking and precise extraction of hand joint coordinates, which serve as input features for the PBRC architecture. The proposed PBRC architecture consists of two echo state network (ESN) based bidirectional reservoir computing (BRC) modules arranged in parallel to capture temporal dependencies, thereby creating a rich feature representation for classification. We trained our PBRC-based SLR system on the Word-Level American Sign
Language (WLASL)
video dataset, achieving top-1, top-5, and top-10 accuracies of 60.85%, 85.86%, and 91.74%, respectively. Training time was significantly reduced to 18.67 seconds due to the intrinsic properties of reservoir computing, compared to over 55 minutes for deep learning based methods such as Bi-GRU. This approach offers a lightweight, cost-effective solution for real-time SLR on edge devices.
Read the full abstract ▾
All authors Nitin Kumar Singh, Arie Rachmad Syulistyo, Yuichiro Tanaka, Hakaru Tamukoh
Robotics · Cross-listed from math.NA
Mixed formulation and structure-preserving discretization of Cosserat rod dynamics in a port-Hamiltonian framework
Philipp L. Kinon, Simon R. Eugster and Peter Betsch
An energy-based modeling framework for the nonlinear dynamics of spatial Cosserat rods undergoing large displacements and rotations is proposed. The mixed formulation features independent displacement, velocity and stress variables and is further objective and locking-free. Finite rotations are represented using a director formulation that avoids singularities and yields a constant mass
matrix. This results in an infinite-dimensional nonlinear port-
Hamiltonian (PH) system governed by partial differential-algebraic equations with a quadratic energy functional. Using a time-differentiated compliance form of the stress-strain relations allows for the imposition of kinematic constraints, such as inextensibility or shear-
rigidity. A structure-preserving finite element discretization leads to a finite-dimensional system with PH structure, thus facilitating the design of an energy-momentum consistent integration scheme. Dissipative material behavior (via the generalized-Maxwell model) and non-standard actuation approaches (via pneumatic chambers or tendons) integrate naturally into the framework. As illustrated by selected numerical examples, the present framework establishes a new approach to energy-momentum consistent formulations in computational mechanics involving finite rotations.
Read the full abstract ▾
Robotics
Real2Edit2Real: Generating Robotic Demonstrations via a 3D Control Interface
Yujie Zhao, Hongwei Fan, Di Chen et al. (+5 more)
Recent progress in
robot learning has been driven by large-scale
datasets and powerful visuomotor policy architectures, yet policy robustness remains limited by the substantial cost of collecting diverse demonstrations, particularly for spatial generalization in manipulation tasks. To reduce repetitive data collection, we present Real2Edit2Real, a framework that generates new demonstrations by bridging 3D editability with 2D visual data through a 3D control interface. Our approach first reconstructs scene geometry from multi-view RGB observations with a metric-scale 3D reconstruction model. Based on the reconstructed geometry, we perform depth-reliable 3D editing on point clouds to generate new manipulation trajectories while geometrically correcting the
robot poses to recover physically consistent depth, which serves as a reliable condition for synthesizing new demonstrations. Finally, we propose a multi-conditional
video generation model guided by depth as the primary control signal, together with action, edge, and ray maps, to synthesize spatially augmented multi-view manipulation
videos. Experiments on four real-world manipulation tasks demonstrate that policies trained on data generated from only 1-5 source demonstrations can match or outperform those trained on 50 real-world demonstrations, improving data efficiency by up to 10-50x. Moreover, experimental results on height and texture editing demonstrate the framework's flexibility and extensibility, indicating its potential to serve as a unified data generation framework. Project website is https://real2edit2real.github.io/.
Read the full abstract ▾
All authors Yujie Zhao, Hongwei Fan, Di Chen, Shengcong Chen, Liliang Chen, Xiaoqi Li, Guanghui Ren, Hao Dong
Robotics
TwinAligner: Visual-Dynamic Alignment Empowers Physics-aware Real2Sim2Real for Robotic Manipulation
Hongwei Fan, Hang Dai, Jiyao Zhang et al. (+7 more)
The robotics field is evolving towards data-driven, end-to-end learning, inspired by
multimodal large models. However, reliance on expensive real-world data limits progress. Simulators offer cost-effective alternatives, but the gap between simulation and reality challenges effective policy transfer. This paper introduces TwinAligner, a novel Real2Sim2Real system that addresses both visual and dynamic gaps. The visual
alignment module achieves pixel-level
alignment through SDF reconstruction and editable 3DGS rendering, while the dynamic
alignment module ensures dynamic consistency by identifying rigid physics from
robot-object interaction. TwinAligner improves
robot learning by providing scalable data collection and establishing a trustworthy iterative cycle, accelerating
algorithm development. Quantitative evaluations highlight TwinAligner's strong capabilities in visual and dynamic real-to-sim
alignment. This system enables policies trained in simulation to achieve strong zero-shot generalization to the real world. The high consistency between real-world and simulated policy performance underscores TwinAligner's potential to advance scalable
robot learning. Code and data will be released on https://twin-aligner.github.io
Read the full abstract ▾
All authors Hongwei Fan, Hang Dai, Jiyao Zhang, Jinzhou Li, Qiyang Yan, Yujie Zhao, Mingju Gao, Jinghang Wu, Hao Tang, Hao Dong
Robotics
OMP: One-step Meanflow Policy with Directional Alignment
Han Fang, Yize Huang, Yuheng Zhao et al. (+3 more)
Robot manipulation has increasingly adopted data-driven generative policy frameworks, yet the field faces a persistent trade-off:
diffusion models suffer from high inference latency, while flow-based methods often require complex architectural constraints. Although in image generation domain, the MeanFlow paradigm offers a path to single-step inference, its direct application to robotics is impeded by critical theoretical pathologies, specifically spectral bias and gradient starvation in low-velocity regimes. To overcome these limitations, we propose the One-step MeanFlow Policy (OMP), a novel framework designed for high-fidelity, real-time manipulation. We introduce a lightweight directional
alignment mechanism to explicitly synchronize predicted velocities with true mean velocities. Furthermore, we implement a Differential Derivation Equation (DDE) to approximate the Jacobian-Vector Product (JVP) operator, which decouples forward and backward passes to significantly reduce
memory complexity. Extensive experiments on the Adroit and Meta-World
benchmarks demonstrate that OMP outperforms state-of-the-art methods in success rate and trajectory accuracy, particularly in high-precision tasks, while retaining the efficiency of single-step generation.
Read the full abstract ▾
All authors Han Fang, Yize Huang, Yuheng Zhao, Paul Weng, Xiao Li, Yutong Ban
Robotics
Comparison and Evaluation of Different Simulation Environments for Rigid Body Systems
Longxiang Shao, Ulrich Dahmen and Juergen Rossmann
Rigid body dynamics simulators are important tools for the design, analysis and optimization of mechanical systems in a variety of technical and scientific applications. This study examines four different simulation environments (Adams, Simscape, OpenModelica, and VEROSIM), focusing in particular on the comparison of the modeling methods, the numerical solvers, and the treatment of numerical problems that arise especially in closed-loop kinematics (esp. redundant boundary conditions and static
equilibrium problem). A novel and complex crane boom of a real forestry machine serves as a practical
benchmark application example. The direct comparison of the different solution approaches in the examined simulation tools supports the user in selecting the most suitable tool for his application.
Read the full abstract ▾
Robotics
Are All Data Necessary? Efficient Data Pruning for Large-scale Autonomous Driving Dataset via Trajectory Entropy Maximization
Zhaoyang Liu, Weitao Zhou, Junze Wen et al. (+4 more)
Collecting large-scale naturalistic driving data is essential for training robust autonomous driving planners. However, real-world
datasets often contain a substantial amount of repetitive and low-value samples, which lead to excessive storage costs and bring limited benefits to policy learning. To address this issue, we propose an information-theoretic data
pruning method that effectively reduces the training data volume without compromising model performance. Our approach evaluates the trajectory distribution information
entropy of driving data and iteratively selects high-value samples that preserve the statistical characteristics of the original
dataset in a model-agnostic manner. From a theoretical perspective, we show that maximizing trajectory
entropy effectively constrains the Kullback-Leibler divergence between the pruned subset and the original data distribution, thereby maintaining generalization ability. Comprehensive experiments on the NuPlan
benchmark with a large-scale imitation learning framework demonstrate that the proposed method can reduce the
dataset size by up to 40% while maintaining closed-loop performance. This work provides a lightweight and theoretically grounded approach for scalable data management and efficient policy learning in autonomous driving systems.
Read the full abstract ▾
All authors Zhaoyang Liu, Weitao Zhou, Junze Wen, Cheng Jing, Qian Cheng, Kun Jiang, Diange Yang
Robotics
Translating Flow to Policy via Hindsight Online Imitation
Yitian Zheng, Zhangchen Ye, Weijun Dong et al. (+5 more)
Recent advances in hierarchical
robot systems leverage a high-level planner to propose task plans and a low-level policy to generate
robot actions. This design allows training the planner on action-free or even non-
robot data sources (e.g.,
videos), providing transferable high-level guidance. Nevertheless, grounding these high-level plans into executable actions remains challenging, especially with the limited availability of high-quality
robot data. To this end, we propose to improve the low-level policy through online interactions. Specifically, our approach collects online rollouts, retrospectively annotates the corresponding high-level goals from achieved outcomes, and aggregates these hindsight-relabeled experiences to update a goal-conditioned imitation policy. Our method, Hindsight Flow-conditioned Online Imitation (HinFlow), instantiates this idea with 2D point flows as the high-level planner. Across diverse manipulation tasks in both simulation and physical world, our method achieves more than $2\times$ performance improvement over the base policy, significantly outperforming the existing methods. Moreover, our framework enables policy acquisition from planners trained on cross-embodiment
video data, demonstrating its potential for scalable and transferable
robot learning.
Read the full abstract ▾
All authors Yitian Zheng, Zhangchen Ye, Weijun Dong, Shengjie Wang, Yuyang Liu, Chongjie Zhang, Chuan Wen, Yang Gao
Robotics · Cross-listed from eess.SY
Vision-Aided Relative State Estimation for Approach and Landing on a Moving Platform with Inertial Measurements
Tarek Bouazza, Alessandro Melis, Soulaimane Berkane et al. (+2 more)
This paper tackles the problem of estimating the relative position, orientation, and velocity between a
UAV and a planar platform undergoing arbitrary 3D motion during approach and landing. The estimation relies on measurements from Inertial Measurement Units (IMUs) mounted on both systems, assuming there is a suitable communication channel to exchange data, together with visual information provided by an onboard
monocular camera, from which the bearing (line-of-sight direction) to the platform's center and the normal vector of its planar surface are extracted. We propose a cascade observer with a complementary filter on $\mathbf{SO}(3)$ to reconstruct the relative attitude, followed by a linear Riccati observer for relative position and velocity estimation. Convergence of both observers is established under persistently exciting conditions, and the cascade is shown to be almost globally asymptotically and locally exponentially stable. We further extend the design to the case where the platform's rotation is restricted to its normal axis and show that its measured linear acceleration can be exploited to recover the remaining unobservable rotation angle. A sufficient condition for local exponential convergence in this setting is provided. The proposed observers are validated through extensive simulations.
Read the full abstract ▾
All authors Tarek Bouazza, Alessandro Melis, Soulaimane Berkane, Robert Mahony, Tarek Hamel
Robotics
Vision-Language-Policy Model for Dynamic Robot Task Planning
Jin Wang, Kim Tien Ly, Jacques Cloete et al. (+3 more)
Bridging the gap between natural
language commands and autonomous execution in unstructured environments remains an open challenge for robotics. This requires
robots to perceive and reason over the current task scene through multiple modalities, and to plan their behaviors to achieve their intended goals. Traditional robotic task-planning approaches often struggle to bridge low-level execution with high-level task
reasoning, and cannot dynamically update task strategies when instructions change during execution, which ultimately limits their versatility and adaptability to new tasks. In this work, we propose a novel
language model-based framework for dynamic
robot task planning. Our
Vision-
Language-Policy (VLP) model, based on a
vision-
language model fine-tuned on real-world data, can interpret semantic instructions and integrate
reasoning over the current task scene to generate behavior policies that control the
robot to accomplish the task. Moreover, it can dynamically adjust the task strategy in response to changes in the task, enabling flexible adaptation to evolving task requirements. Experiments conducted with different
robots and a variety of real-world tasks show that the trained model can efficiently adapt to novel scenarios and dynamically update its policy, demonstrating strong planning autonomy and cross-embodiment generalization.
Videos: https://robovlp.github.io/
Read the full abstract ▾
All authors Jin Wang, Kim Tien Ly, Jacques Cloete, Jin Jin, Nikos Tsagarakis, Ioannis Havoutis
Robotics
A Flexible Field-Based Policy Learning Framework for Diverse Robotic Systems and Sensors
Jose Gustavo Buenaventura Carreon, Floris Erich, Roman Mykhailyshyn et al. (+3 more)
We present a cross
robot visuomotor learning framework that integrates
diffusion policy based control with 3D semantic scene representations from D3Fields to enable category level generalization in manipulation. Its modular design supports diverse
robot camera configurations including UR5 arms with Microsoft Azure Kinect arrays and bimanual manipulators with Intel RealSense sensors through a low latency control stack and intuitive teleoperation. A unified configuration layer enables seamless switching between setups for flexible data collection training and evaluation. In a grasp and lift block task the framework achieved an 80 percent success rate after only 100 demonstration episodes demonstrating robust skill transfer between platforms and sensing modalities. This design paves the way for scalable real world studies in cross robotic generalization.
Read the full abstract ▾
All authors Jose Gustavo Buenaventura Carreon, Floris Erich, Roman Mykhailyshyn, Tomohiro Motoda, Ryo Hanai, Yukiyasu Domae
Robotics
WorldRFT: Latent World Model Planning with Reinforcement Fine-Tuning for Autonomous Driving
Pengxuan Yang, Ben Lu, Zhongpu Xia et al. (+7 more)
Latent World Models enhance scene representation through temporal self-supervised learning, presenting a perception annotation-free paradigm for end-to-end autonomous driving. However, the reconstruction-oriented representation learning tangles perception with planning tasks, leading to suboptimal optimization for planning. To address this challenge, we propose WorldRFT, a planning-oriented latent world model framework that aligns scene representation learning with planning via a hierarchical planning decomposition and local-aware interactive refinement mechanism, augmented by reinforcement learning fine-tuning (RFT) to enhance safety-critical policy performance. Specifically, WorldRFT integrates a
vision-geometry foundation model to improve 3D spatial awareness, employs hierarchical planning task decomposition to guide representation optimization, and utilizes local-aware iterative refinement to derive a planning-oriented driving policy. Furthermore, we introduce Group Relative Policy Optimization (GRPO), which applies trajectory Gaussianization and collision-aware rewards to fine-tune the driving policy, yielding systematic improvements in safety. WorldRFT achieves state-of-the-art (SOTA) performance on both open-loop nuScenes and closed-loop NavSim
benchmarks. On nuScenes, it reduces collision rates by 83% (0.30% -> 0.05%). On NavSim, using camera-only sensors input, it attains competitive performance with the
LiDAR-based SOTA method DiffusionDrive (87.8 vs. 88.1 PDMS).
Read the full abstract ▾
All authors Pengxuan Yang, Ben Lu, Zhongpu Xia, Chao Han, Yinfeng Gao, Teng Zhang, Kun Zhan, XianPeng Lang, Yupeng Zheng, Qichao Zhang
Robotics
CoDrone: Autonomous Drone Navigation Assisted by Edge and Cloud Foundation Models
Pengyu Chen, Tao Ouyang, Ke Luo et al. (+2 more)
Autonomous
navigation for Unmanned Aerial Vehicles faces key challenges from limited onboard computational resources, which restrict deployed deep neural networks to shallow architectures incapable of handling complex environments. Offloading tasks to remote edge servers introduces high latency, creating an inherent trade-off in system design. To address these limitations, we propose CoDrone - the first cloud-edge-end collaborative computing framework integrating foundation models into autonomous
UAV cruising scenarios - effectively leveraging foundation models to enhance performance of resource-constrained unmanned aerial vehicle platforms. To reduce onboard computation and data transmission overhead, CoDrone employs grayscale imagery for the
navigation model. When enhanced environmental perception is required, CoDrone leverages the edge-assisted foundation model Depth Anything V2 for depth estimation and introduces a novel one-dimensional occupancy grid-based
navigation method - enabling fine-grained scene understanding while advancing efficiency and representational simplicity of autonomous
navigation. A key component of CoDrone is a Deep Reinforcement Learning-based neural scheduler that seamlessly integrates depth estimation with autonomous
navigation decisions, enabling real-time adaptation to dynamic environments. Furthermore, the framework introduces a
UAV-specific
vision language interaction module incorporating domain-tailored low-level flight primitives to enable effective interaction between the cloud foundation model and the
UAV. The introduction of VLM enhances open-set
reasoning capabilities in complex unseen scenarios. Experimental results show CoDrone outperforms baseline methods under varying flight speeds and network conditions, achieving a 40% increase in average flight distance and a 5% improvement in average Quality of
Navigation.
Read the full abstract ▾
All authors Pengyu Chen, Tao Ouyang, Ke Luo, Weijie Hong, Xu Chen
Robotics
EGM: Efficiently Learning General Motion Tracking Policy for High Dynamic Humanoid Whole-Body Control
Chao Yang, Yingkai Sun, Peng Ye et al. (+3 more)
Learning a general motion tracking policy from human motions shows great potential for versatile
humanoid whole-body control. Conventional approaches are not only inefficient in data utilization and training processes but also exhibit limited performance when tracking highly dynamic motions. To address these challenges, we propose EGM, a framework that enables efficient learning of a general motion tracking policy. EGM integrates four core designs. Firstly, we introduce a Bin-based Cross-motion Curriculum Adaptive Sampling strategy to dynamically orchestrate the sampling probabilities based on tracking error of each motion bin, eficiently balancing the training process across motions with varying dificulty and durations. The sampled data is then processed by our proposed Composite Decoupled Mixture-of-Experts (CDMoE) architecture, which efficiently enhances the ability to track motions from different distributions by grouping experts separately for upper and lower body and decoupling orthogonal experts from shared experts to separately handle dedicated features and general features. Central to our approach is a key insight we identified: for training a general motion tracking policy, data quality and diversity are paramount. Building on these designs, we develop a three-stage curriculum training flow to progressively enhance the policy's robustness against disturbances. Despite training on only 4.08 hours of data, EGM generalized robustly across 49.25 hours of test motions, outperforming baselines on both routine and highly dynamic tasks.
Read the full abstract ▾
All authors Chao Yang, Yingkai Sun, Peng Ye, Xin Chen, Chong Yu, Tao Chen
Robotics
IndoorUAV: Benchmarking Vision-Language UAV Navigation in Continuous Indoor Environments
Xu Liu, Yu Liu, Hanshuo Qiu et al. (+2 more)
Vision-
Language Navigation (VLN) enables
agents to navigate in complex environments by following natural
language instructions grounded in visual observations. Although most existing work has focused on ground-based
robots or outdoor Unmanned Aerial Vehicles (
UAVs), indoor
UAV-based VLN remains underexplored, despite its relevance to real-world applications such as inspection, delivery, and search-and-rescue in confined spaces. To bridge this gap, we introduce IndoorUAV, a novel
benchmark and method specifically tailored for VLN with indoor
UAVs. We begin by curating over 1,000 diverse and structurally rich 3D indoor scenes from the Habitat simulator. Within these environments, we simulate realistic
UAV flight dynamics to collect diverse 3D
navigation trajectories manually, further enriched through data augmentation techniques. Furthermore, we design an automated annotation pipeline to generate natural
language instructions of varying granularity for each trajectory. This process yields over 16,000 high-quality trajectories, comprising the IndoorUAV-VLN subset, which focuses on long-horizon VLN. To support short-horizon planning, we segment long trajectories into sub-trajectories by selecting semantically salient keyframes and regenerating concise instructions, forming the IndoorUAV-VLA subset. Finally, we introduce IndoorUAV-
Agent, a novel
navigation model designed for our
benchmark, leveraging task decomposition and
multimodal reasoning. We hope IndoorUAV serves as a valuable resource to advance research on
vision-
language embodied AI in the indoor aerial
navigation domain.
Read the full abstract ▾
All authors Xu Liu, Yu Liu, Hanshuo Qiu, Yang Qirong, Zhouhui Lian
Robotics · Cross-listed from cs.CV
Sihao Lin, Zerui Li, Xunyi Zhao et al. (+10 more)
Despite remarkable progress in
Vision-
Language Navigation (VLN), existing
benchmarks remain confined to fixed, small-scale
datasets with naive physical simulation. These shortcomings limit the insight that the
benchmarks provide into sim-to-real generalization, and create a significant research gap. Furthermore, task fragmentation prevents unified/shared progress in the area, while limited data scales fail to meet the demands of modern LLM-based pretraining. To overcome these limitations, we introduce VLNVerse: a new large-scale, extensible
benchmark designed for Versatile,
Embodied, Realistic Simulation, and Evaluation. VLNVerse redefines VLN as a scalable, full-stack
embodied AI problem. Its Versatile nature unifies previously fragmented tasks into a single framework and provides an extensible toolkit for researchers. Its
Embodied design moves beyond intangible and teleporting "ghost"
agents that support full-kinematics in a Realistic Simulation powered by a robust physics engine. We leverage the scale and diversity of VLNVerse to conduct a comprehensive Evaluation of existing methods, from classic models to MLLM-based
agents. We also propose a novel unified multi-task model capable of addressing all tasks within the
benchmark. VLNVerse aims to narrow the gap between simulated
navigation and real-world generalization, providing the community with a vital tool to boost research towards scalable, general-purpose
embodied locomotion
agents.
Read the full abstract ▾
All authors Sihao Lin, Zerui Li, Xunyi Zhao, Gengze Zhou, Liuyi Wang, Rong Wei, Rui Tang, Juncheng Li, Hanqing Wang, Jiangmiao Pang, Anton van den Hengel, Jiajun Liu, Qi Wu
Robotics · Cross-listed from eess.SP
PalpAid: Multimodal Pneumatic Tactile Sensor for Tissue Palpation
Devi Yuliarti, Ravi Prakash, Hiu Ching Cheung et al. (+3 more)
The tactile properties of tissue, such as elasticity and stiffness, often play an important role in
surgical oncology when identifying tumors and pathological tissue boundaries. Though extremely valuable,
robot-assisted surgery comes at the cost of reduced sensory information to the surgeon, with
vision being the primary. Sensors proposed to overcome this sensory desert are often bulky, complex, and incompatible with the
surgical workflow. We present PalpAid, a
multimodal pneumatic tactile sensor to restore touch in
robot-assisted surgery. PalpAid is equipped with a microphone and pressure sensor, converting contact force into an internal pressure differential. The pressure sensor acts as an event detector, while the acoustic signature assists in tissue identification. We show the design, fabrication, and assembly of sensory units with characterization tests for robustness to use, repetition cycles, and integration with a robotic system. Finally, we demonstrate the sensor's ability to classify 3D-printed hard objects with varying infills and soft ex vivo tissues. We envision PalpAid to be easily retrofitted with existing
surgical/general robotic systems, allowing soft tissue palpation.
Read the full abstract ▾
All authors Devi Yuliarti, Ravi Prakash, Hiu Ching Cheung, Amy Strong, Patrick J. Codd, Shan Lin
Robotics
DTCCL: Disengagement-Triggered Contrastive Continual Learning for Autonomous Bus Planners
Yanding Yang, Weitao Zhou, Jinhai Wang et al. (+8 more)
Autonomous buses run on fixed routes but must operate in open, dynamic urban environments. Disengagement events on these routes are often geographically concentrated and typically arise from planner failures in highly interactive regions. Such policy-level failures are difficult to correct using conventional imitation learning, which easily overfits to sparse disengagement data. To address this issue, this paper presents a Disengagement-Triggered Contrastive Continual Learning (DTCCL) framework that enables autonomous buses to improve planning policies through real-world operation. Each disengagement triggers cloud-based data augmentation that generates positive and negative samples by perturbing surrounding
agents while preserving route context. Contrastive learning refines policy representations to better distinguish safe and unsafe behaviors, and continual updates are applied in a cloud-edge loop without human supervision. Experiments on urban bus routes demonstrate that DTCCL improves overall planning performance by 48.6 percent compared with direct retraining, validating its effectiveness for scalable, closed-loop policy improvement in autonomous public transport.
Read the full abstract ▾
All authors Yanding Yang, Weitao Zhou, Jinhai Wang, Xiaomin Guo, Junze Wen, Xiaolong Liu, Lang Ding, Zheng Fu, Jinyu Miao, Kun Jiang, Diange Yang
Robotics
Affordance RAG: Hierarchical Multimodal Retrieval with Affordance-Aware Embodied Memory for Mobile Manipulation
Ryosuke Korekata, Quanting Xie, Yonatan Bisk et al. (+1 more)
In this study, we address the problem of open-vocabulary mobile manipulation, where a
robot is required to carry a wide range of objects to receptacles based on free-form natural
language instructions. This task is challenging, as it involves understanding visual semantics and the affordance of manipulation actions. To tackle these challenges, we propose Affordance RAG, a zero-shot hierarchical
multimodal retrieval framework that constructs Affordance-Aware
Embodied Memory from pre-explored images. The model retrieves candidate targets based on regional and visual semantics and reranks them with affordance scores, allowing the
robot to identify manipulation options that are likely to be executable in real-world environments. Our method outperformed existing approaches in
retrieval performance for mobile manipulation instruction in large-scale indoor environments. Furthermore, in real-world experiments where the
robot performed mobile manipulation in indoor environments based on free-form instructions, the proposed method achieved a task success rate of 85%, outperforming existing methods in both
retrieval performance and overall task success.
Read the full abstract ▾
All authors Ryosuke Korekata, Quanting Xie, Yonatan Bisk, Komei Sugiura
Robotics
A Framework for Deploying Learning-based Quadruped Loco-Manipulation
Yadong Liu, Jianwei Liu, He Liang et al. (+1 more)
Quadruped mobile manipulators offer strong potential for agile loco-manipulation but remain difficult to control and transfer reliably from simulation to reality. Reinforcement learning (RL) shows promise for whole-body control, yet most frameworks are proprietary and hard to reproduce on real hardware. We present an open pipeline for training, benchmarking, and deploying RL-based controllers on the Unitree B1 quadruped with a Z1 arm. The framework unifies sim-to-sim and sim-to-real transfer through ROS, re-implementing a policy trained in Isaac Gym, extending it to MuJoCo via a hardware abstraction layer, and deploying the same controller on physical hardware. Sim-to-sim experiments expose discrepancies between Isaac Gym and MuJoCo contact models that influence policy behavior, while real-world teleoperated object-picking trials show that coordinated whole-body control extends reach and improves manipulation over floating-base baselines. The pipeline provides a transparent, reproducible foundation for developing and analyzing RL-based loco-manipulation controllers and will be released open source to support future research.
Read the full abstract ▾
All authors Yadong Liu, Jianwei Liu, He Liang, Dimitrios Kanoulas
Robotics · Cross-listed from cs.CV
Point What You Mean: Visually Grounded Instruction Policy
Hang Yu, Juntu Zhao, Yufeng Liu et al. (+9 more)
Vision-
Language-Action (VLA) models align
vision and
language with
embodied control, but their object referring ability remains limited when relying solely on text
prompt, especially in cluttered or out-of-distribution (OOD) scenes. In this study, we introduce the Point-VLA, a plug-and-play policy that augments
language instructions with explicit visual cues (e.g., bounding boxes) to resolve referential ambiguity and enable precise object-level grounding. To efficiently scale visually grounded
datasets, we further develop an automatic data annotation pipeline requiring minimal human effort. We evaluate Point-VLA on diverse real-world referring tasks and observe consistently stronger performance than text-only instruction VLAs, particularly in cluttered or unseen-object scenarios, with robust generalization. These results demonstrate that Point-VLA effectively resolves object referring ambiguity through pixel-level visual grounding, achieving more generalizable
embodied control.
Read the full abstract ▾
All authors Hang Yu, Juntu Zhao, Yufeng Liu, Kaiyu Li, Cheng Ma, Di Zhang, Yingdong Hu, Guang Chen, Junyuan Xie, Junliang Guo, Junqiao Zhao, Yang Gao