Robotics
VISTA: Scale-Aware Visual Navigation via Action History Conditioning
Maeva Guerrier, Koki Kobayashi, Simon Roy et al. (+2 more)
Vision Navigation Foundation Models (VNMs) promise end-to-end learned
navigation policies capable of zero-shot deployment across diverse embodiments and environments. To maintain generality, many
vision-based
navigation models predict normalized actions. However, this normalization introduces a critical deployment vulnerability: applying different scaling factors to the same normalized trajectory alters its physical geometry, which degrades
navigation performance and increases collision risks. We address this vulnerability by conditioning the model on normalized action histories alongside image observations, providing explicit context on the relationship between the model's predictions and the
robot's actual physical displacement. Furthermore, current VNMs often struggle in visually repetitive environments that lack distinct features. To resolve this issue, we integrate a DINOv3 encoder, whose richer representations enable our model to capture both spatial and
geometric dimensions between observations. VISTA generalizes robustly to out-of-distribution environments, achieving 100% goal prediction accuracy in zero-shot, real-world deployment in Outdoor, Forest and Office settings, and an average of 95% checkpoints crossed, demonstrating consistent path following in unseen environments.
Read the full abstract ▾
All authors Maeva Guerrier, Koki Kobayashi, Simon Roy, Jana Pavlasek, Giovanni Beltrame
Robotics
Contrastive Action-Image Pre-training for Visuomotor Control
Yuvan Sharma, Dantong Niu, Anirudh Pai et al. (+16 more)
Existing
vision encoders for robotics face a fundamental bottleneck: robotic
datasets lack the scale necessary for large-scale pre-training. Prior work circumvents this data scarcity by turning to internet-scale image and
language data or
egocentric human
video. While these models show promise, neither paradigm learns from paired
vision and action data, which downstream visuomotor control policies require. However,
robot trajectories, the most direct source of this paired signal, are not available at pre-training scale, motivating us to extract action signals from abundant human
video instead. To this end, we introduce CAIP (Contrastive Action-Image Pre-training), a
vision encoder that treats human hand poses from large-scale
egocentric video as a proxy for end-effector actions. By extracting 3D hand keypoints, a representation that aligns naturally with downstream
robot action spaces, CAIP learns a unified action-image representation through a contrastive objective. Leveraging 32,041 hours of
egocentric human
video and only 88 hours of robotic manipulation data, CAIP outperforms state-of-the-art
vision encoders including DINOv2, SigLIP, MVP, and R3M. Evaluated on a challenging real-world dexterous manipulation setup using Dexmate Vega and Sharpa Wave hands, CAIP yields performance gains of more than 30% on tasks involving folding, pouring, and fine-grained manipulation. Our results show that our method of contrastive action-centric pre-training yields a scalable path to achieving robust visual representations better suited for physical interaction.
Read the full abstract ▾
All authors Yuvan Sharma, Dantong Niu, Anirudh Pai, Zekai Wang, Zhuoyang Liu, Baifeng Shi, Stefano Saravalle, Boning Shao, Ruijie Zheng, Jing Wang, Konstantinos Kallidromitis, Yusuke Kato, Fabio Galasso, Yuke Zhu, Danfei Xu, Linxi "Jim" Fan, Jitendra Malik, Trevor Darrell, Roei Herzig
Robotics · Cross-listed from cs.CV
Beyond Benchmarks: Continuous Edge Inference for Fine-Grained Roadside Perception
Aditya Mishra and Haroon Lone
Continuous AI inference on resource-constrained edge hardware introduces deployment effects that are largely invisible to conventional
benchmark evaluation, including temporal instability in streaming
video, thermal throttling under sustained load, and workload-dependent performance variability. We present Edge-TSR, a deployment-oriented continuous edge inference system for sustained roadside perception on the NVIDIA Jetson Orin Nano. Edge-TSR integrates detection, tracking, fine-grained classification, and a lightweight track-aware temporal stabilization mechanism that improves streaming inference consistency with negligible computational overhead. Our central finding is that
benchmark-centric evaluation systematically overstates deployed edge inference performance. Across three state-of-the-art baselines, we observe consistent 20-30% relative degradation when transitioning from static-image evaluation to real-world streaming deployment. Edge-TSR addresses this gap through temporal inference stabilization, recovering up to 10.16% classification accuracy over per-frame inference baselines while maintaining sustained real-time performance under continuous operation. We evaluate the complete system under diverse real-world deployment conditions, jointly characterizing inference quality, latency, throughput, and thermal behavior during long-duration operation. A 55-minute vehicular deployment over a 26 km route demonstrates sustained operation at 16.18 FPS within safe thermal limits on a single embedded device without cloud offload. Our findings show that deployment-aware evaluation and temporal inference stabilization are necessary components of continuously operating edge AI systems intended for real-world sensing deployments. We release a sample annotated streaming
video evaluation
dataset and full system implementation to support reproducible deployment-centric evaluation.
Read the full abstract ▾
Robotics · Cross-listed from cs.MA
Intermittent Strategic Cooperation of Two Selfish Agents on Graphs
Itay Shedlezki and Noa Agmon
We study strategic space- and time-constrained cooperation between two self-interested
agents through the Intermittent Strategic Cooperation-Based Two-
Agent Path Planning (IC2PP) problem, a shortest-path game on graphs in which
agents navigate toward individual targets while optionally cooperating at specific nodes to reduce their own travel times. Although such cooperation can strictly benefit both
agents, it is strategically fragile:
agents may deviate at any point along their paths. Modeled as a 2-player game, we characterize the structure of Pure Nash
Equilibrium (PNE) joint strategies in IC2PP, and show that stable cooperation must follow a highly constrained form. We further prove that at least one PNE exists in every instance of IC2PP, and present a
polynomial-time
algorithm for enumerating all relevant PNEs. When multiple
equilibria arise, we study coordination mechanisms based on bargaining-theoretic selection concepts and empirically compare
equilibrium outcomes in terms of individual travel times and social welfare.
Read the full abstract ▾
Robotics
ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining
Hao Li, Ganlong Zhao, Yufei Liu et al. (+8 more)
Vision-
Language-Action (VLA) models benefit from large-scale and diverse
embodied data, yet scaling
robot trajectory collection is costly and labor-intensive. Recent advances show that large-scale
egocentric human
videos provide complementary real-world supervision in pretraining. However, joint training on human and
robot data remains challenging due to divergences in action spaces, embodiment structures, temporal dynamics, and supervision quality. We introduce ACE-EGO-0, a unified VLA pretraining framework jointly leveraging heterogeneous data sources. To extract large-scale pretraining supervision from
egocentric human
videos, we build a scalable
egocentric video-to-action pipeline that converts raw human
videos into
robot-format pseudo-action trajectories. To make these labels comparable with
robot demonstrations, ACE-EGO-0 uses a unified action representation based on camera-space actions, morphology conditioning, and time-aligned action chunking. To robustly leverage noisy pseudo-action supervision from
egocentric human
videos, we formulate a reliability-aware training objective with a human auxiliary loss that concentrates supervision on reliable signals. We instantiate ACE-EGO-0 on 4.53K hours of
robot and simulation data, together with 1.48K hours of pseudo-action-labeled
egocentric human data. Experiments show that incorporating large-scale human supervision under reliability-aware weighting consistently improves both unified joint pretraining and supervised fine-tuning. ACE-EGO-0 achieves state-of-the-art performance on RoboCasa GR1 TableTop and RoboTwin 2.0, while demonstrating strong transfer to real-world bimanual manipulation.
Read the full abstract ▾
All authors Hao Li, Ganlong Zhao, Yufei Liu, Haotian Hou, Guoquan Ye, Tongyan Fang, Chunxiao Liu, Siyuan Huang, Jianbo Liu, Xiaogang Wang, Hongsheng Li
Robotics
VL-MemKnG: Hybrid Memory with a Spatio-Temporal Knowledge Graph for Question Answering over Long Egocentric Navigation Trajectories
Svetlana Lukina, Mohamad Al Mdfaa, Gloria Haro et al. (+2 more)
Answering
navigation-relevant questions over long
egocentric videos requires retrieving and organizing evidence distributed across distant temporal moments while maintaining spatial and contextual consistency. Although long-context
vision–
language models can achieve strong answer quality, they are computationally expensive for long trajectories and inefficient for repeated querying. Recent graph-based approaches such as VL-KnG address this challenge through persistent spatio-temporal knowledge graphs, but graph-centric
retrieval alone may underrepresent broader temporal continuity and contextual cues. We present VL-MemKnG, a hybrid
memory framework that extends VL-KnG by combining a spatio-temporal knowledge graph with persistent segment-level contextual
memory. The knowledge graph captures structured relational information and long-range object associations, while segment-level
memory preserves broader temporal context for long-horizon evidence
retrieval. A hybrid
retrieval-and-
reasoning module jointly operates over both
memory representations to produce evidence-grounded answers and temporally organized supporting evidence. We also introduce WalkieKnowledgeT+, an extension of WalkieKnowledge for long-horizon
navigation-oriented
video question answering. The
benchmark includes temporally distributed
reasoning tasks requiring evidence aggregation across multiple non-cooccurring moments. On WalkieKnowledgeT+, VL-MemKnG improves Top-1
retrieval accuracy from 58% to 67% and Recall@1 from 34.50% to 40.55%, outperforming all compared methods, including Gemini 2.5 Pro and Qwen 3.5+. The gains are particularly pronounced on temporal-global and temporally scattered aggregation questions, demonstrating the benefits of combining structured relational
memory with segment-level contextual
memory while maintaining efficient query-time inference.
Read the full abstract ▾
All authors Svetlana Lukina, Mohamad Al Mdfaa, Gloria Haro, Sergey Zagoruyko, Gonzalo Ferrer
Robotics
T-Rex: Tactile-Reactive Dexterous Manipulation
Dantong Niu, Zhuoyang Liu, Zekai Wang et al. (+31 more)
The ability to react dynamically to tactile signals has long been considered crucial to agile human-level dexterity. Yet contemporary learning-based
Vision-
Language-Action (VLA) models for robotic manipulation generally either overlook the tactile modality or are limited to encoders with static cues, due in part to the scarcity of diverse training data and standardized evaluation, architectural constraints in current VLA models, and limitations of static tactile encoders. In this paper, we push the frontier of tactile-reactive manipulation by addressing all of these limitations. We propose a large-scale, 100-hour tactile-rich
dataset collected via a novel, data-efficient recipe that prioritizes elementary motor primitives. To effectively exploit naturally high-frequency touch signals without sacrificing the existing capabilities of existing VLAs, we introduce a variable-rate Mixture-of-
Transformers (MoT) architecture equipped with a novel temporal tactile VQ-VAE encoder. We demonstrate the effectiveness of tactile-reactive policies on 12 manipulation tasks requiring delicate force control and deformable object manipulation, achieving over 30% higher average success rate than the strongest baseline.
Read the full abstract ▾
All authors Dantong Niu, Zhuoyang Liu, Zekai Wang, Boning Shao, Zhao-Heng Yin, Anirudh Pai, Yuvan Sharma, Stefano Saravalle, Ruijie Zheng, Jing Wang, Ryan Punamiya, Mengda Xu, Yuqi Xie, Yunfan Jiang, Letian Fu, Konstantinos Kallidromitis, Matteo Gioia, Junyi Zhang, Jiaxin Ge, Haiwen Feng, Fabio Galasso, Wei Zhan, David M. Chan, Yutong Bai, Roei Herzig, Jiahui Lei, Li Fei-Fei, Ken Goldberg, Jitendra Malik, Pieter Abbeel, Yuke Zhu, Danfei Xu, Linxi Fan, Trevor Darrell
Robotics
Human Universal Grasping
Kevin Yuanbo Wu, Tianxing Zhou, Isaac Tu et al. (+5 more)
Humans can grasp objects effortlessly, whereas multi-fingered
robots are far from this level of generality. We argue that the most natural source of
robot grasping data is from humans, who pick up thousands of objects every day. We present HUG, a flow-matching model that generates diverse human grasps for any user-specified object in a single RGB-D image captured from a stereo camera. Using smart glasses, we first collect 1M-HUGs, an
egocentric dataset of human grasps spanning 1M frames (27.8 hrs) and 6,707 object instances across 41 buildings. Next, to model the distribution of natural human grasps, our novel flow-matching model fuses RGB and depth observations to output a grasp parameterized by wrist translation, wrist rotation, and MANO hand pose. Predicted grasps can be retargeted to various
robot hands, enabling zero-shot grasping in everyday scenes. To standardize evaluation, we build a new simulated
benchmark, HUG-Bench, of 90 unseen objects from five
geometric categories and various sizes, with metric-scale 3D meshes. We evaluate HUG in the real world on the 30-object test set of HUG-Bench across multiple stereo cameras,
robot embodiments, and household environments. HUG outperforms the state-of-the-art grasping baselines by +23% and +34% on our challenging object set. Code, data,
benchmark, checkpoints, and an interactive demo are released on our website: https://grasping.io/
Read the full abstract ▾
All authors Kevin Yuanbo Wu, Tianxing Zhou, Isaac Tu, Billy Yan, Irmak Guzey, David Fouhey, Dandan Shan, Lerrel Pinto
Robotics
Geometric Action Model for Robot Policy Learning
Jisang Han, Seonghu Jeon, Jaewoo Jung et al. (+7 more)
Generalist
robot policies must follow user instructions while
reasoning about how objects, cameras, and
robot actions interact in the 3D physical world. Recent
vision-
language-action models (VLAs) and
video world-action models (WAMs) inherit strong semantic or temporal priors from large-scale foundation models, but they still operate primarily on 2D image frames or 2D-derived latent spaces, leaving implicit the 3D geometry required for contact-rich manipulation. We propose the
Geometric Action Model (GAM), a
language-conditioned manipulation policy that directly repurposes a pretrained
geometric foundation model (GFM) as a shared substrate for perception, temporal prediction, and action decoding. GAM splits the GFM at an intermediate layer: the shallow layers serve as an observation encoder, and a causal future predictor inserted at the split layer forecasts future latent
tokens conditioned on
language, proprioception, and action history. The predicted future
tokens are then routed through the remaining GFM blocks for feature propagation and decoding, allowing a single backbone to produce both future geometry and actions. This design equips the GFM with
language-conditioned temporal world modeling through minimal architectural modification while preserving its rich
geometric priors. Across a broad suite of simulation and real-
robot manipulation
benchmarks, GAM is more accurate, more robust, faster, and lighter than current foundation-model-scale baselines.
Read the full abstract ▾
All authors Jisang Han, Seonghu Jeon, Jaewoo Jung, René Zurbrügg, Honggyu An, Tifanny Portela, Marco Hutter, Marc Pollefeys, Seungryong Kim, Sunghwan Hong
Robotics
Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes
Tongyan Fang, Siyuan Huang, Naiyu Fang et al. (+6 more)
When pretrained VLA policies are fine-tuned through online RL, each rollout episode produces only a single binary outcome (success or failure), yet the actor update requires per-transition supervision. Existing approaches commonly reduce this sparse outcome to a single scalar reward or advantage signal, which conflates distinct forms of transition-level feedback and provides limited guidance once basic task success becomes achievable. First, a single scalar signal conflates the two objectives of viability and efficiency; once basic success is achieved, the binary label provides no gradient to distinguish efficient completions from slow ones. Second, real-world rollouts mix autonomous and intervention segments; naively assigning episode outcomes across these boundaries introduces incorrect credit assignment. To address these issues, we propose Hierarchical Advantage-Weighted Behavior Cloning (HABC), which trains separate critic heads for these two objectives on different data subsets and combines their outputs with a state-adaptive balance. A state-adaptive gate $g_t$ merges their one-step advantages, prioritizing viability when success is uncertain and shifting to efficiency only when viability is high, and converts the result into per-transition weights on the actor loss. Intervention-aware credit assignment further restricts outcome labels to segments executed by the current policy, preventing supervision from leaking across intervention boundaries. In real-
robot experiments on three contact-rich bimanual tasks, HABC raises success from supervised fine-tuning (SFT) baselines of 36%, 44%, and 12% to 92%, 88%, and 38%.
Read the full abstract ▾
All authors Tongyan Fang, Siyuan Huang, Naiyu Fang, Ganlong Zhao, Zhongjin Luo, Jianbo Liu, Xiaogang Wang, Ying Dong, Hongsheng Li
Robotics
R2RDreamer: 3D-aware Data Augmentation for Spatially-generalized 2D Manipulation Policies
Xiuwei Xu, Haowen Sun, Angyuan Ma et al. (+7 more)
Spatial generalization is critical for imitation-learned manipulation policies, but achieving it typically requires scaling demonstrations across diverse object poses,
robot configurations, and camera viewpoints. Data augmentation from a few source demonstrations offers a practical alternative to costly real-world collection. Simulation-based augmentation can create controllable variation, but requires complex environment and object setup and may introduce a sim-to-real gap. Recent real-to-real methods avoid these issues by jointly editing 3D observations and action trajectories from real demonstrations, yet they still rely on strong 3D scene parsing and geometry completion, and often produce observations tailored to 3D pointcloud policies rather than RGB-based 2D policies. We propose R2RDreamer, a real-to-real demonstration augmentation framework that preserves the
geometric consistency of 3D action-observation editing while moving visual completion to 2D
video space. Specifically, R2RDreamer first performs lightweight 3D augmentation by editing incomplete object pointclouds and end-effector trajectories in a shared 3D frame; it then projects the edited scene into masked image-space control
videos with occlusion-aware
reasoning and uses a dense-control image-to-
video model to complete temporally coherent RGB observations. Experiments on spatially shifted manipulation tasks with both 2D
diffusion-style policies and
vision-
language-action policies show that R2RDreamer improves spatial generalization from limited source demonstrations, with analyses validating the contributions of 3D editing, occlusion-aware projection, and
video completion.
Read the full abstract ▾
All authors Xiuwei Xu, Haowen Sun, Angyuan Ma, Yiwei Zhang, Zhenyu Wu, Xiaofeng Wang, Bingyao Yu, Zheng Zhu, Jie Zhou, Jiwen Lu
Robotics
ROVE: Unlocking Human Interventions for Humanoid Manipulation via Reinforcement Learning
Wei Xiao, Weiliang Tang, Yuying Ge et al. (+4 more)
Human interventions provide crucial corrective signals for post-training
Vision-
Language-Action (VLA) models. However, enabling seamless
humanoid interventions is a formidable systems challenge due to complex whole-body kinematics and dexterous-hand control. Consequently, the collected intervention trajectories are often suboptimal, and methods that rely on human interventions as expert supervision can absorb hesitant, inefficient, or even erroneous behaviors. To address both the system and algorithmic challenges, we propose ROVE, a reinforcement learning framework for
humanoid VLA post-training with imperfect human interventions. First, ROVE introduces a human-in-the-loop pipeline capable of collecting deployment and intervention data for
humanoid manipulation. Second, it utilizes Optimistic Value Estimation (OVE) to prioritize high-value behaviors from mixed-quality trajectories. To further robustify value estimation, we incorporate cross-embodiment human experience
videos to provide rich supervision for long-tailed failure and recovery modes. The resulting critic yields informative advantage signals, steering the VLA actor to focus on high-value behaviors rather than indiscriminately imitating all actions. On challenging real-world contact-rich and fine-grained
humanoid manipulation tasks, ROVE outperforms experience-learning baselines and consistently improves across multiple rollout-intervention iterations.
Read the full abstract ▾
All authors Wei Xiao, Weiliang Tang, Yuying Ge, Hui Zhou, Yao Mu, Li Zhang, Yixiao Ge
Robotics
Task-Error Residual Learning for Real-Robot Five-Ball Juggling
Kai Ploeger, Alap Kshirsagar and Jan Peters
For residual learning that refines existing behavior, sample efficiency depends on two things: how much information each rollout returns, and how efficiently the learner uses that information. Reinforcement learning's standard scalar reward carries far less information than the directional task error that defines the task. Random exploration further discards whatever information each rollout returns. Through residual learning with directional task-error supervision and a task error model that drives sample selection, we achieve stable three-, four-, and five-ball juggling on anthropomorphic Barrett WAM arms. Despite planning and controlling through a simple, idealized stack, the system converges from the second attempt. The first attempt drops, after which task error decreases monotonically without further failures. In comparison, five-ball juggling typically takes humans years of practice. We compare residual learners across two ternary axes, the directional information in the learning feedback and the commitment of the analytic prior, spanning Newton-style Jacobian updates, Composite
Bayesian Optimization, and
stochastic search methods. Both axes prove necessary for fast convergence: neither directional feedback nor an informative prior alone matches their combination, and the simplest method that combines them, a fixed-Jacobian Newton update, is the fastest and fully reliable. The learned residual tolerates substantial prior misalignment and degraded joint tracking, affecting mainly convergence speed. The bottleneck for residual learning on real
robots is therefore the information content of the supervision signal and how the learner uses it, not the accuracy of the surrounding stack.
Video documentation of all experiments is available at https://kai-ploeger.com/residual-juggling.
Read the full abstract ▾
Robotics
When Should a Robot Replan? Regret-Guided Update Scheduling in Time-Varying MDPs
Negin Musavi, Gokul Puthumanaillam, Ruben Hernandez et al. (+2 more)
Robots operating in non-stationary environments must continually adapt their policies as the dynamics drift, but onboard energy and compute budgets cap how often a full state estimation and re-planning step can be performed. This raises a question: when, along a horizon, should a
robot spend its limited budget? We formulate this problem in time-varying Markov decision processes (TVMDPs) with a known bound on the rate of transition drift. We model execution as a skip-update scheme in which, at chosen update times, the
agent estimates the transition kernel by maximum likelihood and computes a finite-horizon policy, and between updates reuses this policy under a propagated state estimate. We analyze the dynamic regret of this scheme and show how it grows during skip intervals in terms of the properties of the TVMDP and the skip lengths; the resulting bound answers the opening question via an online, regret-guided update rule that allocates the budget adaptively. We evaluate the rule in a simulated Mars-rover
navigation task with time-varying slip dynamics and on a Crazyflie quadrotor in indoor obstacle fields. Adaptive allocation outperforms other budgeted baselines.
Read the full abstract ▾
All authors Negin Musavi, Gokul Puthumanaillam, Ruben Hernandez, William Schafer, Melkior Ornik
Robotics
SidewalkBench: Benchmarking Visual Navigation on Urban Sidewalks
Zhizheng Liu, Honglin He, Vivek Alumootil et al. (+4 more)
Urban sidewalk
navigation presents significant challenges due to complex structural layouts, dynamic pedestrian behaviors, and long distances. While recent visual
navigation models offer a promising solution, the lack of a unified
benchmark hinders quantitative and reproducible evaluation. To bridge this gap, we propose SidewalkBench, a comprehensive
benchmark designed for visual
navigation on urban sidewalks. Built upon NVIDIA Isaac Sim, SidewalkBench brings GPU-accelerated simulation of diverse, high-fidelity sidewalk environments, including both procedurally generated and real-world scanned scenes. We further populate the scenes with rich, reactive event-based pedestrian behaviors and flexible, efficient animation, enabling standardized model evaluation under realistic real-world settings. We conduct a comprehensive evaluation of 9 visual
navigation models on 330 unit-test scenarios, 800 pedestrian-reactive scenarios, and 105 long-horizon scenarios. Our findings highlight that pedestrian interaction and long-horizon robustness remain critical bottlenecks for existing models, and scaling up sidewalk training with synthetic data emerges as a promising solution.
Read the full abstract ▾
All authors Zhizheng Liu, Honglin He, Vivek Alumootil, Akshat Pandya, Brad Squicciarini, Wayne Wu, Bolei Zhou
Robotics
CrossMaps: Confidence-Aware Open-Vocabulary Semantic Mapping for Rover Navigation
Jan-Niklas Klein, Sona Ghahremani, Christian Medeiros Adriano et al. (+1 more)
Rovers rely on perception to maintain spatial maps that encode both objects and sensor quality (e.g., range reliability, lighting artifacts, data density), guiding data fusion,
embedding updates, and
navigation under partial observability. To study these coupled perception-
navigation processes, we present CrossMaps, a real-time confidence-aware open-vocabulary semantic mapping pipeline that constructs
language-queryable maps from RGB-D data. Building on VLMaps-style approaches, CrossMaps integrates multi-scale CLIP
embeddings with confidence-aware fusion and a dual-
memory architecture consisting of Short-Term
Memory (STM) and Long-Term
Memory (LTM). The STM aggregates noisy visual observations using
geometric, semantic, and temporal confidence cues, while confident and coherent
cells are promoted to the LTM as persistent semantic landmarks. Designed for deployment with a Jetson Orin-powered UGV alongside SLAM, CrossMaps runs in real time and produces semantic heatmaps that can be queried with natural
language to guide rover
navigation.
Read the full abstract ▾
All authors Jan-Niklas Klein, Sona Ghahremani, Christian Medeiros Adriano, Holger Giese
Robotics
Unified Motion-Action Modeling for Heterogeneous Robot Learning
Yunhao Cao, Shitong Liu, Chao Feng et al. (+4 more)
We present Unified Motion-Action (UMA) Model, an approach that uses 3D object motion trajectories as a shared interface to bridge visuomotor control and dynamics modeling. UMA treats object motion and
robot actions as co-evolving variables under a masked generative objective, in which the mask pattern determines both the supervision regime during pretraining and the inference mode at deployment. Using hindsight-relabeled motion contexts and a contrastive objective that disentangles task intent from scene geometry, UMA enables multi-task pretraining across heterogeneous data sources without requiring manually annotated task instructions. At deployment, the same pretrained parameters support motion-conditioned visuomotor control, motion-based dynamics modeling, and task adaptation from few-shot demonstrations. Pretrained on a mixture of
robot demonstrations, human
videos, and simulated data, UMA consistently outperforms state-of-the-art baselines specialized for each inference mode.
Read the full abstract ▾
All authors Yunhao Cao, Shitong Liu, Chao Feng, Meryl Zhang, Xuanchen Lu, Andrew Owens, Kuan Fang
Robotics
Binary Tracking for Spatial QA and Navigation with Open Vision-Language Models
Dongbin Na, Chanwoo Kim, Soonbin Rho et al. (+3 more)
This work addresses spatial question answering for service
robots traversing long
egocentric routes. Given a query such as "where can I find a dry cleaner on the way back home?", the system returns a metric coordinate that downstream
navigation components can act on. Prior Spatial Question Answering approaches leverage
retrieval-augmented
agents built on closed-source models such as GPT-4o for path exploration. However,
robots operating in the real world often cannot reliably depend on online closed-source models due to network instability, communication latency, and deployment cost. It creates a need for open-source based Spatial Question Answering approaches that can run onboard the
robot, yet prior research in this direction remains limited. This work proposes BinTrack, a simple yet effective, fully open-source spatial-localization
agent that leverages the temporal ordering of a
robot's trajectory. BinTrack performs a binary search over the trajectory segments between two anchor landmarks identified from a query. It improves overall accuracy by up to 22.8% over other open-source implementations and even matches the reported closed-source model result on the global category of the SpaceLocQA
benchmark, the most challenging setting that has so far required strong
reasoning agents such as GPT-4o. Furthermore, its optimized inference strategy consistently yields more than a 1.5x inference speedup over previous approaches. Finally, this work releases GangnamLoop, a novel and practical multi-trip outdoor
benchmark collected by deploying a real quadruped
robot on public streets with the anonymization policy. It revisits the same locations under different outdoor conditions and pairs the
robot's low viewpoint with the human owner's. The source codes and
datasets are publicly available at https://github.com/ndb796/BinaryTracking
Read the full abstract ▾
All authors Dongbin Na, Chanwoo Kim, Soonbin Rho, Giyun Choi, Gangbok Lee, Dooyoung Hong
Robotics
LOPAL: Local Performance-Aware Active Learning from Imperfect Demonstrations
Johannes Heidersberger, Shail Jadav and Dongheui Lee
Learning from Demonstration (LfD) enables intuitive
robot skill acquisition by allowing
robots to learn directly from human task demonstrations. However, current methods often fail to address the fact that due to suboptimal and inconsistent human behavior, the quality of the demonstration can vary within each demonstration. Therefore, we introduce LOPAL (LOcal Performance-aware Active Learning), an active learning approach that leverages this local demonstration quality information. Our approach consists of two synergistic components. First, a local performance-driven LfD method uses a Gaussian Mixture Model (GMM) to encode both the demonstrated trajectories and their associated local quality assessments. This enables the generation of trajectories that outperform the imperfect demonstrations by utilizing complementary local data of high performance. Second, active data acquisition allows to improve beyond the imperfect demonstrations by collecting additional informative samples. In areas missing good data, the user is actively requested to provide corrections through a shared autonomy (SA) mechanism, while the
robot autonomously executes the learned behavior. The efficacy of LOPAL was validated in both a simulation and a real-world experiment. The results from a real-world pipe inspection task showed that the proposed approach can achieve up to 27.31 % improvement in task performance while also reducing the effort required to collect the demonstrations.
Read the full abstract ▾
Robotics
SGM-SLAM: Scene Graph Matching for Data-Efficient Distributed SLAM
Yewei Huang, Tixiao Shan, Abhinav Rajvanshi et al. (+4 more)
We introduce a data-efficient distributed Simultaneous Localization and Mapping (SLAM) framework for multi-
robot operation in GPS-denied and communication-constrained environments. The framework is designed for teams of
robots equipped with
LiDAR, cameras, and inertial sensors, where
robots may operate from independently initialized local coordinate frames and share only limited spatial overlap. Our approach constructs a scene graph by using fused RGB-
LiDAR point clouds to generate both a semantically segmented point cloud layer, and a layer of discrete bounded objects, to accompany estimated
robot trajectories. Scene graph matching is performed collaboratively through exchanging and matching object data with neighboring
robots. To maximize communication efficiency, we utilize a multi-step data exchange and optimization process. We demonstrate the effectiveness and communication efficiency of the proposed approach using real-world
datasets containing limited trajectory overlap, viewpoint variation, repeated structures, and degraded visual conditions.
Read the full abstract ▾
All authors Yewei Huang, Tixiao Shan, Abhinav Rajvanshi, Niluthpol Chowdhury Mithun, Yaxuan Li, Brendan Englot, Han-Pang Chiu
Robotics
ExoTraj: A General Lower-limb Exoskeleton Assistance Policy for Complex Environments
Xiao-Yin Liu, Guotao Li, Long Sun et al. (+2 more)
Adaptive torque prediction in dynamic exoskeleton scenarios requires expensive motion capture systems, which are infeasible in complex outdoor environments. Trajectory prediction has emerged as one of the effective approaches to address such an issue. However, the core challenges of exoskeleton trajectory prediction are twofold: establishing the mapping from multi-modal features to trajectory information; constructing the mapping from trajectory to torque. For the former, most existing methods perform only single-step prediction and neglect inter-subject trajectory variability, thereby limiting the trajectory optimization space and prediction generalization. To address this, this paper proposes a fast flow matching method that enables accurate trajectory prediction and better generalization for real-time performance, where trajectory generation errors and encoded observations are used to guide the training direction. For the second challenge, due to the high dynamics of the human-
robot system and the strong coupling between perception and control, simple control methods struggle to achieve efficient assistance based on the predicted trajectory. This paper utilizes model predictive control and designs a novel optimization objective to optimize torque, ensuring the exoskeleton achieves comfortable and robust assistance. By integrating the above two components, the unified policy, denoted as ExoTraj, is developed to enable adaptive assistance in complex outdoor scenarios without high data acquisition cost. Experimental results show that compared to traditional methods, ExoTraj reduces cross-subject prediction error by 14.0% during the online phase and maintains robustness against external noise. Relative to the zero torque condition, ExoTraj decreases metabolic rate by 11.5-24.4%, heart rate by 1.7-19.5%, and peak muscle activation levels by 10.9-41.3%, respectively.
Read the full abstract ▾
All authors Xiao-Yin Liu, Guotao Li, Long Sun, Xu Liang, Zeng-Guang Hou
Robotics
Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning
Tung M. Luu, Hwanhee Kim, Younghwan Lee et al. (+1 more)
Conveying complex objectives to reinforcement learning (RL)
agents often requires meticulous reward engineering. Preference-based RL (PbRL) offers a promising alternative by learning reward functions from human feedback, but its scalability is hindered by high labeling costs. Inspired by advances in
Video Foundation Models (ViFMs), we present
Video-based Optimal Transport Preference (VOTP), a semi-supervised framework that learns effective reward functions from only a handful of labels. By leveraging optimal transport to align visual trajectories within the rich representation space of ViFMs, VOTP effectively generates high-fidelity pseudo-labels for large amounts of unlabeled data, substantially reducing human supervision. Extensive experiments across locomotion and manipulation
benchmarks demonstrate the superiority of VOTP, which outperforms state-of-the-art offline PbRL methods under limited feedback budgets. We also showcase the robustness of VOTP in the presence of visual distractors and validate its utility on real robotic tasks, where it learns meaningful rewards with minimal human input.
Read the full abstract ▾
All authors Tung M. Luu, Hwanhee Kim, Younghwan Lee, Chang D. Yoo
Robotics
ATOM-Bench: A Real-World Benchmark for Atomic Skills and Compositional Generalization in Manipulation Policies
Zenan Wu, Bingqing Wei, Lu Liu et al. (+8 more)
Generalist manipulation policies are increasingly presented as foundation models for robotic control, but their real-world generalization remains difficult to diagnose. A policy may succeed on demonstrated tasks while still failing to execute fine-grained atomic skills or recombine learned skills in new task structures. We introduce ATOM-Bench, a real-world
benchmark for evaluating both atomic skills and compositional generalization in manipulation policies. ATOM-Bench factorizes tabletop manipulation into motor atoms and instruction atoms, and contains 30 atomic tasks and 24 held-out compositional tasks across paired single-arm and dual-arm
robot tracks. We collect 3,000 human demonstrations for atomic fine-tuning and release both the demonstration data and evaluation rollout data to support reproducible real-world evaluation. Policies are fine-tuned on atomic tasks and evaluated on both atomic skill acquisition and held-out compositional tasks. We further introduce Atomic Score (AS) and Compositional Failure Share (CFS) to distinguish failures caused by weak atomic skills from failures caused by limited compositional reuse. Through 2,700 physical rollouts on five representative manipulation policies, we find that current policies can acquire simple instruction-grounding skills, but still struggle with fine-grained motor atoms, counting, and logical filtering. More importantly, strong atomic performance does not reliably transfer to held-out compositional tasks. ATOM-Bench provides a diagnostic testbed for studying whether failures arise from weak motor execution, poor instruction grounding, or limited compositional reuse.
Read the full abstract ▾
All authors Zenan Wu, Bingqing Wei, Lu Liu, Zheqi He, Xi Wang, Jiakang Liu, Zehui Li, Guocai Yao, Jing-Shu Zheng, Xi Yang, Yongtao Wang
Robotics
SoK: Security and Privacy of Foundation-Model-Powered Robots
Xueluan Gong, Chen Chen, Jinxin Liu et al. (+2 more)
Foundation models are reshaping robotics by enabling
robots to interpret open-ended instructions, reason over
multimodal contexts, and operate in complex, open-world environments. However, their integration also introduces security and
privacy (S&P) risks that extend beyond the FMs themselves to
embodied execution pipelines, supporting ecosystems, and broader governance impacts. Existing literature reviews provide valuable insights but often focus on specific FM types, risk categories, mitigation strategies, or trust boundaries. Consequently, the field lacks a unified structure for analyzing where risks originate, how they propagate across robotic systems, and where mitigations should intervene. To address this gap, we propose a progressive F-E-S-G structural boundary framework for analyzing the S&P of FM-powered
robots. The framework comprises four layers: the Foundation model layer (F),
Embodied system layer (E), Supporting ecosystem layer (S), and Governance impact layer (G). Building on this structure, we develop a multi-level taxonomy that organizes prior studies along three levels: F-E-S-G trust boundary, security-
privacy concerns, and risk-mitigation perspectives. We further annotate each study using fine-grained coding attributes, including target, lifecycle stage, mechanism, system access, and effect. Guided by this framework and taxonomy, we systematize 96 papers. Our analysis uncovers multiple threat patterns, defense mismatches, and evaluation gaps that are difficult to identify from a single-boundary perspective. Based on these findings, we identify open challenges and future directions to provide a research agenda for developing secure,
privacy-preserving, and responsibly governed FM-powered robotic systems.
Read the full abstract ▾
All authors Xueluan Gong, Chen Chen, Jinxin Liu, Qian Wang, Kwok-Yan Lam
Robotics
DIFF-IPPO: Diffusion-Based Informative Path Planning with Open-Vocabulary Belief Maps
Sausar Karaf, Oleg Sautenkov, Mikhail Martynov et al. (+1 more)
Exploration and object search require
robots to perceive their environment, identify regions of interest, and plan trajectories that improve target-detection likelihood or maximize information gain. Many IPP methods, especially in continuous environmental monitoring, rely on Gaussian-process belief models, while object-search settings often produce complex,
multimodal belief maps from semantic or open-vocabulary perception. Global trajectory generation directly conditioned on such non-Gaussian belief maps remains comparatively underexplored. Although
diffusion-based planners offer strong capabilities for modeling such distributions, their use in informative path planning remains limited. In this work, we propose DIFF-IPPO, a pipeline that integrates an open-vocabulary belief map generator with a
diffusion-based planner for global trajectory generation over belief maps. The method generates trajectories that concentrate sensor coverage over high-belief regions, achieving normalized detection scores between 81.49% and 86.55% across different
dataset scenarios. We validate the system in a simulated search-and-rescue scenario where the planner searches candidate building regions to locate a burning building. In this setting, a team of five drones using batched belief-map-conditioned trajectory generation achieves first detections in 3.5 minutes.
Read the full abstract ▾
All authors Sausar Karaf, Oleg Sautenkov, Mikhail Martynov, Dzmitry Tsetserukou
Robotics
JoyAI-Sim: A Simulation-Enabled Interconversion Toolchain for the Embodied Data Pyramid
Peidong Liu, Yongce Liu, Songyan Guo et al. (+34 more)
Generalist
robot policies require trustworthy evaluation and
robot-usable training data, but both are difficult to scale with physical
robots alone. Real-
robot trials and demonstrations remain the most faithful source of deployment signals, yet they are slow, costly, and hard to reproduce. We present JoyAI-Sim, a simulation-enabled interconversion toolchain for human-
robot aligned model evaluation and data generation, denoted as
Robot $\rightleftharpoons$ Simulation $\rightleftharpoons$ Human. On the one hand, the
Robot $\rightarrow$ Simulation $\rightarrow$ Human pathway supports human-
robot aligned model evaluation by reconstructing real-
robot tabletop organization tasks as calibrated digital twins for scalable evaluation, while using human
embodied feedback to inspect and refine the naturalness of simulated motions. On the other hand, the Human $\rightarrow$ Simulation $\rightarrow$
Robot pathway supports human-
robot aligned data generation: it lifts ego-centric human demonstrations into simulation, checks them under
robot physical constraints, and converts them into
robot-centered trajectories, annotations, and visual observations. Together, these pathways use the JoySim simulator as both a scalable evaluation layer and a physical consistency filter for
robot data generation. We further package the core reconstruction, simulation, rendering, and realism-augmentation modules as cloud services on JD Cloud, turning the system into a reusable and scalable infrastructure for
robot data generation and model evaluation.
Read the full abstract ▾
All authors Peidong Liu, Yongce Liu, Songyan Guo, Fuyuan Ma, Zhihao Yuan, Ao Li, Zengjue Chen, Wenhao Li, Tianle Zhang, Mingyang Li, Jiale Zhang, Junzhe Xiong, Zhiyuan Xiang, Dafeng Chi, Yuzheng Zhuang, Liyi Luo, Wei Tan, Dongjiang Li, Nan Jiang, Yihang Li, Qingrong He, Jiaming Liang, Chen Cai, Mingxi Luo, Hui Zhang, Peng Hao, Song Wang, Ning Qiao, Yince Gao, Lei Kang, Junwu Xiong, Ruodai Li, Jiawei Li, Hui Shen, Yicheng Gong, Nan Duan, Liang Lin
Robotics
Pride and Prejudice: Toward an Information-Theoretic Framework for Mutually Communicative Driver Behavior Modeling
Tingjun Li, Nan Xu, Shuo Feng et al. (+3 more)
Mixed autonomy driving becomes unsafe and inefficient when autonomous vehicles (AVs) and human-driven vehicles (HVs) misread each other's intentions. We study this problem as implicit mutual communication in lane changes. The proposed framework models how the ego vehicle both expresses its intent and probes the other driver's preference under epistemic uncertainty. It combines a level-k
Bayesian persuasion game with virtual features for proactive signaling, information-theoretic rewards for mutual communication, and adaptive weights of communication affordances. We further introduce the Pride-Inquiry (P-I) and Pride-Prejudice (P-P) planes to analyze communication intensity and tendency. The model is calibrated with a Communication-Based Multi-
Agent Inverse Reinforcement Learning
algorithm (C-MIRL) on the naturalistic NGSIM
dataset. Compared with the non-communicative baseline, the proposed model reduces the prediction error of mandatory lane changes by up to 20% while maintaining strong generalization. Driver-In-the-Loop questionnaire scores are positively correlated with the calibrated communication variables, supporting the subjective validity of the model. The learned rewards further show that inquiry and listening affordances contribute more than pride and expression alone, and that inquiry preference varies more strongly across drivers. These results support explicit modeling of mutual communication and epistemic uncertainty in interactive driving.
Read the full abstract ▾
All authors Tingjun Li, Nan Xu, Shuo Feng, Hassan Askari, Bruno Henrique Groenner Barbosa, Konghui Guo
Robotics
VENOM: Versatile Embodied Network for Omni-bodied Motion tracking
Siddharth Padmanabhan, Kazuki Miyazawa and Takato Horii
Achieving expert-level expressive full-body motion tracking across multiple
humanoids solely from demonstration data remains a challenging and relatively an underexplored problem in
humanoid robot learning. Cross-embodiment motion tracking policies are mostly trained by decoupling the control problem into upper and lower body control. This work proposes VENOM, a cross-embodiment full-body motion tracking model for
humanoids in simulation. VENOM is a GPT-based motion tracker trained on multiple
humanoid data that can track the entire body without the requirement to split into upper and lower body control. We curate a multi-
humanoid motion tracking
dataset called the VENOM
dataset that contains states, actions, and rewards and train VENOM and the baselines on this
dataset. In this letter, we evaluate VENOM's performance against baselines and show that we can achieve a stable motion tracker across different
humanoids more capable than an MLP trained on multiple
humanoid data with supervised learning alone, and also show that despite lack of reward feedback, VENOM closely matches the tracking capability of experts that were trained using asymmetric-actor critic reinforcement learning.
Read the full abstract ▾
Robotics
PATCH: Action-Chunk-Conditioned Latent Patch Innovation Monitoring for Robot Manipulation
Yanan Zhou, Ranpeng Qiu, Yincong Chen et al. (+2 more)
Learning-based manipulation policies have made substantial progress in real-world
robot manipulation, particularly for short-horizon action generation. However, deployment in open workspaces remains fragile under unexpected local scene dynamics, such as moving objects, transient occlusions, or disturbances near the intended motion. Existing runtime monitors often rely on global observation anomalies, policy uncertainty, or frame-level visual changes, and struggle to distinguish task-relevant execution risk from benign visual variation. We introduce PATCH, an action-chunk-conditioned latent patch innovation monitor for deployment-time intervention. Given the active action chunk, PATCH defines a projected execution corridor, predicts latent patch
evolution inside it, and accumulates persistent residuals unexplained by the
robot's own motion. These residuals form a localized intervention signal that allows PATCH-Router to pause execution, select an available recovery source, and resume the original policy once localized innovation subsides. Experiments on real
robot rollout data show that PATCH produces more stable and context-relevant triggers than competing runtime monitors. Real-
robot deployment further demonstrates monitor-driven intervention and policy resumption for disturbance-aware manipulation. Project Page: https://yananzhou5555.github.io/PATCH/.
Read the full abstract ▾
All authors Yanan Zhou, Ranpeng Qiu, Yincong Chen, Jiajie Cui, Weiming Zhi
Robotics · Cross-listed from eess.SP
Towards mm-Level Accurate UWB Radar: High-Accuracy Phase-Based Obstacle Detection through Multi-Channel Fusion
Jelle De Moerloose, Adnan Shahid and Eli De Poorter
Accurate, tag-free distance estimation with ultrawideband (UWB) radar is essential for applications such as autonomous guided vehicles, robotics, and environment characterization. For tag-based localization systems, phase-based UWB signal processing techniques have demonstrated sub-wavelength ranging precision, but these approaches are not applicable for passive (tagless) radar setups with weak reflections, mixed multipath conditions, and the absence of a known time-of-flight (ToF) first-path reference. This paper demonstrates for the first time that phase information can be effectively exploited in a fully passive UWB radar setting. We introduce a signal processing framework that extracts reliable distance information by combining coarse amplitude-based estimates with high-resolution phase changes across multiple frequency channels. By referencing phase measurements with the line-of-sight component, the method compensates for hardware-induced phase drift, while the use of multichannel frequency diversity enables disambiguation of periodic phase information and improves robustness against frequencyspecific channel degradation such as Fresnel zones. The proposed approach is validated on a
robot equipped with a bistatic UWB radar using DW3000 devices and evaluated in a realistic metallic industrial environment. Experimental results show that our work consistently achieves centimeter-level accuracy even at high speeds, with a median error of 1.69 cm, significantly outperforming existing ~10cm accuracy UWB radar approaches relying only on amplitude-information. We further show how multi-channel fusion exploits uncorrelated channel degradation to reduce the error by more than 40% compared to single-channel operation, and outline how phase modeling and fusion can be pushed toward sub-centimeter accuracy.
Read the full abstract ▾