Robotics
SCAN-Planner: Spatial Collision-Aware Local Planning for Route-Guided Long-Range Quadruped Navigation
Han Zheng, Zhe Chen, Yiwen Fu et al. (+2 more)
Quadruped
robots are increasingly expected to navigate through narrow passages, cluttered indoor scenes, and large-scale 3D unstructured environments. Existing local planners commonly approximate the
robot using isotropic
geometric inflation or rely on planar and elevation-map representations, leading to conservative motion in tight spaces and limited
reasoning about overhanging structures. This letter presents SCAN-Planner, a spatial collision-aware local planning framework for long-range quadruped
navigation. A yaw-aware twin-cylinder footprint is used to model the elongated
robot body, enabling whole-body collision evaluation through sparse queries in an inflated 3D occupancy map. We further introduce a projected A* search that generates collision-free guidance on an interpolated ground-following surface, with z-gradient suppression to avoid obstacles horizontally while maintaining vertical stability. For large-scale deployment, a
robot-centric sliding map with boundary fallback provides high-resolution local collision checking and recovery from local dead ends. Simulation and real-world experiments demonstrate that SCAN-Planner generates safe, smooth, and efficient trajectories in dense clutter, 3D unstructured scenes, stair traversal, and long-range
navigation tasks.
Read the full abstract ▾
All authors Han Zheng, Zhe Chen, Yiwen Fu, Ming Yang, Tong Qin
Robotics · Cross-listed from cs.CV
ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?
Yuyang Zhang, Wenyao Zhang, Zekun Qi et al. (+7 more)
World Action Models (WAMs) commonly rely on
video generation to bridge visual world modeling and
robot control. However,
video-based WAMs face three coupled limitations: dense multi-frame future
tokens make inference costly, full
video prediction spends capacity on action-irrelevant temporal and appearance details, and long-horizon future imagination may introduce errors that mislead action prediction. These issues raise a simple question: Does world action model really need
video generation? We propose ImageWAM, a simple WAM framework that repurposes pretrained image editing models for
robot action prediction. In contrast to
video generation, image editing provides a better-matched prior: it only needs to model a target-frame transformation, focuses on action-relevant current-to-target visual differences, and grounds task instructions to localized visual changes through edit pretraining. In practice, ImageWAM does not decode the target frame at inference time; instead, it conditions a flow-matching action expert on the KV caches produced by image-editing denoising, using them as a compact world-action context. ImageWAM outperforms standard VLA baselines and matching competitive WAMs without additional policy pretraining across different simulator and real-world experiments. It also reduces FLOPs to 1/6 and latency to 1/4 of
video-based WAMs.
Attention analysis further shows that editing caches focus on task-relevant change regions, supporting image editing as an effective alternative to
video-based world-action modeling.
Read the full abstract ▾
All authors Yuyang Zhang, Wenyao Zhang, Zekun Qi, He Zhang, Haitao Lin, Jingbo Zhang, Yao Mu, Xiaokang Yang, Wenjun Zeng, Xin Jin
Robotics
A Categorial and Sheaf-Theoretic Semantics for Autonomic Component Ensembles
Manuel Hernández and Eduardo Sánchez-Soto
The proliferation of large-scale, decentralized systems of autonomous
agents, such as swarms of
robots and networked cyber-physical systems, presents a formidable challenge to traditional formal methods. The Software Component Ensemble
Language (SCEL) offers a formal model for such systems, but its operational semantics is not ideal for
reasoning about global, structural, and emergent properties. This report proposes a new, multi-layered mathematical model for SCEL using category theory and sheaf theory. We argue that a society of
robots described in SCEL can be formally modeled as a sheaf on a topological space, where components are points, ensembles are open sets, and distributed knowledge forms the sheaf's data. In this framework, computational processes like information sharing become equivalent to the sheaf-theoretic operation of "gluing" local data. System failures can then be understood and quantified as topological obstructions, measurable by sheaf
cohomology. This approach transforms the verification of a complex distributed system into the analysis of the geometry of a mathematical object, providing deep, structural insights for the design of robust autonomic systems.
Read the full abstract ▾
Robotics
Proprioceptive Invariant State Estimation for Humanoid Robots on Non-Inertial Ground
Falak Mandali, Zijian He and Yan Gu
This paper presents an invariant extended Kalman filtering (InEKF) approach for real-time state estimation of
humanoid robots operating on non-inertial ground using only onboard proprioceptive sensing. The proposed approach estimates the
robot's base position and velocity relative to the moving ground frame without requiring direct measurements of ground motion or externally mounted sensors. By exploiting kinematic constraints at the stance foot through foot-mounted IMUs, the filter accounts for ground-induced nonlinearities in the process and measurement models while remaining fully proprioceptive. The
estimator is formulated to admit a right-invariant measurement model, enabling favorable error dynamics under large initial uncertainties. Observability analysis establishes conditions under which the
robot's relative base position and velocity are observable with respect to the non-inertial ground frame. Experiments with the Digit
humanoid robot standing and squatting atop a swaying and pitching ground showcase a 96% speedup in convergence rate and an 80% reduction in position estimate errors over existing InEKFs. Walking experiments on a uni-axially rotating ground achieve an average estimation error of less than 9 cm for an initial error of up to 1 m.
Read the full abstract ▾
Robotics
Simulating Robotic Locomotion in Sand: Resistive Force Theory in an Open-Source Physics Engine
Ryan Walker Brown, Laura K. Treers and Kathryn A. Daltorio
Recent advancements in Resistive Force Theory (RFT) enable approximation of ground reaction forces for locomotion in sand without the computational expense of modeling interactions with individual grains. However, these tools have been absent in 3D physics engines commonly used for
robot simulation. We explore if resistive force approximations are sufficient, when integrated with standard dynamics calculations, to provide a stable substrate for a freely walking
robot. To determine this, we implement 3D Granular Resistive Force Theory (3D RFT) in a physics simulation engine, MuJoCo. We verify simulations in multiple scenarios to demonstrate that key trends due to end effector shape, speed, and loading are preserved. Our implementation predicts both walking distance and foot sinkage of a 12-Degree of Freedom hexapod
robot within 7% of experiments in sand. While RFT has inherent approximations, the open source tool described here has potential to help develop new and improved
robot designs to traverse granular media substrates.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
3D-DLP: Self-Supervised 3D Object-Centric Scene Representation Learning
Ellina Zhang, Madhaven Iyengar, Amir Zadeh et al. (+4 more)
We introduce 3D-DLP, a self-supervised object-centric representation learning model that decomposes scene-level RGB-D or voxel observations into a set of 3D latent particles. Building on the Deep Latent Particles (DLP) framework, each particle encodes disentangled attributes, including 3D keypoint position, bounding box dimensions, and appearance features, and represents a distinct entity in the scene. The model learns interpretable per-particle
segmentation maps through an end-to-end self-supervised reconstruction objective. We demonstrate on both simulated and real-world
datasets that the learned latent space is interpretable and controllable: by manipulating particle positions and decoding, we can generate novel scene configurations. Furthermore, we show that leveraging these compact 3D latent particles for downstream robotic manipulation improves performance over baselines that either lack explicit 3D information or rely on
memory-intensive dense 3D inputs without object-centric structure. Code and
videos are available at https://eubooks3003.github.io/3d-dlp.
Read the full abstract ▾
All authors Ellina Zhang, Madhaven Iyengar, Amir Zadeh, Chuan Li, Deepak Pathak, David Held, Tal Daniel
Robotics
Playful Agentic Robot Learning
Junyi Zhang, Jiaxin Ge, Hanjun Yoo et al. (+17 more)
Current agentic
robot systems can write executable Code-as-Policy programs, observe feedback, and revise behavior across multiple attempts, but they remain largely task-driven: reusable skills are acquired only after explicit instructions. We study Playful Agentic
Robot Learning, where an
embodied coding
agent uses self-directed play as a continual skill-learning stage before downstream tasks arrive. We introduce RATs, Robotics
Agent Teams designed for play-time skill acquisition. During play, RATs proposes novel yet learnable exploratory tasks, plans and executes
robot-code policies, verifies intermediate progress, diagnoses failures, retries with dense, step-level feedback, and distills successful executions into a persistent code skill library. At test time, the
agent reuses relevant skills from this frozen library to help solve new tasks. Experiments in LIBERO-PRO and MolmoSpaces show that play-learned skills improve held-out downstream tasks over no-play and random-play baselines, with 20.6 and 17.0 percentage-point gains over CaP-
Agent0 on LIBERO-PRO and MolmoSpaces, respectively. Moreover, the learned skills can be plugged into other inference-time Code-as-Policy
agents by simply retrieving them into the context, improving RoboSuite and real-world transfer by 8.9 and 8.8 points, respectively, without finetuning the underlying model.
Read the full abstract ▾
All authors Junyi Zhang, Jiaxin Ge, Hanjun Yoo, Letian Fu, Zihan Yang, Yaowei Liu, Raj Saravanan, Shaofeng Yin, Justin Yu, Dantong Niu, Zirui Wang, Roei Herzig, Ken Goldberg, Yutong Bai, David M. Chan, Ion Stoica, Angjoo Kanazawa, Jiahui Lei, Haiwen Feng, Trevor Darrell
Robotics · Cross-listed from cs.LG
FlexLAM: Resolving the Bottleneck Trade-off in Latent Action Learning
Takanori Yoshimoto, Yang Hu, Naruya Kondo et al. (+1 more)
Latent actions provide a compact interface between action-free
video and downstream decision-making, yet existing Latent Action Models (LAMs) force every transition through a fixed-capacity bottleneck. We identify a bottleneck trade-off: overly tight codes can discard transition cues needed for action
alignment, while overly loose codes preserve additional transition variation that must be resolved when
alignment labels are scarce or narrowly distributed. FlexLAM replaces this fixed capacity with variable-length latent actions trained by nested dropout, yielding prefix-valid codes that capture compact transition structure first and add detail only when needed, without new architectures or losses. A single FlexLAM matches or surpasses separately trained fixed-capacity LAMs at every evaluated
token budget under standard scarce-label supervision and under a low-return single-task
alignment stress test, indicating that FlexLAM is not merely adjustable at inference time but learns a better latent-action interface at the same
token budgets. The same model supports inference-time
token-budget adjustment without retraining, and FlexLAM improves Ego4D transition reconstruction. These results suggest that variable-length latent actions are an architecture-free, drop-in upgrade to the fixed-capacity bottleneck in latent action models, latent-action world models, and
video-pretrained action interfaces.
Read the full abstract ▾
All authors Takanori Yoshimoto, Yang Hu, Naruya Kondo, Tatsuya Matsushima
Robotics
DiffusionVS: A Generative Framework for Robust Visual Servoing Based on Diffusion Policy
Hongkang Cui, Rui He and Haoyao Chen
Visual servoing is a fundamental technique in robotic manipulation and
navigation. Regression-based visual servoing frequently experiences trajectory jitter as a result of noise-sensitive single-step mappings and the accumulation of errors during distribution shifts. In contrast,
Diffusion Policy maintains temporal consistency by predicting action sequences and improves robustness through implicit data augmentation. This paper presents a novel
diffusion-based servoing method. Based on
Diffusion Policy, the proposed approach uses normalized image coordinates of observed tag corners as input and generates camera velocity through conditional denoising. To overcome the generalization limitations of models trained on static
datasets, an online training paradigm is adopted, continuously expanding the diversity of training data through interactive experience collection. This strategy substantially enhances both the performance and generalization capability of the model. Comprehensive simulations and real-world experiments demonstrate the effectiveness of the proposed method, achieving success rates of nearly 100% in simulation and 93% in physical experiments. Beyond the specific pipeline, we further validate the generality of the
diffusion mechanism. Experiments show that existing visual servoing networks consistently achieve improved performance when integrated with our
diffusion-based module. These results indicate that the proposed strategy possesses broad applicability and can enhance various visual servoing systems beyond the specific architecture presented here.
Read the full abstract ▾
Robotics
ZeroDex: Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning
Jisoo Kim, Sangwon Baik, Taeksoo Kim et al. (+6 more)
We present ZeroDex, a zero-shot framework for long-horizon dexterous manipulation that grounds
language instructions into executable 3D task plans from calibrated multi-view RGB images. Our framework builds on the grounding ability that
vision-
language models (VLMs) already exhibit in 2D and lifts their predictions into 3D through multi-view fusion, which extends VLM grounding beyond image-space keypoints to the metric 3D quantities that manipulation actually requires: contact points, placement targets, and tool-motion anchors. Because execution is driven by these 3D groundings rather than a learned policy, the system preserves the VLM's open-world generalization, without collecting demonstrations or post-training for a specific task or embodiment. The VLM decomposes a complex instruction into a sequence of atomic tasks, each executed by a primitive from a small reusable library that covers manipulation tasks including pick-and-place and tool-use: for tool-use, we retrieve an object-centric atomic action corresponding to the inferred skill category and align its stored 6D tool trajectory to the grounded keypoints; for dexterous execution, we expand the lifted grasp keypoint into a task-conditioned grasp affordance region and generate feasible grasp-motion pairs with an arm-hand motion generator. Real-world experiments show improved 3D grounding accuracy and higher task success than zero-shot visual-prompting and constraint-based baselines, both of which ground from a single RGB-D view, and our demonstration-free system outperforms VLA policies fine-tuned on 100 task demonstrations. We further demonstrate long-horizon manipulation, where execution verification and replanning recover from intermediate failures, and show that the system generalizes to unseen objects and tool-use tasks in novel scenes.
Read the full abstract ▾
All authors Jisoo Kim, Sangwon Baik, Taeksoo Kim, Sungjoo Kim, Junyoung Lee, Mingi Choi, Kisang Park, Sungjoon Choi, Hanbyul Joo
Robotics
Do as I Do: Dexterous Manipulation Data from Everyday Human Videos
Bhawna Paliwal, Haritheja Etukuru, William Liang et al. (+3 more)
How can we scalably generate data for robotic manipulation, especially on human-like platforms such as dexterous multi-fingered hands? Learning from human
videos has recently emerged as a likely answer to this question. However, difficulties in estimating hand-object interaction and crossing the human-to-
robot embodiment gap have hindered the adoption of abundant
monocular RGB-only human
videos as the primary source of
robot manipulation data. In this work, we present DO AS I DO, an
algorithm to reconstruct and retarget
monocular RGB human
videos to multi-fingered dexterous robotic hands. DO AS I DO reconstructs hand-object interactions from various
egocentric and exocentric in-the-wild
video sources. The
algorithm then retargets these hand-object interaction estimates into a sequence of actions executable in the real world, yielding
robot-complete manipulation data from disparate human
videos. Overall, DO AS I DO outperforms previous state of the art in estimating hand-object interactions and extracting dexterous manipulation trajectories from RGB
videos, as we show in experiments on
datasets with ground truths and on a
dataset of
video clips collected online. Our experiments enable us to propose an efficacy playbook for practitioners collecting human data for manipulation.
Read the full abstract ▾
All authors Bhawna Paliwal, Haritheja Etukuru, William Liang, Pieter Abbeel, Nur Muhammad Mahi Shafiullah, Jitendra Malik
Robotics · Cross-listed from cs.LG
UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning
Mohamed Nabail, Leo Kaixuan Cheng, Jingmin Wang et al. (+1 more)
Preference-based RL provides an approach to learning reward models from pairwise comparisons of behaviors, bypassing the need for explicit reward design. However, existing methods typically rely on passive data collection and suffer from poor sample efficiency, especially during the early stages of learning. We introduce a model-based approach that actively directs exploration by jointly
reasoning over uncertainties in the reward, dynamics, and value functions. Our method, Uncertainty-Balanced Preference Planning (UBP2), uses ensembles of reward, dynamics, and value function models to evaluate candidate trajectories according to a unified score that combines expected reward, terminal value, and epistemic uncertainty. Planning under this objective yields an explicit tradeoff between exploitation and information acquisition without requiring ad hoc exploration heuristics. Under standard regularity assumptions, we establish sublinear regret guarantees for both finite-horizon and infinite-horizon settings. Empirically, experiments on the Meta-World
benchmark show UBP2 achieves substantially higher sample efficiency than model-free preference-based methods and non-optimistic model-based baselines.
Read the full abstract ▾
All authors Mohamed Nabail, Leo Kaixuan Cheng, Jingmin Wang, Nicholas Rhinehart
Robotics
Modeling Branches for Active Manipulation using Iterative Parameter Estimation
Madhav Rijal, Rashik Shrestha, Trevor Smith et al. (+1 more)
This study presents a method for modeling diverse plant branches by iteratively estimating material parameters to support delicate branch manipulation. Branch manipulation is necessary in agricultural robotics for plant repositioning, stabilizing, and clearing visual obstructions in dense foliage. The proposed method builds a tetrahedral branch model from point-cloud data and simulates its behavior using the finite element method. Using real observed deformation data, it iteratively estimates branch parameters and then computes an optimal path with a deformation-aware motion planner to move and stabilize branches within another
robot's field of view. Across 30 trials on branches with varying geometries and material properties, the proposed method reduced the deformation energy by 35.69% while increasing the path length by 8.10% on average.
Read the full abstract ▾
All authors Madhav Rijal, Rashik Shrestha, Trevor Smith, Yu Gu
Robotics
Observability and Consistency Analysis for Visual-Inertial Navigation with Anchored Feature Parameterizations
Mitchell Cohen, Vassili Korotkine and James Richard Forbes
This paper presents an analysis of the observability and consistency properties of filtering-based visual-inertial
navigation systems (VINS) that utilize anchored feature representations. The unobservable subspace of VINS with anchored landmark parameterizations is shown to be independent of the estimated landmark state, which leads to improved
estimator consistency properties without any additional modifications. However, the unobservable subspace is still found to depend on the estimated
navigation state, necessitating additional consistency-enforcing techniques. Two methods to improve the consistency of VINS with anchored feature representations are presented. Simulation results showcase that all
estimators employing anchored feature paramterizations exhibit improved consistency properties compared to
algorithms that estimate features resolved in a global reference frame, especially in scenarios where feature initialization may be poor. Real-world experiments on the TUM-VI
dataset showcase that the use of anchored feature representations alone can yield comparable performance to consistency-improved
estimators employing a global feature representation, demonstrating the benefit of using anchored feature parameterizations for VINS.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models
Nikita Kachaev, Andrey Moskalenko, Matvey Skripkin et al. (+10 more)
Embodied Vision-
Language-Action (VLA) models are typically obtained by fine-tuning powerful pretrained VLMs on robotics data, yet it is unclear how much commonsense and factual knowledge they retain after adaptation. Failures on knowledge-sensitive tasks are ambiguous, conflating missing knowledge with poor generalization of low-level control. We introduce Act2Answer, a lightweight protocol that adapts VLM knowledge
benchmarks to VLA evaluation by requiring
agents to answer through action. Each question becomes a short tabletop episode where the
agent performs a single object-placement action to select among candidate answers, yielding an action-grounded success rate with reduced control confounds. We curate a test suite of such environments across diverse commonsense and world-knowledge categories and introduce layerwise intent probing to localize answer-relevant information across the VLM backbone and action head. In a large-scale study of 7 VLA models and 9 VLM baselines, we systematically rank models across categories, finding that VLAs show solid performance on simple concepts while exhibiting larger gaps on richer semantic categories relative to their source VLMs, that VQA co-training is associated with better knowledge retention, and that answer-relevant signals peak in middle VLA layers but attenuate in upper layers. Act2Answer is available at https://tttonyalpha.github.io/act2answer/.
Read the full abstract ▾
All authors Nikita Kachaev, Andrey Moskalenko, Matvey Skripkin, Nikita Kurlaev, Daria Pugacheva, Albina Burlova, Mikhail Kolosov, Denis Shepelev, Andrey Kuznetsov, Elena Tutubalina, Aleksandr I. Panov, Alexey K. Kovalev, Vlad Shakhuro
Robotics
A Mixed-Reality Testbed for Autonomous Vehicles
H. M. Sabbir Ahmad, Ehsan Sabouni, Emrullah Celik et al. (+4 more)
We propose a mixed-reality, hardware-in-the-loop (HIL) testbed for autonomous vehicles that seamlessly integrates a physical testbed of mobile
robots with a high-fidelity simulation environment. The virtual simulation enables the creation of diverse, safety-critical driving scenarios to validate state-of-the-art perception, planning, and control
algorithms, while augmenting simulations with physical
robots equipped with
multimodal sensors in photorealistic virtual environments further facilitating rigorous validation. Our testbed also features vehicular connectivity using wireless communication and can accommodate a large number of
agents through the combination of physical
robots and virtual simulated
agents, supporting research on multi-
agent systems including Connected and Autonomous Vehicles (CAVs). Finally, we present a safety-guaranteed framework combining perception, planning and a novel online learning-based controller using Control Barrier Functions (CBFs) for CAVs. Experiments using the proposed framework are used to validate and demonstrate the key functionalities and the overall utility of the testbed to bridge the gap between simulation and real-world hardware deployment.
Read the full abstract ▾
All authors H. M. Sabbir Ahmad, Ehsan Sabouni, Emrullah Celik, Zean Wan, Damola Ajeyemi, Christos G. Cassandras, Wenchao Li
Robotics
Shape Sensing of Continuum Robots using Direct Laser Writing
Amber K. Rothe, Nidhi Malhotra and Jaydev P. Desai
Continuum
robots offer a promising approach for minimally invasive and natural-orifice
surgical procedures due to their inherent compliance and dexterity. However, this flexibility also makes estimating the current shape of the
robot challenging. Several approaches have been used to reconstruct the shape of these
robots, including imaging, optical sensing,
magnetic sensing, and resistive sensing. Strain sensors fabricated using direct
laser writing (DLW) could provide an alternative sensing method. This technique involves using a
laser to induce carbonization of certain
polymers to create
graphene patterns, such as strain sensors. In this paper, we demonstrate how a flexible continuum joint and a DLW sensor can be machined as one monolithic structure using the same
laser and the same setup. The fabricated sensors are characterized using linear and nonlinear models, which are used to predict the joint angle with error as low as 1.76 degrees. Furthermore, we demonstrate how a DLW sensor can be used to implement closed-loop control in a robotic joint, achieving tracking error under 3 degrees.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
CABLE: Cloud-Assisted Bandwidth-efficient LMM-based Encoding for V2X Systems
Haohua Que, Zhipeng Bao, Qianyi Wu et al. (+1 more)
Cloud-hosted large
multimodal models (LMMs) can provide strong open-vocabulary perception for Vehicle-to-Everything systems, but naively transmitting full-resolution frames from edge to cloud causes severe communication overhead and high cloud-side prefill latency. We present CABLE, a cloud-assisted bandwidth-efficient LMM-based encoding framework for edge-cloud perception. CABLE propagates the previous cloud
segmentation mask on the edge using ego-motion compensation, refines it with residual-motion cues, and consolidates disconnected regions via a corridor envelope to form a robust region of interest (ROI). Only ROI-masked images are uploaded, while the cloud
segmentation output is fed back as the prior for the next frame, forming a mask-to-ROI-to-LMM feedback loop. Experiments on five
datasets (nuScenes, WOD-ZB, Waymo, KITTI, and CADC) show consistent communication savings while largely preserving perception, achieving $73$–$87\%$ ROI pixel-coverage reduction with $5$–$8\times$ estimated LMM prefill speedup at a modest detection-quality trade-off relative to full-frame inference.
Read the full abstract ▾
All authors Haohua Que, Zhipeng Bao, Qianyi Wu, Handong Yao
Robotics · Cross-listed from cs.CV
OneCanvas: 3D Scene Understanding via Panoramic Reprojection
Bartłomiej Baranowski, Dave Zhenyu Chen and Matthias Nießner
Existing approaches to 3D scene understanding in
Vision-
Language Models (VLMs) either rely on complex, model-specific geometry encoders or large training budgets in pursuit of spatial
reasoning. Instead, OneCanvas aggregates patch features from all views onto a single equirectangular panoramic canvas. Namely, each patch is unprojected to a 3D world coordinate using its depth and camera pose, then placed on the canvas at the continuous longitude and latitude of that point as seen from the canvas origin, with no rasterization or aggregation across overlapping views. A 3D position
embedding of the patch's metric coordinates is added to its feature, restoring the depth lost when collapsing the world position to an angular canvas coordinate. Patches from all frames thus share one spatial coordinate system with no fusion or major architectural modifications of the backbone. The pretrained VLM consumes this representation as if it were an ordinary image. Because the canvas can be centered on any pose of interest, the same representation directly supports situated
reasoning from a specific viewpoint, a common requirement in robotics and
embodied AI. Thanks to this representation, we can also introduce a spatial pretraining curriculum: by procedurally placing patch features of objects, drawn from real images, at chosen 3D world positions on an otherwise empty canvas, we generate on-the-fly supervision spanning a broad range of spatial
reasoning tasks, with answer distributions controlled to reduce spatial
reasoning shortcuts. OneCanvas achieves state-of-the-art results on SQA3D and VSI-Bench and generalizes to out-of-distribution data on SPBench, using an order of magnitude less training compute than its competitors closest in
benchmark performance.
Read the full abstract ▾
Robotics
Seeing Through Occlusion: Deterministic Arm Kinematic Correction for Robot Teleoperation
Thomas M. Kwok, Nicholas Koenig and Yue Hu
Markerless, single-RGB-D-camera motion capture provides a low-cost and non-invasive alternative to conventional marker-based systems for
robot teleoperation; however, depth estimation often degrades in the presence of self-occlusion, particularly during upper-limb motion. This paper presents an Arm Kinematic Correction (AKC) method that improves depth estimation by enforcing
geometric constraints based on constant arm lengths. The proposed approach reconstructs occluded joint depths by leveraging wrist positions and predefined arm lengths via a deterministic formulation based on the Pythagorean
theorem, thereby avoiding the need for complex probabilistic modeling or parameter tuning. Experimental validation against a Vicon reference system demonstrates reliable performance for both static and dynamic joint motions, evaluated using root-mean-square error (RMSE) and Pearson correlation. Furthermore, motion-mapping teleoperation is successfully demonstrated in both simulated and physical
robot environments. The results show that AKC enhances robustness and preserves anatomical consistency under long-duration, severe self-occlusion, even when paired with less reliable temporal filters, highlighting its practicality for real-time applications such as
robot teleoperation and human-
robot interaction.
Read the full abstract ▾
Robotics
Mobile Pedipulation for Object Sliding via Hierarchical Control on a Wheeled Bipedal Robot
Yue Qin, Yulun Zhuang, Zelin Shen et al. (+1 more)
In this letter, we present a hierarchical control framework that enables wheeled bipedal
robots to perform planar object sliding tasks with their wheeled legs. The proposed approach formulates a nonlinear model predictive controller (NMPC) based on a reduced-order three rigid bodies (TRB) dynamical model that explicitly accounts for the hip roll degree of freedom and multiple wheel-environment contact modes, which is essential for lateral stepping and pedipulation tasks. Within this framework, the NMPC simultaneously regulates
robot locomotion and interaction forces, allowing the
robot to stably execute both rolling and object manipulation behaviors. A trajectory-optimization-based
robot-object motion planner is developed to generate reference motions that incorporate stick-slip transitions in ground-object contact. Two representative pedipulation motions, namely scooting and lateral sliding, are validated through real-world hardware experiments, in which the
robot successfully retrieves a 1 kg object from under a desk and slides a 4 kg object over a distance of 0.228 m via scooting.
Read the full abstract ▾
All authors Yue Qin, Yulun Zhuang, Zelin Shen, Yanran Ding
Robotics
Constant Time-Delay Leader Following with Neural Networks and Invariant Extended Kalman Filters for Arbitrary Trajectories
Luka Antonyshyn, Paulo Ricardo Marques de Araujo and Sidney Givigi
This paper proposes a constant time-delay trajectory tracking method for vehicle convoys operating without inter-vehicle communication, a common coordinate system, or global positioning. The method integrates a probabilistic sequence-to-sequence (Seq2Seq) neural network with an invariant extended Kalman filter (IEKF) to warm-start the prediction process, allowing accurate estimation of a leader vehicle's relative trajectory on the SE(2)
manifold. A
geometric model predictive controller is further incorporated to fully exploit the
manifold-based trajectory predictions for improved control performance. The system can handle arbitrary nonlinear trajectories with varying speeds and motion profiles while reducing the need for expert-based domain knowledge for the design of trajectory following systems, even under long trajectory delays. The effectiveness of the method is validated through comparisons with a pure IEKF baseline, learning-based methods, and the ground-truth trajectory in kinematic simulations, as well as in experiments using real robotic vehicles.
Read the full abstract ▾
Robotics
Invertible Neural Network Adapter for One-Step Flow Matching in Robot Manipulation
Yu Zhang, Kangyi Ji, Yongxiang Zou et al. (+3 more)
This paper presents an invertible neural network adapter for general robotic manipulation, designed to generate precise high-dimensional actions conditioned on
multimodal observations, including visual, linguistic, and proprioceptive inputs, through a one-step denoising process. Built upon a flow-matching formulation, the proposed adapter effectively constrains the action generation trajectory within an invertible latent space, thereby enabling efficient and high-quality dexterous action synthesis with only a single inference step. Compared with conventional iterative flow-matching policies, the proposed framework substantially reduces inference complexity while maintaining strong action prediction accuracy and stability. Extensive experiments are conducted across a diverse set of simulation
benchmarks and real-world robotic platforms to evaluate the effectiveness of the proposed method. Across simulation
benchmarks, the proposed adapter consistently demonstrates superior or near state-of-the-art performance on a wide range of manipulation tasks. Furthermore, real-world experiments reveal a significant improvement in inference efficiency for
vision-
language-action (VLA) models, reducing the average inference latency from 110 ms to 61 ms while maintaining strong task performance.
Read the full abstract ▾
All authors Yu Zhang, Kangyi Ji, Yongxiang Zou, Rongtao Xu, Feng Zheng, Long Cheng
Robotics
FAST-LIVGO: A Degeneracy-Robust LiDAR-Inertial-Visual-GNSS Fusion Odometry
Zhiyu Chen, Chunran Zheng, Jiayu Wen et al. (+4 more)
Robust state estimation and mapping in long-term, large-scale, and highly dynamic environments remains a key challenge in robotics. Existing
LiDAR-Inertial-Visual Odometry (LIVO) systems achieve strong local accuracy but suffer from accumulated drift over long distances and may fail in geometrically degraded or textureless scenes. Meanwhile, GNSS-aided fusion frameworks often rely on
LiDAR or visual odometry for state prediction and outlier rejection, making them vulnerable when odometry degenerates. To address these limitations, we propose a tightly coupled
LiDAR-Inertial-Visual-GNSS fusion framework based on an Error-State Iterated Kalman Filter. An online spatiotemporal
alignment module using Dynamic Time Warping is introduced for highly dynamic conditions. To better exploit GNSS precision, we develop observation models based on Doppler shifts and fixed-anchor Time-Differenced Carrier Phase, providing millimeter-level relative constraints without augmenting historical anchor states. We further design a degeneracy-aware dual-mode outlier rejection strategy that switches between LIVO-prior-guided rejection and GNSS-aided recovery according to the LIVO degeneracy level. Experiments on the public M3DGR
dataset and a custom 20~m/s fixed-wing
UAV dataset demonstrate that our system reduces accumulated drift and map ghosting, outperforming state-of-the-art methods in accuracy and robustness.
Read the full abstract ▾
All authors Zhiyu Chen, Chunran Zheng, Jiayu Wen, XiaoLei Zhang, Jiaming Xu, Feng Pan, Yukang Cui
Robotics
Learning to Annotate Delayed and False AEB Events: A Practical System for Extreme Class Imbalance and Asymmetric Label Noise
Mengxiang Hao, Xin Jiang, Xinghao Huang et al. (+10 more)
Autonomous Emergency Braking (AEB) optimization relies on accurately annotated real-world trigger events, particularly rare but critical delayed and false AEB triggers that expose system deficiencies. However, these minority samples comprise less than 5% of thousands of daily triggers, making manual annotation prohibitively expensive at scale. We present the first automated AEB annotation framework to address this problem. During development, we identified two fundamental challenges that severely impair delayed/false trigger annotation accuracy: (1) Extreme class imbalance where delayed/false triggers are overwhelmed by true triggers; (2) Asymmetric label noise where mislabeled majority samples (true triggers) suppress minority samples (delayed/false triggers) learning. To overcome these challenges, we propose two key innovations: (1) Specific data augmentation that synthesizes realistic samples by manipulating focal target attributes, transplanting ego-vehicle dynamics, and masking non-focal
agents; (2) noise suppression using stable hardness estimation and probe-guided adaptive threshold to clean mislabeled true trigger samples. Crucially, we deploy our model as a practical annotation system with full-stack architecture, efficiently identifying critical delayed/false triggers from thousands of daily AEB events. Production results demonstrate 80% improvement in recall of delayed/false triggers and 50% reduction in manual workload. Beyond immediate gains, the system enables continuous self-improvement through accumulated high-quality annotations, establishing a necessary data foundation for on-vehicle AEB system optimization
Read the full abstract ▾
All authors Mengxiang Hao, Xin Jiang, Xinghao Huang, Wenliang Su, Zhiteng Wang, Junjie Rao, Xiaotian Yang, Wei Liao, Chengyu Han, Gen Liang, Yulun Song, Zhitao Xu, Xianpeng Lang
Robotics
Hardware- and Vision-in-the-Loop Validation of Deep Monocular Pose Estimation for Autonomous Maritime UAV Flight
Maneesha Wickramasuriya, Beomyeol Yu, Jaden Shin et al. (+3 more)
Autonomous
UAV operations on ships require reliable
vision-based relative pose estimation, yet at-sea validation is costly, weather-dependent, and risky. This paper presents a hardware-validated
vision-in-the-loop framework that enables fully autonomous indoor flight while emulating photorealistic maritime environments. Rendered maritime views are processed onboard by a deep
transformer-based
monocular pose
estimator. Delayed
vision measurements are fused with high-rate IMU data using a delayed Kalman filter to provide consistent state estimates for
geometric control. The system captures critical embedded effects, including perception latency, asynchronous updates, and computational constraints, that are absent in pure simulation. Autonomous takeoff, trajectory tracking, and landing experiments demonstrate stable closed-loop flight. The results establish a safe and hardware-realistic intermediate stage for developing maritime
UAV autonomy prior to shipboard deployment.
Read the full abstract ▾
All authors Maneesha Wickramasuriya, Beomyeol Yu, Jaden Shin, Mason Huslig, Taeyoung Lee, Murray Snyder
Robotics
HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Vision
Yuzhe Huang, Jiaping Wu, Jiaming Jiang et al. (+8 more)
Establishing a universal
benchmark for tactile representation learning in robotic manipulation remains challenging due to the diversity of tactile sensor designs, data formats, and
robot embodiments. Rather than seeking to establish such, we explore a scalable and promising direction for future development:
egocentric vision paired with full-hand tactile data. To this end, we introduce HT-Bench, a large-scale multi-task
benchmark for dexterous full-hand tactile sensing, comprising 10M RGB frames and 7.8M tactile frames collected across 226 tasks. HT-Bench evaluates tactile representations from three key perspectives: whether they encode meaningful contact geometry, whether they can align tactile observations with visual information, and whether they generalize to unseen tasks. To assess these capabilities, HT-Bench includes four tasks: fine-grained tactile similarity
retrieval, masked tactile inpainting,
vision-to-tactile synthesis, and
multimodal tactile frame prediction. We further propose HandTouch, a vector-quantized
vision–tactile encoder that learns tactile representations through progressive spatial, cross-modal, and temporal training. Across HT-Bench, HandTouch consistently outperforms representative tactile encoder baselines, improving Recall@5 on fine-grained tactile similarity
retrieval from 74.65% to 85.23%, reducing RMSE on masked tactile inpainting from 0.022 to 0.010, and increasing OOD cIoU on
vision-to-tactile synthesis from 0.628 to 0.705. These results demonstrate the effectiveness of HandTouch and suggest that large-scale
egocentric full-hand tactile data provides a scalable basis for evaluating and advancing tactile representation learning in dexterous manipulation.
Read the full abstract ▾
All authors Yuzhe Huang, Jiaping Wu, Jiaming Jiang, Hezhe Lin, Aikebaier Aierken, Yunlong Wang, Kun Cheng, Wanlin Li, Chenxi Xiao, Ziyuan Jiao, Yuanxin Zhong
Robotics
Viking Hill Dataset: A Lidar-Radar-Camera Dataset for Detection and Segmentation in Forest Scenes
Vladimír Kubelka, Oleksandr Kotlyar, Unal Artan et al. (+1 more)
Autonomous
robots operating under forest canopies need robust perception of trees and surrounding vegetation across varying seasonal conditions. Existing forestry
datasets provide
lidar or camera data with per-tree annotations, but none include co-registered 4D imaging radar – a modality of growing interest for its resilience to visual degradation, surface contamination, and vegetation occlusion. We introduce a multi-sensor forest
dataset collected by a mobile
robot equipped with a high-resolution FMCW imaging radar,
lidar, RGB camera, IMU, and RTK-GNSS. The site was recorded in two sessions under contrasting vegetation states, and 3D cuboid annotations – including per-tree diameter estimates – provide shared semantic labels across all three perception modalities. Furthermore, we provide baseline results for semantic
segmentation of the radar and
lidar point clouds using MinkowskiUNet. Radar achieves IoU scores competitive with
lidar for dominant classes (ground 91%, canopy 86%) while lagging on geometrically fine structures such as tree trunks (56% vs. 74%). A cross-modality analysis further compares
lidar and radar trunk
segmentation against an RGB detection model, and a diameter-stratified evaluation reveals how trunk
segmentation quality varies with tree size. Beyond
segmentation, the co-registered multi-modal data and RTK-GNSS-aided reference positioning support research in mapping, localization, and sensor fusion under canopy. The
dataset and annotation tools are publicly available.
Read the full abstract ▾
All authors Vladimír Kubelka, Oleksandr Kotlyar, Unal Artan, Martin Magnusson
Robotics
Monocular 3D Occupancy Perception for Robots on Sidewalks via Hybrid 2D-3D Learning
Yukai Ma, Joe Lin, Liu Liu et al. (+4 more)
Sidewalks in the real world are crowded, cluttered, and less structured than roads, making 3D occupancy prediction a key ingredient for the safe
navigation of mobile
robots such as delivery bots and electric wheelchairs. Existing occupancy learning pipelines are largely designed for on-road autonomous driving and often train on large-scale paired
LiDAR-RGB
datasets with dense 3D supervision and multiple camera inputs, which are costly to collect and do not adequately capture sidewalk-specific characteristics. We propose WalkOCC, a hybrid Ray-marching
monocular 3D occupancy perception framework for
robots operating on sidewalks. WalkOCC explicitly couples
geometric grounding from
LiDAR-RGB paired data with scalable learning from large-scale unpaired
monocular images. It bootstraps pseudo occupancy supervision from paired sequences and jointly learns image-level representations on additional 2D-only data. It yields stable optimization and improved generalization without requiring costly 3D occupancy annotations. Extensive experiments demonstrate consistent gains in prediction accuracy, fine-grained
segmentation of subtle urban structures such as curbs and gutters, and robustness to environmental and cross-embodiment shifts compared with self-supervised image-based baselines. To facilitate evaluation and benchmarking, we also introduce Sidewalk3D, a large-scale sidewalk perception
dataset with
LiDAR-camera paired sequences collected across multiple locations and time periods, along with 3D semantic occupancy annotations for evaluation. Code and data will be made available.
Read the full abstract ▾
All authors Yukai Ma, Joe Lin, Liu Liu, Honglin He, Lulu Ricketts, Brad Squicciarini, Bolei Zhou
Robotics
GCNGrasp-VP: Affordance-Guided View Planning for Efficient Task-Oriented Grasping
Zanjia Tong, Wenlong Dong, Chengjie Zhang et al. (+1 more)
Task-oriented grasping performance degrades significantly when object views suffer from occlusions. Existing task-oriented grasping methods typically assume task-relevant regions are visible in the initial frame, while view planning approaches enable active perception but often ignore task semantics and rely on time-consuming scene reconstruction. To address these limitations, we present GCNGrasp-VP, an efficient framework integrating affordance field prediction with active view planning. Central to this framework is GCNGrasp-v2, a task-oriented grasp model that simultaneously supports grasp evaluation and affordance field prediction, achieving constant-time inference complexity. Leveraging this capability, our Affordance-guided View Planner (Affordance-VP) utilizes the affordance field as an information gain metric to guide camera observation of task-relevant regions without requiring scene reconstruction. View planning results show that our method significantly outperforms scene-uncertainty-driven baselines with only one view adjustment. Real-world validation further confirms substantial improvements in grasp success rates for single-object scenarios while maintaining millisecond-level computational latency. Code and models are available at https://github.com/Instinct323/GCNGrasp-VP.
Read the full abstract ▾
All authors Zanjia Tong, Wenlong Dong, Chengjie Zhang, Hong Zhang