Robotics
Jonathan Crespo, Ramón Barber, O. M. Mozos et al. (+2 more)
Semantic
navigation is the
navigation paradigm in which environmental semantic concepts and their relationships are taken into account to plan the route of a mobile
robot. This paradigm facilitates the interaction with humans and the understanding of human environments in terms of
navigation goals and tasks. At the high level, a semantic
navigation system requires two main components: a semantic representation of the environment, and a reasoner system. This paper is focused on develop a model of the environment using semantic concepts. This paper presents two solutions for the semantic
navigation paradigm. Both systems implement an ontological model. Whilst the first one uses a relational database, the second one is based on KnowRob. Both systems have been integrated in a semantic navigator. We compare both systems at the qualitative and quantitative levels, and present an implementation on a mobile
robot as a proof of concept.
Read the full abstract ▾
All authors Jonathan Crespo, Ramón Barber, O. M. Mozos, Daniel Beßler, Michael Beetz
Robotics
Functionalization of Situated Robots via Vapour
Kadri-Ann Pankratov, Leonid Zinatullin, Adele Metsniit et al. (+2 more)
Tight matching with the environment is key to effective
robot operation in complex settings. Situated
robots that build their bodies in situ (e.g. by spinning) are uniquely positioned to exploit their surroundings, yet functionalization of these structures remains an integration challenge - multimaterial spinning requires complex spinneret multiplexing, and mixture doping is limited by additive availability and chemical stability. We propose instead using materials available in the environment to functionalize in situ spun webs, reducing payload and uniquely matching the structure to its surroundings. As a demonstration, we transform an optically scattering PVDF fiber web into an optically absorbing, polypyrrole-grafted structure via pyrrole vapour exposure. Two activator-delivery strategies are shown:
liquid infusion into a prefabricated web, and activator pre-
embedding in the spinning mixture. Beyond this proof-of-concept, we foresee broader applications including biohybrid
robots that exploit bacterial genomes for specific biomolecule synthesis in situ.
Read the full abstract ▾
All authors Kadri-Ann Pankratov, Leonid Zinatullin, Adele Metsniit, Marie Vihmar, Indrek Must
Robotics
LARD 2.0: Enhanced Datasets and Benchmarking for Autonomous Landing Systems
Yassine Bougacha, Geoffrey Delhomme, Mélanie Ducoffe et al. (+8 more)
This paper addresses key challenges in the development of autonomous landing systems, focusing on
dataset limitations for supervised training of Machine Learning (ML) models for object detection. Our main contributions include: (1) Enhancing
dataset diversity, by advocating for the inclusion of new sources such as BingMap aerial images and Flight Simulator, to widen the generation scope of an existing
dataset generator used to produce the
dataset LARD; (2) Refining the Operational Design Domain (ODD), addressing issues like unrealistic landing scenarios and expanding coverage to multi-runway airports; (3) Benchmarking ML models for autonomous landing systems, introducing a framework for evaluating object detection subtask in a complex multi-instances setting, and providing associated open-source models as a baseline for AI models' performance.
Read the full abstract ▾
All authors Yassine Bougacha, Geoffrey Delhomme, Mélanie Ducoffe, Augustin Fuchs, Jean-Brice Ginestet, Jacques Girard, Sofiane Kraiem, Franck Mamalet, Vincent Mussot, Claire Pagetti, Thierry Sammour
Robotics · Cross-listed from cs.CV
Language-Conditioned World Modeling for Visual Navigation
Yifei Dong, Fengyi Wu, Yilong Dai et al. (+8 more)
Goal-conditioned visual
navigation has been a long-standing testbed for
embodied AI. We study a natural
language-conditioned variant,
language-conditioned visual
navigation (LCVN), in which an
embodied agent must follow a natural
language instruction given only an initial
egocentric observation. Without access to goal images, the
agent must rely on
language to shape its perception and continuous control. We introduce the LCVN
Dataset, a
benchmark of 39,016 trajectories and 117,048 human-verified instructions spanning diverse environments and instruction styles. Building on this
benchmark, we study two complementary paradigms: (i) latent-imagination policy learning, in which a
diffusion-based world model (LCVN-WM) imagines future observations and an actor-critic
agent (LCVN-AC) learns its policy entirely within the imagined latent space; and (ii) unified autoregressive prediction, in which a single
multimodal backbone (LCVN-Uni) jointly predicts actions and observations in one forward pass over a shared
token sequence. Experiments show that two paradigms offer complementary strengths: latent imagination produces more temporally coherent rollouts, whereas unified prediction generalizes better to unseen environments. Targeted ablations further isolate the contributions of
language guidance, conditioning signals, and instruction style, clarifying when
language grounding versus dynamics modeling is the performance bottleneck. Together, these findings position LCVN as a testbed for studying how
language, imagination, and decision-making interact in
embodied agents.
Read the full abstract ▾
All authors Yifei Dong, Fengyi Wu, Yilong Dai, Lingdong Kong, Guangyu Chen, Yetong Sha, Qiyu Hu, Feng Liu, Siyu Huang, Qi Dai, Zhi-Qi Cheng
Robotics · Cross-listed from cs.CV
MAGICIAN: Efficient Long-Term Planning with Imagined Gaussians for Active Mapping
Shiyao Li, Antoine Guédon, Shizhe Chen et al. (+1 more)
Active mapping aims to determine how an
agent should move to efficiently reconstruct unknown environments. Most existing approaches rely on greedy next-best-view prediction, resulting in inefficient exploration and incomplete reconstruction. To address this, we introduce MAGICIAN, a novel long-term planning framework that maximizes accumulated surface coverage gain through Imagined Gaussians, a scene representation based on 3D Gaussian Splatting, derived from a pre-trained occupancy network with strong structural priors. This representation enables efficient coverage gain computation for any novel viewpoint via fast volumetric rendering, allowing its integration into a tree-search
algorithm for long-horizon planning. We update Imagined Gaussians and refine the trajectory in a closed loop. Our method achieves state-of-the-art performance across indoor and outdoor
benchmarks with varying action spaces, highlighting the advantage of long-term planning in active mapping.
Read the full abstract ▾
All authors Shiyao Li, Antoine Guédon, Shizhe Chen, Vincent Lepetit
Robotics
Trajectory Generation for Underactuated Soft Robot Manipulators using Discrete Elastic Rod Dynamics
Beibei Liu, Akua K. Dickson, Ran Jing et al. (+1 more)
Soft
robots are well suited for contact-rich tasks due to their compliance, yet this property makes accurate and tractable modeling challenging. Planning motions with dynamically-feasible trajectories requires models that capture arbitrary deformations, remain computationally efficient, and are compatible with underactuation. However, existing approaches balance these properties unevenly: continuum rod models provide physical accuracy but are computationally demanding, while reduced-order approximations improve efficiency at the cost of modeling fidelity. To address this, our work introduces a control-oriented reformulation of Discrete Elastic Rod (DER) dynamics for soft
robots, and a method to generate trajectories with these dynamics. The proposed formulation yields a control-affine representation while preserving certain first-principles force-deformation relationships. As a result, the generated trajectories are both dynamically feasible and consistent with the underlying actuation assumptions. We present our trajectory generation framework and validate it experimentally on a pneumatic soft robotic limb. Hardware results demonstrate consistently improved trajectory tracking performance over a constant-
curvature-based baseline, particularly under complex actuation conditions.
Read the full abstract ▾
All authors Beibei Liu, Akua K. Dickson, Ran Jing, Andrew P. Sabelhaus
Robotics · Cross-listed from cs.CV
A vision-language model and platform for temporally mapping surgery from video
Dani Kiyasseh
Mapping surgery is fundamental to developing operative guidelines and enabling autonomous robotic surgery. Recent advances in artificial intelligence (AI) have shown promise in mapping the behaviour of surgeons from
videos, yet current models remain narrow in scope, capturing limited behavioural components within single procedures, and offer limited translational value, as they remain inaccessible to practising surgeons. Here we introduce Halsted, a
vision-
language model trained on the Halsted
Surgical Atlas (HSA), one of the most comprehensive annotated
video libraries grown through an iterative self-labelling framework and encompassing over 650,000
videos across eight
surgical specialties. To facilitate benchmarking, we publicly release HSA-27k, a subset of the Halsted
Surgical Atlas. Halsted surpasses previous state-of-the-art models in mapping
surgical activity while offering greater comprehensiveness and computational efficiency. To bridge the longstanding translational gap of
surgical AI, we develop the Halsted web platform (https://halstedhealth.ai/) to provide surgeons anywhere in the world with the previously-unavailable capability of automatically mapping their own procedures within minutes. By standardizing unstructured
surgical video data and making these capabilities directly accessible to surgeons, our work brings
surgical AI closer to clinical deployment and helps pave the way toward autonomous robotic surgery.
Read the full abstract ▾
Robotics
Versatile Exoskeleton Control Supports Elderly Joint Energetics During Hip-Intensive Tasks
Jiefu Zhang, Nikhil V. Divekar, Chandramouli Krishnan et al. (+1 more)
Age-related mobility decline is frequently accompanied by a redistribution of joint kinetics, where older adults compensate for reduced ankle function by increasing demand on the hip. Paradoxically, this compensatory shift typically coincides with age-related reductions in maximal hip power. Although robotic exoskeletons can provide immediate energetic benefits, conventional control strategies have limited previous studies in this population to specific tasks such as steady-state walking, which do not fully reflect mobility demands in the home and community. Here, we implement a task-agnostic hip exoskeleton controller that is inherently sensitive to joint power and validate its efficacy in eight older adults. Across a battery of hip-intensive activities that included level walking, ramp ascent, stair climbing, and sit-to-stand transitions, the exoskeleton matched biological power profiles with high accuracy (mean cosine similarity 0.89). Assistance significantly reduced sagittal plane biological positive work by 24.7% at the hip and by 9.3% for the lower limb, while simultaneously augmenting peak total (biological + exoskeleton) hip power and reducing peak biological hip power and torque. These results suggest that hip exoskeletons can potentially enhance endurance through biological work reduction, and increase functional reserve through total power augmentation, serving as a promising biomechanical intervention to support elderly mobility.
Read the full abstract ▾
All authors Jiefu Zhang, Nikhil V. Divekar, Chandramouli Krishnan, Robert D. Gregg
Robotics
GIFT: Generalizing Intent for Flexible Test-Time Rewards
Fin Amin, Nathaniel Dennler and Andreea Bobu
Robots learn reward functions from user demonstrations, but these rewards often fail to generalize to new environments. This failure occurs because learned rewards latch onto spurious correlations in training data rather than the underlying human intent that demonstrations represent. Existing methods leverage visual or semantic similarity to improve robustness, yet these surface-level cues often diverge from what humans actually care about. We present Generalizing Intent for Flexible Test-Time Rewards (GIFT), a framework that grounds reward generalization in human intent rather than surface cues. GIFT leverages
language models to infer high-level intent from user demonstrations by contrasting preferred with non-preferred behaviors. At deployment, GIFT maps novel test states to behaviorally equivalent training states via intent-conditioned similarity, enabling learned rewards to generalize across distribution shifts without retraining. We evaluate GIFT on tabletop manipulation tasks with new objects and layouts. Across four simulated tasks with over 50 unseen objects, GIFT consistently outperforms visual and semantic similarity baselines in test-time pairwise win rate and state-
alignment F1 score. Real-world experiments on a 7-DoF Franka Panda
robot demonstrate that GIFT reliably transfers to physical settings. Further discussion can be found at https://mit-clear-lab.github.io/GIFT/
Read the full abstract ▾
Robotics
Allometric Scaling Laws for Bipedal Robots
Naomi Oke, Aja M. Carter, Ben Gu et al. (+4 more)
Legged
robots operate across a wide range of physical scales, but how their designs should be adapted as size changes remains unclear. Here, we tackle this question in two ways. First, we survey existing legged
robots to provide a broad context for the key scaling variables,
robot mass m and leg length L. We find the surprising result that bipedal
robot mass generally scales with the length squared, L^2, rather than the isometric prediction L^3. Then, to reduce the variance in design choices, we focus on a pair of previously developed bipeds that share the same quasi-passive morphology but differ by a factor of six in leg length, use different feet and controllers, and achieve different relative speeds. We reconstruct both
robots in a common 3-D simulation environment and scale each design over leg lengths from 0.02 to 1.2 meters under both mass models (mass is proportional to L^2 and is proportional to L^). The controlled comparison shows that velocity follows dynamic similarity, velocity is proportional to L^{1/2}, across designs and mass models, while the torque needed to sustain walking follows that tau is proportional to mL. Consequently, torque scales approximately with L^3 when m is proportional to L^2 and L^4 when mass is proportional to L^3. A 3-D foot-shape sweep further shows that foot dimensions that permit walking scale approximately linearly with leg length, but the speed-maximizing shape and the mechanism by which each
robot moves do not transfer by scaling alone. Overall, the results provide practical insights for rescaling legged systems that leverage natural body dynamics.
Read the full abstract ▾
All authors Naomi Oke, Aja M. Carter, Ben Gu, Steven Man, Cordelia Pride, Sarah Bergbreiter, Aaron M. Johnson
Robotics
Learning Sidewalk Autopilot from Multi-Scale Imitation with Corrective Behavior Expansion
Honglin He, Yukai Ma, Brad Squicciarini et al. (+2 more)
Sidewalk micromobility is a promising solution for last-mile transportation, but current learning-based control methods struggle in complex urban environments. Imitation learning (IL) learns policies from human demonstrations, yet its reliance on fixed offline data often leads to compounding errors, limited robustness, and poor generalization. To address these challenges, we propose a framework that advances IL through corrective behavior expansion and multi-scale imitation learning. On the data side, we augment teleoperation
datasets with diverse corrective behaviors and sensor augmentations to enable the policy to learn to recover from its own mistakes. On the model side, we introduce a multi-scale IL architecture that captures both short-horizon interactive behaviors and long-horizon goal-directed intentions via horizon-based trajectory clustering and hierarchical supervision. Real-world experiments show that our approach significantly improves robustness and generalization in diverse sidewalk scenarios.
Read the full abstract ▾
All authors Honglin He, Yukai Ma, Brad Squicciarini, Wayne Wu, Bolei Zhou
Robotics
Parallel OctoMapping: A Scalable Framework for Enhanced Path Planning in Autonomous Navigation
Yihui Mao, Tian Tan, Xuehui Shen et al. (+2 more)
Mapping is essential in robotics and autonomous systems because it provides the spatial foundation for path planning. Efficient mapping enables planning
algorithms to generate reliable paths while ensuring safety and adapting in real time to complex environments. Fixed-resolution mapping methods often produce overly conservative obstacle representations that lead to suboptimal paths or planning failures in cluttered scenes. To address this issue, we introduce Parallel OctoMapping (POMP), an efficient OctoMap-based mapping technique that maximizes available free space and supports multi-threaded computation. To the best of our knowledge, POMP is the first method that, at a fixed occupancy-grid resolution, refines the representation of free space while preserving map fidelity and compatibility with existing search-based planners. It can therefore be integrated into existing planning pipelines, yielding higher pathfinding success rates and shorter path lengths, especially in cluttered environments, while substantially improving computational efficiency.
Read the full abstract ▾
All authors Yihui Mao, Tian Tan, Xuehui Shen, Warren E. Dixon, Rushikesh Kamalapurkar
Robotics
Energy-Aware Collaborative Exploration for a UAV-UGV Team
Cahit Ikbal Er, Saikiran Juttu and Yasin Yazicioglu
We present an energy-aware collaborative exploration framework for a
UAV-UGV team operating in unknown environments, where the
UAV's energy constraint is modeled as a maximum flight-time limit. The
UAV executes a sequence of energy-bounded exploration tours, while the UGV simultaneously explores on the ground and serves as a mobile charging station. Rendezvous is enforced under a shared time budget so that the vehicles meet at the end of each tour before the
UAV reaches its flight-time limit. We construct a sparsely coupled air-ground roadmap using a density-aware layered probabilistic roadmap (PRM) and formulate tour selection over the roadmap as coupled orienteering problems (OPs) to maximize information gain subject to the rendezvous constraint. The resulting tours are constructed over collision-validated roadmap edges. We validate our method through simulation studies,
benchmark comparisons, and real-world experiments.
Read the full abstract ▾
Robotics
MapForest: A Modular Field Robotics System for Forest Mapping and Invasive Species Localization
Sandeep Zachariah, Francisco Yandun, Sachet Korada et al. (+1 more)
Monitoring and controlling invasive tree species across large forests, parks, and trail networks is challenging due to limited accessibility, reliance on manual scouting, and degraded under-canopy GNSS. We present MapForest, a modular field robotics system that transforms multi-modal sensor data into GIS-ready invasive-species maps. Our system features: (i) a compact, platform-agnostic sensing payload that can be rapidly mounted on
UAV, bicycle, or backpack platforms, and (ii) a software pipeline comprising
LiDAR-inertial mapping, image-based invasive-species detection, and georeferenced map generation. To ensure reliable operation in GNSS-intermittent environments, we enhance a
LiDAR-inertial mapping backbone with covariance-aware GNSS factors and robust loss kernels. We train an object detector to detect the Tree-of-Heaven (Ailanthus altissima) from onboard RGB imagery and fuse detections with the reconstructed map to produce geospatial outputs suitable for downstream decision making. We collected a
dataset spanning six sites across urban environments, parks, trails, and forests to evaluate individual system modules, and report end-to-end results on two sites containing Tree-of-Heaven. The enhanced mapping module achieved a trajectory deviation error of 1.95 m over a 1.2 km forest traversal, and the Tree-of-Heaven detector achieved an F1 score of 0.653. The
datasets and associated tooling are released to support reproducible research in forest mapping and invasive-species monitoring.
Read the full abstract ▾
All authors Sandeep Zachariah, Francisco Yandun, Sachet Korada, Abhisesh Silwal
Robotics
Wake Up to the Past: Using Memory to Model Fluid Wake Effects on Robots
Luca Vendruscolo, Eduardo Sebastián, Amanda Prorok et al. (+1 more)
Autonomous aerial and aquatic
robots that attain mobility by perturbing their medium, such as multicopters and torpedoes, produce wake effects that act as disturbances for adjacent
robots. Wake effects are hard to model and predict due to the chaotic spatio-temporal dynamics of the fluid, entangled with the physical geometry of the
robots and their complex motion patterns. Data-driven approaches using neural networks typically learn a
memory-less function that maps the current states of the two
robots to a force observed by the "sufferer"
robot. Such models often perform poorly in agile scenarios: since the wake effect has a finite propagation time, the disturbance observed by a sufferer
robot is some function of relative states in the past. In this work, we present an empirical study of the properties a wake-effect predictor must satisfy to accurately model the interactions between two
robots mediated by a fluid. We explore seven data-driven models designed to capture the spatio-temporal
evolution of fluid wake effects in four different media. This allows us to introspect the models and analyze the reasons why certain features enable improved accuracy in prediction across predictors and fluids. As experimental validation, we develop a planar rectilinear gantry for two spinning monocopters to test in real-world data with feedback control. The conclusion is that support of history of previous states as input and transport delay prediction substantially helps to learn an accurate wake-effect predictor.
Read the full abstract ▾
All authors Luca Vendruscolo, Eduardo Sebastián, Amanda Prorok, Ajay Shankar
Robotics
CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation
Letian Fu, Justin Yu, Karim El-Refai et al. (+13 more)
"Code-as-Policy" considers how executable code can complement data-intensive
Vision-
Language-Action (VLA) methods, yet their effectiveness as autonomous controllers for
embodied manipulation remains underexplored. We present CaP-X, an open-access framework for systematically studying Code-as-Policy
agents in
robot manipulation. At its core is CaP-Gym, an interactive environment in which
agents control
robots by synthesizing and executing programs that compose perception and control primitives. Building on this foundation, CaP-Bench evaluates frontier
language and
vision-
language models across varying levels of abstraction, interaction, and perceptual grounding. Across 12 models, CaP-Bench reveals a consistent trend: performance improves with human-crafted abstractions but degrades as these priors are removed, exposing a dependence on designer scaffolding. At the same time, we observe that this gap can be mitigated through scaling agentic test-time computation–through multi-turn interaction, structured execution feedback, visual differencing, automatic skill synthesis, and ensembled
reasoning–substantially improves robustness even when
agents operate over low-level primitives. These findings allow us to derive CaP-
Agent0, a training-free framework that recovers human-level reliability on several manipulation tasks in simulation and on real embodiments. We further introduce CaP-RL, showing reinforcement learning with verifiable rewards improves success rates and transfers from sim2real with minimal gap. Together, CaP-X provides a principled, open-access platform for advancing
embodied coding
agents.
Read the full abstract ▾
All authors Letian Fu, Justin Yu, Karim El-Refai, Ethan Kou, Haoru Xue, Huang Huang, Wenli Xiao, Guanzhi Wang, Dantong Niu, Fei-Fei Li, Guanya Shi, Jiajun Wu, Shankar Sastry, Yuke Zhu, Ken Goldberg, Linxi "Jim" Fan
Robotics · Cross-listed from cs.CV
ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model
Haichao Zhang, Yijiang Li, Shwai He et al. (+5 more)
Recent progress in latent world models (e.g., V-JEPA2) has shown promising capability in
forecasting future world states from
video observations. Nevertheless, dense prediction from a short observation window limits temporal context and can bias predictors toward local, low-level extrapolation, making it difficult to capture long-horizon semantics and reducing downstream utility.
Vision–
language models (VLMs), in contrast, provide strong semantic grounding and general knowledge by
reasoning over uniformly sampled frames, but they are not ideal as standalone dense predictors due to compute-driven sparse sampling, a
language-output bottleneck that compresses fine-grained interaction states into text-oriented representations, and a data-regime mismatch when adapting to small action-conditioned
datasets. We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance via a dual-temporal pathway: a dense JEPA branch for fine-grained motion and interaction cues, and a uniformly sampled VLM thinker branch with a larger temporal stride for knowledge-rich guidance. To transfer the VLM's progressive
reasoning signals effectively, we introduce a hierarchical pyramid representation extraction module that aggregates multi-layer VLM representations into guidance features compatible with latent prediction. Experiments on hand-manipulation trajectory prediction show that our method outperforms both a strong VLM-only baseline and a JEPA-predictor baseline, and yields more robust long-horizon rollout behavior.
Read the full abstract ▾
All authors Haichao Zhang, Yijiang Li, Shwai He, Tushar Nagarajan, Mingfei Chen, Jianglin Lu, Ang Li, Yun Fu
Robotics · Cross-listed from cs.CV
DualCoT-VLA: Visual-Linguistic Chain of Thought via Parallel Reasoning for Vision-Language-Action Models
Zhide Zhong, Junfeng Li, Junjie He et al. (+10 more)
Vision-
Language-Action (VLA) models map visual observations and
language instructions directly to robotic actions. While effective for simple tasks, standard VLA models often struggle with complex, multi-step tasks requiring logical planning, as well as precise manipulations demanding fine-grained spatial perception. Recent efforts have incorporated Chain-of-Thought (CoT)
reasoning to endow VLA models with a “thinking before acting” capability. However, current CoT-based VLA models face two critical limitations: 1) an inability to simultaneously capture low-level visual details and high-level logical planning due to their reliance on isolated, single-modal CoT; 2) high inference latency with compounding errors caused by step-by-step autoregressive decoding. To address these limitations, we propose DualCoT-VLA, a visual-linguistic CoT method for VLA models with a parallel
reasoning mechanism. To achieve comprehensive multi-modal
reasoning, our method integrates a visual CoT for low-level spatial understanding and a linguistic CoT for high-level task planning. Furthermore, to overcome the latency bottleneck, we introduce a parallel CoT mechanism that incorporates two sets of learnable query
tokens, shifting autoregressive
reasoning to single-step forward
reasoning. Extensive experiments demonstrate that our DualCoT-VLA achieves state-of-the-art performance on the LIBERO and RoboCasa GR1
benchmarks, as well as in real-world platforms.
Read the full abstract ▾
All authors Zhide Zhong, Junfeng Li, Junjie He, Haodong Yan, Xin Gong, Guanyi Zhao, Yingjie Cai, Jiantao Gao, Xu Yan, Bingbing Liu, Yingcong Chen, Liuqing Yang, Haoang Li
Robotics
UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos
Gu Zhang, Qicheng Xu, Haozhe Zhang et al. (+16 more)
Dexterous manipulation remains challenging due to the cost of collecting real-
robot teleoperation data, the heterogeneity of hand embodiments, and the high dimensionality of control. We present UniDex, a
robot foundation suite that couples a large-scale
robot-centric
dataset with a unified
vision-
language-action (VLA) policy and a practical human-data capture setup for universal dexterous hand control. First, we construct UniDex-
Dataset, a
robot-centric
dataset over 50K trajectories across eight dexterous hands (6–24 DoFs), derived from
egocentric human
video datasets. To transform human data into
robot-executable trajectories, we employ a human-in-the-loop retargeting procedure to align fingertip trajectories while preserving plausible hand-object contacts, and we operate on explicit 3D pointclouds with human hands masked to narrow kinematic and visual gaps. Second, we introduce the Function-Actuator-Aligned Space (FAAS), a unified action space that maps functionally similar actuators to shared coordinates, enabling cross-hand transfer. Leveraging FAAS as the action parameterization, we train UniDex-VLA, a 3D VLA policy pretrained on UniDex-
Dataset and finetuned with task demonstrations. In addition, we build UniDex-Cap, a simple portable capture setup that records synchronized RGB-D streams and human hand poses and converts them into
robot-executable trajectories to enable human-
robot data co-training that reduces reliance on costly
robot demonstrations. On challenging tool-use tasks across two different hands, UniDex-VLA achieves 81% average task progress and outperforms prior VLA baselines by a large margin, while exhibiting strong spatial, object, and zero-shot cross-hand generalization. Together, UniDex-
Dataset, UniDex-VLA, and UniDex-Cap provide a scalable foundation suite for universal dexterous manipulation.
Read the full abstract ▾
All authors Gu Zhang, Qicheng Xu, Haozhe Zhang, Jianhan Ma, Long He, Yiming Bao, Zeyu Ping, Zhecheng Yuan, Chenhao Lu, Chengbo Yuan, Tianhai Liang, Xiaoyu Tian, Maanping Shao, Feihong Zhang, Mingyu Ding, Yang Gao, Hao Zhao, Hang Zhao, Huazhe Xu
Robotics
DexDrummer: In-Hand, Contact-Rich, and Long-Horizon Dexterous Robot Drumming
Hung-Chieh Fang, Amber Xie, Jennifer Grannen et al. (+2 more)
Performing in-hand, contact-rich, and long-horizon dexterous manipulation remains an unsolved challenge in robotics. Prior hand dexterity works have considered each of these three challenges in isolation, yet do not combine these skills into a single, complex task. To further test the capabilities of dexterity, we propose drumming as a testbed for dexterous manipulation. Drumming naturally integrates all three challenges: it involves in-hand control for stabilizing and adjusting the drumstick with the fingers, contact-rich interaction through repeated striking of the drum surface, and long-horizon coordination when switching between drums and sustaining rhythmic play. We present DexDrummer, a hierarchical object-centric bimanual drumming policy trained in simulation with sim-to-real transfer. The framework reduces the exploration difficulty of pure reinforcement learning by combining trajectory planning with residual RL corrections for fast transitions between drums. A dexterous manipulation policy handles contact-rich dynamics, guided by rewards that explicitly model both finger-stick and stick-drum interactions. In simulation, we show our policy can play two styles of music: multi-drum, bimanual songs and challenging, technical exercises that require increased dexterity. Across simulated bimanual tasks, our dexterous, reactive policy outperforms a fixed grasp policy by 1.87x across easy songs and 1.22x across hard songs F1 scores. In real-world tasks, we show song performance across a multi-drum setup. DexDrummer is able to play our training song and its extended version with an F1 score of 1.0.
Read the full abstract ▾
All authors Hung-Chieh Fang, Amber Xie, Jennifer Grannen, Kenneth Llontop, Dorsa Sadigh
Robotics
Make Tracking Easy: Neural Motion Retargeting for Humanoid Whole-body Control
Qingrui Zhao, Kaiyue Yang, Xiyu Wang et al. (+6 more)
Humanoid robots require diverse motor skills to integrate into complex environments, but bridging the kinematic and dynamic embodiment gap from human data remains a major bottleneck. We demonstrate through Hessian analysis that traditional optimization-based retargeting is inherently non-convex and prone to local optima, leading to physical artifacts like joint jumps and self-penetration. To address this, we reformulate the targeting problem as learning data distribution rather than optimizing optimal solutions, where we propose NMR, a Neural Motion Retargeting framework that transforms static
geometric mapping into a dynamics-aware learned process. We first propose Clustered-Expert Physics Refinement (CEPR), a hierarchical data pipeline that leverages VAE-based motion clustering to group heterogeneous movements into latent motifs. This strategy significantly reduces the computational overhead of massively parallel reinforcement learning experts, which project and repair noisy human demonstrations onto the
robot's feasible motion
manifold. The resulting high-fidelity data supervises a non-autoregressive CNN-
Transformer architecture that reasons over global temporal context to suppress reconstruction noise and bypass
geometric traps. Experiments on the Unitree G1
humanoid across diverse dynamic tasks (e.g., martial arts, dancing) show that NMR eliminates joint jumps and significantly reduces self-collisions compared to state-of-the-art baselines. Furthermore, NMR-generated references accelerate the convergence of downstream whole-body control policies, establishing a scalable path for bridging the human-
robot embodiment gap.
Read the full abstract ▾
All authors Qingrui Zhao, Kaiyue Yang, Xiyu Wang, Shiqi Zhao, Yi Lu, Xinfang Zhang, Qiu Shen, Xiao-Xiao Long, Xun Cao
Robotics
Cross-Modal Reinforcement Learning for Navigation with Degraded Depth Measurements
Omkar Sawant, Luca Zanatta, Grzegorz Malczyk et al. (+1 more)
This paper presents a cross-modal learning framework that exploits complementary information from depth and grayscale images for robust
navigation. We introduce a Cross-Modal Wasserstein Autoencoder that learns shared latent representations by enforcing cross-modal consistency, enabling the system to infer depth-relevant features from grayscale observations when depth measurements are corrupted. The learned representations are integrated with a Reinforcement Learning-based policy for collision-free
navigation in unstructured environments when depth sensors experience degradation due to adverse conditions such as poor lighting or reflective surfaces. Simulation and real-world experiments demonstrate that our approach maintains robust performance under significant depth degradation and successfully transfers to real environments.
Read the full abstract ▾
All authors Omkar Sawant, Luca Zanatta, Grzegorz Malczyk, Kostas Alexis
Robotics · Cross-listed from cs.HC
Feasibility of Augmented Reality-Guided Robotic Ultrasound with Cone-Beam CT Integration for Spine Procedures
Tianyu Song, Felix Pabst, Feng Li et al. (+5 more)
Accurate needle placement in spine interventions is critical for effective pain management, yet it depends on reliable identification of anatomical landmarks and careful trajectory planning. Conventional imaging guidance often relies both on CT and X-ray fluoroscopy, exposing patients and staff to high dose of radiation while providing limited real-time 3D feedback. We present an optical see-through augmented reality (OST-AR)-guided robotic system for spine procedures that provides in situ visualization of spinal structures to support needle trajectory planning. We integrate a cone-beam CT (CBCT)-derived 3D spine model which is co-registered with live ultrasound, enabling users to combine global anatomical context with local, real-time imaging. We evaluated the system in a phantom user study involving two representative spine procedures: facet joint injection and lumbar puncture. Sixteen participants performed insertions under two visualization conditions: conventional screen vs. AR. Results show that AR significantly reduces execution time and across-task placement error, while also improving usability, trust, and spatial understanding and lowering cognitive workload. These findings demonstrate the feasibility of AR-guided robotic ultrasound for spine interventions, highlighting its potential to enhance accuracy, efficiency, and user experience in image-guided procedures.
Read the full abstract ▾
All authors Tianyu Song, Felix Pabst, Feng Li, Yordanka Velikova, Miruna-Alexandra Gafencu, Yuan Bi, Ulrich Eck, Nassir Navab
Robotics
VersualRL: Closed-Loop Verbal Reinforcement Learning with Visual Execution Feedback for Task-Level Robot Planning
Dmitrii Plotnikov, Iaroslav Kolomiets, Dmitrii Maliukov et al. (+9 more)
We introduce VersualRL, a closed-loop framework for task-level
robot planning that uses visual execution feedback to iteratively refine executable Behavior Trees through structured natural-
language critique. VersualRL combines a
vision-
language model critic, which analyzes visual observations and Behavior Tree execution traces, with a large
language model actor that performs discrete and interpretable policy updates. During physical deployment, both foundation models remain frozen, while the executed Behavior Tree is updated at the symbolic level without online gradient-based policy optimization. This enables transparent
reasoning, explicit causal feedback, and human-interpretable policy
evolution. We validate VersualRL on a real mobile
robot performing a multi-stage
navigation and manipulation task under execution uncertainty. Experimental results show that the framework supports explainable policy improvements, closed-loop adaptation to execution failures, and successful deployment on physical robotic systems.
Read the full abstract ▾
All authors Dmitrii Plotnikov, Iaroslav Kolomiets, Dmitrii Maliukov, Dmitrij Kosenkov, Daniia Zinniatullina, Artem Trandofilov, Georgii Gazaryan, Kirill Bogatikov, Timofei Kozlov, Mikhail Konenkov, Miguel Altamirano Cabrera, Dzmitry Tsetserukou
Robotics · Cross-listed from eess.SY
From Singleton Obstacles to Clutter: Translation Invariant Compositional Avoid Sets
Prashant Solanki, Jasper Van Beers and Coen De Visser
This paper studies obstacle avoidance under translation invariant dynamics using an avoid-side travel cost Hamilton Jacobi formulation. For running costs that are zero outside an obstacle and strictly negative inside it, we prove that the value function is non-positive everywhere, equals zero exactly outside the avoid set, and is strictly negative exactly on it. Under translation invariance, this yields a reuse principle: the value of any translated obstacle is obtained by translating a single template value function. We show that the pointwise minimum of translated template values exactly characterizes the union of the translated single-obstacle avoid sets and provides a conservative inner certificate of unavoidable collision in clutter. To reduce conservatism, we introduce a blockwise composition framework in which subsets of obstacles are merged and solved jointly. This yields a hierarchy of conservative certificates from singleton reuse to the exact clutter value, together with monotonicity under block merging and an exactness criterion based on the existence of a common clutter avoiding control. The framework is illustrated on a Dubins car example in a repeated clutter field.
Read the full abstract ▾
Robotics
ROBOGATE: Adaptive Failure Discovery for Safe Robot Policy Deployment via Two-Stage Boundary-Focused Sampling
Azuki Kim
Deploying learned
robot manipulation policies in industrial settings requires rigorous pre-deployment validation, yet exhaustive testing across high-dimensional parameter spaces is intractable. We present ROBOGATE, a deployment risk management framework that combines physics-based simulation with a two-stage adaptive sampling strategy to efficiently discover failure boundaries in the operational parameter space. Stage 1 employs Latin Hypercube Sampling (LHS) across an 8-dimensional parameter space; Stage 2 applies boundary-focused sampling concentrated in the 30-70% success rate transition zone. Using NVIDIA Isaac Sim with Newton physics, we evaluate a scripted pick-and-place controller across four
robot embodiments – Franka Panda (7-DOF), UR3e (6-DOF), UR5e (6-DOF), and UR10e (6-DOF) – totaling over 50,000 experiments. Our logistic regression risk model achieves AUC 0.780 and identifies a closed-form failure boundary equation. We further
benchmark eight VLA (
Vision-
Language-Action) policies, including a fine-tuned NVIDIA GR00T N1.6 (3B) trained on LIBERO-Spatial for 20K steps. The same checkpoint achieves 97.65% success rate on LIBERO (MuJoCo) but 0% on RoboGate's 68 industrial scenarios in NVIDIA Isaac Sim – a 97.65 percentage point cross-simulator gap on a single model that underscores the deployment validation challenge. Inspired by the validation-layer paradigm NVIDIA codified for
quantum computing with Ising, ROBOGATE provides this validation layer for Physical AI. Open-source.
Read the full abstract ▾
Robotics
Programming Manufacturing Robots with Imperfect AI: LLMs as Tuning Experts for FDM Print Configuration Selection
Ekta U. Samani and Christopher G. Atkeson
We use fused deposition modeling (FDM) 3D printing as a case study of how manufacturing
robots can use imperfect AI. In FDM, print configuration strongly affects output quality. Yet, novice users typically rely on default configurations, trial-and-error, or direct recommendations from generic AI models (e.g., ChatGPT). These strategies can produce complete prints, but they do not reliably meet specific objectives. We present a modular approach that treats an LLM as a source of tuning expertise. We embed this source of expertise within a
Bayesian optimization loop. An approximate evaluator scores each candidate print configuration and returns structured diagnostics, which the LLM uses to propose natural-
language adjustments that are compiled into machine-actionable guidance for optimization. On 100 Thingi10k parts, our LLM-guided loop achieves the best configuration on 78% objects with 0% likely-to-fail cases, while direct AI model recommendations are rarely best and exhibit 15% likely-to-fail cases. These results suggest that LLMs provide more value as constrained decision modules in optimization loops than as end-to-end oracles for print configuration selection. We expect this result to extend to broader LLM-based
robot programming.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
FreeArtGS: Articulated Gaussian Splatting Under Free-moving Scenario
Hang Dai, Hongwei Fan, Han Zhang et al. (+3 more)
The increasing demand for augmented reality and robotics is driving the need for articulated object reconstruction with high scalability. However, existing settings for reconstructing from discrete articulation states or casual
monocular videos require non-trivial axis
alignment or suffer from insufficient coverage, limiting their applicability. In this paper, we introduce FreeArtGS, a novel method for reconstructing articulated objects under free-moving scenario, a new setting with a simple setup and high scalability. FreeArtGS combines free-moving part
segmentation with joint estimation and end-to-end optimization, taking only a
monocular RGB-D
video as input. By optimizing with the priors from off-the-shelf point-tracking and feature models, the free-moving part
segmentation module identifies rigid parts from relative motion under unconstrained capture. The joint estimation module calibrates the unified object-to-camera poses and recovers joint type and axis robustly from part
segmentation. Finally, 3DGS-based end-to-end optimization is implemented to jointly reconstruct visual textures, geometry, and joint angles of the articulated object. We conduct experiments on two
benchmarks and real-world free-moving articulated objects. Experimental results demonstrate that FreeArtGS consistently excels in reconstructing free-moving articulated objects and remains highly competitive in previous reconstruction settings, proving itself a practical and effective solution for realistic asset generation. The project page is available at: https://freeartgs.github.io/
Read the full abstract ▾
All authors Hang Dai, Hongwei Fan, Han Zhang, Duojin Wu, Jiyao Zhang, Hao Dong
Robotics
Do World Action Models Generalize Better than VLAs? A Robustness Study
Zhanguang Zhang, Zhiyuan Li, Behnam Rahmati et al. (+11 more)
Robot action planning in the real world is challenging as it requires not only understanding the current state of the environment but also predicting how it will evolve in response to actions.
Vision-
language-action (VLA), which repurpose large-scale
vision-
language models for
robot action generation using action experts, have achieved notable success across a variety of robotic tasks. Nevertheless, their performance remains constrained by the scope of their training data, exhibiting limited generalization to unseen scenarios and vulnerability to diverse contextual perturbations. More recently, world models have been revisited as an alternative to VLAs. These models, referred to as world action models (WAMs), are built upon world models that are trained on large corpora of
video data to predict future states. With minor adaptations, their latent representation can be decoded into
robot actions. It has been suggested that their explicit dynamic prediction capacity, combined with spatiotemporal priors acquired from web-scale
video pretraining, enables WAMs to generalize more effectively than VLAs. In this paper, we conduct a comparative study of prominent state-of-the-art VLA policies and recently released WAMs. We evaluate their performance on the LIBERO-Plus and RoboTwin 2.0-Plus
benchmarks under various visual and
language perturbations. Our results show that WAMs achieve strong robustness, with LingBot-VA reaching 74.2% success rate on RoboTwin 2.0-Plus and Cosmos-Policy achieving 82.2% on LIBERO-Plus. While VLAs such as $\pi_{0.5}$ can achieve comparable robustness on certain tasks, they typically require extensive training with diverse robotic
datasets and varied learning objectives. The evaluation code for the RoboTwin2.0-Plus
benchmark is available at: https://
robot-robustness.github.io/RoboTwin2.0-Plus/.
Read the full abstract ▾
All authors Zhanguang Zhang, Zhiyuan Li, Behnam Rahmati, Rui Heng Yang, Yintao Ma, Amir Rasouli, Sajjad Pakdamansavoji, Yangzheng Wu, Lingfeng Zhang, Tongtong Cao, Feng Wen, Xinyu Wang, Xingyue Quan, Yingxue Zhang
Robotics · Cross-listed from cs.GR
MineRobot: An Actuator-Centered Kinematic Modeling and Solving Framework for Underground Mining Robots
Shengzhe Hou, Xinming Lu, Tianyu Zhang et al. (+2 more)
Underground mining
robots are increasingly modeled for planning, operator training, and digital-twin workflows, where reliable actuator-level kinematics is needed to reduce hazardous in situ trials. Unlike typical open-chain industrial manipulators, representative mining machines are often linear-actuator-driven closed-chain mechanisms with planar four-bar linkages, making reusable kinematic modeling and real-time FK/IK solving challenging. We present \textit{\hl{MineRobot}}, an actuator-centered framework for modeling and solving the kinematics of this representative mechanism class. MineRobot introduces the Mining
Robot Description Format (MRDF), a domain-specific representation that parameterizes mining-
robot kinematics with native semantics for actuators and loop closures. It then contracts planar four-bar substructures into generalized joints and extracts, for each actuator, an Independent Topologically Equivalent Path (ITEP) classified into four canonical types. Based on this decomposition, per-type solvers are composed into a sequential forward-kinematics (FK) pipeline, while inverse kinematics (IK) is formulated as a bound-constrained actuator-length optimization solved by a Gauss–Seidel-style update scheme. By converting coupled closed-chain kinematics into small
topology-aware solves, MineRobot reduces
robot-specific hand derivations and supports efficient repeated FK/IK computation without treating each query as a full coupled constraint-solving problem. Experiments on representative underground mining
robots demonstrate real-time FK performance and robust IK convergence within the tested operating ranges, supporting the use of MineRobot as an actuator-centered kinematic layer for planning, training, and digital-twin workflows.
Read the full abstract ▾
All authors Shengzhe Hou, Xinming Lu, Tianyu Zhang, Changqing Yan, Xingli Zhang