Robotics
POrTAL: Plan-Orchestrated Tree Assembly for Lookahead
Evan Conway, David Porfirio, David Chan et al. (+2 more)
When tasking
robots in partially observable environments, these
robots must efficiently and robustly plan to achieve task goals under uncertainty. Although many probabilistic planning
algorithms exist for this purpose, these
algorithms can be inefficient if executed with the
robot's limited computational resources, or may produce policies that take more steps than expected to achieve the goal. We therefore created a new, lightweight, probabilistic planning
algorithm, Plan-Orchestrated Tree Assembly for Lookahead (POrTAL), that combines the strengths of two baseline planning
algorithms, FF-Replan and POMCP. We demonstrate that POrTAL is an anytime
algorithm that generally outperforms these baselines in terms of the final executed plan length given bounded computation time, especially for problems with only moderate levels of uncertainty.
Read the full abstract ▾
All authors Evan Conway, David Porfirio, David Chan, Mark Roberts, Laura M. Hiatt
Robotics
KALIKO: Kalman-Implicit Koopman Operator Learning For Prediction of Nonlinear Dynamical Systems
Albert H. Li, Ivan Dario Jimenez Rodriguez, Joel W. Burdick et al. (+2 more)
Long-horizon dynamical prediction is fundamental in robotics and control, underpinning canonical methods like model predictive control. Yet, many systems and disturbance phenomena are difficult to model due to effects like nonlinearity, chaos, and high-dimensionality. Koopman theory addresses this by modeling the linear
evolution of
embeddings of the state under an infinite-dimensional linear operator that can be approximated with a suitable finite basis of
embedding functions, effectively trading model nonlinearity for representational complexity. However, explicitly computing a good choice of basis is nontrivial, and poor choices may cause inaccurate forecasts or overfitting. To address this, we present Kalman-Implicit Koopman Operator (KALIKO) Learning, a method that leverages the Kalman filter to implicitly learn
embeddings corresponding to latent dynamics without requiring an explicit encoder. KALIKO produces interpretable representations consistent with both theory and prior works, yielding high-quality reconstructions and inducing a globally linear latent dynamics. Evaluated on wave data generated by a high-dimensional PDE, KALIKO surpasses several baselines in open-loop prediction and in a demanding closed-loop simulated control task: stabilizing an underactuated manipulator's payload by predicting and compensating for strong wave disturbances.
Read the full abstract ▾
All authors Albert H. Li, Ivan Dario Jimenez Rodriguez, Joel W. Burdick, Yisong Yue, Aaron D. Ames
Robotics · Cross-listed from cs.CV
Flux4D: Flow-based Unsupervised 4D Reconstruction
Jingkang Wang, Henry Che, Yun Chen et al. (+4 more)
Reconstructing large-scale dynamic scenes from visual observations is a fundamental challenge in computer
vision, with critical implications for robotics and autonomous systems. While recent differentiable rendering methods such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have achieved impressive photorealistic reconstruction, they suffer from scalability limitations and require annotations to decouple actor motion. Existing self-supervised methods attempt to eliminate explicit annotations by leveraging motion cues and
geometric priors, yet they remain constrained by per-scene optimization and sensitivity to hyperparameter tuning. In this paper, we introduce Flux4D, a simple and scalable framework for 4D reconstruction of large-scale dynamic scenes. Flux4D directly predicts 3D Gaussians and their motion dynamics to reconstruct sensor observations in a fully unsupervised manner. By adopting only photometric losses and enforcing an "as static as possible" regularization, Flux4D learns to decompose dynamic elements directly from raw data without requiring pre-trained supervised models or foundational priors simply by training across many scenes. Our approach enables efficient reconstruction of dynamic scenes within seconds, scales effectively to large
datasets, and generalizes well to unseen environments, including rare and unknown objects. Experiments on outdoor driving
datasets show Flux4D significantly outperforms existing methods in scalability, generalization, and reconstruction quality.
Read the full abstract ▾
All authors Jingkang Wang, Henry Che, Yun Chen, Ze Yang, Lily Goli, Sivabalan Manivasagam, Raquel Urtasun
Robotics
GRAND: Guidance, Rebalancing, and Assignment for Networked Dispatch in Multi-Agent Path Finding
Johannes Gaber, Meshal Alharbi, Daniele Gammelli et al. (+1 more)
Large
robot fleets are now common in warehouses and other logistics settings, where small control gains translate into large operational impacts. In this article, we address task scheduling for lifelong Multi-
Agent Pickup-and-Delivery (MAPD) and propose a hybrid method that couples learning-based global guidance with lightweight optimization. A graph neural network policy trained via reinforcement learning outputs a desired distribution of free
agents over an aggregated warehouse graph. This signal is converted into region-to-region rebalancing through a minimum-cost flow, and finalized by small, local assignment problems, preserving accuracy while keeping per-step latency within a 1 s compute budget. We call this approach GRAND: a hierarchical
algorithm that relies on Guidance, Rebalancing, and Assignment to explicitly leverage the workspace Network structure and Dispatch
agents to tasks. On congested warehouse
benchmarks from the League of
Robot Runners (LoRR) with up to 500
agents, our approach improves throughput by up to 10% over the 2024 winning scheduler while maintaining real-time execution. The results indicate that coupling graph-structured learned guidance with tractable solvers reduces congestion and yields a practical, scalable blueprint for high-throughput scheduling in large fleets.
Read the full abstract ▾
All authors Johannes Gaber, Meshal Alharbi, Daniele Gammelli, Gioele Zardini
Robotics
Multi-Agent Reinforcement Learning and Real-Time Decision-Making in Robotic Soccer for Virtual Environments
Aya Taourirte and Md Sohag Mia
The deployment of multi-
agent systems in dynamic,
adversarial environments like robotic soccer necessitates real-time decision-making, sophisticated cooperation, and scalable
algorithms to avoid the curse of dimensionality. While Reinforcement Learning (RL) offers a promising framework, existing methods often struggle with the multi-granularity of tasks (long-term strategy vs. instant actions) and the complexity of large-scale
agent interactions. This paper presents a unified Multi-
Agent Reinforcement Learning (MARL) framework that addresses these challenges. First, we establish a baseline using Proximal Policy Optimization (PPO) within a client-server architecture for real-time action scheduling, with PPO demonstrating superior performance (4.32 avg. goals, 82.9% ball control). Second, we introduce a Hierarchical RL (HRL) structure based on the options framework to decompose the problem into a high-level trajectory planning layer (modeled as a Semi-Markov Decision Process) and a low-level action execution layer, improving global strategy (avg. goals increased to 5.26). Finally, to ensure scalability, we integrate mean-field theory into the HRL framework, simplifying many-
agent interactions into a single
agent vs. the population average. Our mean-field actor-critic method achieves a significant performance boost (5.93 avg. goals, 89.1% ball control, 92.3% passing accuracy) and enhanced training stability. Extensive simulations of 4v4 matches in the Webots environment validate our approach, demonstrating its potential for robust, scalable, and cooperative behavior in complex multi-
agent domains.
Read the full abstract ▾
Robotics
Video2Act: A Dual-System Video Diffusion Policy with Robotic Spatio-Motional Modeling
Yueru Jia, Jiaming Liu, Shengbang Liu et al. (+7 more)
Robust perception and dynamics modeling are fundamental to real-world robotic policy learning. Recent methods employ
video diffusion models (VDMs) to enhance robotic policies, improving their understanding and modeling of the physical world. However, existing approaches overlook the coherent and physically consistent motion representations inherently encoded across frames in VDMs. To this end, we propose
Video2Act, a framework that efficiently guides robotic action learning by explicitly integrating spatial and motion-aware representations. Building on the inherent representations of VDMs, we extract foreground boundaries and inter-frame motion variations while filtering out background noise and task-irrelevant biases. These refined representations are then used as additional conditioning inputs to a
diffusion transformer (DiT) action head, enabling it to reason about what to manipulate and how to move. To mitigate inference inefficiency, we propose an asynchronous dual-system design, where the VDM functions as the slow System 2 and the DiT head as the fast System 1, working collaboratively to generate adaptive actions. By providing motion-aware conditions to System 1,
Video2Act maintains stable manipulation even with low-frequency updates from the VDM. For evaluation,
Video2Act surpasses previous state-of-the-art VLA methods by 7.7% in simulation and 21.7% in real-world tasks in terms of average success rate, further exhibiting strong generalization capabilities.
Read the full abstract ▾
All authors Yueru Jia, Jiaming Liu, Shengbang Liu, Rui Zhou, Wanhe Yu, Yuyang Yan, Xiaowei Chi, Yandong Guo, Boxin Shi, Shanghang Zhang
Robotics · Cross-listed from cs.GR
SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Control
Yuxuan Mu, Ziyu Zhang, Yi Shi et al. (+9 more)
Data-driven motion priors that can guide
agents toward producing naturalistic behaviors play a pivotal role in creating life-like virtual characters.
Adversarial imitation learning has been a highly effective method for learning motion priors from reference motion data. However,
adversarial priors, with few exceptions, need to be retrained for each new controller, thereby limiting their reusability and necessitating the retention of the reference motion data when applied to downstream tasks. In this work, we present Score-Matching Motion Priors (SMP), which leverages pre-trained motion
diffusion models and score
distillation sampling (SDS) to create reusable task-agnostic motion priors. SMPs can be pre-trained on a motion
dataset, independent of any control policy or task. Once trained, SMPs can be kept frozen and reused as general-purpose reward functions to train new policies to produce naturalistic behaviors for downstream tasks. We show that a general motion prior trained on large-scale
datasets can be repurposed into a variety of style-specific priors. Furthermore, SMP can compose different styles to synthesize new styles not present in the original
dataset. Our method can create reusable and modular motion priors that produce high-quality motions comparable to state-of-the-art
adversarial imitation learning methods. In our experiments, we demonstrate the effectiveness of SMP across a diverse suite of control tasks with physically simulated
humanoid characters.
Video available at https://youtu.be/jBA2tWk6vzU
Read the full abstract ▾
All authors Yuxuan Mu, Ziyu Zhang, Yi Shi, Dun Yang, Minami Matsumoto, Kotaro Imamura, Guy Tevet, Chuan Guo, Michael Taylor, Chang Shu, Pengcheng Xi, Xue Bin Peng
Robotics · Cross-listed from cs.CV
SurfFill: Completion of LiDAR Point Clouds via Gaussian Surfel Splatting
Svenja Strobel, Matthias Innmann, Bernhard Egger et al. (+2 more)
LiDAR-captured point clouds are often considered the gold standard in active 3D reconstruction. While their accuracy is exceptional in flat regions, the capturing is susceptible to miss small
geometric structures and may fail with dark, absorbent materials. Alternatively, capturing multiple photos of the scene and applying 3D photogrammetry can infer these details as they often represent feature-rich regions. However, the accuracy of
LiDAR for featureless regions is rarely reached. Therefore, we suggest combining the strengths of
LiDAR and camera-based capture by introducing SurfFill: a Gaussian surfel-based
LiDAR completion scheme. We analyze
LiDAR capturings and attribute
LiDAR beam divergence as a main factor for artifacts, manifesting mostly at thin structures and edges. We use this insight to introduce an ambiguity heuristic for completed scans by evaluating the change in density in the point cloud. This allows us to identify points close to missed areas, which we can then use to grow additional points from to complete the scan. For this point growing, we constrain Gaussian surfel reconstruction to focus optimization and densification on these ambiguous areas. Finally, Gaussian primitives of the reconstruction in ambiguous areas are extracted and sampled for points to complete the point cloud. To address the challenges of large-scale reconstruction, we extend this pipeline with a divide-and-conquer scheme for building-sized point cloud completion. We evaluate on the task of
LiDAR point cloud completion of synthetic and real-world scenes and find that our method outperforms previous reconstruction methods.
Read the full abstract ▾
All authors Svenja Strobel, Matthias Innmann, Bernhard Egger, Marc Stamminger, Linus Franke
Robotics · Cross-listed from cs.CV
U4D: Uncertainty-Aware 4D World Modeling from LiDAR Sequences
Xiang Xu, Alan Liang, Youquan Liu et al. (+4 more)
Modeling dynamic 3D environments from
LiDAR sequences is central to building reliable 4D worlds for autonomous driving and
embodied AI. Existing generative frameworks, however, often treat all spatial regions uniformly, overlooking the varying uncertainty across real-world scenes. This uniform generation leads to artifacts in complex or ambiguous regions, limiting realism and temporal stability. In this work, we present U4D, an uncertainty-aware framework for 4D
LiDAR world modeling. Our approach first estimates spatial uncertainty maps from a pretrained
segmentation model to localize semantically challenging regions. It then performs generation in a "hard-to-easy" manner through two sequential stages: (1) uncertainty-region modeling, which reconstructs high-
entropy regions with fine
geometric fidelity, and (2) uncertainty-conditioned completion, which synthesizes the remaining areas under learned structural priors. To further ensure temporal coherence, U4D incorporates a mixture of spatio-temporal (MoST) block that adaptively fuses spatial and temporal representations during
diffusion. Extensive experiments show that U4D produces geometrically faithful and temporally consistent
LiDAR sequences, advancing the reliability of 4D world modeling for autonomous perception and simulation.
Read the full abstract ▾
All authors Xiang Xu, Alan Liang, Youquan Liu, Linfeng Li, Lingdong Kong, Ziwei Liu, Qingshan Liu
Robotics · Cross-listed from cs.CV
BEVDilation: LiDAR-Centric Multi-Modal Fusion for 3D Object Detection
Guowen Zhang, Chenhang He, Liyi Chen et al. (+1 more)
Integrating
LiDAR and camera information in the bird's eye view (BEV) representation has demonstrated its effectiveness in 3D object detection. However, because of the fundamental disparity in
geometric accuracy between these sensors, indiscriminate fusion in previous methods often leads to degraded performance. In this paper, we propose BEVDilation, a novel
LiDAR-centric framework that prioritizes
LiDAR information in the fusion. By formulating image BEV features as implicit guidance rather than naive concatenation, our strategy effectively alleviates the spatial misalignment caused by image depth estimation errors. Furthermore, the image guidance can effectively help the
LiDAR-centric paradigm to address the sparsity and semantic limitations of point clouds. Specifically, we propose a Sparse Voxel Dilation Block that mitigates the inherent point sparsity by densifying foreground voxels through image priors. Moreover, we introduce a Semantic-Guided BEV Dilation Block to enhance the
LiDAR feature
diffusion processing with image semantic guidance and long-range context capture. On the challenging nuScenes
benchmark, BEVDilation achieves better performance than state-of-the-art methods while maintaining competitive computational efficiency. Importantly, our
LiDAR-centric strategy demonstrates greater robustness to depth noise compared to naive fusion. The source code is available at https://github.com/gwenzhang/BEVDilation.
Read the full abstract ▾
All authors Guowen Zhang, Chenhang He, Liyi Chen, Lei Zhang
Robotics
Experimental Characterization of Fingertip Trajectory following for a 3-DoF Series-Parallel Hybrid Robotic Finger
Nicholas Baiata and Nilanjan Chakraborty
Task-space control of robotic fingers is a critical enabler of dexterous manipulation, as manipulation objectives are most naturally specified in terms of fingertip motions and applied forces rather than individual joint angles. While task-space planning and control have been extensively studied for larger, arm-scale manipulators, demonstrations of precise task-space trajectory tracking in compact, multi-DoF robotic fingers remain scarce. In this paper, we present the physical prototyping and experimental characterization of a three-degree-of-freedom, linkage-driven, series-parallel robotic finger with analytic forward kinematics and a closed-form Jacobian. A resolved motion rate control (RMRC) scheme is implemented to achieve closed-loop task-space trajectory tracking. We experimentally evaluate the fingertip tracking performance across a variety of trajectories, including straight lines, circles, and more complex curves, and report millimeter-level accuracy. To the best of our knowledge, this work provides one of the first systematic experimental demonstrations of precise task-space trajectory tracking in a linkage-driven robotic finger, thereby establishing a
benchmark for future designs aimed at dexterous in-hand manipulation.
Read the full abstract ▾
Robotics
VLA Models Are More Generalizable Than You Think: Revisiting Physical and Spatial Modeling
Weiqi Li, Quande Zhang, Ruifeng Zhai et al. (+2 more)
Vision-
language-action (VLA) models achieve strong in-distribution performance but degrade sharply under novel camera viewpoints and visual perturbations. We show that this brittleness primarily arises from misalignment in Spatial Modeling, rather than Physical Modeling. To address this, we propose a one-shot adaptation framework that recalibrates visual representations through lightweight, learnable updates. Our first method, Feature
Token Modulation (FTM), applies a global affine transformation to visual
tokens and improves Libero viewpoint accuracy from 48.5% to 87.1% with only 4K parameters. Building on this, Feature Linear Adaptation (FLA) introduces low-rank updates to the ViT encoder, achieving 90.8% success with 4.7M parameters – matching LoRA-scale finetuning at far lower cost. Together, these results reveal substantial untapped robustness in pretrained VLA models and demonstrate that targeted, minimal visual adaptation is sufficient to restore viewpoint generalization.
Read the full abstract ▾
All authors Weiqi Li, Quande Zhang, Ruifeng Zhai, Liang Lin, Guangrun Wang
Robotics · Cross-listed from cs.CV
Polar Perspectives: Evaluating 2-D LiDAR Projections for Robust Place Recognition with Visual Foundation Models
Pierpaolo Serio, Giulio Pisaneschi, Andrea Dan Ryals et al. (+4 more)
This work presents a systematic investigation into how alternative
LiDAR-to-image projections affect metric place recognition when coupled with a state-of-the-art
vision foundation model. We introduce a modular
retrieval pipeline that controls for backbone, aggregation, and evaluation protocol, thereby isolating the influence of the 2-D projection itself. Using consistent
geometric and structural channels across multiple
datasets and deployment scenarios, we identify the projection characteristics that most strongly determine discriminative power, robustness to environmental variation, and suitability for real-time autonomy. Experiments with different
datasets, including integration into an operational place recognition policy, validate the practical relevance of these findings and demonstrate that carefully designed projections can serve as an effective surrogate for end-to-end 3-D learning in
LiDAR place recognition.
Read the full abstract ▾
All authors Pierpaolo Serio, Giulio Pisaneschi, Andrea Dan Ryals, Vincenzo Infantino, Lorenzo Gentilini, Valentina Donzella, Lorenzo Pollini
Robotics
EmbodiedDiffusion: End-to-End Traversability-Guided Visual Diffusion for Heterogeneous Robot Navigation
Iana Zhura, Sausar Karaf, Faryal Batool et al. (+7 more)
Visual traversability estimation is central to autonomous
navigation, yet most approaches either rely on
prompt-driven
Vision-
Language Model (VLM) or decouple traversability from trajectory planning, requiring separate planners with heavy mapping, manual tuning, and extended deployment time. We propose EmbodiedDiffusion, a
diffusion-based framework that simultaneously predicts traversability maps and generates feasible trajectories from RGB images using planner-free synthetic supervision and embodiment conditioning for cross-platform transfer. The framework distills category-level traversability semantics from a VLM teacher into a lightweight student model during training, enabling
prompt-free, real-time inference at deployment. A modular FiLM-based conditioning mechanism isolates embodiment-specific
reasoning into a compact trainable subset of the network, allowing rapid adaptation to new
robot platforms without retraining the visual backbone or the trajectory
diffusion model. Across indoor environments with quadruped and aerial
robots, EmbodiedDiffusion achieves 80-100%
navigation success in the full-data regime with real-time inference (90 ms) and adapts to new platforms using only 10 min of visual data collection, demonstrating scalable, unified traversability
reasoning and trajectory generation for heterogeneous
robots.
Read the full abstract ▾
All authors Iana Zhura, Sausar Karaf, Faryal Batool, Nipun Dhananjaya Weerakkodi Mudalige, Valerii Serpiva, Ali Alridha Abdulkarim, Aleksey Fedoseev, Didar Seyidov, Hajira Amjad, Dzmitry Tsetserukou
Robotics
VLM as Strategist: Adaptive Generation of Safety-critical Testing Scenarios via Guided Diffusion
Xinzheng Wu, Junyi Chen, Naiting Zhong et al. (+1 more)
The safe deployment of autonomous driving systems (ADSs) relies on comprehensive testing and evaluation. However, safety-critical scenarios that can effectively expose system vulnerabilities are extremely sparse in the real world. Existing scenario generation methods face challenges in efficiently constructing long-tail scenarios that ensure fidelity, criticality, and interactivity, while particularly lacking real-time dynamic response capabilities to the vehicle under test (VUT). To address these challenges, this paper proposes a safety-critical testing scenario generation framework that integrates the high-level semantic understanding capabilities of
Vision Language Models (VLMs) with the fine-grained generation capabilities of adaptive guided
diffusion models. The framework establishes a three-layer hierarchical architecture comprising a strategic layer for VLM-directed scenario generation objective determination, a tactical layer for guidance function formulation, and an operational layer for guided
diffusion execution. We first establish a high-quality fundamental
diffusion model that learns the data distribution of real driving scenarios. Next, we design an adaptive guided
diffusion method that enables real-time, precise control of background vehicles (BVs) in closed-loop simulation. The VLM is then incorporated to autonomously generate scenario generation objectives and guidance functions through deep scenario understanding and risk
reasoning, ultimately guiding the
diffusion model to achieve VLM-directed scenario generation. Experimental results demonstrate that the proposed method can efficiently generate realistic, diverse, and highly interactive safety-critical testing scenarios. Furthermore, case studies validate the adaptability and VLM-directed generation performance of the proposed method.
Read the full abstract ▾
All authors Xinzheng Wu, Junyi Chen, Naiting Zhong, Yong Shen
Robotics
Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach
Siyuan Yang, Yang Zhang, Haoran He et al. (+4 more)
Vision-
Language-Action (VLA) models, trained via flow-matching or
diffusion objectives, excel at learning complex behaviors from large-scale, multi-modal
datasets (e.g., human teleoperation, scripted policies). However, since VLAs incorporate diverse data modes in the pre-training stage, and the finetuning
dataset often contains demonstration data collected in a kinematically suboptimal or undesirable way, it exists redundant action modes that are irrelevant to the success action modes of the downstream task. Specifically, we observe a critical inference-time fragility among various sampled noises after supervised finetuning of pre-trained VLAs. In this paper, we attribute this instability to the distribution shift between the VLA policy and the policy induced by stable success modes of the downstream task
dataset. Thus, we propose TACO, a test-time-scaling (TTS) framework that applies a lightweight pseudo-count
estimator as a high-fidelity verifier of action chunks. The VLA models integrated with TACO can execute the actions with maximum pseudo-count from all sampled action chunks, thereby preventing distribution shifts while preserving the generalization ability of VLAs since the constraint is applied only during inference. Our method resembles the classical anti-exploration principle in offline reinforcement learning (RL), and being gradient-free, it incurs significant computational benefits compared to RL update, especially for flow or
diffusion-based VLAs which are difficult to perform RL update due to denoising process. Extensive experiments across four simulation
benchmarks (RoboTwin2.0, Robotwin, LIBERO, SimplerEnv) and a dual-arm platform demonstrate that our method significantly improves the inference stability and success rates in downstream-task adaptations.
Read the full abstract ▾
All authors Siyuan Yang, Yang Zhang, Haoran He, Ling Pan, Xiu Li, Chenjia Bai, Xuelong Li
Robotics
Phase-Adaptive LLM Framework with Multi-Stage Validation for Construction Robot Task Allocation: A Systematic Benchmark Against Traditional Optimization Algorithms
Shyam prasad reddy Kaitha and Hongrui Yu
Multi-
robot task allocation in construction automation has traditionally relied on optimization methods such as Dynamic Programming and Reinforcement Learning. This research introduces the LangGraph-based Task Allocation
Agent (LTAA), an LLM-driven framework that integrates phase-adaptive allocation strategies, multi-stage validation with hierarchical retries, and dynamic prompting for efficient
robot coordination. Although recent LLM approaches show potential for construction robotics, they largely lack rigorous validation and benchmarking against established
algorithms. This paper presents the first systematic comparison of LLM-based task allocation with traditional methods in construction scenarios.The study validates LLM feasibility through SMART-LLM replication and addresses implementation challenges using a Self-Corrective
Agent Architecture. LTAA leverages natural-
language reasoning combined with structured validation mechanisms, achieving major computational gains reducing
token usage by 94.6% and allocation time by 86% through dynamic prompting. The framework adjusts its strategy across phases: emphasizing execution feasibility early and workload balance in later allocations.The authors evaluate LTAA against Dynamic Programming, Q-learning, and Deep Q-Network (DQN) baselines using construction operations from the TEACh human-
robot collaboration
dataset. In the Heavy Excels setting, where
robots have strong task specializations, LTAA achieves 77% task completion with superior workload balance, outperforming all traditional methods. These findings show that LLM-based
reasoning with structured validation can match established optimization
algorithms while offering additional advantages such as interpretability, adaptability, and the ability to update task logic without retraining.
Read the full abstract ▾
Robotics
Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols
Xianchao Zeng, Xinyu Zhou, Youcheng Li et al. (+5 more)
Vision-
Language-Action (VLA) models have recently achieved remarkable progress in robotic manipulation, yet they remain limited in failure diagnosis and learning from failures. Additionally, existing failure
datasets are mostly generated programmatically in simulation, which limits their generalization to the real world. In light of these, we introduce ViFailback, a framework designed to diagnose robotic manipulation failures and provide both textual and visual correction guidance. Our framework utilizes explicit visual symbols to enhance annotation efficiency. We further release the ViFailback
dataset, a large-scale collection of 58,126 Visual Question Answering (VQA) pairs along with their corresponding 5,202 real-world manipulation trajectories. Based on the
dataset, we establish ViFailback-Bench, a
benchmark of 11 fine-grained VQA tasks designed to assess the failure diagnosis and correction abilities of
Vision-
Language Models (VLMs), featuring ViFailback-Bench Lite for closed-ended and ViFailback-Bench Hard for open-ended evaluation. To demonstrate the effectiveness of our framework, we built the ViFailback-8B VLM, which not only achieves significant overall performance improvement on ViFailback-Bench but also generates visual symbols for corrective action guidance. Finally, by integrating ViFailback-8B with a VLA model, we conduct real-world robotic experiments demonstrating its ability to assist the VLA model in recovering from failures. Project Website: https://x1nyuzhou.github.io/vifailback.github.io/
Read the full abstract ▾
All authors Xianchao Zeng, Xinyu Zhou, Youcheng Li, Jiayou Shi, Tianle Li, Liangming Chen, Lei Ren, Yong-Lu Li
Robotics
CogDrive: Cognition-Driven Multimodal Prediction-Planning Fusion for Safe Autonomy
Heye Huang, Yibin Yang, Mingfeng Fan et al. (+3 more)
Safe autonomous driving in mixed traffic requires a unified understanding of
multimodal interactions and dynamic planning under uncertainty. Existing learning based approaches struggle to capture rare but safety critical behaviors, while rule based systems often lack adaptability in complex interactions. To address these limitations, CogDrive introduces a cognition driven
multimodal prediction and planning framework that integrates explicit modal
reasoning with safety aware trajectory optimization. The prediction module adopts cognitive representations of interaction modes based on topological motion semantics and nearest neighbor relational encoding. With a differentiable modal loss and
multimodal Gaussian decoding, CogDrive learns sparse and unbalanced interaction behaviors and improves long horizon trajectory prediction. The planning module incorporates an emergency response concept and optimizes safety stabilized trajectories, where short term consistent branches ensure safety during replanning cycles and long term branches support smooth and collision free motion under low probability switching modes. Experiments on Argoverse2 and INTERACTION
datasets show that CogDrive achieves strong performance in trajectory accuracy and miss rate, while closed loop simulations confirm adaptive behavior in merge and intersection scenarios. By combining cognitive
multimodal prediction with safety oriented planning, CogDrive offers an interpretable and reliable paradigm for safe autonomy in complex traffic.
Read the full abstract ▾
All authors Heye Huang, Yibin Yang, Mingfeng Fan, Haoran Wang, Xiaocong Zhao, Jianqiang Wang
Robotics
RoboWheel: A Data Engine from Real-World Human Demonstrations for Cross-Embodiment Robotic Learning
Yuhong Zhang, Zihan Gao, Shengpeng Li et al. (+12 more)
We introduce Robowheel, a data engine that converts human hand object interaction (HOI)
videos into training-ready supervision for cross morphology robotic learning. From
monocular RGB or RGB-D inputs, we perform high precision HOI reconstruction and enforce physical plausibility via a reinforcement learning (RL) optimizer that refines hand object relative poses under contact and penetration constraints. The reconstructed, contact rich trajectories are then retargeted to cross-embodiments,
robot arms with simple end effectors, dexterous hands, and
humanoids, yielding executable actions and rollouts. To scale coverage, we build a simulation-augmented framework on Isaac Sim with diverse domain randomization (embodiments, trajectories, object
retrieval, background textures, hand motion mirroring), which enriches the distributions of trajectories and observations while preserving spatial relationships and physical plausibility. The entire data pipeline forms an end to end pipeline from
video,reconstruction,retargeting,augmentation data acquisition. We validate the data on mainstream
vision language action (VLA) and imitation learning architectures, demonstrating that trajectories produced by our pipeline are as stable as those from teleoperation and yield comparable continual performance gains. To our knowledge, this provides the first quantitative evidence that HOI modalities can serve as effective supervision for robotic learning. Compared with teleoperation, Robowheel is lightweight, a single
monocular RGB(D) camera is sufficient to extract a universal, embodiment agnostic motion representation that could be flexibly retargeted across embodiments. We further assemble a large scale
multimodal dataset combining multi-camera captures,
monocular videos, and public HOI corpora for training and evaluating
embodied models.
Read the full abstract ▾
All authors Yuhong Zhang, Zihan Gao, Shengpeng Li, Ling-Hao Chen, Kaisheng Liu, Runqing Cheng, Xiao Lin, Junjia Liu, Zhuoheng Li, Jingyi Feng, Ziyan He, Jintian Lin, Zheyan Huang, Zhifang Liu, Haoqian Wang
Robotics
SAM2Grasp: Resolve Multi-modal Grasping via Prompt-conditioned Temporal Action Prediction
Shengkai Wu, Jinrong Yang, Wenqiu Luo et al. (+7 more)
Imitation learning for robotic grasping is often plagued by the
multimodal problem: when a scene contains multiple valid targets, demonstrations of grasping different objects create conflicting training signals. Standard imitation learning policies fail by averaging these distinct actions into a single, invalid action. In this paper, we introduce SAM2Grasp, a novel framework that resolves this issue by reformulating the task as a uni-modal,
prompt-conditioned prediction problem. Our method leverages the frozen SAM2 model to use its powerful visual temporal tracking capability and introduces a lightweight, trainable action head that operates in parallel with its native
segmentation head. This design allows for training only the small action head on pre-computed temporal-visual features from SAM2. During inference, an initial
prompt, such as a bounding box provided by an upstream object detection model, designates the specific object to be grasped. This
prompt conditions the action head to predict a unique, unambiguous grasp trajectory for that object alone. In all subsequent
video frames, SAM2's built-in temporal tracking capability automatically maintains stable tracking of the selected object, enabling our model to continuously predict the grasp trajectory from the
video stream without further external guidance. This temporal-prompted approach effectively eliminates ambiguity from the visuomotor policy. We demonstrate through extensive experiments that SAM2Grasp achieves state-of-the-art performance in cluttered, multi-object grasping tasks.
Read the full abstract ▾
All authors Shengkai Wu, Jinrong Yang, Wenqiu Luo, Linfeng Gao, Chaohui Shang, Meiyu Zhi, Mingshan Sun, Fangping Yang, Liangliang Ren, Yong Zhao
Robotics · Cross-listed from cs.HC
Reframing Human-Robot Interaction Through Extended Reality: Unlocking Safer, Smarter, and More Empathic Interactions with Virtual Robots and Foundation Models
Yuchong Zhang, Yong Ma and Danica Kragic
This perspective reframes human-
robot interaction (HRI) through extended reality (XR), arguing that virtual
robots powered by large foundation models (FMs) can serve as cognitively grounded, empathic
agents. Unlike physical
robots, XR-native
agents are unbound by hardware constraints and can be instantiated, adapted, and scaled on demand, while still affording embodiment and co-presence. We synthesize work across XR, HRI, and cognitive AI to show how such
agents can support safety-critical scenarios, socially and cognitively empathic interaction across domains, and outreaching physical capabilities with XR and AI integration. We then discuss how
multimodal large FMs (e.g., large
language model, large
vision model, and
vision-
language model) enable context-aware
reasoning, affect-sensitive situations, and long-term adaptation, positioning virtual
robots as cognitive and empathic mediators rather than mere simulation assets. At the same time, we highlight challenges and potential risks, including overtrust, cultural and representational bias,
privacy concerns around biometric sensing, and data governance and transparency. The paper concludes by outlining a research agenda for human-centered, ethically grounded XR
agents - emphasizing multi-layered evaluation frameworks, multi-user ecosystems, mixed virtual-physical embodiment, and societal and ethical design practices to envision XR-based virtual
agents powered by FMs as reshaping future HRI into a more efficient and adaptive paradigm.
Read the full abstract ▾
Robotics
Robotic capabilities framework: A boundary object and intermediate-level knowledge artifact for co-designing robotic processes
Alessandro Ianniello, Dave Murray-Rust, Sara Muscolo et al. (+6 more)
As
robots become more adaptable, responsive, and capable of interacting with humans, the design of effective human-
robot collaboration becomes critical. Yet, this design process is typically led by monodisciplinary approaches, often overlooking interdisciplinary knowledge and the experiential knowledge of workers who will ultimately share tasks with these systems. To address this gap, we introduce the robotic capabilities framework, a vocabulary that enables transdisciplinary collaborations to meaningfully shape the future of work when robotic systems are integrated into the workplace. Rather than focusing on the internal workings of
robots, the framework centers discussion on high-level capabilities, supporting dialogue around which elements of a task should remain human-led and which can be delegated to
robots. We developed the framework through reflexive and iterative processes, and applied it in two distinct settings: by engaging roboticists in describing existing commercial
robots using its vocabulary, and through a design activity with students working on robotics-related projects. The framework emerges as an intermediate-level knowledge artifact and a boundary object that bridges technical and experiential domains, guiding designers, empowering workers, and contributing to more just and collaborative futures of work.
Read the full abstract ▾
All authors Alessandro Ianniello, Dave Murray-Rust, Sara Muscolo, Olger Siebinga, Nicky Mol, Denis Zatyagov, Eva Verhoef, Deborah Forster, David Abbink
Robotics
AID: Agent Intent from Diffusion for Multi-Agent Informative Path Planning
Jeric Lew, Yuhong Cao, Derek Ming Siang Tan et al. (+1 more)
Information gathering in large-scale or time-critical scenarios (e.g., environmental monitoring, search and rescue) requires broad coverage within limited time budgets, motivating the use of multi-
agent systems. These scenarios are commonly formulated as multi-
agent informative path planning (MAIPP), where multiple
agents must coordinate to maximize information gain while operating under budget constraints. A central challenge in MAIPP is ensuring effective coordination while the belief over the environment evolves with incoming measurements. Recent learning-based approaches address this by using distributions over future positions as "intent" to support coordination. However, these autoregressive intent predictors are computationally expensive and prone to compounding errors. Inspired by the effectiveness of
diffusion models as expressive, long-horizon policies, we propose AID, a fully decentralized MAIPP framework that leverages
diffusion models to generate long-term trajectories in a non-autoregressive manner. AID first performs behavior cloning on trajectories produced by existing MAIPP planners and then fine-tunes the policy using reinforcement learning via
Diffusion Policy Policy Optimization (DPPO). This two-stage pipeline enables the policy to inherit expert behavior while learning improved coordination through online reward feedback. Experiments demonstrate that AID consistently improves upon the MAIPP planners it is trained from, achieving 4x faster execution and up to 17% increased information gain, while scaling effectively to larger numbers of
agents. Our implementation is publicly available at https://github.com/marmotlab/AID.
Read the full abstract ▾
All authors Jeric Lew, Yuhong Cao, Derek Ming Siang Tan, Guillaume Sartoretti
Robotics · Cross-listed from cs.CV
nuScenes Revisited: Progress and Challenges in Autonomous Driving
Whye Kit Fong, Venice Erin Liong, Kok Seang Tan et al. (+1 more)
Autonomous Vehicles (AV) and Advanced Driver Assistance Systems (ADAS) have been revolutionized by Deep Learning. As a data-driven approach, Deep Learning relies on vast amounts of driving data, typically labeled in great detail. As a result,
datasets, alongside hardware and
algorithms, are foundational building blocks for the development of AVs. In this work we revisit one of the most widely used autonomous driving
datasets: the nuScenes
dataset. nuScenes exemplifies key trends in AV development, being the first
dataset to include radar data, to feature diverse urban driving scenes from two continents, and to be collected using a fully autonomous vehicle operating on public roads, while also promoting multi-modal sensor fusion, standardized
benchmarks, and a broad range of tasks including perception, localization & mapping, prediction and planning. We provide an unprecedented look into the creation of nuScenes, as well as its extensions nuImages and Panoptic nuScenes, summarizing many technical details that have hitherto not been revealed in academic publications. Furthermore, we trace how the influence of nuScenes impacted a large number of other
datasets that were released later and how it defined numerous standards that are used by the community to this day. Finally, we present an overview of both official and unofficial tasks using the nuScenes
dataset and review major methodological developments, thereby offering a comprehensive survey of the autonomous driving literature, with a particular focus on nuScenes.
Read the full abstract ▾
All authors Whye Kit Fong, Venice Erin Liong, Kok Seang Tan, Holger Caesar
Robotics
Vehicle Dynamics Embedded World Models for Autonomous Driving
Huiqian Li, Wei Pan, Haodong Zhang et al. (+2 more)
World models have gained significant
attention as a promising approach for autonomous driving. By emulating human-like perception and decision-making processes, these models can predict and adapt to dynamic environments. Existing methods typically map high-dimensional observations into compact latent spaces and learn optimal policies within these latent representations. However, prior work usually jointly learns ego-vehicle dynamics and environmental transition dynamics from the image input, leading to inefficiencies and a lack of robustness to variations in vehicle dynamics. To address these issues, we propose the Vehicle Dynamics embedded Dreamer (VDD) method, which decouples the modeling of ego-vehicle dynamics from environmental transition dynamics. This separation allows the world model to generalize effectively across vehicles with diverse parameters. Additionally, we introduce two strategies to further enhance the robustness of the learned policy: Policy Adjustment during Deployment (PAD) and Policy Augmentation during Training (PAT). Comprehensive experiments in simulated environments demonstrate that the proposed model significantly improves both driving performance and robustness to variations in vehicle dynamics, outperforming existing approaches.
Read the full abstract ▾
All authors Huiqian Li, Wei Pan, Haodong Zhang, Jin Huang, Zhihua Zhong
Robotics · Cross-listed from eess.SY
On the Convergence of Density-Based Predictive Control for Multi-Agent Non-Uniform Area Coverage
Sungjun Seo and Kooktae Lee
This paper presents Density-based Predictive Control (DPC), a novel multi-
agent control strategy for efficient non-uniform area coverage, grounded in optimal transport theory. In large-scale scenarios such as search and rescue or environmental monitoring, traditional uniform coverage fails to account for varying regional priorities. DPC leverages a pre-constructed reference distribution to allocate
agents' coverage efforts, spending more time in high-priority or densely sampled regions. We analyze convergence conditions using the Wasserstein distance, derive an analytic optimal control law for unconstrained cases, and propose a numerical method for constrained scenarios. Simulations on first-order dynamics and linearized quadrotor models demonstrate that DPC achieves trajectories closely matching the non-uniform reference distribution, outperforming existing coverage methods.
Read the full abstract ▾
Robotics
VIGS-SLAM: Visual Inertial Gaussian Splatting SLAM
Zihan Zhu, Wei Zhang, Moyang Li et al. (+3 more)
We present VIGS-SLAM, a visual-inertial 3D Gaussian Splatting SLAM system that achieves robust real-time tracking and high-fidelity reconstruction. Although recent 3DGS-based SLAM methods achieve dense and photorealistic mapping, their purely visual design degrades under challenging conditions such as motion blur, low texture, and exposure variations. Our method tightly couples visual and inertial cues within a unified optimization framework, jointly optimizing camera poses, depths, and IMU states. It features robust IMU initialization, time-varying bias modeling, and loop closure with consistent Gaussian updates. Experiments on five challenging
datasets demonstrate our superiority over state-of-the-art methods. Project page: https://vigs-slam.github.io
Read the full abstract ▾
All authors Zihan Zhu, Wei Zhang, Moyang Li, Norbert Haala, Marc Pollefeys, Daniel Barath