Robotics
Proprioceptive-only State Estimation for Legged Robots with Set-Coverage Measurements of Learned Dynamics
Abhijeet M. Kulkarni, Ioannis Poulakakis and Guoquan Huang
Proprioceptive-only state estimation is attractive for legged
robots since it is computationally cheaper and is unaffected by perceptually degraded conditions. The history of joint-level measurements contains rich information that can be used to infer the dynamics of the system and subsequently produce navigational measurements. Recent approaches produce these estimates with learned measurement models and fuse with IMU data, under a Gaussian noise assumption. However, this assumption can easily break down with limited training data and render the estimates inconsistent and potentially divergent. In this work, we propose a proprioceptive-only state estimation framework for legged
robots that characterizes the measurement noise using set-coverage statements that do not assume any distribution. We develop a practical and computationally inexpensive method to use these set-coverage measurements with a Gaussian filter in a systematic way. We validate the approach in both simulation and two real-world quadrupedal
datasets. Comparison with the Gaussian baselines shows that our proposed method remains consistent and is not prone to drift under real noise scenarios.
Read the full abstract ▾
Robotics
Sparse3DTrack: Monocular 3D Object Tracking Using Sparse Supervision
Nikhil Gosala, B. Ravi Kiran, Senthil Yogamani et al. (+1 more)
Monocular 3D object tracking aims to estimate temporally consistent 3D object poses across
video frames, enabling autonomous
agents to reason about scene dynamics. However, existing state-of-the-art approaches are fully supervised and rely on dense 3D annotations over long
video sequences, which are expensive to obtain and difficult to scale. In this work, we address this fundamental limitation by proposing the first sparsely supervised framework for
monocular 3D object tracking. Our approach decomposes the task into two sequential sub-problems: 2D query matching and 3D geometry estimation. Both components leverage the spatio-temporal consistency of image sequences to augment a sparse set of labeled samples and learn rich 2D and 3D representations of the scene. Leveraging these learned cues, our model automatically generates high-quality 3D pseudolabels across entire
videos, effectively transforming sparse supervision into dense 3D track annotations. This enables existing fully-supervised trackers to effectively operate under extreme label sparsity. Extensive experiments on the KITTI and nuScenes
datasets demonstrate that our method significantly improves tracking performance, achieving an improvement of up to 15.50 p.p. while using at most four ground truth annotations per track.
Read the full abstract ▾
All authors Nikhil Gosala, B. Ravi Kiran, Senthil Yogamani, Abhinav Valada
Robotics
Offload or Overload: A Platform Measurement Study of Mobile Robotic Manipulation Workloads
Sara Pohland, Xenofon Foukas, Ganesh Ananthanarayanan et al. (+4 more)
Mobile robotic manipulation–the ability of
robots to navigate spaces and interact with objects–is a core capability of physical AI. Foundation models have led to breakthroughs in their performance, but at a significant computational cost. We present the first measurement study of mobile robotic manipulation workloads across onboard, edge, and cloud GPU platforms. We find that the full workload stack is infeasible to run on smaller onboard GPUs, while larger onboard GPUs drain
robot batteries several hours faster. Offloading alleviates these constraints but introduces its own challenges, as additional network latency degrades task accuracy, and the bandwidth requirement makes naive cloud offloading impractical. Finally, we quantify opportunities and pitfalls of sharing compute across
robot fleets. We believe our measurement study will be crucial to designing inference systems for mobile
robots.
Read the full abstract ▾
All authors Sara Pohland, Xenofon Foukas, Ganesh Ananthanarayanan, Andrey Kolobov, Sanjeev Mehrotra, Bozidar Radunovic, Ankit Verma
Robotics
SG-CoT: An Ambiguity-Aware Robotic Planning Framework using Scene Graph Representations
Akshat Rana, Peeyush Agarwal, K.P.S. Rana et al. (+1 more)
Ambiguity poses a major challenge to large
language models (LLMs) used as robotic planners. In this letter, we present Scene Graph-Chain-of-Thought (SG-CoT), a two-stage framework where LLMs iteratively query a scene graph representation of the environment to detect and clarify ambiguities. First, a structured scene graph representation of the environment is constructed from input observations, capturing objects, their attributes, and relationships with other objects. Second, the LLM is equipped with
retrieval functions to query portions of the scene graph that are relevant to the provided instruction. This grounds the
reasoning process of the LLM in the observation, increasing the reliability of robotic planners under ambiguous situations. SG-CoT also allows the LLM to identify the source of ambiguity and pose a relevant disambiguation question to the user or another
robot. Extensive experimentation demonstrates that SG-CoT consistently outperforms prior methods, with a minimum of 10% improvement in question accuracy and a minimum success rate increase of 4% in single-
agent and 15% in multi-
agent environments, validating its effectiveness for more generalizable
robot planning.
Read the full abstract ▾
All authors Akshat Rana, Peeyush Agarwal, K.P.S. Rana, Amarjit Malhotra
Robotics
Manufacturing Micro-Patterned Surfaces with Multi-Robot Systems
Annalisa T. Taylor, Malachi Landis, Ping Guo et al. (+1 more)
Applying micro-patterns to surfaces has been shown to impart useful physical properties such as drag reduction and hydrophobicity. However, current manufacturing techniques cannot produce micro-patterned surfaces at scale due to high-cost machinery and inefficient coverage techniques such as raster-scanning. In this work, we use multiple
robots, each equipped with a patterning tool, to manufacture these surfaces. To allow these
robots to coordinate during the patterning task, we use the ergodic control
algorithm, which specifies coverage objectives using distributions. We demonstrate that
robots can divide complicated coverage objectives by communicating compressed representations of their trajectory history both in simulations and experimental trials. Further, we show that
robot-produced patterning can lower the coefficient of friction of metallic surfaces. This work demonstrates that distributed multi-
robot systems can coordinate to manufacture products that were previously unrealizable at scale.
Read the full abstract ▾
All authors Annalisa T. Taylor, Malachi Landis, Ping Guo, Todd D. Murphey
Robotics
Rapid Adaptation of Particle Dynamics for Generalized Deformable Object Mobile Manipulation
Bohan Wu, Roberto Martín-Martín and Li Fei-Fei
We address the challenge of learning to manipulate deformable objects with unknown dynamics. In non-rigid objects, the dynamics parameters define how they react to interactions – how they stretch, bend, compress, and move – and they are critical to determining the optimal actions to perform a manipulation task successfully. In other robotic domains, such as legged locomotion and in-hand rigid object manipulation, state-of-the-art approaches can handle unknown dynamics using Rapid Motor Adaptation (RMA). Through a supervised procedure in simulation that encodes each rigid object's dynamics, such as mass and position, these approaches learn a policy that conditions actions on a vector of latent dynamic parameters inferred from sequences of state-actions. However, in deformable object manipulation, the object's dynamics not only includes its mass and position, but also how the shape of the object changes. Our key insight is that the recent ground-truth particle positions of a deformable object in simulation capture changes in the object's shape, making it possible to extend RMA to deformable object manipulation. This key insight allows us to develop RAPiD, a two-phase method that learns to perform real-
robot deformable object mobile manipulation by: 1) learning a visuomotor policy conditioned on the object's dynamics
embedding, which is encoded from the object's privileged information in simulation, such as its mass and ground-truth particle positions, and 2) learning to infer this
embedding using non-privileged information instead, such as
robot visual observations and actions, so that the learned policy can transfer to the real world. On a mobile manipulator with 22 degrees of freedom, RAPiD enables over 80%+ success rates across two
vision-based deformable object mobile manipulation tasks in the real world, under various object dynamics, categories, and instances.
Read the full abstract ▾
Robotics
ReDAG-RT: Global Rate-Priority Scheduling for Real-Time Multi-DAG Execution in ROS 2
Md. Mehedi Hasan, Rafid Mostafiz, Bikash Kumar Paul et al. (+2 more)
ROS 2 has become a dominant middleware for robotic systems, where perception, estimation, planning, and control pipelines are structured as directed acyclic graphs of callbacks executed under a shared executor. However, default ROS 2 executors use best-effort dispatch without cross-DAG priority enforcement, leading to callback contention, structural priority inversion, and deadline instability under concurrent workloads. These limitations restrict deployment in time-critical and safety-sensitive cyber-physical systems. This paper presents ReDAGRT, a user-space global scheduling framework for deterministic multi-DAG execution in unmodified ROS 2. The framework introduces a Rate-Priority driven global ready queue that orders callbacks by activation rate, enforces per-DAG concurrency bounds, and mitigates cross-graph priority inversion without modifying the ROS 2 API, executor interface, or underlying operating system scheduler. We formalize a multi-DAG task model for ROS 2 callback pipelines and analyze cross-DAG interference under Rate-Priority scheduling. Response-time recurrences and schedulability conditions are derived within classical Rate-Monotonic theory. Experiments in a ROS 2 Humble environment compare ReDAGRT against SingleThreadedExecutor and MultiThreadedExecutor using synthetic multi-DAG workloads. Results show up to 29.7 percent reduction in deadline miss rate, 42.9 percent reduction in 99th percentile response time, and 13.7 percent improvement over MultiThreadedExecutor under comparable utilization. Asymmetric per-DAG concurrency bounds further reduce interference by 40.8 percent. These results demonstrate that deterministic and analyzable multi-DAG scheduling can be achieved entirely in the ROS 2 user-space execution layer, providing a practical foundation for real-time robotic middleware in safety-critical systems.
Read the full abstract ▾
All authors Md. Mehedi Hasan, Rafid Mostafiz, Bikash Kumar Paul, Md. Abir Hossain, Ziaur Rahman
Robotics · Cross-listed from cs.CV
Semantic Segmentation and Depth Estimation for Real-Time Lunar Surface Mapping Using 3D Gaussian Splatting
Guillem Casadesus Vila, Adam Dai and Grace Gao
Navigation and mapping on the lunar surface require robust perception under challenging conditions, including poorly textured environments, high-contrast lighting, and limited computational resources. This paper presents a real-time mapping framework that integrates dense perception models with a 3D Gaussian Splatting (3DGS) representation. We first
benchmark several models on synthetic
datasets generated with the LuPNT simulator, selecting a stereo dense depth estimation model based on Gated Recurrent Units for its balance of speed and accuracy in depth estimation, and a convolutional neural network for its superior performance in detecting semantic segments. Using ground truth poses to decouple the local scene understanding from the global state estimation, our pipeline reconstructs a 120-meter traverse with a
geometric height accuracy of approximately 3 cm, outperforming a traditional point cloud baseline without
LiDAR. The resulting 3DGS map enables novel view synthesis and serves as a foundation for a full SLAM system, where its capacity for joint map and pose optimization would offer significant advantages. Our results demonstrate that combining semantic
segmentation and dense depth estimation with learned map representations is an effective approach for creating detailed, large-scale maps to support future lunar surface missions.
Read the full abstract ▾
Robotics
GoalVLM: VLM-driven Object Goal Navigation for Multi-Agent System
MoniJesu James, Amir Atef Habel, Aleksey Fedoseev et al. (+1 more)
Object-goal
navigation has traditionally been limited to ground
robots with closed-set object vocabularies. Existing multi-
agent approaches depend on precomputed probabilistic graphs tied to fixed category sets, precluding generalization to novel goals at test time. We present GoalVLM, a cooperative multi-
agent framework for zero-shot, open-vocabulary object
navigation. GoalVLM integrates a
Vision-
Language Model (VLM) directly into the decision loop, SAM3 for text-prompted detection and
segmentation, and SpaceOM for spatial
reasoning, enabling
agents to interpret free-form
language goals and score frontiers via zero-shot semantic priors without retraining. Each
agent builds a BEV semantic map from depth-projected voxel splatting, while a Goal Projector back-projects detections through calibrated depth into the map for reliable goal localization. A constraint-guided
reasoning layer evaluates frontiers through a structured
prompt chain (scene captioning, room-type classification, perception gating, multi-frontier ranking), injecting commonsense priors into exploration. We evaluate GoalVLM on GOAT-Bench val_unseen (360 multi-subtask episodes, 1032 sequential object-goal subtasks, HM3D scenes), where each episode requires navigating to a chain of 5-7 open-vocabulary targets. GoalVLM with N=2
agents achieves 55.8% subtask SR and 18.3% SPL, competitive with state-of-the-art methods while requiring no task-specific training. Ablation studies confirm the contributions of VLM-guided frontier
reasoning and depth-projected goal localization.
Read the full abstract ▾
All authors MoniJesu James, Amir Atef Habel, Aleksey Fedoseev, Dzmitry Tsetserokou
Robotics · Cross-listed from cs.LG
R2-Dreamer: Redundancy-Reduced World Models without Decoders or Augmentation
Naoki Morihira, Amal Nahar, Kartik Bharadwaj et al. (+3 more)
A central challenge in image-based Model-Based Reinforcement Learning (MBRL) is to learn representations that distill essential information from irrelevant visual details. While promising, reconstruction-based methods often waste capacity on large task-irrelevant regions. Decoder-free methods instead learn robust representations by leveraging Data Augmentation (DA), but reliance on such external regularizers limits versatility. We propose R2-Dreamer, a decoder-free MBRL framework with a self-supervised objective that serves as an internal regularizer, preventing representation collapse without resorting to DA. The core of our method is a redundancy-reduction objective inspired by Barlow Twins, which can be easily integrated into existing frameworks. On DeepMind Control Suite and Meta-World, R2-Dreamer is competitive with strong baselines such as DreamerV3 and TD-MPC2 while training 1.59x faster than DreamerV3, and yields substantial gains on DMC-Subtle with tiny task-relevant objects. These results suggest that an effective internal regularizer can enable versatile, high-performance decoder-free MBRL. Code is available at https://github.com/NM512/r2dreamer.
Read the full abstract ▾
All authors Naoki Morihira, Amal Nahar, Kartik Bharadwaj, Yasuhiro Kato, Akinobu Hayashi, Tatsuya Harada
Robotics
Final Report for the Workshop on Robotics & AI in Medicine
Juan P Wachs
The CARE Workshop on Robotics and AI in Medicine, held on December 1, 2025 in Indianapolis, convened leading researchers, clinicians, industry innovators, and federal stakeholders to shape a national
vision for advancing robotics and artificial intelligence in healthcare. The event highlighted the accelerating need for coordinated research efforts that bridge engineering innovation with real clinical priorities, emphasizing safety, reliability, and translational readiness with an emphasis on the use of robotics and AI to achieve this readiness goal. Across keynotes, panels, and breakout sessions, participants underscored critical gaps in data availability, standardized evaluation methods, regulatory pathways, and workforce training that hinder the deployment of intelligent robotic systems in
surgical, diagnostic, rehabilitative, and assistive contexts. Discussions emphasized the transformative potential of AI enabled robotics to improve precision, reduce provider burden, expand access to specialized care, and enhance patient outcomes particularly in undeserved regions and high risk procedural domains. Special
attention was given to austere settings, disaster and relief and military settings. The workshop demonstrated broad consensus on the urgency of establishing a national Center for AI and Robotic Excellence in medicine (CARE). Stakeholders identified priority research thrusts including human
robot collaboration, trustworthy autonomy, simulation and digital twins, multi modal sensing, and ethical integration of generative AI into clinical workflows. Participants also articulated the need for high quality
datasets, shared test beds, autonomous
surgical systems, clinically grounded
benchmarks, and sustained interdisciplinary training mechanisms.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
TokenPB: Enabling VLA Action Self-Verification via VLM Token Probability Bucketing
Chen Zhao, Zhuoran Wang, Haoyang Li et al. (+6 more)
Test-time sampling can improve
Vision-
Language-Action (VLA) policies, but only if the model can efficiently select a good action from multiple candidates. Prior work often relies on separately trained external verifiers, adding extra models and inference overhead. We propose
Token Probability Bucketing (TokenPB), a self-verification approach that discretizes continuous action chunks into bucket
tokens and co-trains a single VLA checkpoint with a joint objective: a flow-matching loss for action generation and an auxiliary bucket loss $\mathcal{L}_{\text{bucket}}$ that teaches the VLM backbone to model observation-conditioned distributions over bucket-
token sequences. At inference, TokenPB samples $M$ candidates, scores each via teacher-forced evaluation under the learned distribution, and selects the best—enabling efficient batched scoring with observation-prefix KV-cache reuse. We provide theory connecting the training objective to a monotonic ranking property and bounds on expected selection regret that separate candidate-set effects from calibration. Across multiple backbones and
benchmarks, TokenPB yields consistent success-rate improvements over single-sample inference from the same checkpoint (+2.7 in simulation, +17.1 in real-world, +16.4 under out-of-distribution shifts), and further improves over external-verifier baselines under the same sampling budget. The code is publicly available at https://github.com/RUCKBReasoning/TokenPB.
Read the full abstract ▾
All authors Chen Zhao, Zhuoran Wang, Haoyang Li, Shifeng Bao, Guanlin Li, Youhe Feng, Yang Li, Jie Tang, Jing Zhang
Robotics
Uncovering Latent Phase Structures and Branching Logic in Locomotion Policies: A Case Study on HalfCheetah
Daisuke Yasui, Toshitaka Matsuki and Hiroshi Sato
In locomotion control tasks, Deep Reinforcement Learning (DRL) has demonstrated high performance; however, the decision-making process of the learned policy remains a black box, making it difficult for humans to understand. On the other hand, in periodic motions such as walking, it is well known that implicit motion phases exist, such as the stance phase and the swing phase. Focusing on this point, this study hypothesizes that a policy trained for locomotion control may also represent a phase structure that is interpretable by humans. To examine this hypothesis in a controlled setting, we consider a locomotion task that is amenable to observing whether a policy autonomously acquires temporally structured phases through interaction with the environment. To verify this hypothesis, in the MuJoCo locomotion
benchmark HalfCheetah-v5, the state transition sequences acquired by a policy trained for walking control through interaction with the environment were aggregated into semantic phases based on state similarity and consistency of subsequent transitions. As a result, we demonstrated that the state sequences generated by the trained policy exhibit periodic phase transition structures as well as phase branching. Furthermore, by approximating the states and actions corresponding to each semantic phase using Explainable Boosting Machines (EBMs), we analyzed phase-dependent decision making-namely, which state features the policy function attends to and how it controls action outputs in each phase. These results suggest that neural network-based policies, which are often regarded as black boxes, can autonomously acquire interpretable phase structures and logical branching mechanisms.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
GMT: Goal-Conditioned Multimodal Transformer for 6-DOF Object Trajectory Synthesis in 3D Scenes
Huajian Zeng, Abhishek Saroha, Daniel Cremers et al. (+1 more)
Synthesizing controllable 6-DOF object manipulation trajectories in 3D environments is essential for enabling
robots to interact with complex scenes, yet remains challenging due to the need for accurate spatial
reasoning, physical feasibility, and
multimodal scene understanding. Existing approaches often rely on 2D or partial 3D representations, limiting their ability to capture full scene geometry and constraining trajectory precision. We present GMT, a
multimodal transformer framework that generates realistic and goal-directed object trajectories by jointly leveraging 3D bounding box geometry, point cloud context, semantic object categories, and target end poses. The model represents trajectories as continuous 6-DOF pose sequences and employs a tailored conditioning strategy that fuses
geometric, semantic, contextual, and goaloriented information. Extensive experiments on synthetic and real-world
benchmarks demonstrate that GMT outperforms state-of-the-art human motion and human-object interaction baselines, such as CHOIS and GIMO, achieving substantial gains in spatial accuracy and orientation control. Our method establishes a new
benchmark for learningbased manipulation planning and shows strong generalization to diverse objects and cluttered 3D environments. Project page: https://huajian- zeng.github. io/projects/gmt/.
Read the full abstract ▾
All authors Huajian Zeng, Abhishek Saroha, Daniel Cremers, Xi Wang
Robotics
A Single-Fiber Optical Frequency Domain Reflectometry (OFDR)-Based Shape Sensing of Concentric Tube Steerable Drilling Robots
Yash Kulkarni, Mobina Tavangarifard, Daniyal Maroufi et al. (+4 more)
This paper introduces a novel shape-sensing approach for Concentric Tube Steerable Drilling
Robots (CT-SDRs) based on Optical Frequency Domain Reflectometry (OFDR). Unlike traditional FBG-based methods, OFDR enables continuous strain measurement along the entire fiber length with enhanced spatial resolution. In the proposed method, a Shape Sensing Assembly (SSA) is first fabricated by integrating a single OFDR fiber with a flat NiTi wire. The calibrated SSA is then routed through and housed within the internal channel of a flexible drilling instrument, which is guided by the pre-shaped NiTi tube of the CT-SDR. In this configuration, the drilling instrument serves as a protective sheath for the SSA during drilling, eliminating the need for integration or adhesion to the instrument surface that is typical of conventional optical sensor approaches. The performance of the proposed SSA, integrated within the cannulated CT-SDR, was thoroughly evaluated under free-bending conditions and during drilling along multiple J-shaped trajectories in synthetic Sawbones phantoms. Results demonstrate accurate and reliable shape-sensing capability, confirming the feasibility and robustness of this integration strategy.
Read the full abstract ▾
All authors Yash Kulkarni, Mobina Tavangarifard, Daniyal Maroufi, Mohsen Khadem, Justin E. Bird, Jeffrey H. Siewerdsen, Farshid Alambeigi
Robotics
Specification-Aware Distribution Shaping for Robotics Foundation Models
Sad{\i}k Bera Yüksel and Derya Aksaray
Robotics foundation models have demonstrated strong capabilities in executing natural
language instructions across diverse tasks and environments. However, they remain largely data-driven and lack formal guarantees on safety and satisfaction of time-dependent specifications during deployment. In practice,
robots often need to comply with operational constraints involving rich spatio-temporal requirements such as time-bounded goal visits, sequential objectives, and persistent safety conditions. In this work, we propose a specification-aware action distribution optimization framework that enforces a broad class of Signal Temporal Logic (STL) constraints during execution of a pretrained robotics foundation model without modifying its parameters. At each decision step, the method computes a minimally modified action distribution that satisfies a hard STL feasibility constraint by
reasoning over the remaining horizon using forward dynamics propagation. We validate the proposed framework in simulation using a state-of-the-art robotics foundation model across multiple environments and complex specifications.
Read the full abstract ▾
Robotics
RoboForge: Physically Optimized Text-guided Whole-Body Locomotion for Humanoids
Xichen Yuan, Zhe Li, Bofan Lyu et al. (+4 more)
While generative models have become effective at producing human-like motions from text, transferring these motions to
humanoid robots for physical execution remains challenging. Existing pipelines are often limited by retargeting, where kinematic quality is undermined by physical infeasibility, contact-transition errors, and the high cost of real-world dynamical data. We present a unified latent-driven framework that bridges natural
language and whole-body
humanoid locomotion through a retarget-free, physics-optimized pipeline. Rather than treating generation and control as separate stages, our key insight is to couple them bidirectionally under physical constraints.We introduce a Physical Plausibility Optimization (PP-Opt) module as the coupling interface. In the forward direction, PP-Opt refines a teacher-student
distillation policy with a plausibility-centric reward to suppress artifacts such as floating, skating, and penetration. In the backward direction, it converts reward-optimized simulation rollouts into high-quality explicit motion data, which is used to fine-tune the motion generator toward a more physically plausible latent distribution. This bidirectional design forms a self-improving cycle: the generator learns a physically grounded latent space, while the controller learns to execute latent-conditioned behaviors with dynamical integrity.Extensive experiments on the Unitree G1
humanoid show that our bidirectional optimization improves tracking accuracy and success rates. Across IsaacLab and MuJoCo, the implicit latent-driven pipeline consistently outperforms conventional explicit retargeting baselines in both precision and stability. By coupling
diffusion-based motion generation with physical plausibility optimization, our framework provides a practical path toward deployable text-guided
humanoid intelligence.
Read the full abstract ▾
All authors Xichen Yuan, Zhe Li, Bofan Lyu, Kuangji Zuo, Yanshuo Lu, Gen Li, Jianfei Yang
Robotics
DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation
Xitong Chen, Yifeng Pan, Min Li et al. (+1 more)
Large-scale, high-quality
multimodal demonstrations are essential for
robot learning of contact-rich dexterous manipulation. While human-centric data collection systems lower the barrier to scaling, they struggle to capture the tactile information during physical interactions. Motivated by this, we present DexViTac, a portable, human-centric data collection system tailored for contact-rich dexterous manipulation. The system enables the high-fidelity acquisition of first-person
vision, high-density tactile sensing, end-effector poses, and hand kinematics within unstructured, in-the-wild environments. Building upon this hardware, we propose a kinematics-grounded tactile representation learning
algorithm that effectively resolves semantic ambiguities within tactile signals. Leveraging the efficiency of DexViTac, we construct a
multimodal dataset comprising over 2,400 visuo-tactile-kinematic demonstrations. Experiments demonstrate that DexViTac achieves a collection efficiency exceeding 248 demonstrations per hour and remains robust against complex visual occlusions. Real-world deployment confirms that policies trained with the proposed
dataset and learning strategy achieve an average success rate exceeding 85% across four challenging tasks. This performance significantly outperforms baseline methods, thereby validating the substantial improvement the system provides for learning contact-rich dexterous manipulation. Project page: https://xitong-c.github.io/DexViTac/.
Read the full abstract ▾
All authors Xitong Chen, Yifeng Pan, Min Li, Xiaotian Ding
Robotics
ProbeFlow: Training-Free Adaptive Flow Matching for Vision-Language-Action Models
Zhou Fang, Jiaqi Wang, Wenzhuo Li et al. (+2 more)
Vision-
Language-Action (VLA) policies increasingly use Flow Matching (FM) action heads to generate action chunks as continuous trajectories. At inference, an FM head solves an ordinary differential equation (ODE): starting from a prior action latent, it repeatedly queries a learned velocity field to advance the state toward the decoded action chunk. Because each query depends on the preceding state, this iterative decoding can impose substantial sequential action-head cost at deployment. We propose ProbeFlow, a training-free framework that routes each action chunk to an integration schedule using a Lookahead Consistency Probe. Optional oscillation-conditioned (OSC) threshold adaptation reuses the same velocity evaluations and can trade part of the acceleration for potentially better task performance. On MetaWorld, the fixed-threshold configuration uses 2.60 scheduled steps (2.92 action-head evaluations) and reports 15.9 ms action-head latency, versus 235.7 ms for 50-step Euler, while obtaining 83.2% success versus 82.5%. On LIBERO, it uses 4.53 scheduled steps (5.09 action-head evaluations), reports 32.7 ms, and obtains 88.7% success, compared with 278.7 ms and 92.5% for 50-step Euler. We also evaluate OSC on three real-world manipulation tasks with 50 trials per method and task. These evaluations characterize action-decoding compute-performance trade-offs without additional training.
Read the full abstract ▾
All authors Zhou Fang, Jiaqi Wang, Wenzhuo Li, Yi Zhou, Qiongfeng Shi
Robotics
Generative Control as Optimization: Time Unconditional Flow Matching for Adaptive and Robust Robotic Control
Zunzhe Zhang, Runhan Huang, Yicheng Liu et al. (+3 more)
Diffusion models and flow matching have become a cornerstone of robotic imitation learning, yet they suffer from a structural inefficiency where inference is often bound to a fixed integration schedule that is agnostic to state complexity. This paradigm forces the policy to expend the same computational budget on trivial motions as it does on complex tasks. We introduce Generative Control as Optimization (GeCO), a time-unconditional framework that transforms action synthesis from trajectory integration into iterative optimization. GeCO learns a stationary velocity field in the action-sequence space where expert behaviors form stable attractors. Consequently, test-time inference becomes an adaptive process that allocates computation based on convergence–exiting early for simple states while refining longer for difficult ones. Furthermore, this stationary geometry yields an intrinsic, training-free safety signal, as the field norm at the optimized action serves as a robust out-of-distribution (OOD) detector, remaining low for in-distribution states while significantly increasing for anomalies. We validate GeCO on standard simulation
benchmarks and demonstrate seamless scaling to pi0-series
Vision-
Language-Action (VLA) models. As a plug-and-play replacement for standard flow-matching heads, GeCO improves success rates and efficiency with an optimization-native mechanism for safe deployment.
Video and code can be found at https://hrh6666.github.io/GeCO/
Read the full abstract ▾
All authors Zunzhe Zhang, Runhan Huang, Yicheng Liu, Shaoting Zhu, Linzhan Mou, Hang Zhao
Robotics
EVA: Aligning Video World Models with Executable Robot Actions via Inverse Dynamics Rewards
Ruixiang Wang, Qingming Liu, Yueci Deng et al. (+3 more)
Video generative models offer a promising basis for robotic planning by predicting future visual rollouts from observations and task instructions. However, their training objectives typically lack explicit constraints on
robot executability. Generated rollouts can exhibit embodiment-related artifacts, including arm deformation and temporal discontinuities, while the actions decoded from these rollouts may be unstable or violate
robot motion limits. We term this mismatch between generated
robot behavior and embodiment-specific execution requirements the executability gap. We introduce Executable
Video Alignment (EVA), a post-training framework that uses action-space feedback to improve both the visual integrity of generated
robot motion and the feasibility of decoded actions. EVA trains an inverse dynamics model (IDM) on
robot demonstrations and freezes it to decode actions from generated
videos. We compute a reward from these actions that penalizes non-smooth motion and violations of velocity and acceleration limits, and use it to guide reinforcement-learning updates to the
video generator. By feeding decoded-action constraints back into
video generation, EVA links visual and action-level improvements through a shared
alignment signal. Experiments on RoboTwin, LIBERO, and a real bimanual
robot show that EVA reduces embodiment-related visual artifacts and improves task success when executing IDM-decoded actions. Further experiments show that
video models fine-tuned with EVA generate more effective
video–action pairs for training downstream VLAs.
Read the full abstract ▾
All authors Ruixiang Wang, Qingming Liu, Yueci Deng, Guiliang Liu, Zhen Liu, Kui Jia
Robotics
Huddle: Parallel Shape Assembly using Decentralized, Minimalistic Robots
Khai Yi Chin, Tingwei Meng, Zhe Chen et al. (+2 more)
We propose a novel
algorithm for forming arbitrarily shaped assemblies using decentralized
robots. By relying on local interactions, the
algorithm ensures there are no unreachable states or gaps in the assembly, which are global properties. The in-assembly
robots attract passing-by
robots into expanding the assembly via a simple implementation of signaling and
alignment. Our approach is minimalistic, requiring only communication between attached, immediate neighbors. It is motion-agnostic and requires no pose localization, enabling asynchronous and order-independent assembly. We prove the
algorithm's correctness and demonstrate its effectiveness in forming a 107-
robot assembly.
Read the full abstract ▾
All authors Khai Yi Chin, Tingwei Meng, Zhe Chen, Daniel Bassett, Yuri Ivanov
Robotics
Multisource human-in-the-loop digital twin testbed for connected and autonomous vehicles in mixed traffic flow
Jianghong Dong, Chunying Yang, Mengchi Cai et al. (+5 more)
In the emerging mixed traffic environments, Connected and Autonomous Vehicles (CAVs) have to interact with surrounding human-driven vehicles (HDVs). This paper introduces MSH-MCCT (Multi-Source Human-in-the-Loop Mixed Cloud Control Testbed), a novel CAV testbed that captures complex interactions between various CAVs and HDVs. Utilizing the Mixed Digital Twin concept, which combines Mixed Reality with Digital Twin, MSH-MCCT integrates physical, virtual, and mixed platforms, along with multi-source control inputs. Bridged by the mixed platform, MSH-MCCT allows human drivers and CAV
algorithms to operate both physical and virtual vehicles within multiple fields of view. Particularly, this testbed facilitates the coexistence and real-time interaction of physical and virtual CAVs & HDVs, significantly enhancing the experimental flexibility and scalability. Experiments on vehicle platooning in mixed traffic showcase the potential of MSH-MCCT to conduct CAV testing with multi-source real human drivers in the loop through driving simulators of diverse fidelity. The
videos for the experiments are available at our project website: https://dongjh20.github.io/MSH-MCCT.
Read the full abstract ▾
All authors Jianghong Dong, Chunying Yang, Mengchi Cai, Chaoyi Chen, Qing Xu, Jianqiang Wang, Jiawei Wang, Keqiang Li
Robotics
VolumeDP: Modeling Volumetric Representation for Manipulation Policy Learning
Tianxing Zhou, Feiyang Xue, Zhangchen Ye et al. (+3 more)
Imitation learning is a prominent paradigm for robotic manipulation. However, existing visual imitation methods map 2D image observations directly to 3D action outputs, imposing a 2D-3D mismatch that hinders spatial
reasoning and degrades robustness. We present VolumeDP, a policy architecture that restores spatial
alignment by explicitly
reasoning in 3D. VolumeDP first lifts image features into a Volumetric Representation via cross-
attention. It then selects task-relevant voxels with a learnable module and converts them into a compact set of spatial
tokens, markedly reducing computation while preserving action-critical geometry. Finally, a multi-
token decoder conditions on the entire
token set to predict actions, thereby avoiding lossy aggregation that collapses multiple spatial
tokens into a single descriptor. VolumeDP achieves a state-of-the-art average success rate of 88.8% on the LIBERO simulation
benchmark, outperforming the strongest baseline by a substantial 14.8% improvement. It also delivers large performance gains over prior methods on the ManiSkill and LIBERO-Plus
benchmarks. Real-world experiments further demonstrate higher success rates and robust generalization to novel spatial layouts, camera viewpoints, and environment backgrounds. Code and
videos are available on the project page: https://yzc0731.github.io/VolumeDP/
Read the full abstract ▾
All authors Tianxing Zhou, Feiyang Xue, Zhangchen Ye, Tianyuan Yuan, Hang Zhao, Tao Jiang
Robotics
AERR-Nav: Adaptive Exploration-Recovery-Reminiscing Strategy for Zero-Shot Object Navigation
Jingzhi Huang, Junkai Huang, Haoyang Yang et al. (+2 more)
Zero-Shot Object
Navigation (ZSON) in unknown multi-floor environments presents a significant challenge. Recent methods, mostly based on semantic value greedy waypoint selection, spatial
topology-enhanced
memory, and
Multimodal Large
Language Model (MLLM) as a decision-making framework, have led to improvements. However, these architectures struggle to balance exploration and exploitation for ZSON when encountering unseen environments, especially in multi-floor settings, such as
robots getting stuck at narrow intersections, endlessly wandering, or failing to find stair entrances. To overcome these challenges, we propose AERR-Nav, a Zero-Shot Object
Navigation framework that dynamically adjusts its state based on the
robot's environment. Specifically, AERR-Nav has the following two key advantages: (1) An Adaptive Exploration-Recovery-Reminiscing Strategy, enables
robots to dynamically transition between three states, facilitating specialized responses to diverse
navigation scenarios. (2) An Adaptive Exploration State featuring Fast and Slow-Thinking modes helps
robots better balance exploration, exploitation, and higher-level
reasoning based on evolving environmental information. Extensive experiments on the HM3D and MP3D
benchmarks demonstrate that our AERR-Nav achieves state-of-the-art performance among zero-shot methods. Comprehensive ablation studies further validate the efficacy of our proposed strategy and modules.
Read the full abstract ▾
All authors Jingzhi Huang, Junkai Huang, Haoyang Yang, Haoang Li, Yi Wang
Robotics
Consistency-Driven Dual LSTM Models for Kinematic Control of a Wearable Soft Robotic Arm
Xingyu Chen, Yi Xiong and Li Wen
In this paper, we introduce a consistency-driven dual LSTM framework for accurately learning both the forward and inverse kinematics of a pneumatically actuated soft robotic arm integrated into a wearable device. This approach effectively captures the nonlinear and hysteretic behaviors of soft pneumatic actuators while addressing the one-to-many mapping challenge between actuation inputs and end-effector positions. By incorporating a cycle consistency loss, we enhance physical realism and improve the stability of inverse predictions. Extensive experiments-including trajectory tracking, ablation studies, and wearable demonstrations-confirm the effectiveness of our method. Results indicate that the inclusion of the consistency loss significantly boosts prediction accuracy and promotes physical consistency over conventional approaches. Moreover, the wearable soft robotic arm demonstrates strong human-
robot collaboration capabilities and adaptability in everyday tasks such as object handover, obstacle-aware pick-and-place, and drawer operation. This work underscores the promising potential of learning-based kinematic models for human-centric, wearable robotic systems.
Read the full abstract ▾
Robotics
Zihao Xin, Wentong Li, Yixuan Jiang et al. (+6 more)
Vision-and-
Language Navigation (VLN) requires an
embodied agent to ground complex natural-
language instructions into long-horizon
navigation in unseen environments. While
Vision-
Language Models (VLMs) offer strong 2D semantic understanding, current VLN systems remain constrained by limited spatial perception, 2D-3D representation mismatch, and
monocular scale ambiguity. In this paper, we propose AgentVLN, a novel and efficient
embodied navigation framework that can be deployed on edge computing platforms. We formulate VLN as a Partially Observable Semi-Markov Decision Process (POSMDP) and introduce a VLM-as-
Brain paradigm that decouples high-level semantic
reasoning from perception and planning via a plug-and-play skill library. To resolve multi-level representation inconsistency, we design a cross-space representation mapping that projects perception-layer 3D topological waypoints into the image plane, yielding pixel-aligned visual
prompts for the VLM. Building on this bridge, we integrate a context-aware self-correction and active exploration strategy to recover from occlusions and suppress error accumulation over long trajectories. To further address the spatial ambiguity of instructions in unstructured environments, we propose a Query-Driven Perceptual Chain-of-Thought (QD-PCoT) scheme, enabling the
agent with the metacognitive ability to actively seek
geometric depth information. Finally, we construct AgentVLN-Instruct, a large-scale instruction-tuning
dataset with dynamic stage routing conditioned on target visibility. Extensive experiments show that AgentVLN consistently outperforms prior state-of-the-art methods (SOTA) on long-horizon VLN
benchmarks, offering a practical paradigm for lightweight deployment of next-generation
embodied navigation models. Code: https://github.com/Allenxinn/AgentVLN.
Read the full abstract ▾
All authors Zihao Xin, Wentong Li, Yixuan Jiang, Ziyuan Huang, Bin Wang, Piji Li, Jianke Zhu, Jie Qin, Shengjun Huang
Robotics
REAL: Robust Extreme Agility via Spatio-Temporal Policy Learning and Physics-Guided Filtering
Jialong Liu, Dehan Shen, Yanbo Wen et al. (+2 more)
Extreme legged parkour demands rapid terrain assessment and precise foot placement under highly dynamic conditions. While recent learning-based systems achieve impressive agility, they remain fundamentally fragile to perceptual degradation, where even brief visual noise or latency can cause catastrophic failure. To overcome this, we propose Robust Extreme Agility Learning (REAL), an end-to-end framework for reliable parkour under sensory corruption. Instead of relying on perfectly clean perception, REAL tightly couples
vision, proprioceptive history, and temporal
memory. We distill a cross-modal teacher policy into a deployable student equipped with a FiLM-modulated Mamba backbone to actively filter visual noise and build short-term terrain
memory actively. Furthermore, a physics-guided
Bayesian state
estimator enforces rigid-body consistency during high-impact maneuvers. Validated on a Unitree Go2 quadruped, REAL successfully traverses extreme obstacles even with a 1-meter visual blind zone, while strictly satisfying real-time control constraints with a bounded 13.1 ms inference time.
Read the full abstract ▾
All authors Jialong Liu, Dehan Shen, Yanbo Wen, Zeyu Jiang, Changhao Chen
Robotics
VectorWorld: Efficient Streaming World Model via Diffusion Flow on Vector Graphs
Chaokang Jiang, Desen Zhou, Jiuming Liu et al. (+1 more)
Closed-loop evaluation of autonomous-driving policies requires interactive simulation beyond log replay. However, existing generative world models often degrade in closed loop due to (i) history-free initialization that mismatches policy inputs, (ii) multi-step sampling latency that violates real-time budgets, and (iii) compounding kinematic infeasibility over long horizons. We propose VectorWorld, a streaming world model that incrementally generates ego-centric $64 \mathrm{m}\times 64\mathrm{m}$ lane–
agent vector-graph tiles during rollout. VectorWorld aligns initialization with history-conditioned policies by producing a policy-compatible interaction state via a motion-aware gated VAE. It enables real-time outpainting via solver-free one-step masked completion with an edge-gated relational DiT trained with interval-conditioned MeanFlow and JVP-based large-step supervision. To stabilize long-horizon rollouts, we introduce $\Delta$Sim, a physics-aligned non-ego (NPC) policy with hybrid discrete–continuous actions and differentiable kinematic logit shaping. On Waymo open motion and nuPlan, VectorWorld improves map-structure fidelity and initialization validity, and supports stable, real-time $1\mathrm{km}+$ closed-loop rollouts (\href{https://github.com/jiangchaokang/VectorWorld}{code}).
Read the full abstract ▾
All authors Chaokang Jiang, Desen Zhou, Jiuming Liu, Kevin Li Sun
Robotics · Cross-listed from eess.SY
Real-Time Online Learning for Model Predictive Control using a Spatio-Temporal Gaussian Process Approximation
Lars Bartels, Amon Lahr, Andrea Carron et al. (+1 more)
Learning-based model predictive control (MPC) can enhance control performance by correcting for model inaccuracies, enabling more precise state trajectory predictions than traditional MPC. A common approach is to model unknown residual dynamics as a Gaussian process (GP), which leverages data and also provides an estimate of the associated uncertainty. However, the high computational cost of online learning poses a major challenge for real-time GP-MPC applications. This work presents an efficient implementation of an approximate spatio-temporal GP model, offering online learning at constant computational complexity. It is optimized for GP-MPC, where it enables improved control performance by learning more accurate system dynamics online in real-time, even for time-varying systems. The performance of the proposed method is demonstrated by simulations and hardware experiments in the exemplary application of autonomous miniature racing.
Read the full abstract ▾
All authors Lars Bartels, Amon Lahr, Andrea Carron, Melanie N. Zeilinger