Robotics · Cross-listed from cs.CV
Flame3D: Zero-shot Compositional Reasoning of 3D Scenes with Agentic Language Models
Sagar Bharadwaj, Ziyong Ma, Anurag Ghosh et al. (+2 more)
3D scene understanding spans
reasoning about free space, object grounding, hypothetical object insertions, complex
geometric relationships, and integrating all of these with external tools and data sources. Existing 3D understanding methods typically rely on large-scale 3D-
language training or focus on object grounding and simple spatial relationships. We argue that the broad generalization that motivates 3D-
language training can be achieved at inference time, without 3D-specific training. We propose Flame3D, a training-free framework that represents scenes as editable visual-textual 3D
memories and exposes them to an off-the-shelf MLLM through composable spatial tools. Flame3D also lets the
agent synthesize custom spatial programs at inference time, enabling open-ended
reasoning over layouts, empty space, and objects not yet present in the scene. External data and corrections can be added to the
memory without retraining. In addition to showing competitive performance to finetuned 3D-LMM methods on ScanQA, we study multi-hop 3D
reasoning capabilities of Flame3D by evaluating it on a curated compositional spatial-
reasoning benchmark, Compose3D. We find that fixed tools fall short and that the
agent's ability to synthesize spatial operations at inference time is essential. These results invite the question: should future progress in 3D scene understanding focus on richer scene
memories and expressive compositional abstractions?
Read the full abstract ▾
All authors Sagar Bharadwaj, Ziyong Ma, Anurag Ghosh, Srinivasan Seshan, Anthony Rowe
Robotics
Continuum Robot Modeling with Action Conditioned Flow Matching
Jiong Lin, Jinchen Ruan and Hod Lipson
Predicting the shape of tendon driven continuum
robots (TDCRs) at steady state from actuation remains challenging due to continuous deformation, complex tendon routing, compliance, friction, and fabrication variability. In this paper, we address this problem as kinematic self modeling conditioned on action. We present a lightweight 3D printed TDCR hardware platform and an RGB-D data collection pipeline with multiple cameras, and we learn a point cloud flow matching model that maps motor actuation states to the
robot's settled 3D geometry. The model is trained from randomly sampled quasi static configurations and evaluated on test motor commands within the same TDCR design family and actuation range. We compare against prior 3D deformable object and
robot self modeling approaches in both MuJoCo simulation and real hardware experiments. Experiments on simulated 2-, 3-, and 5-module TDCRs and real 2- and 3-module
robots show improved shape prediction accuracy under CD and EMD metrics. We further show in simulation that the same conditional formulation generalizes to tip payload as a conditioning input, enabling payload conditioned steady-state shape prediction. These results demonstrate a data driven self modeling framework for quasi static TDCR geometry prediction.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
RigidFormer: Learning Rigid Dynamics using Transformers
Zhiyang Dou, Minghao Guo, Haixu Wu et al. (+3 more)
Learning-based simulation of multi-object rigid-body dynamics remains difficult because contact is discontinuous and errors compound over long horizons. Most existing methods remain tied to mesh connectivity and vertex-level message passing, which limits their applicability to mesh-free inputs such as point clouds and leads to high computational cost. Efficiently modeling high-fidelity rigid-body dynamics from mesh-free representations, therefore, remains challenging. We introduce RigidFormer, an object-centric
Transformer-based model that learns mesh-free rigid-body dynamics with controllable integration step sizes. RigidFormer reasons at the object level and advances each object through compact anchors; Anchor-Vertex Pooling enriches these anchors with local vertex features, retaining contact-relevant geometry without dense vertex-level interaction. We propose Anchor-based RoPE to inject anchor geometry into
attention while respecting the unordered nature of objects and anchors: object-
token processing is permutation-equivariant, and the mean-pooled anchor descriptor is invariant to anchor reindexing while preserving shape extent. RigidFormer further enforces
rigidity by projecting updates onto the rigid-body
manifold using differentiable Kabsch
alignment. On standard
benchmarks, RigidFormer outperforms or matches mesh-based baselines using point inputs, runs faster, generalizes to unseen point resolutions and across
datasets, and scales to 200+ objects; we also show a preliminary extension to command-conditioned articulated bodies by treating body parts as interacting object-level components.
Read the full abstract ▾
All authors Zhiyang Dou, Minghao Guo, Haixu Wu, Doug Roble, Tuur Stuyck, Wojciech Matusik
Robotics
SHIELD: Scalable Optimal Control with Certification using Duality and Convexity
Hansung Kim, Siddharth H. Nair and Francesco Borrelli
We present SHIELD, a hierarchical
algorithm that reduces both the decision-variable dimension and the constraint set in $\ell_1$-regularized convex programs. From strong convexity and Lagrangian duality, we derive certificates that safely discard constraints and decision variables while guaranteeing that all removed constraints remain satisfied and all removed variables are null. To further accelerate the proposed
algorithm, we propose a
transformer-based deep neural network to guide the dual certificate inference. We validate SHIELD on
stochastic model predictive control (SMPC) in complex, multi-modal traffic scenarios, comparing against a full-dimensional SMPC policy. Numerical simulations demonstrate order-of-magnitude computational speedups while preserving feasibility and closed-loop safety, highlighting the practicality of certifiably safe, lightweight MPC in complex driving scenes.
Read the full abstract ▾
Robotics
Beyond Self-Play: Hierarchical Reasoning for Continuous Motion in Closed-Loop Traffic Simulation
Weifan Zhang, Xiaofeng Zhao, Adel Bazzi et al. (+3 more)
Closed-loop traffic simulation requires
agents that are both scalable and behaviorally realistic. Recent self-play reinforcement learning approaches demonstrate strong scalability, but their
equilibrium strategies fail to capture the socially aware behaviors of real human drivers. We propose a hierarchical architecture that goes beyond self-play by combining high-level multi-
agent interaction
reasoning with low-level continuous trajectory realization. Specifically, a Stackelberg-style Multi-
Agent Reinforcement Learning (MARL) module generates interaction-aware intention commands. These commands condition a low-level continuous motion module, translating the strategic intent into physically consistent, scene-responsive control sequences. To mitigate distribution shift in closed-loop deployment, we introduce a hybrid co-training scheme combining MARL with auxiliary recovery supervision. Experiments on a SUMO-based urban network demonstrate that the proposed framework achieves superior control smoothness and safety compared to self-play and passive imitation baselines, while maintaining competitive traffic efficiency.
Read the full abstract ▾
All authors Weifan Zhang, Xiaofeng Zhao, Adel Bazzi, Mingrui Li, Yifan Wei, Dengfeng Sun
Robotics
IMPACT: An Implicit Active-Set Augmented Lagrangian for Fast Contact-Implicit Trajectory Optimization
Jiayun Li, Dejian Gong and Georgia Chalvatzaki
Contact-implicit trajectory optimization (CITO) has attracted growing
attention as a unified framework for planning and control in contact-rich robotic tasks. Recent approaches have demonstrated promising results in manipulation and locomotion without requiring a prescribed contact-mode schedule. It is well known that the underlying mathematical programs with complementarity constraints (MPCCs) remain numerically ill-conditioned, and systematic, scalable solution strategies for CITO remain an active area of research. More efficient and principled solvers that can handle contact constraints are therefore essential to broaden the applicability of CITO. In this work, we develop an augmented-Lagrangian approach to CITO for solving MPCC-based CITO with stationarity guarantees. The method can be interpreted as identifying the implicit contact-mode branches on the fly during the trajectory optimization (TO) iterations; we call this approach IMPACT (IMPlicit contact ACtive-set Trajectory optimization). We provide an efficient C++ implementation tailored to trajectory-optimization workloads and evaluate it on the open-source CITO and contact-implicit model predictive control (CI-MPC)
benchmarks. On CITO, IMPACT achieves 2.9x-70x speedups over strong baselines (
geometric mean 13.8x). On CI-MPC, we show improved control quality for contact-rich trajectories on dexterous manipulation tasks in simulation. Finally, we demonstrate the proposed method on real robotic hardware on a T-shaped object pushing task.
Read the full abstract ▾
Robotics
HyDRA Scorpion: A Cost-effective and Modular ROV for Real-Time Underwater Inspection, Intervention, and Object Detection
Anika Tabassum Orchi, Md Farhan Zaman, Md Darain Khan et al. (+16 more)
A Remotely Operated Vehicle (ROV) is a tethered
underwater robot used for tasks like inspection and intervention. While essential tools for
underwater science, the high cost of commercial ROVs and a persistent gap between mechanically capable platforms and those with integrated intelligence create a significant barrier to access. HyDRA Scorpion differs from conventional systems by addressing these challenges, integrating an advanced, AI-driven perception stack with in-situ measurement capabilities onto a low-cost, locally manufacturable platform. The system combines 4-DoF maneuverability, dual manipulators, and a custom pressure-tested housing. Experimental results validate the system's robustness and performance. Leak-free operation was confirmed through prolonged pressure testing of the electronics housing to 4 bar, equivalent to the pressure of a 304.8-meter water depth approximately in a simulated environment, with no moisture ingress detected. The vehicle also demonstrated stable station-keeping, maintaining its position within a tight tolerance of $\(\pm\)0.15$ meters under external disturbances. The onboard AI module achieved
underwater object detection mean Average Precision (mAP) of 0.89 with real-time inference, length and 3D-mapping based distance measurement. Also, 4-DoF manipulator arm can grip and maintain dual-function manipulator feature which support 360 degree tangle-free rotation.
Read the full abstract ▾
All authors Anika Tabassum Orchi, Md Farhan Zaman, Md Darain Khan, Md Alamgir, Mahbubul Islam, Md. Jobayer Rahman, A. M. Zayed Abdullah, Md Mehrab Hossain Khan, Md. Kutub Al Baki, Iftekharul Islam, Shakil Ahmed, Md Sadique Hossain, Md Muzahidul Islam, Shah Mohammad Seaman, Nusrat Jahan Piyal, Shekh Md. Saifur Rahman, Fahim Hafiz, A.K.M. Muzahidul Islam, M. Rezwan Khan
Robotics
Smoothing Out the Edges: Continuous-Time Estimation with Gaussian Process Motion Priors on Factor Graphs
Connor Holmes, Sven Lilge, Zi Cong Guo et al. (+2 more)
Continuous-time state estimation is gaining in popularity due to its abilities to provide smooth solutions, handle asynchronous sensors, and interpolate between data points. While there are two main paradigms, parametric (e.g., temporal basis functions, splines) and nonparametric (Gaussian processes), the latter has seen less adoption despite its technical advantages and relative ease of implementation. In this article, we seek to rectify this situation by providing a new simplified explanation of GP continuous-time estimation rooted in the
language of factor graphs, which have become the de facto estimation paradigm in much of robotics. To simplify onboarding, we also provide three working examples implemented in the popular GTSAM estimation framework.
Read the full abstract ▾
All authors Connor Holmes, Sven Lilge, Zi Cong Guo, Frank Dellaert, Timothy D. Barfoot
Robotics
Octopus Protocol: One-Shot Hardware Discovery and Control for AI Agents via Infrastructure-as-Prompts
Quilee Simeon, Justin M. Wei and Yile Fan
Bringing a previously unintegrated device under the control of an AI
agent still requires device-specific engineering: driver selection, dependency resolution, interface design, and deployment, repeated per device and per platform. We present Octopus, a hardware onboarding framework in which a coding
agent, rather than a shipped integration, is the runtime that produces the required infrastructure. Given shell access and a model API key, a single bootstrap command drives the
agent through a five-stage pipeline that enumerates operating-system- visible hardware, infers device identity and capabilities, generates typed Model Context Protocol tools and the hardware-facing code behind them, and activates the result as a live endpoint. A persistent daemon then maintains the result, repairing defined classes of failure in the deployment it produced. Across four hosts spanning two processor architectures, three operating- system families, and two device-access paths, identical prose specifications produced working interfaces with no per-host edits and no hand-written integration code. Five consecutive runs on the reference host completed end to end on first attempt. We report both the resulting capability and a failure mode of unattended repair loops observed over eleven hours of continuous operation.
Read the full abstract ▾
Robotics
Automated Robotic Moisture Monitoring in Agricultural Fields
Senthil Palanisamy and Akila I.S
Monitoring moisture level of land in a large-scale plantation is tedious. The main objective of this project is to use a robotic kit in collaboration with the on-field moisture sensor
circuits, thereby creating an efficient and economical moisture monitoring system. A large agriculture field is divided into smaller grids. Each grid is placed with a moisture sensor. Whenever a sensor reports the soil to be dry, the
robot goes to the concerned field for inspection. The path to the concerned field is found by applying Dijkstra's shortest path
algorithm on the aerial image of the field. Then the total moisture content of the field is calculated by the
robot using suitable image processing
algorithms and reported accordingly. For developing and testing this work, a small study field was set up above which a camera was mounted at an appropriate height to capture its aerial view. Thus a prototype for an automated system of monitoring agricultural fields' moisture has been developed through this work.
Read the full abstract ▾
Robotics
Terminal Matters: Kinodynamic Planning with a Terminal Cost and Learned Uncertainty in Belief State-Cost Space
Zhuoyun Zhong, Seyedali Golestaneh and Constantinos Chamzas
In many real-world robotic tasks,
robots must generate dynamically feasible motions that reliably reach desired goals even under uncertainty. Yet existing sampling-based kinodynamic planners typically optimize accumulated trajectory costs and treat goal reaching as a feasibility check, rather than explicitly optimizing terminal-state quality, such as goal preference or goal-reaching reliability. In this work, we introduce a terminal-cost formulation for kinodynamic planning that allows terminal-state quality to be optimized alongside accumulated trajectory cost. We prove that AO-RRT, an asymptotically optimal kinodynamic planner, preserves its asymptotic optimality under this augmented objective. We further extend the formulation to belief space and prove that minimizing the Wasserstein distance between the terminal belief and the goal improves a lower bound on the probability of reaching the goal region. The resulting planner, KiTe, uses this terminal-cost objective to encode goal preferences and improve reliability under uncertainty. To support systems without analytical uncertainty models, we learn dynamics and process uncertainty directly from data and integrate the learned belief dynamics into planning. Experiments on Flappy Bird, Car Parking, and Planar Pushing show that KiTe consistently improves goal-reaching success under uncertainty. Real-world Planar Pushing experiments further demonstrate that KiTe can plan effectively with learned dynamics and uncertainty. Source code is available at https://github.com/elpis-lab/KiTe.
Read the full abstract ▾
Robotics
Towards Backdoor-Based Ownership Verification for Vision-Language-Action Models
Ming Sun, Rui Wang, Xingrui Yu et al. (+5 more)
Vision-
Language-Action models (VLAs) support generalist robotic control by enabling end-to-end decision policies directly from multi-modal inputs. As trained VLAs are increasingly shared and adapted, protecting model ownership becomes essential for secure deployment and responsible open-source usage. In this paper, we present GuardVLA, the first
backdoor-based ownership verification framework specifically designed for VLAs. GuardVLA embeds a stealthy and harmless
backdoor watermark into the protected model during training by injecting secret messages into
embodied visual data. For post-release verification, we propose a swap-and-detect mechanism, in which the trigger projector and an external classifier head are used to activate and detect the embedded
backdoor based on prediction probabilities. Extensive experiments across multiple
datasets, model architectures, and adaptation settings demonstrate that GuardVLA enables reliable ownership verification while preserving benign task performance. Further results show that the embedded watermark remains detectable under post-release model adaptation.
Read the full abstract ▾
All authors Ming Sun, Rui Wang, Xingrui Yu, Lihua Jing, Hangyu Du, Zhenglin Wan, Xu Pan, Ivor Tsang
Robotics
A low-cost mockup to simulate robotic laser cutting in nuclear decommissioning
Frederico Fernandes Afonso Silva, Murilo Marques Marinho and Bruno Vilhena Adorno
This paper introduces a low-cost experimental mockup to simulate the
laser cutting process of containers in
nuclear decommissioning. It is composed of a three-axis table supporting a cuboid container with ultraviolet-sensitive faces, a six-degree-of-freedom serial manipulator holding an ultraviolet torch that simulates the
laser, and a visual system based on cameras and fiducial markers. The system employs a constrained task-space adaptive motion controller that compensates for inaccurate parameters and eliminates the need to calibrate the system. Furthermore, as the motion controller explicitly accounts for
geometric constraints, the
robot reactively avoids collisions with obstacles while handling the ultraviolet torch. To improve the tracking of the
laser-cutting path, we control the ultraviolet beam, which requires only four degrees of freedom, instead of the full end-effector pose. Experiments show that, despite an initially uncalibrated system, the overall system is capable of tracking different trajectories with an overall mean accuracy of 3.9 (sd 2.5) mm when the end-effector pose is controlled and 2.4 (sd 1.3) mm when the ultraviolet beam is controlled.
Read the full abstract ▾
Robotics
Raymoval: Raycasting-based Dynamic Object Removal for Static 3D Mapping
Daebeom Kim, Seungjae Lee, Seoyeon Jang et al. (+2 more)
Static mapping is fundamental to
robot navigation, providing a persistent
geometric prior and a consistent reference for long-term autonomy. However, dynamic objects leave residual traces and cause surface loss, which reduces map consistency. We propose a raycasting-based module for dynamic object removal in static 3D mapping. Each scan is projected onto an azimuth-elevation grid, and for every viewing direction we compare the bin-wise minimum range with the map's first-hit distance computed by raycasting. Furthermore, we apply a raycast consistency test that separates dynamic from static points. Finally, a spatial consistency validation step refines labels, producing static maps with lower residual dynamics and reduced over-removal. We evaluate our approach quantitatively and qualitatively on SemanticKITTI and a challenging custom
dataset, and show consistent static mapping results.
Read the full abstract ▾
All authors Daebeom Kim, Seungjae Lee, Seoyeon Jang, Kevin Christiansen Marsim, Hyun Myung
Robotics · Cross-listed from eess.IV
VISTA: A Benchmark for Real-Time Video Streaming under Network Impairments in Surgical Teleoperation
Zexin Deng, Zhenhui Yuan, Tian Lu et al. (+3 more)
Real-time
video streaming is crucial in
surgical teleoperation, yet reproducible evaluation under realistic network impairments remains limited. This paper presents VISTA, a
benchmark designed to study how impairments along the forward
video path affect received
video quality, temporal continuity, and human task performance. VISTA employs Linux Traffic Control with NetEm and a Gilbert-Elliott loss model to emulate five network conditions: Hospital LAN, 5G Urban, 4G Rural, LEO Satellite, and GEO Satellite. The
benchmark integrates a standardised peg transfer task with synchronized measurements of network quality of service (QoS), objective
video quality (PSNR, SSIM, and VMAF), and temporal continuity through freeze rate, while maintaining a stable reverse control channel. Across 375 experimental trials, network degradation substantially reduced teleoperation performance: success rate decreased from 97% in Hospital LAN to 79% in 5G Urban, 35% in 4G Rural, 71% in LEO Satellite, and 12% in GEO Satellite, while mean task completion time for successful trials increased from 80 s in Hospital LAN to 117 s in 5G Urban, 211 s in 4G Rural, 152 s in LEO Satellite, and 255 s in GEO Satellite. These findings show that network impairments have a direct impact on task completion and success in
surgical teleoperation, and provide a reproducible basis for evaluating teleoperation
video under realistic network constraints. Source code available at https://github.com/Dzxx623/VISTA.
Read the full abstract ▾
All authors Zexin Deng, Zhenhui Yuan, Tian Lu, Gaofeng Li, Meipeng Huang, Longhao Zou
Robotics
Preserving Foundational Capabilities in Flow-Matching VLAs through Conservative SFT
Tianyi Zhang, Shaopeng Zhai, Haoran Zhang et al. (+2 more)
Unconstrained fine-tuning of flow-matching
Vision-
Language-Action (VLA) models drives dense parameter overwrites, degrading pre-trained capabilities. We present Conservative Supervised Fine-Tuning (ConSFT), an optimization objective that adapts to target distributions while mitigating catastrophic forgetting, requiring zero prior data or architectural overhead. By dynamically scaling learning signals based on model confidence, ConSFT suppresses excessive gradients from low-confidence samples to prevent disproportionate parameter updates, thereby bounding the intrinsic parameter disruption risk. Inspired by reinforcement learning's trust-region clipping, this formulation establishes a progressive learning dynamic to secure target convergence and prior capability retention, maintaining sparse parameter updates without relying on the parallel reference networks required by explicit regularization. We evaluate ConSFT on the LIBERO and RoboTwin
benchmarks across state-of-the-art flow-matching VLAs ($\pi_0$, $\pi_{0.5}$, and GR00T-N1.6-3B). The method outperforms vanilla SFT in capability retention by an average absolute margin of over 20%, matching the efficacy of data-heavy Experience Replay in a prior-data-free regime. Real-world robotic deployments confirm that ConSFT precludes spatial overfitting during downstream adaptation, preserving pre-trained physical skills while acquiring sequential target tasks.
Read the full abstract ▾
All authors Tianyi Zhang, Shaopeng Zhai, Haoran Zhang, Fuxian Huang, Qi Zhang
Robotics
AssemPlanner: A Multi-Agent Based Task Planning Framework for Flexible Assembly System
Chenhao Zhang, Chaoran Zhang, Zhaobo Xu et al. (+3 more)
In flexible assembly systems, existing task planning methods require a time-consuming configuration process by multiple experts to establish a production line for a new product. To address this challenge, we propose a multi-
agent based task planning framework for flexible assembly systems, denoted as AssemPlanner. It takes tasks described in natural
language as input, which are then converted into actionable sequential production operations. It comprises several specialized
agents, including SchedAgent , KnowledgeAgent, LineBalanceAgent, and a scene graph. Within the proposed framework, SchedAgent serves as the central
reasoning engine. Departing from traditional static pipelines, AssemPlanner utilizes a ReAct-based SchedAgent to adaptively adjust actions via multi-
agent feedback. By observing the feedback from KnowledgeAgent, LineBalanceAgent, and the scene graph, it autonomously resolves complex industrial process constraints. To facilitate reproducibility, all code and
datasets are released at https://github.com/chz332/Assemplanner.
Read the full abstract ▾
All authors Chenhao Zhang, Chaoran Zhang, Zhaobo Xu, Yongbo Yang, Pingfa Feng, Long Zeng
Robotics · Cross-listed from cs.CV
VECTOR-Drive: Tightly Coupled Vision-Language and Trajectory Expert Routing for End-to-End Autonomous Driving
Rui Zhao, Jianlin Yu, Zhenhai Gao et al. (+2 more)
End-to-end autonomous driving requires models to understand traffic scenes, infer driving intent, and generate executable motion plans. Recent
vision-
language-action (VLA) models inherit semantic priors from large-scale
vision-
language pretraining, yet still face a coupling trade-off: fully shared backbones preserve
multimodal interaction but may entangle
language reasoning and trajectory prediction, whereas decou pled
reasoning-action pipelines reduce task conflict but weaken semantic-motion coupling. We propose VECTOR-DRIVE, a tightly coupled VLA framework built on Qwen2.5-VL-3B. VECTOR-DRIVE keeps all
tokens coupled through shared self
attention and routes feed-forward computation according to
token semantics.
Vision and
language tokens are processed by a
Vision-
Language Expert to preserve semantic priors, while target-point, ego-state, and noisy action
tokens are routed to a Trajectory Expert for motion-specific computation. On the action-
token pathway, a flow-matching planner refines noisy action
tokens into future waypoints and speed profiles. This design couples semantic
reasoning and motion planning within a single
multimodal Transformer while separating task-specific FFN computation. On Bench2Drive, VECTOR-DRIVE achieves 88.91 Driving Score and outperforms representative end-to end and VLA-based baselines. Qualitative results and ablations further validate the benefits of shared
attention, semantic-aware expert routing, progressive training, and flow-based action de coding.
Read the full abstract ▾
All authors Rui Zhao, Jianlin Yu, Zhenhai Gao, Jiaqiao Liu, Fei Gao
Robotics
Constraint-Aware Diffusion Priors for High-Fidelity and Versatile Quadruped Locomotion
Jianhui Chen, Ruixin Zhan, Liu Liu et al. (+2 more)
Reinforcement learning combined with imitation learning has significantly advanced biomimetic quadrupedal locomotion. However, scaling these frameworks to massive, multi-source
datasets exposes fundamental bottlenecks. First, traditional GAN-based discriminators are prone to mode collapse, struggling to capture diverse motion distributions from uncurated
datasets. Second, existing kinematic priors suffer from out-of-distribution (OOD) tracking conflicts, leading to severe unintended heading drifts during complex maneuvers. Furthermore, deploying unconstrained priors to physical hardware poses critical safety risks by disregarding actuator dynamics. To overcome these challenges, we propose Diff-CAST (
Diffusion-guided Constraint-Aware Symmetric Tracking), a novel motion prior framework leveraging the multi-modal distribution modeling capabilities of
diffusion models for stylistic rewards. Diff-CAST effectively replaces traditional GAN discriminators, unlocking robust data scaling on heterogeneous collections. To ensure high-fidelity intent execution and reliable real-world deployment, we introduce a comprehensive Sim2Real architecture integrating Symmetric Augmented Command Conditioning (SACC) for drift-free tracking, and Constrained RL for hardware safety. Experiments on a quadruped demonstrate that Diff-CAST mitigates mode collapse, enables seamless transitions between diverse skills, and ensures robust, hardware-compliant locomotion.
Read the full abstract ▾
All authors Jianhui Chen, Ruixin Zhan, Liu Liu, Yang Cai, Ziqiao Li
Robotics
ElasticFlow: One-Step Physics-Consistent Policy with Elastic Time Horizons for Language-Guided Manipulation
Kewei Chen, Yayu Long, Shuai Li et al. (+1 more)
Diffusion policies have demonstrated exceptional performance in
embodied AI. However, their iterative denoising process results in high latency, and existing acceleration methods often sacrifice physical consistency. To address this, we propose ElasticFlow, a
distillation-free, physics-consistent one-step policy framework. We reconstruct the Mean Field Theory by directly modeling the average velocity field, enabling a direct single-step mapping from noise to action. Addressing the Temporal Heterogeneity of robotic tasks, we introduce the Elastic Time Horizons mechanism. This mechanism effectively overcomes Spectral Bias by explicitly encoding control granularity, achieving efficient
alignment between semantic instructions and physical execution horizons. Experiments on
benchmarks such as LIBERO, CALVIN, and RoboTwin demonstrate that ElasticFlow achieves efficient 1-NFE inference (approximately 71Hz). Furthermore, it outperforms state-of-the-art methods, including OpenVLA and $\pi_0$, on long-horizon tasks, highlighting its potential for efficient, robust, and semantically aligned control.
Read the full abstract ▾
All authors Kewei Chen, Yayu Long, Shuai Li, Mingsheng Shang
Robotics
ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulation
Youhe Feng, Hansen Shi, Haoyang Li et al. (+7 more)
Long-horizon robotic manipulation requires dense feedback that reflects how a task advances through its procedural stages, not merely whether the final outcome is successful. Existing reward models often rely on trajectory-level success labels or time-based interpolation, which can conflate elapsed time with true task progress and therefore fail to capture unfinished steps, stagnation, and failure states. We present ProcVLM, a progress-aware
vision-
language model that learns procedure-grounded progress as a dense reward signal for manipulation. Rather than deriving progress from terminal outcomes or temporal proxies, ProcVLM grounds progress estimation in procedural structure and intra-stage visual change, and further adopts a
reasoning-before-estimation paradigm that infers the remaining atomic actions before estimating task progress. Specifically, we construct this supervision by synthesizing frame-level subtask-semantic annotations, assigning progress budgets according to subtask structure, and distributing each budget based on intra-subtask visual change. To train ProcVLM at scale, we build a standardized procedural supervision synthesis pipeline and construct ProcCorpus-60M from 30
embodied datasets with 60M annotated frames, from which we derive ProcVQA for procedure-aware pretraining, with progress estimation as the central task alongside action
segmentation and future planning. Experiments on ProcVQA and reward-model
benchmarks show that ProcVLM improves
embodied procedural
reasoning and yields more discriminative trajectory-internal progress estimates than representative baselines, supporting its use as a dense reward model for downstream reward-guided policy optimization. Project page: https://procvlm.github.io/
Read the full abstract ▾
All authors Youhe Feng, Hansen Shi, Haoyang Li, Xinlei Guo, Yang Wang, Chengyang Zhang, Jinkai Zhang, Xiaohan Zhang, Jie Tang, Jing Zhang
Robotics
Omni-scale Learning-based Sequential Decision Framework for Order Fulfillment of Tote-handling Robotic Systems
Jiaxin Liu, Peng Yang, Yuping Li et al. (+1 more)
Driven by the rapid expansion of e-commerce and small-batch production, the size of the intralogistics load unit of finished goods, semi-finished goods and raw materials is steadily shrinking. Totes are gradually replacing pallets as the primary handling and storage container. This shift has propelled tote-handling robotic systems to the forefront of automation order fulfillment centers. The order-fulfillment decisions of tote-handling robotic systems share a common order-tote-
robot sequential decision-making nature. Existing studies primarily focus on decision mechanisms tailored to particular systems, making it difficult to generalize or transfer them to other contexts. We propose an Omni-scale Learning-based Sequential Decision Framework for Order Fulfillment of Tote-handling Robotic Systems (OLSF-TRS), a generalized and scalable sequential decision framework that combines structured combinatorial optimization with multi-
agent reinforcement learning to coordinate order,tote, and
robot decisions. On small-scale tote-handling robotic systems, OLSF-TRS achieves near-optimal performance with average optimality gaps below 3.5% across two distinct system configurations. In large-scale scenarios, OLSF-TRS consistently outperforms heuristic baselines across two different system types, reducing total tote movements by 8-12% and over 30% compared to SOTA rule-based approaches, while maintaining real-time responsiveness. These improvements translate into tangible operational benefits, including cost reduction, lower energy consumption, and enhanced throughput stability. The proposed framework delivers an efficient and unified order fulfillment decision-making framework for widely deployed tote-handling robotic systems,supporting high-quality order fulfillment in both e-commerce and industrial logistics sectors.
Read the full abstract ▾
All authors Jiaxin Liu, Peng Yang, Yuping Li, Xinyue Xie
Robotics
A Visuo-Tactile Data Collection System with Haptic Feedback for Coarse-to-Fine Imitation Learning
Yeseung Kim, Nayoung Oh, Jun Park et al. (+2 more)
We present a visuo-tactile data-collection system that generates temporally structured, contact-rich demonstrations for imitation learning. Conventional systems often decouple the operator from contact forces, which hinders the demonstration of subtle force modulation. Our system introduces a direct-drive gripper that the operator actuates with the fingers, preserving natural haptic feedback. Integrated visual sensors and custom tactile arrays capture image streams and contact geometry. A handle-mounted push button enables the operator to annotate the task's temporal structure in real time by marking task-critical regions. By fusing in-hand force perception with in-situ temporal annotation, the system produces
multimodal datasets designed for coarse-to-fine learning
algorithms that exploit structural task knowledge, enabling the development of high-quality manipulation policies.
Read the full abstract ▾
All authors Yeseung Kim, Nayoung Oh, Jun Park, Teetat Thamronglak, Daehyung Park
Robotics
Latent Geometry Beyond Search: Amortizing Planning in World Models
Hoang Nguyen, Xiaohao Xu and Xiaonan Huang
Modern
vision-based world models can represent observations as compact yet expressive latent
manifolds, but fast goal-oriented planning in these spaces remains challenging. This raises a central question: when does a learned representation simplify control, rather than merely enabling prediction? We study this question in a pretrained LeWorldModel, whose latent geometry is regularized for smoothness and uniformity. Our key insight is that, under such geometry, planning can be amortized into a latent inverse-dynamics mapping instead of requiring online search. We therefore replace iterative planning with a lightweight Goal-Conditioned Inverse Dynamics Model (GC-IDM) that maps the current latent state, goal latent state, and remaining horizon directly to the next action. Empirically, across four
benchmark environments spanning
navigation, contact-rich manipulation, and continuous control, our controller matches or exceeds CEM in seven of eight environment-protocol settings while reducing per-decision cost by 100-130x. A broader sweep over test-time planners (CEM, MPPI, iCEM, and gradient-based methods) shows that this result is not specific to a particular optimizer. These findings suggest that much of the structure recovered by test-time planning is already locally encoded in the latent representation. More broadly, our results indicate that sufficiently structured latent spaces can shift part of the planning burden from online optimization to learned inference. Our code is publicly available at https://github.com/hdnndh/Latent-Geometry-Beyond-Search-Amortizing-Planning-in-World-Models .
Read the full abstract ▾
Robotics
HULK: Large-scale Hierarchical Coordination under Continual and Uncertain Temporal Tasks
Qingyuan Luo, Jie Li and Meng Guo
Multi-
agent systems can be extremely efficient when working concurrently and collaboratively, e.g., for delivery, surveillance, search and rescue. Coordination of such teams often involves two aspects: selecting appropriate subteams for different tasks in various areas, and coordinating
agents in the subteams to execute the associated subtasks. Existing work often assumes that the tasks are static and known beforehand, where an integer program can be formulated and solved offline. However, in many applications, the team-wise tasks are generated online continually by external requests, and the amount of subtasks within each task is uncertain, e.g., the number of packages to deliver or victims to rescue. The aforementioned offline solution becomes inadequate as it would require constant re-computation for the whole team and global communication to broadcast the results. Thus, this work tackles the large-scale coordination problem under continual and uncertain temporal tasks, specified as temporal logic formulas over collaborative actions. The proposed hierarchical framework, HULK, consists of two interleaved layers: the rolling assignment of currently known tasks to subteams within a certain horizon, and the dynamic coordination within a subteam given the detected subtasks during online execution. Thus, coordination is performed hierarchically at different granularities and triggering conditions, improving computational efficiency and robustness. The method is validated rigorously over large-scale heterogeneous systems under various temporal tasks and environment uncertainties.
Read the full abstract ▾
Robotics
REAP: Reinforcement-Learning End-to-End Autonomous Parking with Gaussian Splatting Simulator for Real2Sim2Real Transfer
Changze Li, Zhe Chen, Shaoyu Chen et al. (+7 more)
In recent years, autonomous parking has made significant advances, yet parking tasks still face challenges in extreme scenarios such as mechanical and dead-end parking slots, often resulting in failures. This is mainly due to traditional parking methods adopting a multistage approach, lacking the ability to optimize the parking problem as a whole. End-to-end methods enable joint optimization across perception and planning modules to eliminate the accumulation of errors, enhancing
algorithm performance in extreme scenarios. Although several end-to-end parking methods use imitation or reinforcement learning, the former is limited by data cost and distribution coverage, while the latter suffers from inefficient exploration. To address these challenges, we propose a Reinforcement learning End-to-end Autonomous Parking method (REAP). REAP employs Soft Actor-Critic (SAC) within an asymmetric reinforcement learning framework to improve training efficiency and inference performance. To accelerate model convergence, we distill the capabilities of a rule-based planner into the end-to-end network through behavior cloning. We further introduce a soft predictive collision penalty mechanism to reduce collision rates by penalizing obstacle-approaching actions. To ensure that the trained reinforcement learning network can directly transfer to real-world scenarios, we have established a Real2Sim2Real simulator. In the Real2Sim step, we use 3D Gaussian Splatting (3DGS) to transform real-world scenes into digital scenes. In the Sim2Real step, we deploy the end-to-end model onto the vehicle to bridge the Sim2Real gap. Trained in the 3DGS simulator and deployed on physical vehicles, REAP successfully parks in various types of parking spaces, especially demonstrating the feasibility of end-to-end RL parking in extremely narrow mechanical slots.
Read the full abstract ▾
All authors Changze Li, Zhe Chen, Shaoyu Chen, Lisen Mu, Yijian Li, Yuelong Yu, Qian Zhang, Qing Su, Ming Yang, Tong Qin
Robotics
Geometry Guided Self-Consistency for Physical AI
Yinwei Dai, Zhuofu Chen, Lijie Yang et al. (+1 more)
State-of-the-art physical AI models generate a chunk of actions per inference through
diffusion or flow matching, iteratively refining an initial noise sample into an action trajectory. Because this inference process is inherently
stochastic, committing to a single trajectory per round is brittle, and this brittleness compounds across the many sequential rounds that comprise a complete episode. We introduce KeyStone, an inference-time self-consistency method for
diffusion-based action generation that draws $K$ candidate action chunks in parallel from a shared model context, clusters them in continuous action space, and returns the medoid of the largest cluster – no additional model required. Two properties make this practical. First, the compact nature of action trajectories makes
diffusion inference
memory-bandwidth bound, leaving spare compute capacity to run $K$ chains in parallel with no additional wall-clock latency. Second, unlike
token or pixel spaces where distance carries no semantic meaning and selection requires a learned judge, action chunks are geometrically structured such that Euclidean distance directly reflects physical similarity, making selection principled and judge-free. Across diverse
vision-
language-action models (VLAs) and world-action models (WAMs), KeyStone improves task success rates by up to 13.3% over single-trajectory sampling with negligible latency overhead, while having on par accuracy with model-based selectors at no training cost. We open source KeyStone at https://github.com/dywsjtu/keystone.
Read the full abstract ▾
All authors Yinwei Dai, Zhuofu Chen, Lijie Yang, Ravi Netravali
Robotics
ATAAT: Adaptive Threat-Aware Adversarial Tuning Framework against Backdoor Attacks on Vision-Language-Action Models
Kewei Chen, Yayu Long, Shuai Li et al. (+1 more)
Addressing the escalating security vulnerabilities in
Vision-
Language-Action (VLA) models, this study investigates
backdoor attacks targeting the visual pathway. We identify a core obstacle causing the failure of traditional attack paradigms: "Gradient Interference." This phenomenon represents an optimization failure triggered by conflicting strategies during end-to-end training. To resolve this, we propose an Adaptive Threat-Aware
Adversarial Tuning (ATAAT) framework. Through its core "Threat-Method Adaptive Mapping" mechanism, ATAAT intelligently selects the optimal gradient decoupling strategy based on the adversary's capabilities. Extensive experiments demonstrate that ATAAT exhibits significant advantages, achieving a highly robust Targeted Attack Success Rate (TASR > 80%) while maintaining extreme stealthiness with merely a 5% poisoning rate. It efficiently handles complex semantic-level triggers and achieves implicit decoupled attacks in data poisoning scenarios for the first time. This work reveals a critical security vulnerability in VLAs and provides theoretical and methodological support for future defense architectures.
Read the full abstract ▾
All authors Kewei Chen, Yayu Long, Shuai Li, Mingsheng Shang
Robotics
BEACON: Cross-Domain Co-Training of Generative Robot Policies via Best-Effort Adaptation
Antong Zhang, Han Qi and Heng Yang
We introduce BEACON–Best-Effort Adaptation for Cross-Domain Co-Training–a theory-guided framework for training generative
robot policies with abundant source demonstrations and limited target demonstrations. BEACON casts cross-domain co-training as a discrepancy-aware importance-reweighting problem, jointly learning a
diffusion-based visuomotor policy and per-sample source weights through an objective inspired by the Best-Effort Adaptation (BEA) bound. To make this approach practical for high-dimensional sequence policies, we develop scalable instance-level discrepancy proxies,
stochastic alternating updates for policy and weights, and a multi-source extension that balances heterogeneous source domains. Across sim-to-sim, sim-to-real, and multi-source manipulation settings, BEACON improves robustness and data efficiency over target-only, fixed-ratio co-training, and feature-
alignment baselines. Without an explicit
alignment objective, BEACON also brings source and target representations closer together as an implicit result of discrepancy-aware co-training.
Read the full abstract ▾