Robotics
Neural Barriers: An Online Certifiable Learning-enhanced Adaptive High Order Safety Critical Control
Lishuo Pan, Mattia Catellani, An Cao et al. (+2 more)
Control barrier functions are an effective model-based tool to formally certify the safety of a system. However, transferring their theoretical guarantees to real-world robotics systems requires high model fidelity. For example, payloads or wind disturbances can cause significant model perturbations to an aerial vehicle, leading to safety compromises. In this work, we propose a certifiable online learning-enhanced robust adaptive control barrier function, which adapts to disturbances using a Neural ODE and quantifies its adaptation uncertainty with conformal prediction. Our approach guarantees safety at all time under unknown time-varying model disturbances. It adopts a conservative strategy when the adaptation uncertainty is high; and efficiently adapts to reduce controller conservativeness as it receives more data. Our approach provides a provable safety guarantee with a probability bound under suitable Lipschitz smoothness assumptions on the underlying model and trajectory. These results demonstrate the potential of our method as a practical safety controller for robotics system operating under model perturbations.
Read the full abstract ▾
All authors Lishuo Pan, Mattia Catellani, An Cao, Lorenzo Sabattini, Nora Ayanian
Robotics · Cross-listed from cs.CV
Deep Prior Learning for Embodied Perception
Yimou Wu, Jiaxin Guo, Yun-hui Liu et al. (+1 more)
Embodied systems need
geometric perception that exploits available observations beyond images alone. Recent feed-forward 3D models incorporate
geometric priors, including camera poses, intrinsics, and depth. However, handling noisy poses, preserving accurate priors, and recovering physical scale require more than simply accepting these inputs. We introduce
Vision-Prior Geometry Grounded
Transformer (VPGGT), a VGGT-based framework that extends OmniVGGT for prior-aware embodied perception. We formulate sensor-motivated pose corruptions from ground-truth trajectories for training and introduce a parameter-free prior residual connection (PRC) to mitigate prior dilution, where predictions are less accurate than their supplied pose priors. Our noise formulation targets camera poses; supplied intrinsics and depth receive no additional corruption. We further introduce Metric Global
Attention, which conditions a global scale
token on available pose and depth scales and predicts a shared metric scaling factor for the
geometric outputs. Experiments across four
datasets show that PRC improves translation-direction accuracy and joint pose AUC over a matched training baseline when camera priors are provided for all views, under both exact and corrupted poses. These results support explicit prior access during refinement as a useful addition to feature-level conditioning.
Read the full abstract ▾
All authors Yimou Wu, Jiaxin Guo, Yun-hui Liu, Zheng Li
Robotics
Robust 2D Traversability Mapping for Construction AMRs via Failure-Mode-Aware Fusion of LiDAR Geometry and Monocular Semantics
Manoj Karnekar, Om Mandhane and Gautham Ramkumar
Autonomous Mobile
Robots (AMRs) on active construction sites face severe navigational challenges: geometry-based traversability mapping (e.g.,
LiDAR) misses visually hazardous but geometrically flat surfaces like wet mud and ponding concrete, while abrupt geometry on drivable speed-breakers and inclines produces phantom obstacles. We propose a real-time, failure-mode-aware
multimodal traversability pipeline on an NVIDIA Jetson AGX Orin, where
LiDAR is the primary
geometric safety estimate and monocular semantics act as a selective, class- and confidence-gated corrective signal. The representation retains distinct traversable classes, namely flat road, terrain, and rocky terrain, while flagging construction hazards. We also release a
multimodal construction-site
dataset from a custom AMR: four closed-loop ROS 2 sequences from two active sites (RGB, depth,
LiDAR, IMU, GPS-RTK, odometry) plus 506 annotated frames across 28 semantic classes. By projecting
LiDAR onto dense semantic masks, resolving sparsity via morphological in-painting, and applying failure-mode-aware fusion with Patchwork++, the system corrects complementary
geometric failure modes for a local AMR costmap.
Read the full abstract ▾
Robotics
Robust Surgical Robotic Instrument Tracking via Sequential Multi-Cue Fusion and Sim-to-Real Self-Training
Hanyang Hu, Zekai Liang, Florian Richter et al. (+1 more)
Efficient and robust tracking of surgical robotic instruments is important for
robot-assisted minimally invasive surgery, yet remains challenging due to the complexity of surgical scenes and the unconventional geometry of surgical instruments. Keypoint-based approaches are efficient, but their performance depends on reliable feature detection. Improving these detectors with real-world supervision is difficult because accurate real-world annotations are costly to obtain at scale. To address this limitation, we introduce a tracker-guided self-training framework that adapts a model pretrained on synthetic images to unlabeled real-world
videos. Given measured
robot joint states, an uncertainty-aware EKF recursively corrects the instrument pose and the observable joint angles by comparing projected model features with detected keypoints, shaft boundaries, and mask-derived cues. An RTS smoother subsequently refines the resulting trajectory, which is projected into pseudo-labels for fine-tuning the feature detector without laborious pose annotations. Experiments on real-world
videos demonstrate consistent improvements from self-training across all evaluated keypoint metrics, and the resulting model outperforms prior approaches in both accuracy and runtime. The code and data will be released upon publication.
Read the full abstract ▾
All authors Hanyang Hu, Zekai Liang, Florian Richter, Michael C. Yip
Robotics
FLEX-WAM: Flexible Block-Causal World-Action Models for Long-Horizon Imagination and Planning
R. Khorrambakht, Joseph Amigo, Félix Lebel et al. (+4 more)
World–action models (WAMs) promise a unified model that predicts action-conditioned futures, generates feasible actions, and supports planning in imagination. However, existing joint
video–action models often use computationally heavy, fixed-horizon backbones ill-suited to streaming inference and stable long-horizon open-loop rollouts. We introduce FLEX-WAM, a Flexible and Efficient Block-Causal World–Action Model for unified simulation and policy inference. FLEX-WAM supports variable-length contexts and non-causal prediction horizons, as well as infinite autoregressive generation frame by frame or block by block. Its block-causal, KV-cacheable architecture combines axial
attention and blockwise
diffusion forcing to enable efficient real-time rollout and deployment-time latency–throughput tradeoffs without retraining. Joint training can nevertheless produce plausible futures that weakly respond to commanded actions. We address this failure mode by balancing state and action flow-matching gradient contributions across the state–action
diffusion-noise grid and regulating world-model sampling using Forward-Dynamics (FD) elasticity, an efficient training-time proxy for action responsiveness. Across simulated and real-world
datasets, FLEX-WAM achieves superior multi-step prediction quality and latency while producing stable joint state–action rollouts for thousands of steps. As a joint action proposer and simulator within MCTS, it solves long-horizon PushT and all five OGBench Puzzle-4x4 tasks entirely in imagination. On a bimanual OpenArm-based
robot, a single checkpoint jointly serves as a play policy and expected-outcome predictor, enabling real-time identification and collection of model–reality mismatches for future self-improvement.
Read the full abstract ▾
All authors R. Khorrambakht, Joseph Amigo, Félix Lebel, Leon Seetoo, Jean Ponce, Zhenzhen Li, Ludovic Righetti
Robotics
Social Navigation for Tour-guide Robot
Vikram Shree and Jose Nino
We propose a force-based model for social navigation of a tour-guide
robot. Social forces due to various factors like obstacles, user position and heading, have been accounted for in the model. We claim that each one of these forces makes the
robot more sociable to the user and we design an experimental setup for evaluation. In the experiment, the user follows an autonomous
robot to a destination in a known map, while undertaking a few simple sub-tasks in the middle, which serve as distractions. For each participant, we run several rounds of the experiment, each with different forces and a shortest path, A*-search baseline model. Using per-round subjective indicators, we propose to study the effect of our force model on constructs such as: follow-ability, perceived safety, and perceived intelligence.
Read the full abstract ▾
Robotics
EvoMem-VLA: State-Evolution Memory for Long-Horizon Robot Manipulation
Yuheng Na, Zhide Zhong, Junjie He et al. (+7 more)
Most
vision-
language-action (VLA) models rely on current observations and lose task-relevant evidence once it leaves view, limiting performance on long-horizon,
memory-dependent tasks. Existing efforts incorporate compressed historical features or sparse visual keyframes. However, isolated snapshots can leave the policy uncertain about what changed during past interactions and which action should follow. To overcome this limitation, we propose EvoMem-VLA, which constructs state-
evolution memory by explicitly encoding and retaining observed changes between historical states. These change representations preserve evidence of interaction outcomes, allowing the policy to track task progress beyond isolated snapshots. Specifically, we introduce conditional delta tokenization to encode ordered frame pairs into directional, source-conditioned delta
tokens, each associated with its corresponding state evidence. A shared VLM backbone supports task-adaptive routing: normal long-horizon tasks follow a direct action route, whereas multi-stage tasks use a subtask route that generates an executable subtask as an additional input for action generation. With a single jointly trained policy for each simulation
benchmark, EvoMem-VLA achieves success rates of 80.7% on RMBench, 82.0% on RoboMME and 83.8% across four real-world tasks spanning two
robot embodiments. These results represent substantial improvements over the previous state of the art in all three evaluation settings.
Read the full abstract ▾
All authors Yuheng Na, Zhide Zhong, Junjie He, Junfeng Li, Haodong Yan, Jiaan Wang, Jiaguan Zhu, Yangyang Zheng, Tianyu Huang, Haoang Li
Robotics
RMMBench: A Comprehensive Benchmark for Robotic Mobile Manipulation
Huapeng Li, Fuxiang Feng, Jinqiu Fan et al. (+5 more)
Although the advancement of
vision-
language models (VLMs) has endowed
robots with enhanced environmental understanding and task
reasoning, a comprehensive evaluation methodology is important to advance the integration of VLMs in robotic navigation and manipulation. However, current
benchmarks lack a comprehensive method to evaluate diverse robotic tasks, and evaluation metrics remain relatively constrained, making it difficult to assess the embodied capabilities of VLMs in a thorough and fine-grained manner. To address this issue, we propose RMMBench, an evaluation
benchmark that requires
robots to understand
language instructions and perform long-horizon tasks in continuous spaces. RMMBench seamlessly integrates high- and low-level embodied tasks into a unified framework, constructing a "navigation-manipulation" task suite comprising 70 canonical task scenarios that range from localized manipulation to long-horizon composite navigation. The results reveal that leading VLMs still face major challenges in spatial localization when performing mobile manipulation tasks, and also highlight the necessity of enhancing the spatial perception capability of
robots during long-horizon interactions. RMMBench can be accessed at https://mxxq-stack.github.io/rmmbench-project/
Read the full abstract ▾
All authors Huapeng Li, Fuxiang Feng, Jinqiu Fan, Shuo Yang, Fengjiao Chen, Xuezhi Cao, Ran Song, Wei Zhang
Robotics
TUCO: Curating Simulation Demonstrations for Sim-to-Real Robot Policy Co-Training
Ning Zhu, Mengfei Zhao, Yikai Tang et al. (+5 more)
Simulation demonstrations can supplement scarce real-world data for
robot policy co-training. However, the value of using data curation to actively select these demonstrations for sim-to-real co-training remains underexplored. Existing curation methods also lack a unified criterion for measuring trajectory-level utility and set-level coverage from closed-loop target behavior. To address these gaps, we present the first systematic study of data curation for sim-to-real
robot policy co-training and propose Trajectory-level Utility and set-level Coverage Optimization (TUCO). TUCO uses influence functions to trace how each source demonstration affects target-domain scoring rollouts. Our key insight is that these effects can be decomposed into an overall contribution to target return and variation across rollouts, providing a common closed-loop basis for measuring trajectory utility and set coverage. We further propose a performance-aligned subset optimizer that combines these measures in a unified curation objective to reduce redundancy and select complementary demonstrations. Extensive experiments on RoboMimic and OmniReset establish the value of active simulation data curation for sim-to-real policy co-training and show that TUCO achieves state-of-the-art performance across single-simulator, sim-to-sim, and sim-to-real settings.
Read the full abstract ▾
All authors Ning Zhu, Mengfei Zhao, Yikai Tang, Zhangyujie Sun, Peihao Li, Dongyue Ni, Jindou Jia, Jianfei Yang
Robotics
Optimal Control with Learned Critics under Unmodeled State Dependencies
Philipp Schoch and Markus Ryll
Model Predictive Control (MPC) provides a structured and constraint-aware mechanism for decision-making, but its reliance on optimization-friendly analytical dynamics models limits its use in tasks with contacts and other hard-to-model state dependencies. Model-free reinforcement learning avoids explicit modeling assumptions but typically requires large amounts of interaction data. We present a learning-based MPC framework that combines the data efficiency and structure of local model-based planning with learned components that compensate for incomplete dynamics and finite-horizon myopia. The method augments a nominal analytical model with a residual dynamics network that learns missing state-dependent effects from data and combines the resulting planner with a learned action-value critic that injects long-horizon MDP structure into the local iLQR optimization. To make this practical at reinforcement-learning scale, we develop a GPU-accelerated batched iLQR solver that evaluates learned dynamics and critic networks inside the optimal-control loop and solves thousands of trajectory-optimization problems in parallel. The complete system is integrated into a robotics simulator, enabling scalable model-based reinforcement learning under incomplete dynamics. Experiments on biased and incompletely modeled control tasks show that the approach improves closed-loop control performance while preserving the model-based structure needed for efficient constrained trajectory optimization.
Read the full abstract ▾
Robotics
VAMPS: Visual and Motor Policies from Sampling-Based Planning
Mohamed Yassine Kabouri, Pietro Noah Crestaz, Quang-Nam Nguyen et al. (+3 more)
Learning
robot policies directly on physical systems remains difficult because data collection is costly and policy exploration can be unsafe. We introduce Visual and Motor Policies from Sampling-Based Planning (VAMPS), a framework that uses Model Predictive Path Integral (MPPI) control to train reusable policies without human demonstrations. VAMPS supports two training modes. For one-step proprioceptive policies, it operates iteratively in simulation: the policy warm-starts MPPI, and the refined trajectories provide new supervision as the policy changes. A learned terminal value improves short-horizon planning, while an Implicit Q-Learning (IQL) critic guides the policy update. Iterative refinement outperforms training once on frozen MPPI data, and we transfer the learned locomotion policy to a Unitree Go2. For visuomotor policies, VAMPS operates directly from real-
robot data. MPPI uses task-specific state estimates to plan and execute trajectories while recording RGB and sensor observations on a Flexiv Rizon 10S. Action Chunking with
Transformers predicts action chunks, reducing the effective prediction horizon, and is trained offline on this fixed
dataset. We demonstrate visuomotor pick-and-place and force-aware whiteboard erasing. In the latter task, the policy additionally observes the measured $6$-D wrench and desired normal force. These results show that VAMPS can learn policies either in simulation followed by hardware transfer or directly from autonomously collected real-
robot data.
Read the full abstract ▾
All authors Mohamed Yassine Kabouri, Pietro Noah Crestaz, Quang-Nam Nguyen, Qilong Cheng, Ludovic Righetti, Nicolas Mansard
Robotics
CoDance: Learning Reactive and Compliant Human-Humanoid Interaction from Video
Zhuoqun Chen, Shucheng Jia and Boyuan Chen
Partnered human-humanoid interaction couples locomotion with continuous physical contact. A humanoid needs to coordinate with a person's motion while responding to interaction forces and maintaining stable and natural movement. We present CoDance, a framework for learning reactive and compliant human-humanoid interaction from
video. We study partnered dancing as a challenging instantiation, where a humanoid coordinates its footsteps with a moving partner and maintains continuous two-hand contact. Given a single
video of two human dancers, CoDance retargets their motions into a
robot reference and a moving partner. We introduce a multi-link compliance augmentation that transforms the kinematic demonstration into force-aware training data by adapting the
robot reference under structured forces at both hands. Policies trained on this data follow the observed partner while preserving the demonstrated locomotion style and responding compliantly to physical interaction. In simulation, the policies adapt their footsteps to changes in the partner and reproduce approximately 80% of the wrist displacement encoded by the augmented demonstrations. On a physical humanoid, CoDance enables sustained two-hand dancing with a human partner including repeated transitions between forward and backward motions.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
When and What to Prune? Stage-Aware Visual Token Pruning for Efficient VLA
Tianjun Shi, Haotian Xiong, Ziyu Gong et al. (+2 more)
Visual
token pruning is an effective way to accelerate
vision-
language models and is especially useful for
vision-
language-action (VLA) inference, where many visual
tokens must be processed before predicting
robot actions. Existing
pruning methods usually estimate which
tokens can be pruned based on
attention scores or feature diversity, retaining
tokens that are either highly attended or visually different from others. However, most of them use fixed
pruning schedules, such as
pruning once at a preset layer or
pruning at uniformly spaced layers. Such schedules can be risky for VLA models, because the model may not know which visual regions matter for the action in early layers.
Tokens that look unimportant at first may become useful after the model combines visual observations with the
language instruction. In this work, we propose SAPrune, a training-free visual
token pruning framework for efficient VLA inference. Instead of
pruning at fixed layers, SAPrune uses a small calibration set to observe how action-to-visual
attention changes across layers, and chooses
pruning layers only after the
attention pattern becomes more reliable. At each selected layer, SAPrune applies a dual-path
pruning rule: one path protects strongly attended visual
tokens from
pruning, while the other prevents useful surrounding context from being discarded. Experiments on LIBERO, SIMPLER, and real-world robotic tasks show that SAPrune prunes 87.5% of visual
tokens and achieves up to 1.718x inference speedup while maintaining competitive task success rates.
Read the full abstract ▾
All authors Tianjun Shi, Haotian Xiong, Ziyu Gong, Qi Lu, Li Li
Robotics
Safe Ergodic Control for Multi-Robot Systems via Quadratic Programming
Yo Toyomoto, Mohamed Elobaid, Bryce L. Ferguson et al. (+2 more)
Ergodic control drives
robots to spend time in each region in proportion to a spatial distribution of interest, making it well suited for dense spatiotemporal environmental monitoring. Existing safe ergodic controllers rely on offline trajectory optimization or hierarchical architectures, which limit real-time applicability and decouple the ergodicity objective from the safety constraint. This paper presents a quadratic programming (QP)-based controller that treats ergodicity and safety jointly. We first introduce a Gaussian-kernel ergodic metric that, unlike the classical indicator-based metric, is time differentiable. This allows the exponential decay of the metric to be imposed as a time-varying control barrier function (CBF) constraint, relaxed by a slack variable, alongside hard CBF constraints for region containment and inter-
robot collision avoidance. We establish that the resulting QP remains feasible from any safe initial configuration and that, up to the slack term, the ergodic metric decays exponentially. Simulations show improved performance over baseline methods, and experiments with three aerial vehicles validate the work on real hardware.
Read the full abstract ▾
All authors Yo Toyomoto, Mohamed Elobaid, Bryce L. Ferguson, Alberto Quattrini Li, Shinkyu Park
Robotics
Vela: Scaling Vision-Language-Action Models with Adaptive Action Curve Parametrization
Yifan Li, Jiaxu Wang, Dongming Wu et al. (+4 more)
Most
vision-
language-action models represent future motion as fixed-rate action chunks, tying temporal resolution and prediction horizon to a fixed output budget. This pointwise representation wastes capacity on highly correlated neighboring actions, leaves temporal continuity and smoothness to be learned implicitly, and forces a tradeoff between long-horizon coverage and the local precision required for contact-rich manipulation. To address these limitations, we introduce Vela, a
vision-
language-action foundation model that represents future
robot behavior as continuous trajectories. Vela combines a compact spline-based action representation with motion-dependent temporal support and a shared action interface for heterogeneous embodiments, allowing a fixed output budget to adapt its temporal resolution across motions. We pretrain Vela on large-scale multi-embodiment
robot data and evaluate it on LIBERO-X, EBench, and two real-world long-horizon tasks, egg-cake cooking and potato shredding, obtaining promising results across simulation and physical manipulation. These results highlight the potential of continuous action representations as a foundation for future embodied foundation models. Project page and more results: https://clementine24.github.io/Vela/ .
Read the full abstract ▾
All authors Yifan Li, Jiaxu Wang, Dongming Wu, Yicheng Jiang, Ryan Ji, Xiangyu Yue, Yanwei Fu
Robotics
VICON: Visual-Inertial-Contact based Hand-Object Tracking for Manipulation Datasets
Youbeen Jeon, Uiseong Shin, Hwanchul La et al. (+3 more)
Learning dexterous manipulation benefits from human demonstration
datasets that capture diverse and natural hand-object interactions. In particular, contact points and forces provide supervision on where and how strongly to interact, which cannot be fully captured by motion trajectories alone. However, methods for jointly capturing hand and object motion, contact points, and forces remain limited. Moreover, severe occlusion from hand-object interaction challenges accurate tracking of both hands and objects. To address these limitations, we present a Visual-Inertial-CONtact based hand-object tracking (VICON) framework. It holistically captures both hand and object motion along with contact information during manipulation, even under severe occlusion. First, we adopt a visual-inertial glove and an RGB-D camera for accurate hand tracking, and redesign the glove to incorporate contact sensing. Specifically, force-sensitive resistors (FSRs) are placed on the glove based on human grasp frequency to synchronously record contact states and calibrated normal forces. Second, without requiring pre-existing CAD models, we estimate object poses using RGB-D images and a mesh reconstructed from a monocular
video. We propose factor-graph-based object trajectory estimation that fuses object-pose estimates weighted by visibility under hand-object occlusion, FSR measurements, and a hand-motion prior. Across 40 motion-capture sessions with five objects, VICON achieves a 2.5% failed-frame rate compared with 50.9-64.6% for the baselines, with median errors of 3.9 mm and 3.0 degrees under occlusion. Using VICON, we construct a
dataset containing synchronized hand-object motion, contact points, and normal forces, and will publicly release an expanded
dataset covering 10 object categories at https://github.com/VICON-
dataset/
dataset.
Read the full abstract ▾
All authors Youbeen Jeon, Uiseong Shin, Hwanchul La, Jaeseong Kang, Hyelim Choi, Yongseok Lee
Robotics · Cross-listed from cs.AI
A Safe Action Is Not Enough: Feasible-Future Decoding for Vision-Language-Action Policies
Tu Nguyen, Matthieu Zimmer, Vu Anh Vu et al. (+4 more)
A safe action is not necessarily a viable one. A frozen
vision-
language-action (VLA) policy can favor a locally admissible move that leaves no policy-supported route to safe task completion. We call this the feasibility-likelihood gap: likelihood ranks the next move, while feasibility depends on the futures it leaves open. To bring those futures into the decision, we derive the exact next-block marginal of the history-conditioned policy-environment trajectory law restricted to safe task completion. The derivation reveals a candidate-dependent feasible-future mass: its support records whether safe completion remains possible under the frozen continuation process, while its magnitude measures how much weighted safe-completion mass remains. Since exact evaluation is impractical online, we develop a selective finite-candidate approximation and establish conditions for recovering the best retained viable candidate. Our alarm-triggered, training-free reranker VICS-G lowers mean cumulative safety cost by 1.9%-57.5% across six Safety-CHORES settings while remaining within 2.5 percentage points of policy sampling in success and 0.82 steps in mean episode length. Our approach offers a promising and practical path toward safer task completion, grounded in an exact policy-relative target yet requiring neither policy retraining nor online rollouts.
Read the full abstract ▾
All authors Tu Nguyen, Matthieu Zimmer, Vu Anh Vu, Ziyi Wang, Jannik Hammel Nielsen, Xuebing Zhou, Haitham Bou Ammar
Robotics
Recursive Self-Improvement of Visuomotor Policies through Local Recovery Supervision
Yuzhi Zhang, Xinyu Liu and Yu Zhang
Visuomotor policies can execute familiar tasks yet lack the corrective behavior needed after their own mistakes. We present a framework for recursive self-improvement through local recovery supervision. Each round audits the current policy, generates corrective demonstrations at supported failure states, and uses them to update the policy that drives the next round of collection. An offline auditor locates unresolved failures using coarse and dense temporal evidence and specifies observable repair goals. A fixed
multimodal agent acts as a tool-using teacher, generating recovery actions through observation, computation, execution, and feedback. The frozen student tests whether each teacher endpoint supports further progress. If continuation fails, the system restores that endpoint and extends the demonstration. Action-level quality assessment then defines continuous training windows with aligned observations, quality weights, and validity masks. Only the student is deployed. In a preliminary LIBERO-Goal study, recovery-augmented post-training achieves 88 successful episodes out of 100 validation scenes, compared with 78 for original-data continuation from the same $\pi_0$ checkpoint. An earlier BC-RNN study on robomimic Can improves success from 102/130 to 112/130 using 26 local recovery segments. Both comparisons match 2,000 additional optimization steps.
Read the full abstract ▾
Robotics
PB-STDG: A Prediction-Based Short-Term Decentralized Greedy Guidance Algorithm for a Drone Road System
Zhouyu Qu, Andreas Willig and Xiaobing Wu
In recent years, Unmanned Aerial Vehicles (
UAVs) or drones have been increasingly adopted in urban environments for applications such as parcel delivery, infrastructure inspection, emergency response, and drone light shows. A non-negligible issue is how to manage the increasing number of drones operated by different entities, to enable them to cooperatively avoid potential collisions and determine conflict-free short-term flight paths in urban airspace. This paper presents a Prediction-Based Short-Term Decentralized Greedy (PB-STDG) guidance
algorithm for a structured Drone Road System (DRS). PB-STDG extends the original STDG
algorithm by introducing a prediction mechanism that enables drones to anticipate the decisions of neighboring drones using additional information shared in beacon packets, aiming to address the over-conservative behavior observed in the STDG
algorithm. Several simulation scenarios are conducted to evaluate and compare the proposed
algorithm with STDG. The results show that PB-STDG improves traffic efficiency while maintaining a safety level comparable to that of STDG.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
Direction-Conditioned Policies for Online Goal-Conditioned Reinforcement Learning
S K Swaminathan, Damiya Gondha, Theyanesh Eswaramoorthy Rajahkrishnan et al. (+1 more)
Contrastive Reinforcement Learning (CRL) learns representations that estimate goal reachability, yet its policy remains conditioned on raw goals and therefore does not directly exploit the geometry encoded by its critic. We introduce Direction-Conditioned Policies (DCP), a method built around a small modification to CRL: DCP selects previously visited states as waypoints during online training and conditions the policy on their direction and distance in representation space. At deployment, DCP applies the same interface directly to the final goal, requiring neither waypoint selection nor planning. Across nine navigation and manipulation tasks, DCP attains higher final success rates than CRL on seven tasks and spends more time near the goal on seven. Controlled maze experiments further show that DCP captures shortest-path geometry more accurately and that the supplied direction causally influences the actor's behavior. We identify waypoint coverage and ranking as limits to exploration, and show that learned candidate generation improves goal reaching in two controlled mazes.
Read the full abstract ▾
All authors S K Swaminathan, Damiya Gondha, Theyanesh Eswaramoorthy Rajahkrishnan, Aritra Hazra
Robotics
Now You Feel It, Now You See Me: Digital-Twin-based Teleoperation Interface for Dexterous Manipulation
Youngchan Shim, Kyutae Lee, JooYun Kim et al. (+3 more)
Teleoperation is becoming increasingly important for collecting high-quality demonstrations to teach
robots dexterous manipulation skills. For dexterous manipulation, bare-hand tracking provides a practical way to control robotic hands and demonstrate coordinated finger movements without gloves or exoskeletons. However, this type of teleoperation faces two key feedback limitations: a lack of force feedback and visual feedback of occluded region. The absence of force feedback hinders precise and safe manipulation, as operators must infer contact force visually rather than feel them directly. Occlusion by objects or other
robot parts impairs the assessment of hand positions, approach distances, and pre-grasp configurations suited to the object's shape. To address these limitations, we propose a digital-twin-based augmented reality (AR) teleoperation interface that integrates bare-hand tracking, bimanual robotic hands, and virtual representations of the
robot and task objects. The interface renders tactile measurements on
robot hand meshes as visuo-force feedback. Furthermore, it offers a user-controlled virtual view panel or occlusion-aware transparency rendering to provide occlusion-mitigating visual feedback. We evaluated the interface through user studies involving Task 1 and Task 2, examining how force visualization and visual assistance support demonstration collection in tasks requiring careful force regulation and manipulation under occlusion.
Read the full abstract ▾
All authors Youngchan Shim, Kyutae Lee, JooYun Kim, Jaeseong Hwang, Harim Ji, Yongseok Lee
Robotics · Cross-listed from cs.AR
Beyond LLM Serving: Characterizing Vision-Language-Action Workloads for Embodied AI System Design
Seonghun Jung, Sieun Moon, Jiyoung Jeong et al. (+2 more)
Vision-
language-action (VLA) models translate
multimodal observations into low-level
robot actions. During
robot operation, each control period sets an inference deadline, and overruns leave the
robot acting on stale observations, reducing task success. Meeting this deadline motivates on-device or nearby edge execution, where a single
robot requires batch-1 inference outside the design point of LLM serving systems. Although VLA architectures combine familiar
vision-
language, autoregressive, and
diffusion-style components, their runtime behavior in this batch-1 control setting remains uncharacterized. We characterize four representative VLA models on an edge GPU server and two onboard SoCs, using single-inference profiling and 43,200 closed-loop episodes. Action tensor dimensionality determines whether a stage is
memory- or compute-bound, platform balance can shift that bottleneck, and GPU frequency scaling yields a platform-dependent energy-latency sweet spot. In closed-loop operation, overlapping inference with action execution creates an accuracy-speed-energy tradeoff, and no configuration is Pareto-dominant across deployment SLOs. These results guide joint design of VLA model architectures, hardware, and runtime policies.
Read the full abstract ▾
All authors Seonghun Jung, Sieun Moon, Jiyoung Jeong, Jimin Lee, Jaehyuk Huh
Robotics · Cross-listed from cs.CV
GeoBridge-VLA: Geometry-Aware Residual Adaptation for Vision-Language-Action Models
Hyun Song, Kangmin Kim, Loren Jinsoo Um et al. (+4 more)
Vision-
language-action (VLA) models encode semantic information from
vision-
language pretraining, but manipulation also requires precise spatial
reasoning. We present GeoBridge-VLA, a two-stage method for learning
geometric features from a pretrained VLA's frozen visual encoder and using them for action prediction. Stage I trains a feature bridge and geometry decoder with depth supervision. Stage II freezes these modules and trains a gated residual interface together with the action-side projections and action expert. The residual augments the existing visual
tokens without adding a second image encoder or increasing the
token count. Deployment requires RGB,
robot state, and
language, but no depth observations. Under matched evaluation conditions, GeoBridge-VLA achieves 70.9% success on LIBERO, compared with 60.0% for SmolVLA. Disabling the residual in the same trained checkpoint reduces success from 70.90% to 69.85%, with mixed effects across suites. On a physical ROBOTIS OMY
robot, GeoBridge-VLA succeeds in 148 of 200 trials (74.0%) across four tasks, compared with 108 of 200 (54.0%) for SmolVLA.
Read the full abstract ▾
All authors Hyun Song, Kangmin Kim, Loren Jinsoo Um, Minhui Han, Jaehyeok Park, Taewan Cho, Andrew Jaeyong Choi
Robotics · Cross-listed from cs.CV
LightVLN: Efficient Aerial Vision-and-Language Navigation with Compact Memory and History-Guided Local Aggregation
Yiming Zhao, Tianshun Li, Jingle He et al. (+2 more)
Aerial
vision-and-
language navigation (VLN) enables unmanned aerial vehicles to execute long-horizon natural-
language instructions from visual observations in complex three-dimensional environments. However, recent aerial VLN models often rely on large-scale
vision-
language backbones and dense visual histories, imposing substantial computation and
memory costs that hinder onboard deployment. We propose LightVLN, a lightweight history-aware aerial VLN framework that combines a compact 0.5B
language backbone with compact representations of both historical and current observations. LightVLN compresses each historical frame into a single
token using visual features already computed by the policy. It further introduces history- and instruction-conditioned local aggregation to reduce the current observation from 256 to 32 visual
tokens while preserving navigation-relevant spatial information. With up to 16 historical frames, the policy uses at most 48 observation-derived
tokens. On the public OpenFly
dataset, LightVLN achieves 50.93% Test-Seen and 36.14% Test-Unseen success rates (SR), outperforming the evaluated 7B
language-backbone baselines on most reported metrics. It also achieves 25.83% SR on AerialVLN-S Val-Seen. In a reconstructed unseen campus, we deploy LightVLN on a DJI M350 RTK with an external Jetson Orin NX 16 GB for closed-loop onboard-compute real-to-sim hardware-in-the-loop (HIL) evaluation, achieving 14.61 Hz model inference and 11.13 Hz end-to-end decision updates. These results demonstrate the effectiveness and efficiency of LightVLN for aerial navigation.
Read the full abstract ▾
All authors Yiming Zhao, Tianshun Li, Jingle He, Ruonan Chai, Xinhu Zheng
Robotics
FreeLoc: Online Floorplan Localization via Diffusion-Aided Pose Refinement
Haocheng Peng, Boyang Zhou, Jiarui Hu et al. (+7 more)
Floorplans provide compact and widely available
geometric maps for indoor localization, but existing high-performing floorplan-based methods still convert them into dense scene-specific offline databases, tying accuracy, storage, and runtime to the sampling resolution of the discretized pose space. We present FreeLoc, an online RGB-based floorplan localization framework that treats the floorplan as a directly queryable
geometric map. FreeLoc introduces an efficient online
geometric querying and
diffusion-aided refinement scheme, which retrieves plausible pose anchors through on-the-fly floorplan ray querying and refines them into accurate continuous pose estimates. For sequential localization, FreeLoc develops an online likelihood construction strategy that bridges single-frame localization and probabilistic temporal fusion by constructing likelihoods from coarse-sampled candidates and refined pose hypotheses, enabling histogram-filter-based temporal fusion without offline databases. Experiments demonstrate real-time online inference and state-of-the-art performance in both single-frame and sequential localization, while real-world results validate practical deployability in indoor robotic localization scenarios.
Read the full abstract ▾
All authors Haocheng Peng, Boyang Zhou, Jiarui Hu, Xiyue Guo, Ziyang Zhang, Boming Zhao, Yifan Gao, Xiao Li, Hujun Bao, Zhaopeng Cui
Robotics
A Unified Dynamics Framework for Reinforcement Learning and Classical Control of a Six-DOF Pipeline-Tracking ROV in NVIDIA Isaac Sim
Cheng Siong Chin, M. Venkateshkumar and Jianhua Zhang
Reinforcement learning controllers for underwater vehicles are usually trained against one physics representation and deployed against another, so reported performance does not always describe behavior outside training. This paper presents a pipeline-tracking architecture for a six-degree-of-freedom remotely operated vehicle (ROV) in which one Universal Scene Description (USD) scene supplies the real BlueROV2-Heavy mass, added-mass, damping, buoyancy, and thruster parameters to both halves of the system: a vectorized NumPy implementation of Fossen's marine-craft equations, and an interactive NVIDIA Isaac Sim deployment applying the identical equations as PhysX forces at every step. The Coriolis-centripetal term is the primary dynamics model in both branches; a controlled ablation on PPO and TRPO shows that including it does not destabilize either
algorithm and modestly improves tracking, about 19 percent lower standoff RMS error for TRPO. Five reinforcement learning
algorithms, PPO, soft actor-critic, TD3, DDPG, and TRPO, are trained against one environment, reward, and randomized evaluation harness through a checkpoint-compatibility layer scoring any policy with the same code. The pipeline is extended with six classical baselines, PID, sliding-mode, fuzzy logic, feedback linearization, model predictive control, and an adaptive neuro-fuzzy inference system, driven by the same guidance geometry and thruster allocation as the learned policies. Under Coriolis-enabled dynamics, PPO, TRPO, and feedback linearization reach the strongest combination of 100 percent success and competitive accuracy; PID, fuzzy control, and the neuro-fuzzy baseline also reach 100 percent success with looser tracking; DDPG and TD3 each show a specific, explainable failure mode rather than a general weakness of off-policy learning; and classical control remains a strong baseline against the best learned policies.
Read the full abstract ▾
Robotics · Cross-listed from cs.MA
Orchestrating Level-$K$ Policies Against Unknown Opponents in Partially-Observable Dynamic Games
Addison Kalanther, Sanika Bharvirkar, Daniel Bostwick et al. (+2 more)
Level-$K$
reasoning generates a hierarchy of policies specialized to opponents with different
reasoning levels. When an opponent's level is unknown, a common deployment rule estimates that level and selects the corresponding response. In a dynamic, partially-observed game, this selection is repeated, with each choice shaping subsequent states and observations. The response associated with the most likely opponent level need not maximize expected return from the current history. We formulate this deployment problem as dynamic orchestration of a fixed, pretrained policy library in a partially observable Markov game. We compare classification-based orchestrators (CBOs) trained using offline data or on-policy data aggregation with a reinforcement-learning-based orchestrator (RLBO) trained to maximize expected discounted return. In pursuit-evasion experiments, on-policy training improves classification and return, yet RLBO achieves higher return than the on-policy and offline CBOs. Given a pretrained library, RLBO also reaches performance comparable to a policy trained directly over the pursuers' action space with fewer training timesteps. These findings support treating a hierarchy of level-$K$ policies as a resource for orchestration, not a prescription for deployment.
Read the full abstract ▾
All authors Addison Kalanther, Sanika Bharvirkar, Daniel Bostwick, Chinmay Maheshwari, Shankar Sastry
Robotics
DiVeR: Decision-Critical Verifier Learning for VLA Test-Time Scaling
Seongheon Park, Heecheol Kim, Shulin Tian et al. (+5 more)
Scaling
robot data and model capacity has improved
Vision-
Language-Action (VLA) policies, but further progress is constrained by the high cost of robotic data. Verifier-guided test-time scaling offers an efficient alternative by sampling multiple action candidates and selecting the one most likely to lead to task success at inference time. Existing classification-based verifiers learn from trajectory-level outcomes but treat all visited states equally, even though their value for candidate discrimination can vary across a trajectory. At many states, plausible actions are similar and provide limited discrimination signal, while only a sparse set of decision-critical states admits meaningfully different actions that can substantially affect downstream outcomes. To address this, we propose DiVeR, which estimates decision criticality from the dispersion of sampled action representations. DiVeR then uses this signal to reweight verifier learning toward states where action selection is most consequential, without requiring step-level annotations or additional environment interaction. Across LIBERO, RoboCasa, and real-world experiments on a Franka Research 3
robot, DiVeR consistently improves task success through more effective verifier-guided action selection, while adding negligible verifier inference overhead.
Read the full abstract ▾
All authors Seongheon Park, Heecheol Kim, Shulin Tian, Lilika Makabe, Namiko Saito, Katsushi Ikeuchi, Sharon Li, Yasuyuki Matsushita
Robotics
RobotUse: Allocating Computation, Context, and Decisions
Junhoo Lee, Injun Baek, Seungyeon Kim et al. (+4 more)
Robot agents must connect their intended actions to observed outcomes while retaining the context needed to revise their choices over repeated attempts. Existing interfaces often leave these choices inside predefined tools or require
agents to manage detailed execution code and its growing history. We introduce RobotUse, a
robot agent harness that organizes computation, context, and decisions around specifying and revising physical actions.
Agents visually select targets and poses, while the backend handles geometry, motion planning, and control. Subagents retain detailed interactions within each subgoal and return the information needed for subsequent decisions. Continual harnessing lets
agents learn from execution by updating a persistent playbook. On RoboLab, RobotUse achieves 45% task success, outperforming CaP-X by 6.7 percentage points while maintaining compact decision contexts and reducing reliance on predefined action abstractions. Furthermore, we show that RobotUse learns from real-world execution despite imperfect feedback and transfers what it learns to subsequent tasks. Project page is available at https://robotuse-team.github.io/.
Read the full abstract ▾
All authors Junhoo Lee, Injun Baek, Seungyeon Kim, Suhyun Jeon, Minkyu Kim, Baekseung Kim, Nojun Kwak
Robotics
Nudge Before You Push: Physics-Aware Navigation via Tactile Probing
Xianyao Li, Fang Xu, Ruitong Tian et al. (+4 more)
Visually identical containers can conceal loads that require different handling decisions. We present TANav, which uses a brief nudge to measure push resistance for navigation under a site-defined handling boundary. TacPhys reads the force sequence, with optional RGB-D and kinematics, into a mass estimate for push authorization. A repeated-patrol planner weighs probe and route costs, requests a second contact when useful, and reuses observations across visits. In simulation, TacPhys approaches a resistance-only Bayes reference and reduces missed pushes from 28.7% to 5.5% relative to peak-force thresholding at comparable low-risk operating points. In repeated-patrol simulation, TANav recovers 90% of the oracle's path saving, more than halves human interventions relative to always-detour, and reduces boundary violations from 4.3% to 2.9% relative to RGB-D-only probing. On a quadruped manipulator with a Hall-array fingertip, offline zero-shot MAE is 0.28-1.07 kg on containers up to 2.82 kg. Force-rise calibration at 3 kg gives 93.5% pooled offline accuracy (86.4% on non-cube episodes); a separate raw-peak rule gives 15 of 20 correct online decisions on unseen boxes.
Read the full abstract ▾
All authors Xianyao Li, Fang Xu, Ruitong Tian, Bowen Sun, Xiao Hu, Yang Ye, Jing Du