Robotics
Robustness-Based Synthesis for Time Window Temporal Logic Specifications via Mixed-Integer Linear Programming
Philip Smith, Ahmad Ahmad and Kevin Leahy
Time Window Temporal Logic (TWTL) is a rich specification
language for cyber-physical systems that can compactly express sequential tasks with explicit timing constraints. In this paper, we consider the problem of synthesizing control inputs for discrete-time linear systems subject to TWTL task specifications. Building on the quantitative semantics (robustness) recently introduced for TWTL in [1], we encode the robust satisfaction of a TWTL formula as a set of Mixed-Integer Linear constraints and pose synthesis as a Mixed Integer Linear Program (MILP) that maximizes the robustness degree. We prove that any feasible solution with positive objective value guarantees Boolean satisfaction of the specification. We address two synthesis settings: an open-loop formulation that optimizes the full control sequence from the initial state, and a closed-loop receding-horizon Model Predictive Controller (MPC) formulation that re-solves the MILP at each step using the current measured state. A key feature of our MPC formulation is a task-adaptive horizon that exploits the TWTL Deterministic Finite Automaton (DFA) to determine the active sub-task at each step, limiting the prediction horizon to the remaining window of the current task rather than the full formula horizon, this makes each re-solve significantly cheaper than the initial open-loop solve.
Read the full abstract ▾
Robotics
TAPE: Tether-Aware Path Planning for Autonomous Exploration of Unknown 3D Cavities Using a Tangle-Compatible Tethered Aerial Robot
Louis Petit and Alexis Lussier Desbiens
This letter presents the first method for autonomous exploration of unknown cavities in three dimensions (3D) that focuses on minimizing the distance traveled and the length of tether unwound. Considering that the tether entanglements are little influenced by the global path, our approach employs a 2-level hierarchical architecture. The global frontier-based planning solves a Traveling Salesman Problem (TSP) to minimize the distance. The local planning attempts to minimize the path cost and the tether length using an adjustable decision function whose parameters play on the trade-off between these two values. The proposed method, TAPE, is evaluated through detailed simulation studies as well as field tests. On average, our method generates a 4.1% increase in distance traveled compared to the TSP solution without our local planner, with which the length of the tether remains below the maximum allowed value in 53% of the simulated cases against 100% with our method.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
GaussLite: Online Task-Conditioned 3D Gaussian Splatting for Real-Time Robotic Mapping
Annika Thomas, Mason Peterson and Jonathan P. How
Existing 3D Gaussian Splatting (3DGS) systems distribute representation capacity uniformly across a scene, ignoring the fact that many downstream robotic tasks engage only a fraction of the reconstructed geometry. This causes valuable onboard compute to be allocated towards optimizing irrelevant parts of the scene, either limiting online capacity or under-optimizing the most relevant parts of the scene. We introduce GaussLite, a task-driven 3DGS mapping system that conditions its representation density on a natural-
language task specification. Given a posed RGB-D stream and a task such as "prepare to pick up the object on the desk," GaussLite uses a one-shot LLM parser to extract target and anchor objects, which are grounded per-frame by an open-vocabulary detector and segmented to produce per-pixel relevance masks in real time. The mapper allocates seeding density, gradient flow and scaling by task relevance. At matched Gaussian budget and real-time mapping at 4 Hz on resource-constrained hardware, GaussLite outperforms baselines on ROI PSNR on the Replica
Dataset by an average +2.72 dB and on a real-hardware demonstration in indoor and outdoor settings by +2.23 dB. We further show that two task-specialized
agents' maps can be fused into a single shared map via per-voxel voting on active-optimization counts in real time, outperforming concatenation by +3.42 dB while only sharing an average 7.08% of the map.
Read the full abstract ▾
Robotics
Off the Rails: Hijacking the Scoring Head in Generative End-to-End Driving Planners with Safety-Violating Adversarial Perturbations
Halima Bouzidi, Mboutidem Ekemini Mkpong, Haoyu Liu et al. (+1 more)
Generative models have recently seen rapid adoption in End-to-End (E2E) autonomous driving (AD), with
diffusion-based denoising and vocabulary-based
retrieval becoming the dominant trajectory-decoding paradigms. Despite their architectural diversity, current generative AD planners share a common inference pattern: a fixed set of candidate trajectories (anchors, vocabulary entries, or proposal queries) is scored by one or more learned heads conditioned on the Bird's-Eye-View (BEV) features, and the highest-scored candidate is returned as the final trajectory. Under this design, the scoring head is the only barrier between perception and the motion command, and its decision margins between competing candidates are often small. We introduce Derail, an
adversarial framework that exploits this scoring-head attack surface. Evaluated on various generative planners, Derail flips the trajectory selection from a safe to an unsafe candidate, with score drops of $39$–$80\%$ and collision rates of up to $50\%$, consistently outperforming generic loss-maximization and feature-divergence attacks. Our analysis suggests that safety-violating objectives govern attack effectiveness against generative AD planners, and that the scoring-head inference pattern itself is a recurring attack surface worth explicit defensive consideration.
Read the full abstract ▾
All authors Halima Bouzidi, Mboutidem Ekemini Mkpong, Haoyu Liu, Mohammad Abdullah Al Faruque
Robotics
Wind and State Estimation on SE(3): Comparative Evaluation of EKF and UKF with Continuous and Discrete Quadrotor Models
Hiranya Udagedara, Adam Bigsby and Mahdis Bisheban
Use of quadrotor
UAVs for wind velocity estimation is gaining popularity in recent studies, leveraging their maneuverability, compact size and low cost. Among available approaches, model-based wind velocity estimation is most commonly used, since it relies only on onboard sensors. However, as the quadrotor is a highly nonlinear system, thus making this task challenging. This study evaluate the use of both discrete and continuous dynamic equations of the quadrotor
UAV for wind velocity estimation on SE(3), rather than commonly adapted continuous or discretized form. Lie Group Variational Integrator, developed on discrete Lagrangian is used as the discrete model without any approximation or discritization. The study assess both the discrete and continuous form of the quadrotor dynamics on SE(3) using Extended Kalman filter (EKF), and Unscented Kalman filter (UKF). The quadrotor
UAV performance is evaluated in both MATLAB-based numerical simulations and free outdoor flight. The numerical simulations are conducted during both hovering and trajectory-tracking flights. Results demonstrate that, by using discrete SE(3) dynamics coupled with UKF, the quadrotor achieves higher estimation accuracy while maintaining trajectory tracking, even with low-cost sensors. These findings highlight the potential of discrete quadrotor models with UKF not only for wind velocity estimation but also for other high-accuracy tasks, even when relying on low-cost onboard sensors.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Streaming Gaussian Encoding for 4D Panoptic Occupancy Tracking
Maximilian Luz, Thomas Nürnberg, Yakov Miron et al. (+1 more)
Camera-based 4D panoptic occupancy tracking (4D-POT) is a promising paradigm for holistic scene understanding from multi-view imagery, enabling joint
reasoning about geometry, semantics, and object identities across time. Recent mask-based pipelines achieve strong performance by propagating instance queries across frames. However, their underlying volumetric representations are typically recomputed at each timestep, limiting
geometric temporal consistency, particularly under occlusion and for static scene elements. To address this limitation, we propose a streaming Gaussian encoder that maintains a persistent volumetric scene representation for 4D-POT. Our method models the scene as a fixed-size set of latent Gaussian queries that are propagated via ego-motion compensation and refreshed under a confidence-guided budget constraint. Crucially, we shape Gaussian opacities through depth-based supervision to serve as proxy for visibility, enabling confidence to accumulate as a temporally aggregated measure of persistent scene support. Together with a warmup-based multi-frame training strategy, this yields representation-level temporal coherence beyond decoder-only tracking. Extensive experiments on Occ3D-extended nuScenes and Waymo establish a new state-of-the-art for camera-based 4D-POT, improving tracking consistency with negligible computational overhead while remaining fully compatible with existing mask-based pipelines. We provide code and models at https://sge.cs.uni-freiburg.de.
Read the full abstract ▾
All authors Maximilian Luz, Thomas Nürnberg, Yakov Miron, Abhinav Valada
Robotics
From Grasps to Dexterity: Large-Scale Grasp Pretraining for Dexterous Manipulation
Ying Yuan, Xinyu Liu, Sriram Krishna et al. (+1 more)
Large-scale dexterous grasp
datasets encode rich priors over hand-object interaction, but their use has largely been confined to grasp generation and pick-and-place manipulation. We study whether such data can instead support functional dexterity in articulated tool use, where a
robot must acquire a tool, maintain contact, and operate its functional moving parts. We adapt a hierarchical imitation learning framework that combines high-level hand sub-goal prediction with a low-level goal-conditioned controller. We construct a 355k-trajectory grasp-pretraining
dataset from large-scale dexterous grasp annotations and use it to pretrain the low-level controller. The controller is then fine-tuned on downstream task demonstrations. To evaluate this setting, we introduce DexCraft, a simulation
benchmark with six articulated tool-use tasks requiring coordinated finger motion. Across simulation and real-world experiments, our approach outperforms end-to-end
diffusion policy baselines and hierarchical policies trained from scratch. In the real world, it improves full-task success by 33.3 percentage points over DP3. These results show that grasp
datasets can serve not only as resources for grasp synthesis, but also as scalable pretraining data for contact-rich dexterous manipulation.
Videos are shown on https://yingyuan0414.github.io/grasp2dexterity/ .
Read the full abstract ▾
All authors Ying Yuan, Xinyu Liu, Sriram Krishna, David Held
Robotics
Vision-Language Procedural Reasoning for Context-Aware Reward Modeling of Robotic Endovascular Guidewire Navigation
Wentong Tian, Jiyuan Zhao, Tianliang Yao et al. (+4 more)
Robotic-assisted endovascular interventions demand accurate, stable, and context-aware guidewire
navigation in complex and patient-specific vascular anatomies. Despite recent advances in robotic precision and learning-based control, existing autonomous
navigation methods remain limited by their reliance on static reward functions and the lack of explicit procedural
reasoning regarding anatomical context and task progression. To address these challenges, this paper proposes a
vision-
language procedural
reasoning (VL-PR) framework for autonomous guidewire
navigation. The framework integrates a
multimodal large
language model (MLLM) as a procedural
reasoning module that interprets real-time visual observations to infer high-level
navigation contexts. Instead of generating low-level control commands, the inferred procedural insights enable context-aware reward adaptation by dynamically adjusting the importance of reward components across different
navigation phases. This approach allows a single policy to resolve competing objectives and handle complex transitions while preserving a consistent global task goal. Experiments on a physical robotic platform across diverse vascular scenarios demonstrate enhanced task reliability and streamlined navigational efficiency, highlighting the advantages over static-reward methods and offering a scalable solution for complex and multi-task robotic endovascular procedures.
Read the full abstract ▾
All authors Wentong Tian, Jiyuan Zhao, Tianliang Yao, Yuxiang Fan, Zhengyu Shi, Dong Liu, Peng Qi
Robotics
Kaier Liang, Hengde Dai and Cristian-Ioan Vasile
Enabling
robots to follow natural
language commands to complete zero-shot long-horizon tasks remains challenging. It requires extracting implicit temporal and logical constraints from natural
language commands and executing multiple sub-tasks accordingly. Recent zero-shot object
navigation methods use
vision-
language models (VLMs) to guide frontier-based exploration in unknown environments, but they are limited to single-target tasks. Real-world commands such as "Clean either the chair or the couch, then turn on the tv." require navigating to multiple targets in a temporally constrained order, which no existing zero-shot system can handle. We present ViTL, a framework that addresses this gap at two levels. At the task level, we use a large
language model (LLM) to compile natural
language commands into Linear Temporal Logic (LTL) formulas, which are then converted into Deterministic Finite Automata~(DFA) that coordinate multi-channel value maps and trigger dynamic replanning when new objects are detected. At the
navigation level, we introduce directional score: rather than producing a direction-agnostic value across the entire field of view, we label frontier directions on the observation image and extract per-direction scores from the VLM. Experiments on Habitat-Matterport 3D (HM3D) show that the full framework enables zero-shot long-horizon completion of natural
language navigation tasks with temporal constraints, and that directional score improves single-target
navigation accuracy and efficiency over the baseline.
Read the full abstract ▾
Robotics
DSIP: A Dynamic Coordination Planner for Signal-Free Intersections using Diffusion-Model-Based Multi-Agent Motion Planning
Qian Hu, Haoyang Peng, Songan Zhang et al. (+2 more)
Traffic signal control at urban intersections inherently introduces stop-and-go behavior, resulting in increased delays and reduced traffic efficiency, especially under high traffic demand. With the emergence of connected and automated vehicles (CAVs), trajectory-level coordination has emerged as a high-potential strategy to augment or transcend conventional phase-based management. This paper proposes DSIP (
Diffusion-model-based Signal-free Intersection Planner), a multi-
agent motion planning framework driven by a generative
diffusion process. DSIP shifts the intersection management paradigm from discrete temporal phasing to continuous multi-vehicle trajectory optimization. This work evaluates the theoretical upper-bound performance of this coordination strategy under idealized communication and execution conditions to isolate the core benefits of the
diffusion-driven approach. Using the SUMO platform, we evaluate DSIP across diverse four-leg intersection configurations. Experimental results demonstrate that DSIP significantly reduces average delay and maintains higher average speed compared to both fixed-time signal control and state-of-the-art reinforcement-learning-based controllers, particularly in medium- to high-density traffic. These findings suggest that
diffusion-based trajectory planning provides a scalable and robust foundation for future autonomous intersection management. By unlocking latent intersection capacity through software-defined coordination, this approach offers a cost-effective pathway to improve urban traffic flow efficiency without requiring physical infrastructure expansion.
Read the full abstract ▾
All authors Qian Hu, Haoyang Peng, Songan Zhang, Ming Yang, Hongtei Eric Tseng
Robotics
VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes
Yen-Jen Wang, Jiaman Li, Sirui Chen et al. (+9 more)
Perception-based
humanoid loco-manipulation requires connecting
egocentric observations and task instructions to whole-body motion. Learning this mapping requires synchronized
egocentric images,
language commands, and
robot-compatible kinematic trajectories, yet no existing data source provides this complete tuple at scale. We address this bottleneck by generating
vision-
language-kinematics (VLK) supervision synthetically in reconstructed scenes. Our pipeline leverages 3D Gaussian Splatting to reconstruct metric-scale indoor environments, synthesizes
navigation and object-interaction trajectories using privileged scene information, and renders paired
egocentric observations after the fact. We produce 48,000 paired trajectories with no human intervention and train a VLK policy that predicts short-horizon whole-body kinematic trajectories. A whole-body tracker converts these predictions into actions on the physical
humanoid. We evaluate on the physical Unitree G1 performing
navigation and single-object transport, demonstrating that synthesized interactions in reconstructed scenes provide effective supervision for sim-to-real perception-based
humanoid loco-manipulation. Project Website: https://
vision-
language-kinematics.github.io/
Read the full abstract ▾
All authors Yen-Jen Wang, Jiaman Li, Sirui Chen, Takara E. Truong, Pei Xu, Pieter Abbeel, Rocky Duan, Koushil Sreenath, Angjoo Kanazawa, Carmelo Sferrazza, Guanya Shi, Karen Liu
Robotics
GROW$^2$: Grounding Which and Where for Robot Tool Use
Yuhong Deng, Yuyao Liu and David Hsu
Can the
robot use a plate to cut a cake if no knife is available? Tool use greatly expands
robot capabilities, but to use tools creatively beyond their intended functions, the
robot faces the challenge of $\textit{open-world affordance grounding}$: select an open-category object to act as a tool and localize its specific region of action. To this end, we introduce GROW$^2$ (GROunding Which and Where), which leverages object parts as a natural abstraction to split the grounding process hierarchically into semantic and
geometric levels, thus bypassing the need for data-heavy, end-to-end training. Semantically, GROW$^2$ harnesses the commonsense
reasoning of
Vision-
Language Models (VLMs) to parse a natural-
language task instruction, select a suitable object as the tool, and identify task-relevant parts on the tool and the target object. Geometrically,
vision foundation models then ground the selected parts into precise 3D regions from a single RGB-D image. Experiments on established
benchmarks show that GROW$^2$ outperforms state-of-the-art baselines on affordance prediction
benchmarks. Further, it achieves zero-shot generalization over open-category objects and outperforms baselines in both simulated and real-world
robot tool use experiments.
Read the full abstract ▾
Robotics
Sequential Planning via Anchored Robotic Keypoints
Bryce Grant, Aryeh Rothenberg, Logan Senning et al. (+3 more)
Changes in a
robot's observations do not automatically imply changes in the task it is solving, but
language-conditioned manipulation systems often fail to separate the two. If an object moves, for example, the locations at which actions should be executed may change while the underlying manipulation plan remains valid. We introduce Sequential Planning via Anchored Robotic Keypoints (SPARK), which separates a task's symbolic plan from its execution-time perceptual grounding, and we experimentally isolate whether a fixed symbolic plan can survive substantial changes to the scene through execution-time re-grounding. SPARK represents the task as a symbolic behavior tree whose object references are grounded to current 3D observations during execution, allowing the
robot to update where actions are performed while preserving the plan. Runtime verification then enables local recovery and renewed grounding, only resorting to a new plan if necessary. To test the separation between plan and perceptual-grounding, we perform a set of manipulation experiments with a fixed plan while moving objects during execution. Under a 10 cm displacement, verification and adaptation maintain $66.7\%$ task success, compared with $2.2\%$ when disabled, demonstrating the importance of the underlying principle. Without task-specific model training and with an open-weight planner, SPARK also achieves $66.8\%$ mean success across six LIBERO-PRO settings, and $68\%$ across eleven physical task-embodiment settings on three
robot platforms. Together, these results show that robustness does not require re-planning the task and for an important class of failures, it is sufficient to preserve the plan and update the system's perception of a changing world. Code and
videos can be found at \url{https://cwru-aism.github.io/spark-page/}
Read the full abstract ▾
All authors Bryce Grant, Aryeh Rothenberg, Logan Senning, Zonghe Chua, Zach Patterson, Peng Wang
Robotics
Realtime Wind Estimation using Low Cost Quadrotor Uncrewed Aerial Vehicles
Hiranya Udagedara and Mahdis Bisheban
In environmental monitoring as well as emergency response applications such as wildfires, wind velocity measurement is essential. Quadrotor
UAVs have become popular platforms for wind velocity estimation due to their maneuverability, compact size, and cost-effectiveness. Numerous studies use the Extended Kalman Filter (EKF) to estimate the wind velocity based on the quadrotor dynamic model. However, most of them use hovering quadrotors only for wind estimation, others use a near-linear trajectory to estimate near-constant velocities. Furthermore, EKF performance is constrained by its reliance on linearized approximations of the nonlinear quadrotor dynamics around current states, limiting accuracy in highly nonlinear scenarios, including windy conditions. This study proposes the use of an Unscented Kalman Filter (UKF), a nonlinear
estimator to provide accurate wind estimations while maintaining the trajectory of the quadrotor
UAV. The quadrotor is modeled on the Special Euclidean group SE(3) and the approach is evaluated through numerical simulations using a
geometric controller to maintain quadrotor flight paths. The results indicate that as the nonlinearity of the simulation increases, the UKF consistently outperforms the EKF. This demonstrates the potential of the UKF as a reliable
estimator for highly nonlinear scenarios, capable of maintaining the trajectory with minimal deviation while providing accurate wind velocity estimations.
Read the full abstract ▾
Robotics
MOAR Planner: Multi-Objective and Adaptive Risk-Aware Path Planning for Infrastructure Inspection with a UAV
Louis Petit and Alexis Lussier Desbiens
The problem of autonomous
navigation for
UAV inspection remains challenging as it requires effectively navigating in close proximity to obstacles, while accounting for dynamic risk factors such as weather conditions, communication reliability, and battery autonomy. This paper introduces the MOAR path planner which addresses the complexities of evolving risks during missions. It offers real-time trajectory adaptation while concurrently optimizing safety, time, and energy. The planner employs a risk-aware cost function that integrates pre-computed cost maps, the new concepts of damage and insertion costs, and an adaptive speed planning framework. With that, the optimal path is searched in a graph using a discrete representation of the state and action spaces. The method is evaluated through simulations and real-world flight tests. The results show the capability to generate real-time trajectories spanning a broad range of evaluation metrics: around 90% of the range occupied by popular
algorithms. The proposed framework contributes by enabling
UAVs to navigate more autonomously and reliably in critical missions.
Read the full abstract ▾
Robotics
Training Vision-Language-Action Models with Dense Embodied Chain-of-Thought Supervision
Haoyang Li, Guanlin Li, Youhe Feng et al. (+9 more)
Cross-embodiment transfer in
vision-
language-action (VLA) models remains challenging because low-level state and action spaces differ fundamentally across
robot platforms. We observe that the high-level cognitive process underlying manipulation, including scene perception, object identification, task planning, and sub-task decomposition, is largely shared across embodiments. Based on this observation, we present ZR-0, a 2.6 billion parameter end-to-end VLA model that uses dense
Embodied Chain-of-Thought (ECoT) supervision to align cross-embodiment representations within the
vision-
language model (VLM). ZR-0 adopts a dual-stream architecture: a pre-trained VLM (System 2) generates structured ECoT
reasoning during training, while a
Diffusion Transformer-based action expert (System 1) produces continuous action chunks via flow matching. The two components are coupled through cross-
attention, with an
attention mask that restricts the action expert to input
prompt features only, enabling ECoT generation to be entirely skipped at inference without any performance loss. ZR-0 is pre-trained on ProcCorpus-60M, a large-scale
dataset comprising approximately 60 million frames (approximately 1,000 hours) from over 400K trajectories, with dense ECoT annotations covering 96.8% of all frames. We evaluate ZR-0 on three simulation
benchmarks spanning single-arm (LIBERO), bimanual (RoboTwin 2.0), and
humanoid (RoboCasa GR-1 Tabletop) embodiments, as well as real-world experiments on the xArm platform, demonstrating strong performance across all settings. Code and model checkpoints are available at https://github.com/RUCKBReasoning/ZR-0.
Read the full abstract ▾
All authors Haoyang Li, Guanlin Li, Youhe Feng, Chen Zhao, Zhuoran Wang, Yang Li, Qizhe Wei, Shifeng Bao, Haitao Shen, Yihan Zhao, Tong Yang, Jing Zhang
Robotics
Learning from Mistakes: Rollout-Retrieval Lifelong Policy Learning for Autonomous Driving
Cheng Gong, Haoyang Wang, Chao Lu et al. (+2 more)
Autonomous driving policies should be able to improve continually as deployment exposes them to increasingly diverse and long-tail traffic situations. However, most learning-based policies are trained or fine-tuned on expert demonstrations and then rely largely on generalization to handle challenging closed-loop scenarios, lacking an explicit mechanism to correct and retain the mistakes exposed in these scenarios. This paper studies autonomous driving policy improvement from a lifelong learning perspective: Can a pretrained policy improve continually by accumulating corrective knowledge derived from its own mistakes, while retaining previously acquired driving competence? To answer this question, we propose Rollout-
Retrieval Lifelong Policy Learning (R$^2$LPL), a policy learning framework that retrieves corrective targets from recoverable policy-induced mistakes and retains the resulting knowledge through lifelong policy learning. R^2LPL addresses a key bottleneck in continual policy improvement: closed-loop mistakes reveal where the policy is weak, but do not directly specify what the policy should learn. By filtering recoverable mistake-related states and retrieving feasible corrective targets, R$^2$LPL turns sparse failure evidence into compact supervised knowledge for stable and sample-efficient policy improvement. We evaluate R$^2$LPL on large-scale closed-loop nuPlan
benchmarks. With only a few rollout and continual-learning cycles, R$^2$LPL elevates a learning-based planner with moderate initial performance to state-of-the-art performance across the evaluated
benchmarks, especially on the challenging and long-tail Test14-hard split. These results demonstrate the effectiveness of R$^2$LPL in converting recoverable closed-loop mistakes into corrective knowledge for sustained policy improvement.
Read the full abstract ▾
All authors Cheng Gong, Haoyang Wang, Chao Lu, Zirui Li, Jianwei Gong
Robotics · Cross-listed from cs.CV
PS-MOT: Cultivating Instance Awareness from Point Seeds for Multi-Object Tracking
Kai Luo, Fei Teng, Mengfei Duan et al. (+6 more)
We introduce Point-supervised Multi-Object Tracking (PS-MOT) as a cost-effective alternative to traditional bounding box supervision, shifting the focus from spatial fitting to topological center-driven representation. However, PS-MOT faces challenges, e.g., spatial ambiguity and identity drift due to the lack of explicit
geometric structure and scale constraints. To address these, we propose PS-Track, a hierarchical pipeline transitioning from points to instances across data, model, and loss levels. At the data level, we introduce Temporal-Feedback Prompting (TFP) to evolve points into temporally consistent pseudo-labels using negative spatial cues and motion priors. At the model level, we design the Point-Excited Wavelet
Attention (PEWA) module, which leverages semantic correlations to activate high-frequency components, “hallucinating” object boundaries. At the loss level, Uncertainty-Guided Gaussian Learning (UGL) models pseudo-labels as probabilistic distributions, dynamically calibrating supervision intensity. Experiments on DanceTrack, EmboTrack, SportsMOT, and JRDB demonstrate that PS-Track provides a feasible and effective point-supervised alternative across diverse tracking scenarios, establishing a new state-of-the-art for point-supervised tracking. The source code is available at https://github.com/xifen523/PS-MOT.
Read the full abstract ▾
All authors Kai Luo, Fei Teng, Mengfei Duan, Wanjun Jia, Xu Wang, Hao Shi, Kunyu Peng, Zhiyong Li, Kailun Yang
Robotics
Grasp-Oriented Non-Prehensile Manipulation via Learning a Graspability Field
Licheng Zhong and Gim Hee Lee
Non-prehensile manipulation is often used as a preparatory step for robotic grasping, yet existing approaches typically require a predefined target object pose. In practice, however, objects admit multiple graspable configurations and the desired pose is not known in advance. We reformulate non-prehensile manipulation for grasping as optimizing an object centric graspability objective rather than reaching a specific pose. We construct a graspable set from synthesized grasps and define a graspability field that measures how suitable an object configuration is for successful grasp execution. The scalar measure provides a dense learning signal for reinforcement learning and determines when to terminate manipulation. This yields a closed-loop manipulation-to-grasp pipeline driven by a single policy. Experiments in simulation and on a real
robot show that the policy reliably reconfigures objects into graspable states and transitions to grasping without external planners or manually specified stopping conditions. The predicted graspability distance correlates with real world grasp success, which indicates that the learned representation captures grasp feasibility of object configurations.
Read the full abstract ▾
Robotics
What Enables In-Context Behavior Prompting for Manipulation?
Austin Patel, Ben Pekarek, Joel Enrique Castro Hernandez et al. (+1 more)
Behavior prompting is a paradigm in which a sensorimotor
robot demonstration, called a behavior
prompt, serves as an in-context
prompt for performing new tasks at test time. While prior work has shown that this capability is possible, the conditions that enable it remain poorly understood. We present an empirical study of when, how, and why behavior prompting works. To support this study, we introduce DrawAnything and LIBERO-Gen,
benchmarks with up to 2000 procedurally generated tasks that evaluate test-time adaptation to unseen drawing and tabletop manipulation tasks. We also present Behavior Prompting Policy (BPP), an in-context visuomotor architecture, and iPhUMI, a handheld interface to demonstrate behavior
prompts at test time. Our main finding is that task diversity, rather than demonstrations per task, is a key driver of prompting capability. Given sufficient diversity, a behavior
prompt improves adaptation to unseen tasks, reducing drawing error by 80.7% over goal-image conditioning and improving success on chained manipulation tasks by up to 20.8% over
language conditioning. Given insufficient diversity in a real-world laundry experiment, behavior prompting has weaker task conditioning than a
language baseline. An
attention analysis shows how the
prompt is used: the policy follows it step by step as a source of dense sub-goals.
Prompt ablations show that dense sensorimotor detail matters: removing actions or downsampling the
prompt hurts fine-grained action adaptation. We have open-sourced all components to enable reproducible research on behavior prompting without needing industrial-scale data collection or compute.
Read the full abstract ▾
All authors Austin Patel, Ben Pekarek, Joel Enrique Castro Hernandez, Shuran Song
Robotics
Vision-Language-Action Models: Experimental Insights from a Real-World UR5 Platform
Mathilde Hochedel and Marc Lalonde
This project investigates whether recent
Vision-
Language-Action (VLA) models can be transferred from controlled research
benchmarks to a real-world robotic platform, specifically a UR5e manipulator, in a reproducible and operationally meaningful manner. The work integrates real-
robot data acquisition,
dataset engineering (compatible with the RLDS format), and the fine-tuning and deployment of OpenVLA and OpenVLA-OFT models, with systematic validation of action representations and control interfaces. The project resulted in several foundational assets: (i) a complete real-
robot data acquisition pipeline, (ii) a
dataset conversion workflow aligned with RLDS standards, (iii) an initial fine-tuning and inference infrastructure for VLA models, and (iv) a structured set of experimental observations grounded in real-
robot trials. These elements collectively establish a reproducible framework for evaluating learning-based manipulation systems beyond simulation. Empirically, the experiments reveal a consistent gap between promising offline indicators and unstable closed-loop behavior on the physical system: this gap cannot be attributed solely to model limitations, it is strongly influenced by action semantics, coordinate frame conventions, temporal
alignment between modalities, image preprocessing consistency, and
dataset coverage and quality. These observations lead to a key interpretation: the successful deployment of VLA systems in real-world settings depends less on incremental improvements in model capacity and more on precise control of the entire data-model-control pipeline. The project reframes VLA-based robotics from a primarily model-centric challenge to a system-level problem; it highlights the difficulty of running robust task execution on the real
robot and provides a clear, experimentally grounded understanding of the conditions required for reliable deployment.
Read the full abstract ▾
Robotics
HUMEMBR: Learning Human Routines for Predictive Embodied Navigation
Samira Huber, Klaas Pelzer, Duc M. Nguyen et al. (+2 more)
Understanding and navigating human-centered environments over extended periods of time while considering human behavior and routines remains a fundamental challenge in robotics. In real-world settings,
robots may be asked to locate a specific individual, predict where that person is likely to be, or estimate when they typically leave a building. Addressing such queries requires
reasoning over extensive histories of observations and capturing long-term behavioral patterns. To this end, we introduce Human-Centered
Memory for
Embodied Robots (HUMEMBR), a system designed for
embodied question answering and routine-conditioned
navigation. HUMEMBR integrates a continuous
memory construction process with a parallel
retrieval and querying mechanism, enabling the system to accumulate structured representations of human routines while supporting interactive, user-driven queries. Our experimental results indicate that HUMEMBR improves long-horizon
reasoning about human behavior relative to full-context LLM baselines, while using substantially fewer
tokens. Furthermore, we deploy HUMEMBR on a physical
robot in two distinct environments, showing its ability to handle diverse queries and
navigation tasks under real-world conditions.
Read the full abstract ▾
All authors Samira Huber, Klaas Pelzer, Duc M. Nguyen, Xuesu Xiao, Sören Pirk
Robotics
FutureNav: Unified World-Action Modeling for Vision-and-Language Navigation
Lingfeng Zhang, Zeying Gong, Xiaoshuai Hao et al. (+7 more)
Vision-and-
language navigation (VLN) in continuous environments requires an
agent to ground instructions in
egocentric observations while maintaining spatial understanding across long action sequences. Recent
navigation foundation models have shown strong progress by scaling
vision-
language models, but they often learn
navigation primarily as direct action generation, without explicitly modeling world states or predicting their future
evolution. We introduce FutureNav, a VLM-based unified world-action modeling framework for
vision-and-
language navigation. Specifically, FutureNav jointly encodes text, visual, and spatial features and feeds them into the LLM, and optimizes four objectives for simultaneous world and action modeling: an action policy objective for
navigation action prediction, inverse and forward dynamics objectives for modeling state transitions, and a future generation objective for predicting future spatial states. This unified architecture strengthens action prediction while explicitly modeling the world, without sacrificing inference speed. Extensive experiments show that, with only a 4B-scale backbone, FutureNav achieves state-of-the-art performance on multiple VLN
benchmarks and substantially outperforms prior VLN methods, paving the way toward future world-action models for VLN. We will release the code and models to support future research.
Read the full abstract ▾
All authors Lingfeng Zhang, Zeying Gong, Xiaoshuai Hao, Haoxiang Fu, Qiang Zhang, Mingliang Zhou, Hangjun Ye, Xiaojun Liang, Junwei Liang, Wenbo Ding
Robotics
ReactiveBFM: Reactive Closed-Loop Motion Planning Towards Universal Humanoid Whole-Body Control
Xiao Chen, Weishuai Zeng, Xiaojie Niu et al. (+12 more)
While current Behavior Foundation Models (BFMs) provide robust control priors for
humanoids, they only execute pre-defined reference motions. As a result, they are vulnerable to environmental shifts and incapable of reactive whole-body coordination. Naively cascading them with generative motion planners fails to achieve true reactivity, as inevitable tracking discrepancies induce fatal cumulative exposure bias. To bridge this gap, we propose ReactiveBFM, a real-time closed-loop planning-control framework. At its core, we effectively mitigate exposure bias via a scheduled prefix sampling curriculum, forcing the generative planner to actively learn error-recovery behaviors from imperfect physical states rather than ground-truth trajectories. Systematically, to reconcile the severe latency mismatch between auto-regressive planning and high-frequency tracking, we introduce an asynchronous replanning mechanism. Combined with trajectory chunking to temporally ensemble spatial references, our system guarantees spatio-temporally fluid execution without physical jitter. Deployed on the Unitree G1
humanoid, ReactiveBFM demonstrates unprecedented physical agility across a vast repertoire of text-conditioned closed-loop motions. Notably, ReactiveBFM achieves zero-shot moving target reaching, showcasing intricate whole-body coordination and on-the-fly replanning. In sim-to-sim benchmarking under severe perturbations, ReactiveBFM achieves a 93.1% success rate, significantly outperforming cascaded open-loop baselines by 28.6%.
Read the full abstract ▾
All authors Xiao Chen, Weishuai Zeng, Xiaojie Niu, Zirui Wang, Jianan Li, Huayi Wang, Furui Xu, Jiahe Chen, Weixiang Zhong, Lihe Ding, Kailin Li, Jiangmiao Pang, Tai Wang, Tianfan Xue, Jingbo Wang
Robotics
Chronos: A Physics-Informed Full-History Framework for Non-Markovian Long-Horizon Manipulation
Yulin Zhou, Yimeng Wang, Nengyu Wang et al. (+9 more)
General-purpose
robot policies should be modeled as dynamical systems, yet many VLA and generative imitation policies still rely on present observations or short windows. This Markovian shortcut fails in
memory-dependent manipulation: identical observations can demand different actions after different histories. We present Chronos, a physics-informed full-history framework for non-Markovian long-horizon manipulation. The key idea is to elevate observation history from auxiliary context to the latent state of the policy dynamics. At each physical control step, Chronos forms one state-representative
token by fusing observation and proprioception, so the
token sequence is aligned one-to-one with physical time. A selective state space model propagates this causal historical state, which conditions a
multimodal coarse action prior through implicit maximum likelihood estimation (IMLE). This prior is then refined by a second-order Schrodinger-inspired bridge that predicts acceleration fields, yielding smoother and more physically grounded
robot motion. Across 16 simulated tasks and 4 real-world experiments, Chronos is evaluated on precision insertion, general manipulation, and
memory-dependent long-horizon control. On RMBench, where success requires remembering task phase, Chronos achieves 73.6% average success, outperforming Markovian VLA baseline pi0.5 by +62.4 percentage points, a 6.6x relative gain, while using 10x fewer parameters. It also surpasses the
memory VLA Mem-0 by 22.8 points while using over 30x fewer parameters. In real-world dual-arm experiments using a single RGB camera, Chronos achieves 78% average success over four tasks, including 72% on the three
memory-dependent tasks, whereas pi0.5 achieves 7% overall and 0% on the
memory-dependent subset. These results suggest that history should not be treated as auxiliary context, but as the latent state of the manipulation policy.
Read the full abstract ▾
All authors Yulin Zhou, Yimeng Wang, Nengyu Wang, Shaojia Xing, Shiyun Tu, Xiang Li, Jingkai Zhang, Ningbo Jiang, Yuankai Lin, Hua Yang, Xiangrui Zeng, Zhouping Yin
Robotics
CSAR: Containerized System Architecture for Robotics
Gregorio Ambrosio-Cestero, Cipriano Galindo, Javier Gonzalez-Jimenez et al. (+1 more)
Robotic applications increasingly rely on distributed computational infrastructures that combine embedded devices, edge servers, and cloud resources. This
evolution, together with the collaborative nature of robotics projects, has made the development, integration, deployment, and long-term operation of robotic systems significantly more complex. In practice, multi-user robotics software teams face persistent challenges related to dependency isolation, compatibility, reproducibility, efficient sharing of specialized hardware, and deployment across heterogeneous environments. In this paper, we present CSAR (Containerized System Architecture for Robotics), a container-centric architectural framework designed specifically for robotics teams and the edge-cloud continuum. CSAR combines LXC/LXD-based system containerization, ROS 2/DDS-based communication, and a three-layer edge infrastructure to organize computation into hardware-affine, persistent execution environments that remain decoupled from the
volatility of experimental workloads. Through its Infrastructure Core, Platform and Multi-User Orchestration, and Compute and Acceleration layers, CSAR provides strong isolation, controlled resource sharing, and
topology-aware networking for distributed robotic applications. To demonstrate its validity, we describe a real deployment of CSAR in an academic robotics laboratory and evaluate it through representative use cases involving edge-offloaded 3D SLAM and GPU-accelerated semantic mapping. The results indicate that CSAR simplifies software integration, improves the utilization of shared computational resources, and facilitates safe prototyping, as well as reproducible and collaborative experimentation in robotics teams. The implementation described in this paper, including deployment templates, configuration files, and documentation, is available at https://github.com/goyoambrosio/CSAR.
Read the full abstract ▾
All authors Gregorio Ambrosio-Cestero, Cipriano Galindo, Javier Gonzalez-Jimenez, Jose-Raul Ruiz-Sarmiento
Robotics
X-Morph: Human Motion Priors for Scalable Robot Learning Across Morphologies
Ritwik Sharma, Shivam Sood, Arhaan Jain et al. (+3 more)
Recent progress in
humanoid behavior models has been driven in large part by abundant human motion data, but comparable motion data is scarce for non-
humanoid legged
robots such as quadrupeds, hexapods, and quadruped manipulators. A promising alternative is to repurpose human motion across embodiments; however, direct retargeting often produces motions that are visually plausible yet physically inconsistent or difficult to track under
robot dynamics. We present X-Morph, a human-motion-to-
robot-behavior pipeline that converts human motion into deployable locomotion and loco-manipulation policies for diverse non-
humanoid legged morphologies. A cross-morphology retargeting stage converts human motions into kinematically plausible, intent-preserving
robot references, which are then tracked by a privileged RL policy and distilled into a causal student policy. We evaluate X-Morph on three morphologically distinct platforms: a quadruped, a hexapod, and a quadruped equipped with a manipulator. The resulting policies track diverse retargeted motions, generalize to unseen human motions, and support downstream use cases including
video-based teleoperation, behavior-prior control, and text-conditioned motion generation. These results suggest that large-scale human motion can serve as a substrate for learning broad, reusable behavior priors beyond
humanoid robots. Project page: https://maker-rat.github.io/morph/
Read the full abstract ▾
All authors Ritwik Sharma, Shivam Sood, Arhaan Jain, Shyam Charan Kesavamoorthi, Chengyang He, Guillaume Sartoretti
Robotics
ActiveVital: Geometry-Aware Embodied Vital Signs Monitoring for Home Healthcare Robots
Yuxuan Hu, Shihao Li, Yang Xiao et al. (+3 more)
Home
robots require reliable vital signs monitoring to support long-term companionship and safety in daily environments, yet obtaining respiration and heart rate without physical contact remains challenging in unconstrained home settings. Millimeter-wave (mmWave) radar offers a promising solution due to its phase sensitivity to sub-millimeter motions. However, mmWave measurements are fundamentally constrained by observation geometry, since only the radial component of motion is observable. Consequently, arbitrary
robot-human orientations often introduce angular misalignment that destabilizes vital signs estimation. To address this limitation, we reformulate vital signs monitoring from passive signal recovery to active
geometric regulation. We propose ActiveVital, a
vision-guided sensing framework that treats sensing geometry as an explicit control variable for
robots. It localizes the chest anchor via visual keypoints and converts
alignment errors into control commands. This steers the
robot-mounted radar toward near-normal incidence to the thoracic surface, maximizing radial observability within a perception-action loop. A differential phase enhancement module further stabilizes signal extraction under motion. Experiments show that ActiveVital reduces respiration interval error from 0.87 s to 0.14 s and heart rate error from 13.59 bpm to 2.22 bpm, achieving accuracy comparable to controlled static sensing while remaining robust under unconstrained
robot-human configurations.
Read the full abstract ▾
All authors Yuxuan Hu, Shihao Li, Yang Xiao, Gen Li, Feng Xu, Jianfei Yang
Robotics
ConCent: Contact-Centric Real-to-Sim-to-Real Learning from One Demonstration
Heecheol Kim, Namiko Saito, Katsushi Ikeuchi et al. (+1 more)
Sim-to-real policy transfer – deploying policies trained in simulation in the real world – is a promising paradigm for scaling
robot manipulation without large-scale real-world data. However, transferring simulation-trained policies remains challenging due to discrepancies in contact dynamics – particularly in contact-rich tasks where subtle differences can alter task outcomes entirely. Because interaction between the manipulated object and the environment is mediated through contact, task success depends on accurately reproducing task-relevant contacts. Accordingly, in manipulation, contact-centric fidelity – reproducing both the contact event sequence (when, where, and how contacts occur) and the local contact dynamics (how forces and motions evolve at each contact) – is a necessary condition for task success. Based on this insight, we propose a contact-centric real-to-sim-to-real RL framework that uses task-relevant contact event sequences extracted from real demonstrations as the learning objective. We approximate objects as groups of primitives and optimize their contact geometry in simulation so that the resulting local contact dynamics explain the observed state transitions. The contact event sequence is automatically extracted by replaying the demonstration. This sequence serves as a structured reward signal, guiding the policy toward physically plausible contact regimes validated in reality and preventing exploitation of unrealistic simulator contacts. The signal is obtained automatically, requiring no per-task reward design. Experiments on contact-rich manipulation tasks demonstrate more stable and robust sim-to-real policy transfer compared to unconstrained RL baselines.
Read the full abstract ▾
All authors Heecheol Kim, Namiko Saito, Katsushi Ikeuchi, Yasuyuki Matsushita
Robotics
KYON: Semi-Modular Wheel-Legged Quadruped With Agile Bimanual Capability
Luca Rossini, Arturo Laurenzi, Francesco Ruscelli et al. (+8 more)
This paper presents KYON, a hybrid wheel-legged quadruped
robot equipped with a bimanual upper body for loco-manipulation tasks. The platform features a semi-modular design with a reconfigurable lower legs, enabling both wheeled and legged locomotion depending on the environment. A design approach that places actuators in the base and uses transmission mechanisms reduces distal inertia, improving agility and dynamic performance. The
robot integrates a whole-body control framework together with a reinforcement learning based policy to handle nonlinear dynamics and enhance robustness to disturbances for the execution of locomotion and manipulation tasks, independently. Experimental results demonstrate effective dynamic locomotion and bimanual manipulation, validating the platform's capability to operate in complex and unstructured scenarios.
Read the full abstract ▾
All authors Luca Rossini, Arturo Laurenzi, Francesco Ruscelli, Yifang Zhang, Jingcheng Jiang, Giovanbattista Gravina, Lorenzo Baccelliere, Corrado Burchielli, Stefano Cordasco, Luca Muratore, Nikos Tsagarakis