Robotics · Cross-listed from cs.CV
Synthesizing the Kill Chain: A Zero-Shot Framework for Target Verification and Tactical Reasoning on the Edge
Jesse Barkley, Abraham George and Amir Barati Farimani
Deploying autonomous edge robotics in dynamic military environments is constrained by both scarce domain-specific training data and the computational limits of edge hardware. This paper introduces a hierarchical, zero-shot framework that cascades lightweight object detection with compact
Vision-
Language Models (VLMs) from the Qwen and Gemma families (4B-12B parameters). Grounding DINO serves as a high-recall, text-promptable region proposer, and frames with high detection confidence are passed to edge-class VLMs for semantic verification. We evaluate this pipeline on 55 high-fidelity synthetic
videos from Battlefield 6 across three tasks: false-positive filtering (up to 100% accuracy), damage assessment (up to 97.5%), and fine-grained vehicle classification (55-90%). We further extend the pipeline into an agentic Scout-Commander workflow, achieving 100% correct asset deployment and a 9.8/10
reasoning score (graded by GPT-4o) with sub-75-second latency. A novel "Controlled Input" methodology decouples perception from
reasoning, revealing distinct failure phenotypes: Gemma3-12B excels at tactical logic but fails in visual perception, while Gemma3-4B exhibits
reasoning collapse even with accurate inputs. These findings validate hierarchical zero-shot architectures for edge autonomy and provide a diagnostic framework for certifying VLM suitability in safety-critical applications.
Read the full abstract ▾
Robotics
Solving Geodesic Equations with Composite Bernstein Polynomials for Trajectory Planning
Nick Gorman, Gage MacLin, Maxwell Hammond et al. (+1 more)
This work presents a trajectory planning method based on composite Bernstein
polynomials for autonomous systems navigating complex environments. The method is implemented in a symbolic optimization framework that enables continuous paths and precise control over trajectory shape. Trajectories are planned over a cost surface that encodes obstacles as continuous fields rather than discrete boundaries. Regions near obstacles are assigned higher costs, naturally encouraging the trajectory to maintain a safe distance while still allowing efficient routing through constrained spaces. The use of composite Bernstein
polynomials preserves continuity while enabling fine control over local
curvature to satisfy geodesic constraints. The symbolic representation supports exact derivatives, improving optimization efficiency. The method applies to both two- and three-dimensional environments and is suitable for ground, aerial,
underwater, and space systems. In spacecraft trajectory planning, for example, it enables the generation of continuous, dynamically feasible trajectories with high numerical efficiency, making it well suited for orbital maneuvers, rendezvous and proximity operations, cluttered gravitational environments, and planetary exploration missions with limited onboard computational resources. Demonstrations show that the approach efficiently generates smooth, collision-free paths in scenarios with multiple obstacles, maintaining clearance without extensive sampling or post-processing. The optimization incorporates three constraint types: (1) a Gaussian surface inequality enforcing minimum obstacle clearance; (2) geodesic equations guiding the path along locally efficient directions on the cost surface; and (3) boundary constraints enforcing fixed start and end conditions. The method can serve as a standalone planner or as an initializer for more complex motion planning problems.
Read the full abstract ▾
All authors Nick Gorman, Gage MacLin, Maxwell Hammond, Venanzio Cichella
Robotics · Cross-listed from cs.LG
Confounding Robust Continuous Control via Automatic Reward Shaping
Mateo Juliani, Mingxuan Li and Elias Bareinboim
Reward shaping has been applied widely to accelerate Reinforcement Learning (RL)
agents' training. However, a principled way of designing effective reward shaping functions, especially for complex continuous control problems, remains largely under-explained. In this work, we propose to automatically learn a reward shaping function for continuous control problems from offline
datasets, potentially contaminated by unobserved confounding variables. Specifically, our method builds upon the recently proposed causal Bellman equation to learn a tight upper bound on the optimal state values, which is then used as the potentials in the Potential-Based Reward Shaping (PBRS) framework. Our proposed reward shaping
algorithm is tested with Soft-Actor-Critic (SAC) on multiple commonly used continuous control
benchmarks and exhibits strong performance guarantees under unobserved confounders. More broadly, our work marks a solid first step towards confounding robust continuous control from a causal perspective. Code for training our reward shaping functions can be found at https://github.com/mateojuliani/confounding_robust_cont_control.
Read the full abstract ▾
Robotics
A Human-in-the-Loop Confidence-Aware Failure Recovery Framework for Modular Robot Policies
Rohan Banerjee, Krishna Palempalli, Bohan Yang et al. (+5 more)
Robots operating in unstructured human environments inevitably encounter failures, especially in
robot caregiving scenarios. While humans can often help
robots recover, excessive or poorly targeted queries impose unnecessary cognitive and physical workload on the human partner. We present a human-in-the-loop failure-recovery framework for modular robotic policies, where a policy is composed of distinct modules such as perception, planning, and control, any of which may fail and often require different forms of human feedback. Our framework integrates calibrated estimates of module-level uncertainty with models of human intervention cost to decide which module to query and when to query the human. It separates these two decisions: a module selector identifies the module most likely responsible for failure, and a querying
algorithm determines whether to solicit human input or act autonomously. We evaluate several module-selection strategies and querying
algorithms in controlled synthetic experiments, revealing trade-offs between recovery efficiency, robustness to system and user variables, and user workload. Finally, we deploy the framework on a
robot-assisted bite acquisition system and demonstrate, in studies involving individuals with both emulated and real mobility limitations, that it improves recovery success while reducing the workload imposed on users. Our results highlight how explicitly
reasoning about both
robot uncertainty and human effort can enable more efficient and user-centered failure recovery in collaborative
robots. Supplementary materials and
videos can be found at: http://emprise.cs.cornell.edu/modularhil
Read the full abstract ▾
All authors Rohan Banerjee, Krishna Palempalli, Bohan Yang, Jiaying Fang, Alif Abdullah, Tom Silver, Sarah Dean, Tapomayukh Bhattacharjee
Robotics
Adaptive Time Step Flow Matching for Autonomous Driving Motion Planning
Ananya Trivedi, Anjian Li, Mohamed Elnoor et al. (+7 more)
Autonomous driving requires
reasoning about interactions with surrounding traffic. A prevailing approach is large-scale imitation learning on expert driving
datasets, aimed at generalizing across diverse real-world scenarios. For online trajectory generation, such methods must operate at real-time rates.
Diffusion models require hundreds of denoising steps at inference, resulting in high latency. Consistency models mitigate this issue but rely on carefully tuned noise schedules to capture the
multimodal action distributions common in autonomous driving. Adapting the schedule, typically requires expensive retraining. To address these limitations, we propose a framework based on conditional flow matching that jointly predicts future motions of surrounding
agents and plans the ego trajectory in real time. We train a lightweight variance
estimator that selects the number of inference steps online, removing the need for retraining to balance runtime and imitation learning performance. To further enhance ride quality, we introduce a trajectory post-processing step cast as a convex quadratic program, with negligible computational overhead. Trained on the Waymo Open Motion
Dataset, the framework performs maneuvers such as lane changes, cruise control, and navigating unprotected left turns without requiring scenario-specific tuning. Our method maintains a 20 Hz update rate on an NVIDIA RTX 3070 GPU, making it suitable for online deployment. Compared to
transformer,
diffusion, and consistency model baselines, we achieve improved trajectory smoothness and better adherence to dynamic constraints. Experiment
videos and code implementations can be found at https://flow-matching-self-driving.github.io/.
Read the full abstract ▾
All authors Ananya Trivedi, Anjian Li, Mohamed Elnoor, Yusuf Umut Ciftci, Avinash Singh, Jovin D'sa, Sangjae Bae, David Isele, Taskin Padir, Faizan M. Tariq
Robotics · Cross-listed from physics.soc-ph
Transforming Police-Car Swerving for Mitigating Isolated Stop-and-Go Traffic Waves: A Practice-Oriented Jam-Absorption Driving Strategy
Zhengbing He
Stop-and-go traffic waves, a major form of freeway congestion, impose severe and persistent adverse impacts, including reduced traffic efficiency, increased safety risks, and elevated vehicle emissions. Among various freeway traffic management strategies, jam-absorption driving (JAD), in which a dedicated vehicle performs "slow-in" and "fast-out" maneuvers before being captured by a stop-and-go wave, has been proposed as a promising approach to suppressing the propagation of such waves. However, most existing JAD strategies remain impractical, primarily due to the lack of consideration of implementation vehicles and operational conditions. Inspired by real-world observations of police-car swerving behavior, this paper first introduces the Single-Vehicle Double-Detector Jam-Absorption Driving (SD-JAD) problem and then proposes a practical JAD strategy based on a definition of the JAD Triangle, transforming such behavior into a traffic control strategy capable of suppressing the propagation of an isolated stop-and-go wave. Five key parameters that significantly affect the proposed strategy, namely JAD speed, inflow traffic speed, wave width, wave speed, and in-wave speed, are identified and systematically analyzed. Using a SUMO-based simulation as an illustrative example, we further demonstrate how these parameters can be measured in practice using only two stationary roadside traffic detectors. The results show that the proposed JAD strategy successfully suppresses the propagation of a stop-and-go wave without triggering secondary waves. This paper is expected to take a significant step toward the practical implementation of JAD, advancing it from a theoretical concept to a feasible and deployable traffic management strategy.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
SAGE: Scalable Agentic 3D Scene Generation for Embodied AI
Hongchi Xia, Xuan Li, Zhaoshuo Li et al. (+9 more)
Real-world data collection for
embodied agents remains costly and unsafe, calling for scalable, realistic, and simulator-ready 3D environments. However, existing scene-generation systems often rely on rule-based or task-specific pipelines, yielding artifacts and physically invalid scenes. We present SAGE, an agentic framework that, given a user-specified
embodied task (e.g., "pick up a bowl and place it on the table"), understands the intent and automatically generates simulation-ready environments at scale. The
agent couples multiple generators for layout and object composition with critics that evaluate semantic plausibility, visual realism, and physical stability. Through iterative
reasoning and adaptive tool selection, it self-refines the scenes until meeting user intent and physical validity. The resulting environments are realistic, diverse, and directly deployable in modern simulators for policy training. Policies trained purely on this data exhibit clear scaling trends and generalize to unseen objects and layouts, demonstrating the promise of simulation-driven scaling for
embodied AI. Code, demos, and the SAGE-10k
dataset can be found on the project page here: https://research.nvidia.com/labs/dir/sage/.
Read the full abstract ▾
All authors Hongchi Xia, Xuan Li, Zhaoshuo Li, Qianli Ma, Jiashu Xu, Ming-Yu Liu, Yin Cui, Tsung-Yi Lin, Wei-Chiu Ma, Shenlong Wang, Shuran Song, Fangyin Wei
Robotics
MR-STORM: Scalable Multi-Arm Control By Distributed MPC
Dan Evron, Elias Goldsztejn, Dan R. Suissa et al. (+1 more)
Centralized motion planning
algorithms struggle to scale in multi-arm manipulation, where inter-arm coupling and tight workspaces amplify complexity. We introduce MR-STORM, a distributed sampling-based MPC framework that leverages massively parallel GPU sampling for arm control. Our approach integrates inter-arm collision avoidance via plan-sharing and a dynamic priority scheme to maintain task efficiency. We show empirically that MR-STORM outperforms existing baselines in complex simulation environments and that it transfers effectively to physical hardware. Code, demos and data are available at: https://roboworkshop.github.io/multi-
robot-mpc/
Read the full abstract ▾
All authors Dan Evron, Elias Goldsztejn, Dan R. Suissa, Ronen I. Brafman
Robotics
Learning Agile Quadrotor Flight in the Real World
Yunfan Ren, Zhiyuan Zhu, Jiaxu Xing et al. (+1 more)
Learning-based controllers have achieved impressive performance in agile quadrotor flight but typically rely on massive training in simulation, necessitating accurate system identification for effective Sim2Real transfer. However, even with precise modeling, fixed policies remain susceptible to out-of-distribution scenarios, ranging from external aerodynamic disturbances to internal hardware degradation. To ensure safety under these evolving uncertainties, such controllers are forced to operate with conservative safety margins, inherently constraining their agility outside of controlled settings. While online adaptation offers a potential remedy, safely exploring physical limits remains a critical bottleneck due to data scarcity and safety risks. To bridge this gap, we propose a self-adaptive framework that eliminates the need for precise system identification or offline Sim2Real transfer. We introduce Adaptive Temporal Scaling (ATS) to actively explore platform physical limits, and employ online residual learning to augment a simple nominal model. {Based on the learned hybrid model, we further propose Real-world Anchored Short-horizon Backpropagation Through Time (RASH-BPTT) to achieve efficient and robust in-flight policy updates. Extensive experiments demonstrate that our quadrotor reliably executes agile maneuvers near actuator saturation limits. The system evolves a conservative base policy with a peak speed of 1.9 m/s to 7.3 m/s within approximately 100 seconds of flight time. These findings underscore that real-world adaptation serves not merely to compensate for modeling errors, but as a practical mechanism for sustained performance improvement in aggressive flight regimes.
Read the full abstract ▾
All authors Yunfan Ren, Zhiyuan Zhu, Jiaxu Xing, Davide Scaramuzza
Robotics
ST4VLA: Spatially Guided Training for Vision-Language-Action Models
Jinhui Ye, Fangjing Wang, Ning Gao et al. (+9 more)
Large
vision-
language models (VLMs) excel at
multimodal understanding but fall short when extended to
embodied tasks, where instructions must be transformed into low-level motor actions. We introduce ST4VLA, a dual-system
Vision-
Language-Action framework that leverages Spatial Guided Training to align action learning with spatial priors in VLMs. ST4VLA includes two stages: (i) spatial grounding pre-training, which equips the VLM with transferable priors via scalable point, box, and trajectory prediction from both web-scale and
robot-specific data, and (ii) spatially guided action post-training, which encourages the model to produce richer spatial priors to guide action generation via spatial prompting. This design preserves spatial grounding during policy learning and promotes consistent optimization across spatial and action objectives. Empirically, ST4VLA achieves substantial improvements over vanilla VLA, with performance increasing from 66.1 -> 84.6 on Google
Robot and from 54.7 -> 73.2 on WidowX
Robot, establishing new state-of-the-art results on SimplerEnv. It also demonstrates stronger generalization to unseen objects and paraphrased instructions, as well as robustness to long-horizon perturbations in real-world settings. These results highlight scalable spatially guided training as a promising direction for robust, generalizable
robot learning. Source code, data and models are released at https://internrobotics.github.io/internvla-m1.github.io/
Read the full abstract ▾
All authors Jinhui Ye, Fangjing Wang, Ning Gao, Junqiu Yu, Yangkun Zhu, Bin Wang, Jinyu Zhang, Weiyang Jin, Yanwei Fu, Feng Zheng, Yilun Chen, Jiangmiao Pang
Robotics
EgoHumanoid: Unlocking In-the-Wild Loco-Manipulation with Robot-Free Egocentric Demonstration
Modi Shi, Shijia Peng, Jin Chen et al. (+6 more)
Human demonstrations offer rich environmental diversity and scale naturally, making them an appealing alternative to
robot teleoperation. While this paradigm has advanced
robot-arm manipulation, its potential for the more challenging, data-hungry problem of
humanoid loco-manipulation remains largely unexplored. We present EgoHumanoid, the first framework to co-train a
vision-
language-action policy using abundant
egocentric human demonstrations together with a limited amount of
robot data, enabling
humanoids to perform loco-manipulation across diverse real-world environments. To bridge the embodiment gap between humans and
robots, including discrepancies in physical morphology and viewpoint, we introduce a systematic
alignment pipeline spanning from hardware design to data processing. A portable system for scalable human data collection is developed, and we establish practical collection protocols to improve transferability. At the core of our human-to-
humanoid alignment pipeline lies two key components. The view
alignment reduces visual domain discrepancies caused by camera height and perspective variation. The action
alignment maps human motions into a unified, kinematically feasible action space for
humanoid control. Extensive real-world experiments demonstrate that incorporating
robot-free
egocentric data significantly outperforms
robot-only baselines by 51%, particularly in unseen environments. Our analysis further reveals which behaviors transfer effectively and the potential for scaling human data.
Read the full abstract ▾
All authors Modi Shi, Shijia Peng, Jin Chen, Haoran Jiang, Tianyu Li, Di Huang, Ping Luo, Hongyang Li, Li Chen
Robotics
DexImit: Learning Bimanual Dexterous Manipulation from Monocular Human Videos
Juncheng Mu, Sizhe Yang, Yiming Bao et al. (+6 more)
Data scarcity fundamentally limits the generalization of bimanual dexterous manipulation, as real-world data collection for dexterous hands is expensive and labor-intensive. Human manipulation
videos, as a direct carrier of manipulation knowledge, offer significant potential for scaling up
robot learning. However, the substantial embodiment gap between human hands and robotic dexterous hands makes direct pretraining from human
videos extremely challenging. To bridge this gap and unleash the potential of large-scale human manipulation
video data, we propose DexImit, an automated framework that converts
monocular human manipulation
videos into physically plausible
robot data, without any additional information. DexImit employs a four-stage generation pipeline: (1) reconstructing hand-object interactions from arbitrary viewpoints with near-metric scale; (2) performing subtask decomposition and bimanual scheduling; (3) synthesizing
robot trajectories consistent with the demonstrated interactions; (4) comprehensive data augmentation for zero-shot real-world deployment. Building on these designs, DexImit can generate large-scale
robot data based on human
videos, either from the Internet or
video generation models. DexImit is capable of handling diverse manipulation tasks, including tool use (e.g., cutting an apple), long-horizon tasks (e.g., making a beverage), and fine-grained manipulations (e.g., stacking cups).
Read the full abstract ▾
All authors Juncheng Mu, Sizhe Yang, Yiming Bao, Hojin Bae, Tianming Wei, Linning Xu, Boyi Li, Huazhe Xu, Jiangmiao Pang
Robotics
Robo3R: Enhancing Robotic Manipulation with Accurate Feed-Forward 3D Reconstruction
Sizhe Yang, Linning Xu, Hao Li et al. (+4 more)
3D spatial perception is fundamental to generalizable robotic manipulation, yet obtaining reliable, high-quality 3D geometry remains challenging. Depth sensors suffer from noise and material sensitivity, while existing reconstruction models lack the precision and metric consistency required for physical interaction. We introduce Robo3R, a feed-forward, manipulation-ready 3D reconstruction model that predicts accurate, metric-scale scene geometry directly from RGB images and
robot states in real time. Robo3R jointly infers scale-invariant local geometry and relative camera poses, which are unified into the scene representation in the canonical
robot frame via a learned global similarity transformation. To meet the precision demands of manipulation, Robo3R employs a masked point head for sharp, fine-grained point clouds, and a keypoint-based Perspective-n-Point (PnP) formulation to refine camera extrinsics and global
alignment. Trained on Robo3R-4M, a curated large-scale synthetic
dataset with four million high-fidelity annotated frames, Robo3R consistently outperforms state-of-the-art reconstruction methods and depth sensors. Across downstream tasks including imitation learning, sim-to-real transfer, grasp synthesis, and collision-free motion planning, we observe consistent gains in performance, suggesting the promise of this alternative 3D sensing module for robotic manipulation.
Read the full abstract ▾
All authors Sizhe Yang, Linning Xu, Hao Li, Juncheng Mu, Jia Zeng, Dahua Lin, Jiangmiao Pang
Robotics
VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model
Jingwen Sun, Wenyao Zhang, Zekun Qi et al. (+6 more)
Pretraining
Vision-
Language-Action (VLA) policies on internet-scale
video is appealing, yet current latent-action objectives often learn the wrong thing: they remain anchored to pixel variation rather than action-relevant state transitions, making them vulnerable to appearance bias, nuisance motion, and information leakage. We introduce VLA-JEPA, a JEPA-style pretraining framework that sidesteps these pitfalls by design. The key idea is leakage-free state prediction: a target encoder produces latent representations from future frames, while the student pathway sees only the current observation – future information is used solely as supervision targets, never as input. By predicting in latent space rather than pixel space, VLA-JEPA learns dynamics abstractions that are robust to camera motion and irrelevant background changes. This yields a simple two-stage recipe – JEPA pretraining followed by action-head fine-tuning – without the multi-stage complexity of prior latent-action pipelines. Experiments on LIBERO, LIBERO-Plus, SimplerEnv and real-world manipulation tasks show that VLA-JEPA achieves consistent gains in generalization and robustness over existing methods.
Read the full abstract ▾
All authors Jingwen Sun, Wenyao Zhang, Zekun Qi, Shaojie Ren, Zezhi Liu, Hanxin Zhu, Guangzhong Sun, Xin Jin, Zhibo Chen
Robotics
UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking
Baijun Chen, Weijie Wan, Tianxing Chen et al. (+13 more)
Robotic manipulation has seen rapid progress with
vision-
language-action (VLA) policies. However, visuo-tactile perception is critical for contact-rich manipulation, as tasks such as insertion are difficult to complete robustly using
vision alone. At the same time, acquiring large-scale and reliable tactile data in the physical world remains costly and challenging, and the lack of a unified evaluation platform further limits policy learning and systematic analysis. To address these challenges, we propose UniVTAC, a simulation-based visuo-tactile data synthesis platform that supports three commonly used visuo-tactile sensors and enables scalable and controllable generation of informative contact interactions. Based on this platform, we introduce the UniVTAC Encoder, a visuo-tactile encoder trained on large-scale simulation-synthesized data with designed supervisory signals, providing tactile-centric visuo-tactile representations for downstream manipulation tasks. In addition, we present the UniVTAC
Benchmark, which consists of eight representative visuo-tactile manipulation tasks for evaluating tactile-driven policies. Experimental results show that integrating the UniVTAC Encoder improves average success rates by 17.1% on the UniVTAC
Benchmark, while real-world robotic experiments further demonstrate a 25% improvement in task success. Our webpage is available at https://univtac.github.io/.
Read the full abstract ▾
All authors Baijun Chen, Weijie Wan, Tianxing Chen, Xianda Guo, Congsheng Xu, Yuanyang Qi, Haojie Zhang, Longyan Wu, Tianling Xu, Zixuan Li, Yizhe Wu, Rui Li, Xiaokang Yang, Ping Luo, Wei Sui, Yao Mu
Robotics
Humanoid Factors: Design Principles for AI Humanoids in Human Worlds
Xinyuan Liu, Eren Sadikoglu, Ransalu Senanayake et al. (+1 more)
Human factors research has long focused on optimizing environments, tools, and systems to account for human performance. Yet, as
humanoid robots begin to share our workplaces, homes, and public spaces, the design challenge expands. We must now consider not only factors for humans but also factors for
humanoids, since both will coexist and interact within the same environments. Unlike conventional machines,
humanoids introduce expectations of human-like behavior, communication, and social presence, which reshape usability, trust, and safety considerations. In this article, we introduce the concept of
humanoid factors as a framework structured around four pillars - physical, cognitive, social, and ethical - that shape the development of
humanoids to help them effectively coexist and collaborate with humans. This framework characterizes the overlap and divergence between human capabilities and those of general-purpose
humanoids powered by AI foundation models. To demonstrate our framework's practical utility, we then apply the framework to evaluate a real-world
humanoid control
algorithm, illustrating how conventional task completion metrics in robotics overlook key human cognitive and interaction principles. We thus position
humanoid factors as a foundational framework for designing, evaluating, and governing sustained human-
humanoid coexistence.
Read the full abstract ▾
All authors Xinyuan Liu, Eren Sadikoglu, Ransalu Senanayake, Lixiao Huang
Robotics
A Collision-Free Sway Damping Model Predictive Controller for Safe and Reactive Forestry Crane Navigation
Marc-Philip Ecker, Christoph Fröhlich, Johannes Huemer et al. (+4 more)
Forestry cranes operate in dynamic, unstructured outdoor environments where simultaneous collision avoidance and payload sway control are critical for safe
navigation. Existing approaches address these challenges separately, either focusing on sway damping with predefined collision-free paths or performing collision avoidance only at the global planning level. We present the first collision-free, sway-damping model predictive controller (MPC) for a forestry crane that unifies both objectives in a single control framework. Our approach integrates
LiDAR-based environment mapping directly into the MPC using online Euclidean distance fields (EDF), enabling real-time environmental adaptation. The controller simultaneously enforces collision constraints while damping payload sway, allowing it to (i) replan upon quasi-static environmental changes, (ii) maintain collision-free operation under disturbances, and (iii) provide safe stopping when no bypass exists. Experimental validation on a real forestry crane demonstrates effective sway damping and successful obstacle avoidance. A
video can be found at https://youtu.be/tEXDoeLLTxA.
Read the full abstract ▾
All authors Marc-Philip Ecker, Christoph Fröhlich, Johannes Huemer, David Gruber, Bernhard Bischof, Tobias Glück, Wolfgang Kemmetmüller
Robotics · Cross-listed from cs.CV
Perception with Guarantees: Certified Pose Estimation via Reachability Analysis
Tobias Ladner, Yasser Shoukry and Matthias Althoff
Agents in cyber-physical systems are increasingly entrusted with safety-critical tasks. Ensuring safety of these
agents often requires localizing the pose for subsequent actions. Pose estimates can, e.g., be obtained from various combinations of
lidar sensors, cameras, and external services such as GPS. Crucially, in safety-critical domains, a rough estimate is insufficient to formally determine safety, i.e., guaranteeing safety even in the worst-case scenario, and external services might additionally not be trustworthy. We address this problem by presenting a certified pose estimation in 3D solely from a camera image and a well-known target geometry. This is realized by formally bounding the pose, which is computed by leveraging recent results from reachability analysis and formal neural network verification. Our experiments demonstrate that our approach efficiently and accurately localizes
agents in both synthetic and real-world experiments.
Read the full abstract ▾
Robotics
RoboSubtaskNet: Temporal Sub-task Segmentation for Human-to-Robot Skill Transfer in Real-World Environments
Dharmendra Sharma, Archit Sharma, John Rebeiro et al. (+4 more)
Temporally locating and classifying fine-grained sub-task segments in long, untrimmed
videos is crucial to safe human-
robot collaboration. Unlike generic activity recognition, collaborative manipulation requires sub-task labels that are directly
robot-executable. We present RoboSubtaskNet, a multi-stage human-to-
robot sub-task
segmentation framework that couples
attention-enhanced I3D features (RGB plus optical flow) with a modified MS-TCN employing a Fibonacci dilation schedule to capture better short-horizon transitions such as reach-pick-place. The network is trained with a composite objective comprising cross-
entropy and temporal regularizers (truncated MSE and a transition-aware term) to reduce over-
segmentation and to encourage valid sub-task progressions. To close the gap between
vision benchmarks and control, we introduce RoboSubtask, a
dataset of healthcare and industrial demonstrations annotated at the sub-task level and designed for deterministic mapping to manipulator primitives. Empirically, RoboSubtaskNet outperforms MS-TCN and MS-TCN++ on GTEA and our RoboSubtask
benchmark (boundary-sensitive and sequence metrics), while remaining competitive on the long-horizon Breakfast
benchmark. Specifically, RoboSubtaskNet attains F1 @ 50 = 79.5%, Edit = 88.6%, Acc = 78.9% on GTEA; F1 @ 50 = 30.4%, Edit = 52.0%, Acc = 53.5% on Breakfast; and F1 @ 50 = 94.2%, Edit = 95.6%, Acc = 92.2% on RoboSubtask. We further validate the full perception-to-execution pipeline on a 7-DoF Kinova Gen3 manipulator, achieving reliable end-to-end behavior in physical trials (overall task success approx 91.25%). These results demonstrate a practical path from sub-task level
video understanding to deployed robotic manipulation in real-world settings.
Read the full abstract ▾
All authors Dharmendra Sharma, Archit Sharma, John Rebeiro, Vaibhav Kesharwani, Peeyush Thakur, Narendra Kumar Dhar, Laxmidhar Behera
Robotics
Learning Force-Regulated Robotic Manipulation with a Low-Cost Tactile-Force-Controlled Gripper
Xuhui Kang, Tongxuan Tian, Sung-Wook Lee et al. (+3 more)
Successfully manipulating many everyday objects, such as potato chips, requires precise force regulation. Failure to modulate force can lead to task failure or irreversible damage to the objects. Humans can precisely achieve this by adapting force from tactile feedback, even within a short period of physical contact. We aim to give
robots this capability. However, commercial grippers exhibit high cost or high minimum force, making them unsuitable for studying force-controlled policy learning with everyday force-sensitive objects. We introduce TF-Gripper, a low-cost (~$150) force-controlled parallel-jaw gripper that integrates tactile sensing as feedback. It has an effective force range of 0.45-45 N and is compatible with different
robot arms. Additionally, we designed a teleoperation device paired with TF-Gripper to record human-applied grasping forces. While we can train standard low-frequency policies with the collected force data, achieving reliable performance remains challenging due to the reactive and contact-dependent nature of force-regulated manipulation. To overcome this, we propose RETAF (REactive Tactile Adaptation of Force), a framework that decouples grasping force control from arm pose prediction. RETAF regulates force at high frequency using wrist images and tactile feedback, while a base policy predicts end-effector pose and gripper open/close action. Our experiments show that, compared to position control, direct force control with TF-Gripper improves grasp stability and overall task performance across six real-world tasks. We further show that tactile feedback is essential for force regulation, and that RETAF consistently outperforms baselines and can be integrated with various base policies. We hope this work opens a path for scaling the learning of force-controlled policies in robotic manipulation. Project page: https://force-gripper.github.io .
Read the full abstract ▾
All authors Xuhui Kang, Tongxuan Tian, Sung-Wook Lee, Binghao Huang, Yunzhu Li, Yen-Ling Kuo
Robotics
A Collaborative Safety Shield for Safe and Efficient CAV Lane Changes in Congested On-Ramp Merging
Bharathkumar Hegde and Melanie Bouroche
Lane changing in dense traffic is a significant challenge for Connected and Autonomous Vehicles (CAVs). Existing lane change controllers primarily either ensure safety or collaboratively improve traffic efficiency, but do not consider these conflicting objectives together. To address this, we propose the Multi-
Agent Safety Shield (MASS), designed using Control Barrier Functions (CBFs) to enable safe and collaborative lane changes. The MASS enables collaboration by capturing multi-
agent interactions among CAVs through interaction
topologies constructed as a graph using a simple
algorithm. Further, a state-of-the-art Multi-
Agent Reinforcement Learning (MARL) lane change controller is extended by integrating MASS to ensure safety and defining a customised reward function to prioritise efficiency improvements. As a result, we propose a lane change controller, known as MARL-MASS, and evaluate it in a congested on-ramp merging simulation. The results demonstrate that MASS enables collaborative lane changes with safety guarantees by strictly respecting the safety constraints. Moreover, the proposed custom reward function improves the stability of MARL policies trained with a safety shield. Overall, by encouraging the exploration of a collaborative lane change policy while respecting safety constraints, MARL-MASS effectively balances the trade-off between ensuring safety and improving traffic efficiency in congested traffic. The code for MARL-MASS is available with an open-source licence at https://github.com/hkbharath/MARL-MASS
Read the full abstract ▾
Robotics
Acoustic Drone Package Delivery Detection
François Marcoux and François Grondin
In recent years, the illicit use of unmanned aerial vehicles (
UAVs) for deliveries in restricted area such as prisons became a significant security challenge. While numerous studies have focused on
UAV detection or localization, little
attention has been given to delivery events identification. This study presents the first acoustic package delivery detection
algorithm using a ground-based microphone array. The proposed method estimates both the drone's propeller speed and the delivery event using solely acoustic features. A deep neural network detects the presence of a drone and estimates the propeller's rotation speed or blade passing frequency (BPF) from a mel spectrogram. The
algorithm analyzes the BPFs to identify probable delivery moments based on sudden changes before and after a specific time. Results demonstrate a mean absolute error of the blade passing frequency
estimator of 16 Hz when the drone is less than 150 meters away from the microphone array. The drone presence detection
estimator has a accuracy of 97%. The delivery detection
algorithm correctly identifies 96% of events with a false positive rate of 8%. This study shows that deliveries can be identified using acoustic signals up to a range of 100 meters.
Read the full abstract ▾
Robotics
RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation
Hao Li, Ziqin Wang, Zi-han Ding et al. (+9 more)
Advances in large
vision-
language models (VLMs) have stimulated growing interest in
vision-
language-action (VLA) systems for
robot manipulation. However, existing manipulation
datasets remain costly to curate, highly embodiment-specific, and insufficient in coverage and diversity, thereby hindering the generalization of VLA models. Recent approaches attempt to mitigate these limitations via a plan-then-execute paradigm, where high-level plans (e.g., subtasks, trace) are first generated and subsequently translated into low-level actions, but they critically rely on extra intermediate supervision, which is largely absent from existing
datasets. To bridge this gap, we introduce the RoboInter Manipulation Suite, a unified resource including data,
benchmarks, and models of intermediate representations for manipulation. It comprises RoboInter-Tool, a lightweight GUI that enables semi-automatic annotation of diverse representations, and RoboInter-Data, a large-scale
dataset containing over 230k episodes across 571 diverse scenes, which provides dense per-frame annotations over more than 10 categories of intermediate representations, substantially exceeding prior work in scale and annotation quality. Building upon this foundation, RoboInter-VQA introduces 9 spatial and 20 temporal
embodied VQA categories to systematically
benchmark and enhance the
embodied reasoning capabilities of VLMs. Meanwhile, RoboInter-VLA offers an integrated plan-then-execute framework, supporting modular and end-to-end VLA variants that bridge high-level planning with low-level execution via intermediate supervision. In total, RoboInter establishes a practical foundation for advancing robust and generalizable robotic learning via fine-grained and diverse intermediate representations.
Read the full abstract ▾
All authors Hao Li, Ziqin Wang, Zi-han Ding, Shuai Yang, Yilun Chen, Yang Tian, Xiaolin Hu, Tai Wang, Dahua Lin, Feng Zhao, Si Liu, Jiangmiao Pang
Robotics
Hydra-Nav: Object Navigation via Adaptive Dual-Process Reasoning
Zixuan Wang, Huang Fang, Shaoan Wang et al. (+4 more)
While large
vision-
language models (VLMs) show promise for object goal
navigation, current methods still struggle with low success rates and inefficient localization of unseen objects–failures primarily attributed to weak temporal-spatial
reasoning. Meanwhile, recent attempts to inject
reasoning into VLM-based
agents improve success rates but incur substantial computational overhead. To address both the ineffectiveness and inefficiency of existing approaches, we introduce Hydra-Nav, a unified VLM architecture that adaptively switches between a deliberative slow system for analyzing exploration history and formulating high-level plans, and a reactive fast system for efficient execution. We train Hydra-Nav through a three-stage curriculum: (i) spatial-action
alignment to strengthen trajectory planning, (ii)
memory-
reasoning integration to enhance temporal-spatial
reasoning over long-horizon exploration, and (iii) iterative rejection fine-tuning to enable selective
reasoning at critical decision points. Extensive experiments demonstrate that Hydra-Nav achieves state-of-the-art performance on the HM3D, MP3D, and OVON
benchmarks, outperforming the second-best methods by 11.1%, 17.4%, and 21.2%, respectively. Furthermore, we introduce SOT (Success weighted by Operation Time), a new metric to measure search efficiency across VLMs with varying
reasoning intensity. Results show that adaptive
reasoning significantly enhances search efficiency over fixed-frequency baselines.
Read the full abstract ▾
All authors Zixuan Wang, Huang Fang, Shaoan Wang, Yuanfei Luo, Heng Dong, Wei Li, Yiming Gan
Robotics
Instruct2Act: From Human Instruction to Actions Sequencing and Execution via Robot Action Network for Robotic Manipulation
Archit Sharma, Dharmendra Sharma, John Rebeiro et al. (+3 more)
Robots often struggle to follow free-form human instructions in real-world settings due to computational and sensing limitations. We address this gap with a lightweight, fully on-device pipeline that converts natural-
language commands into reliable manipulation. Our approach has two stages: (i) the instruction to actions module (Instruct2Act), a compact BiLSTM with a multi-head-
attention autoencoder that parses an instruction into an ordered sequence of atomic actions (e.g., reach, grasp, move, place); and (ii) the
robot action network (RAN), which uses the dynamic adaptive trajectory radial network (DATRN) together with a
vision-based environment analyzer (YOLOv8) to generate precise control trajectories for each sub-action. The entire system runs on a modest system with no cloud services. On our custom proprietary
dataset, Instruct2Act attains 91.5% sub-actions prediction accuracy while retaining a small footprint. Real-
robot evaluations across four tasks (pick-place, pick-pour, wipe, and pick-give) yield an overall 90% success; sub-action inference completes in < 3.8s, with end-to-end executions in 30-60s depending on task complexity. These results demonstrate that fine-grained instruction-to-action parsing, coupled with DATRN-based trajectory generation and
vision-guided grounding, provides a practical path to deterministic, real-time manipulation in resource-constrained, single-camera settings.
Read the full abstract ▾
All authors Archit Sharma, Dharmendra Sharma, John Rebeiro, Peeyush Thakur, Narendra Dhar, Laxmidhar Behera
Robotics
TaCo: A Benchmark for Lossless and Lossy Codecs of Heterogeneous Tactile Data
Zhengxue Cheng, Yan Zhao, Keyu Wang et al. (+2 more)
Tactile sensing is crucial for
embodied intelligence, providing fine-grained perception and control in complex environments. However, efficient tactile data compression, which is essential for real-time robotic applications under strict bandwidth constraints, remains underexplored. The inherent heterogeneity and spatiotemporal complexity of tactile data further complicate this challenge. To bridge this gap, we introduce TaCo, the first comprehensive
benchmark for Tactile data Codecs. TaCo evaluates 30 compression methods, including off-the-shelf compression
algorithms and neural codecs, across five diverse
datasets from various sensor types. We systematically assess both lossless and lossy compression schemes on four key tasks: lossless storage, human visualization, material and object classification, and dexterous robotic grasping. Notably, we pioneer the development of data-driven codecs explicitly trained on tactile data, TaCo-LL (lossless) and TaCo-L (lossy). Results have validated the superior performance of our TaCo-LL and TaCo-L. This
benchmark provides a foundational framework for understanding the critical trade-offs between compression efficiency and task performance, paving the way for future advances in tactile perception.
Read the full abstract ▾
All authors Zhengxue Cheng, Yan Zhao, Keyu Wang, Hengdi Zhang, Li Song
Robotics
TriPilot-FF: Coordinated Whole-Body Teleoperation with Force Feedback
Zihao Li, Yanan Zhou, Ranpeng Qiu et al. (+3 more)
Mobile manipulators broaden the operational envelope for
robot manipulation. However, the whole-body teleoperation of such
robots remains a problem: operators must coordinate a wheeled base and two arms while
reasoning about obstacles and contact. Existing interfaces are predominantly hand-centric (e.g., VR controllers and joysticks), leaving foot-operated channels underexplored for continuous base control. We present TriPilot-FF, an open-source whole-body teleoperation system for a custom bimanual mobile manipulator that introduces a foot-operated pedal with
lidar-driven pedal haptics, coupled with upper-body bimanual leader-follower teleoperation. Using only a low-cost base-mounted
lidar, TriPilot-FF renders a resistive pedal cue from proximity-to-obstacle signals in the commanded direction, shaping operator commands toward collision-averse behaviour without an explicit collision-avoidance controller. The system also supports arm-side force reflection for contact awareness and provides real-time force and visual guidance of bimanual manipulability to
prompt mobile base repositioning, thereby improving reach. We demonstrate the capability of TriPilot-FF to effectively “co-pilot” the human operator over long time-horizons and tasks requiring precise mobile base movement and coordination. Finally, we incorporate teleoperation feedback signals into an Action Chunking with
Transformers (ACT) policy and demonstrate improved performance when the additional information is available. We release the pedal device design, full software stack, and conduct extensive real-world evaluations on a bimanual wheeled platform. The project page of TriPilot-FF is http://bit.ly/46H3ZJT.
Read the full abstract ▾
All authors Zihao Li, Yanan Zhou, Ranpeng Qiu, Hangyu Wu, Guoqiang Ren, Weiming Zhi