Robotics
Safe Multi-Robot Coordination via VLM-LLM Reasoning and Reachability Analysis
Mohamed Dwedar, Ahmad Hafez, Alexander Jesser et al. (+1 more)
Safe coordination in heterogeneous machine-to-machine (M2M) robotic systems is challenging when
robots differ in sensing capabilities, environmental awareness, and motion execution roles. This paper presents a centralized safety-aware M2M framework for cooperative goal-directed navigation in a heterogeneous mobile
robot team comprising a
vision-capable quadruped and a camera-less robotic vehicle. The objective is to guide both platforms toward a goal region while avoiding static and dynamic obstacles and preventing unsafe inter-
robot interactions. Under the principle of shared perception, the
vision-capable
robot provides semantic environmental awareness through a centralized server over an MQTT broker, enabling the camera-less platform to navigate using this shared scene representation alongside its own odometry, IMU, and state feedback. A
vision-
language model (VLM) interprets the visual stream, and the extracted semantic data is mapped into conservative metric
geometric constraints, including inflated obstacle sets, safe corridors, and goal regions. A large
language model (LLM) proposes high-level task allocations, while physical command authority is restricted to a
robot-specific zonotope reachability gate. This verification engine propagates independent reachable tubes to evaluate obstacle avoidance, safe-corridor containment, and inter-
robot separation predicates before approving commands. Online experiments across clear-path and dynamic-obstacle scenarios show that the pipeline reliably approves safe motion, triggers conservative replanning or holding maneuvers upon constraint violation, and enforces a strict architectural separation between advisory semantic
reasoning and formally verified motor execution.
Read the full abstract ▾
All authors Mohamed Dwedar, Ahmad Hafez, Alexander Jesser, Amr Alanwar
Robotics
Task-Prototype Guided Flow Matching for Few-Shot Generalization in Vision-Language Robot Manipulation
Yizhao Wang, Guantao Zhang and Jingbo Wang
Vision-
language robot manipulation policies can follow semantic instructions, but adapting them to a new procedure from only a few demonstrations remains difficult because
language underspecifies contact timing, motion phases, corrective behavior, and execution style. This paper presents Task-Prototype Guided Flow Matching (TP-Flow), a few-shot manipulation framework that converts support demonstrations into structured task-prototype
tokens and uses them to guide both the initial flow prior and the velocity field. TP-Flow employs symmetric cross-
attention with learnable queries to extract phase-level prototypes, parameterizes a task-adaptive initial distribution, and injects prototype information through gated adaptive normalization. It is trained with an episodic support-query objective and prototype contrastive regularization, so few-shot adaptation is simulated during training while nuisance information is suppressed. On the LEROBOT-ARM-SO101 platform, TP-Flow achieves 66.8%, 79.6%, and 82.1% success rates under 1-, 4-, and 6-shot settings, with a 75.5% few-shot AUC. At 1-shot, it improves over CFM, Pooled-Demo CFM, and In-Context Flow by 29.8, 14.5, and 9.9 percentage points. It also improves held-out target-group generalization across novel-object transfer, goal recombination, long-horizon composition, and contact/correction tasks. TP-Flow maintains real-time execution with six online prototype
tokens, 54.3 ms latency, 3.9 GB peak
memory, and a 10 Hz control rate, while reducing the noisy-support success drop to 6.2%. Theoretical diagnostics show that prototype distance aligns with action-distribution distance, the adaptive prior reduces transport cost, and gated modulation keeps measured trajectory deviations below the derived ODE bound. The code repository is omitted for anonymous review.
Read the full abstract ▾
Robotics
Constructing Predictive Surgical Path for AI-based Capsulorhexis Skill Transfer
Mohammad Javad Ahmadi and Hamid D. Taghirad
Automated training of surgeons is one of the most crucial factors that significantly minimize surgical training risks and expenses. With recent advances in artificial intelligence (AI) knowledge and available data from various surgeries, AI's involvement in surgical training is becoming very promising. It is recommended that at the early stages of AI development, it interferes in the surgery as a third
agent alongside the trainer. As trust in AI increases, this process will lead to an AI
agent acting as a trainer in the future. The first phase in which AI can intervene in the training process is to suggest an improved surgical path to the trainer. A platform must be constructed in the first step, to accomplish this task and to enhance the movement path of trainee surgeons. This paper introduces this platform along with an annotated capsulorhexis surgery
dataset called the ARAS-Farabi
dataset. In this research, a deep convolutional neural network is pre-trained with JIGSAWS and ARAS-Farabi surgical
datasets that can extract surgical skill characteristics from surgery tool tip motion data. The proposed platform develops a reference model from the feature space of an expert surgeon's movement trajectory and proposes an improved path to enhance the skill of a novice surgeon. An optimization with two loss functions is utilized to create a path that raises the skill level of the novice surgeon's path while simultaneously predicting and preserving his/her intent. The results of this study reveal that, with the assistance of an AI
agent, the trainee surgeon's movement path can be enhanced by at least 20 percent while maintaining his intentional objective. In addition to the recommended deep network, various tangible indicators have also been developed in this research to verify the level of trainee improvement.
Read the full abstract ▾
Robotics
Geo-VLA: Geometry-Aware Vision-Language-Action Planning via Internalization of Map Semantics
Ran Chen, Jiaxing Ren, Zhikun Zhang et al. (+3 more)
Vision-
language-action (VLA) models have advanced end-to-end autonomous driving by leveraging foundation models for semantic
reasoning and long-tail generalization. However, their planning performance remains limited in complex driving environments because image-only representations inadequately capture planning-relevant road geometry and
topology. In this paper, we propose Geo-VLA, a plug-and-play framework that enhances VLA models by learning geometry-aware visual representations. During training, Geo-VLA internalizes
geometric map semantics to strengthen road-structure representations, while requiring no HD maps or additional lane information during inference. To support this approach, we introduce Geo-QA, a geometry-focused question-answering
dataset that injects road geometry into
vision-
language representations through contrastive learning and instruction tuning. Experiments on NAVSIM v1 demonstrate that Geo-VLA consistently improves VLA planners with distinct action-generation architectures, achieving 92.1 PDMS and establishing a new state-of-the-art among single-camera VLA planners.
Read the full abstract ▾
All authors Ran Chen, Jiaxing Ren, Zhikun Zhang, Yunhao Hou, Junbao Zhuo, Bochao Zou
Robotics
A Task-Agnostic Control Strategy for Dynamic Assistance with Pneumatically Actuated Soft Exosuits
Anoush Sepehri, Zachary Huang, Raymond de Callafon et al. (+2 more)
Pneumatic artificial muscles have provided new opportunities to develop upper-extremity soft exosuits for reha- bilitation, augmentation, and assisted daily living. However, the complex dynamics and limited bandwidth of these actuators has made providing responsive assistance based on user intention a longstanding challenge. In this work, we present an inverse-plant control strategy for pneumatically actuated soft exosuits that only relies on kinematic sensing for task-agnostic and dynamic assistance during daily living. We model the human-
robot system using a Hammerstein dynamic model, consisting of a Preisach hysteresis model and a linear time-invariant filter, to capture the static and dynamic behavior of the system. We personalize our model to each user using 140 s of data and approximate an inverse to integrate into our control loop. When evaluated on a test rig that emulated a soft assistive exosuit for the wrist, our controller reduced the interaction torque by up to 73% and the activation of key flexor and extensor muscles by up to 47% relative to the condition with no assistance for speeds ranging from 8{\deg}/s to 120{\deg}/s. Overall, this work presents a control strategy that can provide task-agnostic, dynamic assistance with pneumatically actuated soft exosuits without the need for physiological or force sensors to interpret user intention.
Read the full abstract ▾
All authors Anoush Sepehri, Zachary Huang, Raymond de Callafon, Michael T. Tolley, Tania K. Morimoto
Robotics · Cross-listed from cs.CV
Reproducible Multimodal Affordance Prediction
Tommaso Apicella, Alessio Xompero and Andrea Cavallaro
Affordance prediction is the identification of potential actions an
agent can perform on a target object from
multimodal inputs. Affordance prediction methods are difficult to evaluate and compare due to heterogeneous problem formulations, inconsistent
dataset annotations, incomplete reporting of experimental protocols, and limited information about deployment conditions. These limitations challenge fair benchmarking and performance comparison. To promote transparency, we propose the Affordance Sheet, a documentation detailing task formulation with its input modalities, model architectures and training information,
datasets, and experimental protocols. Affordance Sheets enable reproducible benchmarking and reliable evaluation of affordance models for real-world scenarios, including generalisation to novel conditions and human safety.
Read the full abstract ▾
Robotics
GuideFetch: A Task Coordination Framework for Concurrent Navigation and Object Retrieval in Assistive Robot Dogs
Qian Yin, Ruiping Liu, Kunyu Peng et al. (+7 more)
Consider one
robot guide dog escorting a blind user to a seat while a second retrieves and delivers an object. We introduce GuideFetch, a framework for coordinating this concurrent guide-and-fetch mission with heterogeneous
robots. A large
language model (LLM) instantiates a schedule-conditioned four-action schema; deterministic normalization and validation enforce registered targets,
robot capabilities, and the selected schedule, while
robot and object states govern execution and completion. We record 360 simulator runs over 90 scene–seed combinations under scripted and online plan-provenance conditions. All 180 online responses validate on the first request and match their scripted references, so the plan-provenance comparison tests normalized-plan agreement rather than a distinct execution factor. A simulator-free mutation test accepts two valid controls and rejects all 32 rule-violating variants. Across 90 scene–seed cases per schedule, sequential and parallel execution achieve $72/90$ and $71/90$ operational successes. Among 56 common successes, the implemented role-reassigned parallel protocol reduces mean makespan by 41.3%. This system-level gain combines role assignment, action overlap, and scene geometry; state checks distinguish plan validity from verified mission completion.
Read the full abstract ▾
All authors Qian Yin, Ruiping Liu, Kunyu Peng, Jianxiang Man, Isik Baran Sandan, Junwei Zheng, Yufan Chen, Di Wen, Kailun Yang, Rainer Stiefelhagen
Robotics
Transferable Tool-Tissue Contact Detection from Stereo Depth in Robot-Assisted Surgery
Mingyeung Wu, Zhonghao Zhang, Hao Yang et al. (+2 more)
Reliable tool–tissue contact detection can support interaction-aware control and downstream force estimation in
robot-assisted surgery. Most existing methods learn a contact classifier from RGB appearance, which is hard to generalize. In this work, we use the depth image generated from a stereo pair to give more information about tool–tissue contact. For each depth frame, we localize a spatially supported minimum-distance patch around the tool boundary and reduce it to a single scalar, $-\log_{10}|d|$; this signal rises and falls in step with ground-truth contact. We formalize this observation with a fully supervised two-state hidden Markov model. We fit this model as a six-fold leave-one-session-out (LOSO) ensemble on six palpation sessions against a single silicone cup-like phantom, with the decision threshold selected from the pooled out-of-fold predictions. It is evaluated on four held-out sessions of three categories: 1. same task on same phantom; 2. same task on different phantom; 3. different task on different phantom. This model reaches held-out macro F1 $0.927$ and AUPRC $0.980$. We further compare against a reproduction of an RGB-based contact classifier from prior work. This RGB-based model achieves high performance on the first category (F1 $0.965$), but substantially lower performance on the other two, resulting in macro F1 $0.320$ across all four sessions. These results indicate that the tool–tissue distance is a strong, transferable cue for contact detection in
robot-assisted surgery.
Read the full abstract ▾
All authors Mingyeung Wu, Zhonghao Zhang, Hao Yang, Alan Kuntz, Jie Ying Wu
Robotics
VERAGMIL: Virtual Environment for Scooping Granular Foods with Imitation Learning Models
Amanuel Ergogo, Diego Dall'Alba and Przemyslaw Korzeniowski
Robot-Assisted Feeding (RAF) systems are essential for assisting individuals with disabilities or motor impairments in eating tasks. Manipulating granular food items, such as rice and beans, poses significant challenges due to their dynamic physical properties. Learning from human demonstrations offers a promising solution, but acquiring high-quality demonstrations is complex. To address this, we present VERAGMIL, a framework that combines a high-fidelity simulator with an intuitive Virtual Reality (VR) interface for recording demonstrations and supporting different imitation learning methods. VERAGMIL provides a realistic environment for training RAF systems to handle granular materials, including
robots, sensors, and various food items with distinct physical characteristics. We evaluate VERAGMIL by training three imitation learning models, BC, BC-RNN, and BCQ, on granular scooping and transporting tasks using both VR interface and 3D space mouse demonstrations, comparing them with a human-expert baseline. The models are assessed on success rate, spillage, generalization to unseen food items, and task completion time. Results show that VR-based demonstrations significantly outperform 3D space mouse data, with BCQ achieving the best overall performance, particularly in reducing spillage and approaching human performance. These findings underscore the effectiveness of our framework for training RAF systems in granular material handling. The code for our framework is publicly available at: https://github.com/AmanuelErgogo/VERAGMIL.git.
Read the full abstract ▾
Robotics
GAPL: Grounded Action-effect Policy Learning for LLM-Based Trajectory Planning
Zhihong Cui, Hengyu Liu, Zhangkai Wu et al. (+5 more)
Trajectory planning for autonomous driving requires both high-level
reasoning and precise low-level control. Large
Language Models (LLMs) offer semantic-rich planning capabilities, however, their application is limited by hallucinated
reasoning, poor grounding in environment dynamics, and limited numerical precision in control. We propose GAPL (Grounded Action-effect Policy Learning), a unified framework that integrates LLM-based effect estimation, simulation-based effect grounding, and policy optimization into a closed-loop system. GAPL consists of three modules: (1) an LLM-based Effect Evaluator for structured multi-dimensional action-effect estimation; (2) a Simulation-based Effect Grounder that predicts dynamics-consistent effects from simulator rollouts; and (3) an Effect-Aware Decision Maker that grounds LLM effect estimates against simulation via a distiller to guide Proximal Policy Optimization (PPO)-based policy learning. Experiments on four Highway-env scenarios demonstrate that GAPL consistently outperforms baselines, achieving average reductions of {0.76, 0.86, 2.00} in collision rate, average displacement error (ADE), and final displacement error (FDE), and an average reward gain of 1.44.
Read the full abstract ▾
All authors Zhihong Cui, Hengyu Liu, Zhangkai Wu, Yushuai Li, Tianyi Li, Peiyuan Guan, Amir Taherkordi, Tor Skeie
Robotics · Cross-listed from cs.CV
Zero-Shot Transfer of Force Map Estimation Across GelSight Mini Sensors
Julio Castaño Amoros and Pablo Gil
Despite the rapid industrialization of the touch sensor manufacturing process, most of these sensors are still handmade in research laboratories. This complicates standardizing their performance, requiring the repetition of data collection and training models for each unit produced. To address this problem, this paper presents a method that can generalize the estimation of 3D force maps across different GelSight Mini sensor units, regardless of the sensor version. Specifically, the method consists of two stages: a domain adaptation stage, in which the input tactile image is reconstructed as a general tactile image using a UniT-based model; and a stage for estimating 3D force maps employing a U-Net network. Our proposal achieves promising results in both steps, such as an SSIM of 0.9338 +- 0.0358 in the image reconstruction phase and an MAE_F of 1.1294 +- 1.5934(N) in the force estimation phase.
Read the full abstract ▾
Robotics
GigaBrain-WBC-0.5: A Behavior World Model for Robust Humanoid Whole-Body Tracking with Environment Interaction
Ziyang Cheng, Tianshu Tang, Jinxin Lan et al. (+19 more)
General-purpose motion trackers enable humanoid
robots to follow diverse whole-body motions while maintaining balance, but are trained only on flat ground, failing to exploit bipedal mobility over complex terrain. Cross-terrain controllers, meanwhile, are task-specific or accept only low-dimensional locomotion commands. We introduce InterTrack, the first behavior world model (BWM) for robust whole-body tracking with environment interaction. Its
Transformer jointly predicts the next action, state, and behavior distribution, learning environment-conditioned dynamics. To scale interaction training data, an automatic annotation pipeline reconstructs 3D support geometry from retargeted motions. At deployment, the policy handles commands implausible in the current environment in a "best-effort" manner. Quantitatively, InterTrack achieves an 81.3% success rate on terrain interaction (4.3 times the best evaluated baseline) and a 99.3% fall-recovery rate, while also improving free-space tracking and outperforming three leading tracking baselines across all of these regimes. To our knowledge, we provide the first demonstration of real-time cross-terrain whole-body teleoperation on a humanoid
robot, alongside object interaction, stable responses to missing supports, and robust recovery from falls.
Read the full abstract ▾
All authors Ziyang Cheng, Tianshu Tang, Jinxin Lan, Xinze Chen, Yuhan Gong, Zhichao Liu, Changzhong Wu, Yahao Mao, Zongyan Deng, Mingxuan Ma, Huasen Xi, Yilong Liu, Yutong Wu, Xiaofeng Wang, Borui Zhang, Bingyao Yu, Yang Wang, Yun Ye, Guan Huang, Xiaojie Jin, Zheng Zhu, Jiwen Lu
Robotics
Revisiting the "Push-T" Robot Manipulation Task with Agentic Robotics
Shuangyu Xie, Kaiyuan Chen and Ken Goldberg
Push-T is an iconic
benchmark for learning manipulation policies from human demonstrations. The
robot must use a single point of contact to push a T-shaped block into a target pose. In this short paper, we revisit the Push-T task in the context of emerging advances in Agentic Robotics where an LLM coding
agent – Claude Code with Fable 5 – is prompted to create an algorithmic solution that does not require any demonstration data. We study how effective the agentic coding loop can solve the Push-T task, and compare the resulting code as policy with the visuomotor imitation learning policy. Results suggest that the
agent found the 2D gym simulation online, and used sim experiments to learn push mechanics, iteratively optimizing to achieve 100% success rate using 46% fewer steps than the best
diffusion policy trained with 200 human demonstrations. The coding
agent also solve extensions from T to the full alphabet (Push-A to Push-Z) using a self generated curriculum and generated simulation code for the Franka and UR5
robot arms in 3D cross-embodiment simulations with visual feedback.
Videos, policies and details will be posted online.
Read the full abstract ▾
Robotics
Trust as a Field: A Macroscopic Representation for Vehicular Networks
Md Mahmudul Islam and Shaurya Agarwal
Trust assessment is a fundamental component of cooperative and connected vehicle systems. However, existing approaches operate primarily at the level of individual vehicles, making it difficult to reason about trust
evolution across road segments. In this paper, we propose a spatio-temporal trust-field framework that aggregates microscopic vehicle-level trust into a continuous representation over space and time. The trust field is formally defined on road segments. We conducted simulation-based experiments using synthetic trajectories generated under controlled conditions, enabling analysis of trust-field behavior in simple road scenarios. Beyond theoretical modeling, we study an implication of the trust-field concept: reconstructing the full trust field from sparse roadside-unit (RSU) measurements. We compare (i) a coordinate-based deep learning baseline that learns a generic trust field from sparse samples and (ii) a field-informed deep learning method that treats trust as a latent quantity carried by vehicles and enforces measurement consistency through the aggregation mechanism. The field-informed approach more accurately recovers trajectory-aligned low-trust patterns and yields improved reconstruction error.
Read the full abstract ▾
Robotics
Hydra-0: Action Flow for Generalist World Modeling and Control
Hongyu Li, Bowen Wen, Xinghao Zhu et al. (+8 more)
We introduce Hydra-0, a generalist world model conditioned on action flow, which represents
robot actions as pixel motion. This shared visual interface enables generalist world modeling and control by learning action consequences across embodiments, tasks, environments, and
video-generation backbones. Our best configuration achieves 90.4% lower
robot-motion error and 60.2% lower object-motion error than our action-conditioned baseline, while supporting zero-shot composition and data-efficient adaptation. On the RoboLab
benchmark, Hydra-0 achieves a Pearson correlation of r=0.96 between replayed and reference success rates. Finally, we uncover an emergent inverse mode of this interface: a world action model that predicts compatible
robot motion from desired object flow transferred from a human demonstration. A trained action head maps the resulting latent features to executable actions without requiring task-specific expert
robot demonstrations. Together, these results demonstrate the potential of action flow as a shared control interface connecting heterogeneous training data, open-loop policy evaluation, and
robot control.
Read the full abstract ▾
All authors Hongyu Li, Bowen Wen, Xinghao Zhu, Yixuan Wang, Yilun Du, Yunzhu Li, George Konidaris, Stan Birchfield, Soha Pouya, Chenran Li, Yan Chang
Robotics
PRISM: Precision and contact-rich Real-world Industrial Skill dataset with Multimodal sensing
Tengbo Yu, Jiahao Wu, Hanning Wang et al. (+4 more)
Recent progress in robotic learning has been fueled by large-scale
datasets collected in everyday environments. However, most existing
datasets emphasize short-horizon, low-contact tasks such as pick-and-place, and therefore do not capture the precision control, force/torque or tactile regulation, and
multimodal feedback required for industrial assembly. To address this gap, we introduce PRISM, a large-scale
multimodal dataset for contact-rich industrial operations. The
dataset spans more than 25 manipulation tasks (e.g., electronic components plug/unplug, conveyor-based sorting) and covers diverse mechanical constraints. PRISM includes more than 5,000 trajectories totaling 45 hours of teleoperated demonstrations, recorded using synchronized multi-view RGB-D, force/torque, tactile, and
robot-state measurements. In contrast to
datasets collected in household or laboratory settings, PRISM provides a realistic
benchmark for
multimodal perception and control under high-precision industrial constraints, and serves as a foundation for contact-rich, generalizable manipulation in real-world manufacturing environments. The
dataset is open-sourced at: https://tengbo-yu.github.io/PRISM/
Read the full abstract ▾
All authors Tengbo Yu, Jiahao Wu, Hanning Wang, Rui Chen, Chuanhou Liu, Chuang Sun, Hangxin Liu
Robotics · Cross-listed from cs.MA
A Theoretical Framework for Parallel Lifelong MAPF Using Group Decentralized Planning
Alex DeWeese, Jiaoyang Li and Guannan Qu
In the Lifelong Multi-
Agent Path Finding (L-MAPF) problem,
agents must repeatedly move from one destination to another while avoiding obstacles and inter-
agent collisions. Widely regarded as one of the highest-performing solutions to this problem is the Rolling-Horizon Collision Resolution (RHCR) framework. However, commensurate with its quality solutions, it incurs a computational cost that limits its applicability to even modest
agent counts. In this paper, leveraging theoretical methods from the Locally Interdependent Multi-
Agent MDP literature, we first theoretically prove the near-optimality of RHCR in a discounted MDP formulation of the L-MAPF problem. Then, we leverage these results to naturally motivate an extended framework called Group Decentralized RHCR (GD-RHCR) which incorporates a group decentralized structure that partitions
agents based on a transitive communication scheme and plans for each partition of
agents in parallel. We show that both RHCR and GD-RHCR achieve similar exponentially close to optimal guarantees, establishing a theoretical duality between the time based restrictions performed by vanilla RHCR and the additional space based partitioning performed by GD-RHCR. Lastly, we show that across varying maps, GD-RHCR is able to attain high throughput that scales into higher
agent counts while maintaining a significantly lower per plan cost.
Read the full abstract ▾
Robotics
ControlledShifts: Towards Standardizing Robustness Evaluation in Trajectory Prediction Under Distribution Shifts
Ingrid Navarro, Pablo Ortega-Kral, Yutong Duan et al. (+2 more)
Trajectory prediction is central to safety in autonomous driving, yet learning-based predictors tend to degrade sharply when encountering scenarios poorly represented by their training data. Many methods attempt to mitigate distribution shift degradation through data-centric or test-time adaptation approaches; however, they are typically validated along fragmented axes of generalization, leaving the field without a standardized way to compare robustness across shifts a model may encounter. To address this, we introduce ControlledShifts, a framework and
benchmark suite that systematically re-splits existing trajectory
datasets into in-distribution (seen) and out-of-distribution (unseen) partitions, via a shared characterization-and-splitting formulation, in which a characterization function fixes the axis of variation a
benchmark probes and a splitting function fixes how the tail of that axis is withheld. The suite comprises three
benchmarks targeting key topological and behavioral distribution shifts. Furthermore, to aggregate multi-dimensional performance metrics across these
benchmarks, we propose a unified robustness score that evaluates models along two complementary dimensions: prediction quality (relative performance gain) and prediction stability (performance preservation under shift). We showcase ControlledShifts by benchmarking prominent
transformer-based architectures, exposing critical differences in how models of varying capacities handle latent relevance and environmental structure.
Read the full abstract ▾
All authors Ingrid Navarro, Pablo Ortega-Kral, Yutong Duan, Jonathan Francis, Jean Oh
Robotics
Jetson-ORB-SLAM3: Accuracy-Preserving GPU Implementation for Edge Computing Devices
Rajat Roy, Aditya Arun Kumar Yadav and Hardik Jain
Visual-inertial SLAM on low-power edge platforms is constrained by the cost of dense feature extraction and loop closure. Prior GPU ports of ORB-SLAM trade accuracy for speed by approximating the ORB detector, altering the feature set and therefore the estimated trajectory. We present an accuracy-preserving GPU implementation of ORB-SLAM3 for the NVIDIA Jetson Orin Nano, whose GPU ORB front end reproduces the reference CPU detector algorithmically to 94.7% exact keypoint agreement and 99.9% descriptor bit agreement. This work also makes CNN-based loop closure edge-viable through native TensorRT. The visual front end (feature extraction) is offloaded to the GPU while the mapping and optimization back end is kept on the CPU, matching each computation to the hardware it suits. The accuracy is verified by comparing four configurations: the GPU pipeline and the unmodified CPU reference, each run on both the Jetson Orin Nano and a desktop. On EuRoC
dataset, all four agree to within 0.10cm in mean absolute trajectory error (SE(3)), so neither the GPU port nor the change of hardware shifts the estimated trajectory. The GPU-versus-CPU comparison is reproducible on TUM-VI and KITTI
datasets, so the acceleration is accuracy-preserving rather than approximate. The proposed implementation is competitive with published ORB-SLAM3 on EuRoC, attains sub-centimeter accuracy on five of the six TUM-VI room sequences, and reaches sub-1% relative translation error on nine of eleven KITTI sequences. For loop closure, the generic ONNX-Runtime CUDA/TensorRT execution providers are unusable with our CosPlace ResNet-50 on the embedded platform, whereas a native libnvinfer FP16 engine reduces per-query inference to 2.2ms, a 180x speedup. Learned place recognition therefore runs concurrently with tracking on a 7W device. In monocular-inertial mode the system sustains 32FPS mean over the eleven EuRoC sequences.
Read the full abstract ▾
Robotics
Effector-Centric NMPC of Tiltable-Multirotors for Offset-Free Omnidirectional Aerial Manipulation
Jinjie Li, Yicheng Chen, Johannes Kübel et al. (+3 more)
Aerial manipulation extends robotic operations to previously inaccessible aerial environments. Unlike arm-equipped aerial systems, tiltable-multirotors can directly generate six-degree-of-freedom wrenches through their flight bases, enabling both efficient movement and omnidirectional operation by tilting the thrust direction. This work presents a design analysis and a wrench-based control framework for tiltable-multirotors in aerial manipulation. We show that a four-rotor tiltable configuration provides a balance between interference-free propeller sizing and hovering efficiency across different attitudes, and its null-space redundancy is crucial for traversing singular configurations under physical constraints. We further show that an upward end-effector placement yields a favorable trade-off between
geometric clearance and available wrench. To address disturbances, we propose a dual strategy consisting of a modified integral term for model error and an acceleration-based
estimator for external wrenches. Building on these insights, we develop an effector-centric nonlinear model predictive control (NMPC) framework that integrates design choices, singularity handling, and disturbance compensation into a unified formulation. The proposed framework runs fully onboard at 100 Hz on a custom-built tiltable-quadrotor. Real-world experiments, including a 90-deg step cartwheel rotation, whiteboard pushing, and continuous 360-deg valve turning, demonstrate the feasibility of wrench-based omnidirectional manipulation with singularity traversal on a one-DoF-per-arm tiltable-quadrotor.
Read the full abstract ▾
All authors Jinjie Li, Yicheng Chen, Johannes Kübel, Haokun Liu, Junichiro Sugihara, Moju Zhao
Robotics · Cross-listed from cs.CV
Training with synthetic data for drone detection in thermal imagery
Tanel Liiv, Sander Soodla, Nzamba Bignoumba et al. (+2 more)
Ground-to-Air (G2A) drone detection in medium- and long-wave infrared (MWIR/LWIR) imagery is challenging due to reduced texture information, sensor noise, weak thermal contrast, and the scarcity of annotated data. This work investigates a synthetic-first training strategy that combines synthetic scene generation with fine-tuning on real data. We show that synthetic data provides an effective basis for learning initial object representations, while real in-domain thermal imagery is still essential for reliable deployment. Even small amounts of real IR data substantially reduce domain gaps. Our experiments indicate that
dataset alignment has a stronger impact on performance than model scale. Finally, our analysis of the
dataset suggests that semantic
alignment in feature space is the strongest predictor of model performance, while radiometric properties such as
entropy and dynamic range also contribute to detection robustness. This work provides a foundation for combining synthetic and real IR data for effective G2A drone detection.
Read the full abstract ▾
All authors Tanel Liiv, Sander Soodla, Nzamba Bignoumba, Alma M. Liezenga, Toomas Pruuden
Robotics
Stability Control for Real World Testing in Autonomous Racing
Phillip Pitschi, Simon Sagmeister, Frederik Werner et al. (+2 more)
Controlling an autonomous vehicle at the limits of handling is a challenging task. Due to external influences, such as road conditions or weather, a vehicle can easily become unstable. Since most control
algorithms assume stable vehicle behavior, they might fail in these situations. Especially when operating expensive vehicles without a safety driver on board, as in autonomous racing, this poses a significant challenge. To enable safe operation at the vehicle's dynamic limits, we present a comprehensive stability control system that safeguards motion control
algorithms in autonomous driving. The proposed system consists of an electronic stability control (ESC), a slip control (SC), and a countersteer system (CS), which collectively adapt steering and brake commands from the motion controller to maintain vehicle stability. We validate our approach through both simulation and experiments on a real-world, full-scale vehicle. The results show that the stability control system maintains vehicle stability in critical situations and extends the operational feasible region. To simplify integration, we provide an open-source implementation at github.com/TUMFTM/tam-stability-control.
Read the full abstract ▾
All authors Phillip Pitschi, Simon Sagmeister, Frederik Werner, Markus Lienkamp, Boris Lohmann
Robotics · Cross-listed from cs.CL
Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See
Ayoub Kirouane and Christos Petrocheilos
Take three frontier mixture-of-experts models (Alibaba, OpenAI, NVIDIA; 3.6-4.0B active parameters each) and fine-tune them to reason in a low-resource
language. On accuracy
benchmarks almost nothing happens, and the
benchmark itself is noise at this scale: changing only the random seed moves the score by 7.7 points, more than every data and recipe effect we measured. That null is our first result. The real changes live where accuracy cannot see. Base models never think in Greek: 0 of 1,000
reasoning traces, even when the question is Greek, so the model answers correctly while
reasoning in a form its user cannot read, audit, or correct. After supervised fine-tuning (SFT), every released checkpoint reasons in the
language of the question on ~98% of items, one family at 3x fewer
tokens, with judged grammaticality improving on all four models and general ability within a few points of each base: nothing was forgotten, and fluency was gained. We propose six behavioural dimensions that make such changes measurable, each gated to reject any metric that correlates with output length, and we report how our own instruments lied: six failures, each caught by a control. What SFT cannot do is fix its own defects: a quarter of answers skip the requested format, answers leak into the
reasoning channel, and an explicit "think in English" is obeyed under half the time. Reinforcement learning with verifiable rewards, pre-registered before training, fixes the first two outright (fallback 24% to 2.5%, leak 3.5% to 0.0%, both against a flat random-reward control) and moves the third (+9.1pp), while the Greek
reasoning habit survives an accuracy-only gradient untouched. We release five checkpoints. The instruments, the controls and the pre-registration travel to any low-resource
language; Greek is the case that let us measure them.
Read the full abstract ▾
Robotics
CompCPZ: Preserving Multi-Modal Intent in Language-Guided Robot Manipulation
Zhen Zhang, Ahmad Hafez, Peng Xie et al. (+3 more)
A
robot asked to "place the cup near the red plate or the blue plate" may reach the centroid between them and appear geometrically successful, while satisfying neither disjunct of the instruction. This silent semantic failure exposes a structural limitation of
language-conditioned
robot policies: representations that collapse a disjunctive instruction into a single connected set cannot preserve all feasible modes, and planners that commit to one action degrade under run-time mode uncertainty. We address this limitation with CompCPZ, a sound algebraic layer that
language-conditioned learning systems wrap to recover multi-modal disjunctive representation, recursively composing per-primitive constrained
polynomial zonotope enclosures along the
language parse tree with distribution-free conformal coverage and sub-millisecond runtime. On a closed-loop ManiSkill3 tabletop-manipulation
benchmark, CompCPZ outperforms convex set baselines, multi-peak decoders, and a zero-shot
vision-
language-action model (1,900/1,918 paired wins, p << 10^(-30)); the same compiler also transfers without retuning to planar real-
robot trials on a Unitree Go2 quadruped under motion capture. These results suggest that compositional
language grounding should be evaluated not only by reaching a decoded target, but by whether the represented feasibility set preserves the connected-component structure of the user's intent.
Read the full abstract ▾
All authors Zhen Zhang, Ahmad Hafez, Peng Xie, Yanliang Huang, Wenyuan Wu, Amr Alanwar