Robotics
TacForeSight: Force-Guided Tactile World Model for Contact-Rich Manipulation
Yujie Zang, Yuhang Zheng, Xian Nie et al. (+7 more)
Contact-rich manipulation requires
robots to continuously perceive and regulate evolving physical interactions under dynamic contact transitions or complex surface geometries. Recent imitation learning methods improve contact-aware control by incorporating tactile or force feedback, but they rarely model the asymmetric spatiotemporal roles of global force and local tactile sensing. To address this, we propose TacForeSight, a lightweight force-conditioned tactile foresight framework for real-time manipulation. The core component is TacForceWM, a tactile world model that predicts short-horizon tactile latent dynamics from dual-finger tactile observations conditioned on high-frequency wrist force and torque signals. Another key component, the Predictive Tactile-Conditioned Policy, leverages the predicted latents as anticipatory contact priors, models the current-to-future tactile
evolution via cross-
attention, and adaptively fuses visuo-tactile features through a tactile-guided gating module. By
forecasting purely within a compact latent space, TacForeSight enables proactive contact
reasoning with efficient real-time inference suitable for high-frequency manipulation control. Real-
robot experiments on five representative tasks and three in-process perturbation settings show that TacForeSight consistently outperforms existing baselines, particularly under dynamic contact disturbances. All models and
datasets will be made publicly available on the project website at https://tacforesight.github.io/ProjectPage.
Read the full abstract ▾
All authors Yujie Zang, Yuhang Zheng, Xian Nie, Yupeng Zheng, Shuai Tian, Songen Gu, Chen Gao, Zining Wang, Shuicheng Yan, Wenchao Ding
Robotics
JOIN: Anchor-Grasp-Conditioned Joining via Opposition, Inference, and Navigation for Bimanual Assistive Manipulation
Drake Moore, Matt Cheng, Xiang Zhi Tan et al. (+1 more)
Assistive mobility and manipulation platforms have received increasing
attention as a means of restoring independence to individuals with disabilities. While effective for many basic activities of daily living (ADLs), a significant percentage of everyday tasks such as opening a jar, pouring a
liquid, lifting a tray, or basic meal preparation, is fundamentally bimanual and remains out of reach for any single-arm system. Adding a second arm to a wheelchair is impractical, due to the additional power draw, cost, and the loss of space required for transfers and mobility. We instead propose a heterogeneous, on-demand bimanual system, in which a wheelchair-mounted anchor arm is joined when needed by a summoned mobile manipulator that serves as a complement arm. The central technical problem, which we call bimanual joining, is conditional: the anchor has already committed to a grasp, and the complement arm must choose where to stand and what to grasp to complete the task. We formulate bimanual joining as a three-phase decomposition (plan, drive, grasp) and show that a
vision-
language model (VLM), coupled with standard
geometric tools, provides task-level knowledge sufficient to solve a representative class of bimanual ADLs. Our system JOIN, contributes (i) a wheelchair-referenced opposition score, and (ii) task-conditioned directional manipulability. We evaluate JOIN on a Kinova Gen3 anchor and a Hello
Robot Stretch~3 complement on representative same-object and different-object tasks. JOIN accomplished more attempts (19/20) than state-of-the-art methods (14/20) and required markedly less correction by the operator.
Read the full abstract ▾
All authors Drake Moore, Matt Cheng, Xiang Zhi Tan, Taşk{\i}n Pad{\i}r
Robotics
EM-Fall: Embodied mmWave Sensing for Day-and-Night Fall Detection on Humanoid Robots
Yanshuo Lu, Yuxuan Hu, Shenghai Yuan et al. (+5 more)
Falls are one of the leading causes of injury and hospitalization among elderly individuals, making reliable fall awareness an essential capability for safety monitoring in residential environments. However, existing fall detection systems often rely on wearable devices or fixed sensing installations, which may suffer from low user compliance, limited spatial coverage, or degraded performance under occlusion and poor lighting conditions. In this work, we propose EM-Fall, an
embodied fall detection framework deployed on a mobile
humanoid robot. The system integrates millimeter-wave (mmWave) sensing with robotic mobility, allowing the
robot to actively adjust its sensing viewpoint and maintain target observability across rooms and under occlusion. To address interference in complex residential environments, including pet motion and multipath artifacts, we design a human-centered perception pipeline combined with lightweight temporal modeling to capture motion
evolution before, during, and after fall events. We evaluate the proposed system across eight real indoor environments with four participants and construct an in-home mmWave fall detection
dataset. Experimental results show that the
embodied mobile sensing paradigm improves monitoring continuity and maintains robust fall detection performance under diverse environmental conditions. The proposed framework provides a practical solution for
robot-assisted safety monitoring in home environments.
Read the full abstract ▾
All authors Yanshuo Lu, Yuxuan Hu, Shenghai Yuan, Xinyu Zhou, Kuangji Zuo, Bofan Lyu, XiChen Yuan, Jianfei Yang
Robotics
RoboNaldo: Accurate, Stable and Powerful Humanoid Soccer Shooting via Motion-Guided Curriculum Reinforcement Learning
Yichao Zhong, Yidan Lu, Yuhang Lu et al. (+9 more)
Elite
humanoid soccer shooting requires whole-body stability, high-impulse whole-body interactions, and accuracy to targets. Motion tracking-driven reinforcement learning (RL) provides stability in whole-body movement coordination, but a fixed reference makes it hard to adapt to varied ball positions and strike timings; in contrast, task reward-driven RL struggles to explore and discover valid kicks from scratch. We therefore introduce RoboNaldo, a three-stage motion-guided curriculum RL framework for high-impulse
humanoid interaction. A single human-kick reference is used as a scaffold and progressively shifts optimization towards shooting performance. The curriculum first learns a stable whole-body kicking prior, then adapts the kick to free-kick settings where the ball is stationary at random positions, and finally extends it to moving-ball shooting through a locomotion-command and kick-trigger interface. A high-level heuristic planner controls this interface during training, while alternative high-level controllers can drive the same low-level policy at inference. In simulation, RoboNaldo demonstrates free-kick shot error 48.6% lower and shoot velocity 2.96x than prior work baselines. In real world on a Unitree G1 with onboard perception, RoboNaldo attains 0.73 m and 0.86 m average target shooting error from 3 m away in free-kick and moving-ball cases, accordingly. And the post-contact ball velocity reaches 13.10 m/s, which is 59-71% of reported professional open-play shot speed. Project page: https://opendrivelab.com/RoboNaldo.
Read the full abstract ▾
All authors Yichao Zhong, Yidan Lu, Yuhang Lu, Tianyang Tang, Haoguang Mai, Yixuan Pan, Tianyu Li, Li Chen, Jingbo Wang, Zhongyu Li, Peng Lu, Hongyang Li
Robotics
A Distributed Multi-UGV Exploration Framework With Loop-Aware Planning and Descriptor-Aided Localization in Resource-Limited Environments
Zhiwei Li, Haiou Liu, Xijun Zhao et al. (+3 more)
Robust and efficient cooperative exploration with multiple unmanned ground vehicles (UGVs) in unknown, GPSdenied, and bandwidth-limited environments without prior maps remains challenging, as localization drift degrades map consistency and induces redundant coverage. This paper presents a fully distributed exploration framework that couples descriptoraided inter-UGV loop closure with loop-aware hierarchical planning while enabling autonomous localization and exploration. We develop a lightweight
LiDAR global descriptor with range-image prealignment to enable robust cross-UGV place recognition under large yaw and lateral variations, and use verified loop closures to maintain globally consistent trajectories and a sparse topological representation. We further introduce an uncertainty-aware crossUGV loop-closure selection module that scores candidate loop closures under pose uncertainty and retains high-utility loop closures as planning anchors for global task allocation and local route refinement. Simulations and real-UGV experiments show that the loop-closure module achieves AR@1/AR@1% of 89.9%/95.5%, distributed optimization reduces absolute trajectory error, the system substantially reduces two-way communication volume, and the overall framework reduces exploration time and travel distance by 15% and 14%, respectively, compared with an mTSP baseline.
Read the full abstract ▾
All authors Zhiwei Li, Haiou Liu, Xijun Zhao, Ji Li, Yingze Wang, Boyang Wang
Robotics
Generation of Diverse and Functional Robot Designs using Superquadrics Parametrisation and Quality-Diversity
Leni Le Goff, Simon Smith and Emma Hart
Generative design of
robots requires navigating a vast search-space, encompassing physical configurations and behavioural parameters. Evolutionary
Algorithms (EAs) have shown promising results, but often converge prematurely to a small set of sub-optimal designs. Most EAs fail to maintain sufficient diversity in the population that would allow the discovery of distinct functional
robots. To counter premature convergence, we introduce a superquadrics-based representation (SQs) for
robot bodies. SQs are interpretable, compact and computationally efficient mathematical representations of 3D geometrical shapes that can be tuned to specific design-spaces. To encourage morphological diversity, we combine this representation with a quality-diversity (QD)
algorithm (MAP-Elites). We compare SQs and Compositional Pattern Producing Networks representations as generators of morphologies, combining them with standard EAs and MAP-Elites. In two test environments, we find that using SQs to generate morphology in conjunction with the MAP-Elites
algorithm reaches the highest QD-score across both environments, maximising diversity of design and functionality of generated
robots. The findings highlight the benefits of using a compact and interpretable
geometric representation for exploring a complex design-space and suggest that combining SQs with an explicit diversity mechanism increases the quality and number of designs generated.
Read the full abstract ▾
Robotics
A Spiking Neural Architecture for Coordinating Arm and Locomotor Control
Lea Steffen, Kathryn Simone, Graeme Damberger et al. (+3 more)
Spiking Neural Networks (SNNs) coupled with neuromorphic hardware offer energy-efficient solutions for
humanoid robot control. However, existing SNN-based motor control systems address bipedal locomotion and arm control in isolation, leaving integrated control of both unaddressed. We present a spiking architecture that coordinates force-based arm control and bipedal locomotion in a simulated
humanoid, using the Neural Engineering Framework (NEF) and Semantic Pointer Architecture (SPA). High-level action selection between locomotor and arm control is mediated by a biologically grounded spiking basal ganglia model. We validate the system through co-simulation of Nengo, for the neural control, and Isaac Sim, demonstrating successful target reaching, continuous digit drawing, path-following locomotion, and finally, switching between walking and arm control via basal ganglia disinhibition. To our knowledge, this is the first integrated spiking controller to combine bipedal locomotion and arm control on a full-scale
humanoid platform. The full spike-based implementation enables future deployment on low-power neuromorphic hardware.
Read the full abstract ▾
All authors Lea Steffen, Kathryn Simone, Graeme Damberger, Travis DeWolf, Hudson Ly, Chris Eliasmith
Robotics
Diffusion Forcing Planner: History-Annealed Planning with Time-Dependent Guidance for Autonomous Driving
Zehan Zhang, Neng Zhang, Yaoyi Li et al. (+2 more)
Learning-based motion planners, despite recent progress, often suffer from temporal inconsistency. Small perturbations across frames can accumulate into unstable trajectories, degrading comfort and safety in closed-loop driving. Several methods attempt to inject history as a static conditioning signal to stabilize outputs, only to induce the planner to copy historical patterns instead of adapting to environment contexts. To address this limitation, we propose
Diffusion Forcing Planner (DFP), a
diffusion-based planning framework driven by history-guided control. Specifically, DFP decomposes the full trajectory into history, current and future segments, and assign independent noise levels to each segment. The model jointly denoises the historical and the future segments, enforcing a heterogeneous joint
diffusion process. At inference, classifier-free guidance (CFG) is applied to steer future sampling using annealed history in a controllable manner. Closed-loop evaluation and comprehensive ablations on nuPlan show that DFP achieves competitive performance while producing continuous, stable, and controllable motion plans in complex driving scenarios.
Read the full abstract ▾
All authors Zehan Zhang, Neng Zhang, Yaoyi Li, Jia Cai, Zhiling Wang
Robotics
Multi-UAV Active Sensing with Information Gain-based Planning and Belief Fusion
S. Habibi and L. Marques
Unmanned aerial vehicles (
UAVs) are increasingly used for active sensing and information gathering in spatially distributed environments. Their performance, however, is constrained by limited flight time, sensing uncertainty, and the trade-off between spatial coverage and observation accuracy. This paper presents a real-world validation of a multi-
UAV active sensing framework for probabilistic binary terrain mapping, with precision agriculture used as the application case. The environment is represented as a probabilistic belief map, where spatial dependencies are modeled through a factor-graph formulation.
UAV decision making is guided by Information Gain based Informative Path Planning (IGbIPP), and the approach is compared with Random Walk and Sweep coverage path planning baselines using both synthetic terrains and real
UAV-derived agricultural imagery. The study also evaluates spatial correlation weights and several probabilistic belief-fusion rules for multi-
UAV information sharing. Results show that IGbIPP reduces
entropy and mapping error more effectively than the baselines, while a wider field of view improves real-world coverage and map accuracy. The results further show that simple equal or biased spatial weights can be more robust than adaptive weights, and that
Bayesian, log-odds, and Dempster–Shafer fusion achieve the best cooperative mapping performance. These findings highlight the importance of uncertainty-driven planning, sensing geometry, spatial modeling, and probabilistic fusion for real-world
UAV-based active sensing.
Read the full abstract ▾
Robotics
Language-Driven Cost Optimization for Autonomous Driving
Diego Martinez-Baselga, Khaled Mustafa and Javier Alonso-Mora
The driving behavior of autonomous vehicles is typically governed by the cost function of their motion planner, which encodes objectives such as speed tracking, smoothness, lane keeping, and collision avoidance. However, tuning the parameters that shape this cost function is a challenging task that requires technical expertise, limiting the vehicle's ability to adapt to evolving traffic scenarios or end-user preferences. This work presents a
language-driven framework for adaptive cost design in autonomous driving. A Large
Language Model (LLM) interprets structured scenario descriptions and natural
language user queries to generate the parameters applied to a risk-aware Model Predictive Path Integral (MPPI) controller. The system incorporates a human-in-the-loop validation stage in which the proposed behavioral changes are described in non-technical
language and confirmed prior to deployment. Users may additionally provide feedback either before or after deployment, enabling iterative refinement of the vehicle's motion behavior. The framework is evaluated across multiple queries in realistic driving scenarios to assess its effectiveness. Simulation results demonstrate that the method successfully induces behavioral changes that align with the intended requirements in an intuitive manner, thereby bridging the gap between intelligent vehicle control systems and end users.
Read the full abstract ▾
Robotics
Resilient Navigation for Autonomous Farm Robots by Leveraging Jerk-Augmented Models with IMU-Only Disturbance Rejection
Batu Candan, Mohammed Atallah, Simone Servadio et al. (+1 more)
Precise state estimation for
navigation of autonomous agricultural
robots is often compromised by sensor outages (GNSS/
LiDAR/Visual) and high-frequency vibrations inherent in off-road environments. This paper proposes a robust
navigation algorithm based on a jerk-augmented Extended Kalman Filter (EKF) integrated with a Multiple Tuning Factor (MTF) adaptation method. Unlike standard EKF approaches that assume constant measurement noise, our method dynamically adjusts the measurement covariance
matrix in real-time, allowing the system to cope with sudden disturbances and sensor outliers. We evaluate the
algorithm using real-world data from a Salin247 autonomous
robot. Results demonstrate that jerk-augmentation combined with MTF adaptation significantly reduces 3D position Root Mean Square Error (RMSE) compared to baseline EKF models, providing superior dead-reckoning capabilities.
Read the full abstract ▾
All authors Batu Candan, Mohammed Atallah, Simone Servadio, Saeed Arabi
Robotics
AllDayNav: Lifelong Navigation via Real-World Reinforcement Learning
Hang Yin, Yinan Liang, Jiazhao Zhang et al. (+4 more)
Lifelong
embodied navigation in dynamic environments requires
robots to form persistent scene understanding from fragmentary observations, which remains difficult for existing methods that rely on explicit maps or scene graphs and struggle to generalize beyond structured settings. We propose AllDayNav, a lifelong self-learning
navigation framework that implicitly encodes scene dynamics into the billion-scale parameters of a large model via reinforcement learning, powered by a self-evolving
multimodal memory that maintains and updates visual keyframes, semantic descriptions, and temporal context while autonomously generating open-vocabulary instructions, image goals, and structured rewards. Experiments in both synthetic and real-world environments across cross-room, cross-episode, and cross-task scenarios show that AllDayNav achieves success rates approaching $100\%$ and consistently surpasses strong map-based, VLM, and RL baselines in path efficiency and robustness, demonstrating implicit,
memory-driven reinforcement learning as a scalable alternative to explicit mapping for reliable lifelong
navigation.
Read the full abstract ▾
All authors Hang Yin, Yinan Liang, Jiazhao Zhang, Jiahang Liu, Minghan Li, Zhizheng Zhang, He Wang
Robotics
Task Robustness via Re-Labelling Vision-Action Robot Data
Artur Kuramshin, Özgür Aslan, Cyrus Neary et al. (+1 more)
The recent trend in scaling models for
robot learning has resulted in impressive policies that can perform various manipulation tasks and generalize to novel scenarios. However, these policies continue to struggle with following instructions, likely due to the limited linguistic and action sequence diversity in existing robotics
datasets. This paper introduces Task Robustness via Re-Labelling
Vision-Action
Robot Data (TREAD), a scalable framework that leverages large
Vision-
Language Models (VLMs) to augment existing robotics
datasets without additional data collection, harnessing the transferable knowledge embedded in these models. Our approach leverages a pretrained VLM through three stages: generating semantic sub-tasks from original instruction labels and initial scenes, segmenting demonstration
videos conditioned on these sub-tasks, and producing diverse instructions that incorporate object properties, effectively decomposing longer demonstrations into grounded
language-action pairs. We further enhance robustness by augmenting the data with linguistically diverse versions of the text goals. Evaluations on LIBERO demonstrate that policies trained on our augmented
datasets exhibit improved performance on novel, unseen tasks and goals. Our results show that TREAD enhances both planning generalization through trajectory decomposition and
language-conditioned policy generalization through increased linguistic diversity.
Read the full abstract ▾
All authors Artur Kuramshin, Özgür Aslan, Cyrus Neary, Glen Berseth
Robotics
AgniNav: Configuration-Driven Cross-Embodiment Local Planning for Robot Navigation
Tianhao Zang, Siwei Cheng, Haidong Huang et al. (+2 more)
Monocular local
navigation is attractive for lightweight
robots, but existing
vision-based policies often couple perception to a specific body, camera height, and footprint, making transfer from wheeled bases to legged platforms dependent on retraining or active depth hardware. This paper introduces AgniNav, a configuration-driven local
navigation framework that standardizes cross-embodiment transfer at the collision-envelope level. Each
robot is specified by a measurable four-parameter safety envelope: collision-relevant height, front length, rear length, and half width. The height parameter conditions an image-to-scan network to predict a one-dimensional, collision-relevant pseudo-laserscan from a
monocular color image, while the remaining footprint parameters configure a dimension-aware local planner for collision checking. Training uses height-conditioned column-minimum scan labels generated from paired color-depth data, allowing the same image to supervise different safety envelopes without collecting
robot-specific data. To the best of our knowledge, AgniNav is the first
monocular local-
navigation framework that jointly conditions perception and planning on a shared collision-envelope configuration for zero-retraining deployment across wheeled, quadruped, and
humanoid platforms. Real-
robot experiments on a Turtlebot2, Unitree Go2, and Accelerated
Evolution K1 achieve 39/40, 18/20, and 18/20 successes with 0/40, 1/20, and 2/20 collisions, respectively, while running at 30 Hz on Jetson Orin.
Read the full abstract ▾
All authors Tianhao Zang, Siwei Cheng, Haidong Huang, Shanze Wang, Wei Zhang
Robotics
MV-Actor: Aligning Multi-View Semantics and Spatial Awareness for Bimanual Manipulation
Yinchen Tian, Huan Li, Muyao Peng et al. (+3 more)
Robotic manipulation has been widely applied in industrial scenarios. Compared with single-arm manipulation, bimanual manipulation is equipped with multiple cameras to capture information from different viewpoints. However, existing multi-view policies encode each view independently or fuse view features shallowly, resulting in limited sharing semantic perception and unreliable spatial awareness. In this paper, we propose MV-Actor, a multi-view perception framework that builds a unified semantic-spatial representation for bimanual manipulation. First, MV-Actor performs Multi-view Semantic Interaction to share semantic perception across views. Then it uses Semantic-Spatial
Token Interaction to ground visual semantics with feed-forward reconstruction model features and acquire reliable spatial awareness. Finally, a Guided Metric Depth Repair module refines degraded sensor depth to provide more reliable metric anchors under consumer-grade depth noise. In simulation experiments conducted on the PerAct2 bimanual
benchmark, MV-Actor achieves a state-of-the-art average success rate of 87.8%. In real-world evaluations with more frequent viewpoint changes and unstable consumer-grade depth, MV-Actor outperforms both RGB and RGB-D baselines, further demonstrating the benefit of sharing semantic perception and reliable spatial awareness for bimanual manipulation.
Read the full abstract ▾
All authors Yinchen Tian, Huan Li, Muyao Peng, Xi Wang, Yan Wang, You Yang
Robotics
Embodiment-conditioned Generalist Control for Multirotor Aerial Robots
Orestis Konstantaropoulos, Welf Rehberg, Mihir Kulkarni et al. (+1 more)
We present a generalist position control policy capable of controlling arbitrary multirotor configurations of a certain rotor count (e.g., hexarotors or quadrotors) with a single set of network weights. The policy is conditioned on a physics-grounded embodiment descriptor: a mass and inertia-normalized control allocation
matrix that captures how mass-normalized motor thrusts generate linear and angular accelerations in the body-frame. To train the policy, we sample from a broad distribution of arbitrary multirotor configurations, including non-planar and asymmetric systems, and optimize a single, compact network using Proximal Policy Optimization. Training requires only five minutes on an RTX 3090 GPU using a custom NVIDIA Warp-based dynamics simulator. Through extensive simulation experiments, we show that embodiment conditioning enables robust generalist control across arbitrary morphologies. We demonstrate zero-shot real-world transfer of this generalist policy on three diverse hexarotor systems, including a planar
robot, a partially symmetric non-planar system, and a random asymmetric, non-planar configuration.
Read the full abstract ▾
All authors Orestis Konstantaropoulos, Welf Rehberg, Mihir Kulkarni, Kostas Alexis
Robotics
An Exposure-Time-Aligned Primary-Path Architecture for Autonomous-Driving ECUs
Toru Saito, Yuki Hagura, Tatsuya Konishi et al. (+2 more)
While end-to-end (E2E) autonomous driving has become the dominant research direction, production vehicles continue to rely on modular multi-NN pipelines for a non-trivial transitional period. The subject of this paper is the design of an architecture that, during this phase, supports a modular pipeline and an E2E path side by side and embeds a path for staged migration. Transplanted to a production SoC, egalitarian late fusion is compute-inefficient and offers no natural unit for staged E2E substitution. As an alternative, we propose three design principles: (i) Primary-Path, which selects a primary perception-to-planner chain and prioritizes its enclosure within a single SoC over the non-critical paths, (ii) Exposure-Time-Aligned, which propagates the primary sensor's exposure time $\tau_{\mathrm{exp}}$ as a tag along the chain and event-drives the fusion node on matched $\tau_{\mathrm{exp}}$ rather than a fixed cycle, and (iii) Co-Path Coexistence, which, building on (i) and (ii), lets an E2E output path co-run with the modular pipeline within the same $\tau_{\mathrm{exp}}$ cycle. On a Dual-SoC production AD-ECU, the implementation achieves a camera-shutter to planner-output latency of 296 ms on average within a 350 ms design budget, with the modular pipeline and the E2E path coexisting within the shared $\tau_{\mathrm{exp}}$ cycle.
Read the full abstract ▾
All authors Toru Saito, Yuki Hagura, Tatsuya Konishi, Satoru Mizusawa, Takumi Yajima
Robotics
Gradient based Bilevel for Inverse Optimal Control, a Riemannian approach
Ahmed-Manaf Dahmani, Vincent Bonnet, David Daney et al. (+1 more)
Inverse Optimal Control (IOC) aims to recover the cost function that explains observed trajectories as solutions of an optimal control problem. Classical IOC formulations rely on bilevel optimization, which repeatedly solves a nested optimal control problem and quickly becomes computationally prohibitive for realistic systems. Recent projection-based approaches offer a promising alternative but suffer from numerical instability when solved with gradient-based methods due to violations of standard constraint qualifications. In this paper, we show that these difficulties stem from the
geometric structure of the IOC feasible set. We demonstrate that the set of trajectories satisfying the optimality conditions naturally forms a
manifold and reformulate IOC as an optimization problem on this
manifold. Based on this insight, we propose a Riemannian Inverse Optimal Control (RIOC) method that projects observed trajectories onto the
manifold of optimal solutions while preserving feasibility by construction. Experiments on real human arm trajectories show that the proposed method achieves comparable or better reconstruction accuracy than classical bilevel IOC while reducing computation time by about a factor of four. These results highlight the potential of
geometric optimization methods to improve the scalability and reliability of IOC for robotics and human motion analysis.
Read the full abstract ▾
All authors Ahmed-Manaf Dahmani, Vincent Bonnet, David Daney, François Charpillet
Robotics
GUIDE: Goal-Initialized Directional Understanding for End-to-End Legged Navigation
Liang Wang, Jin Jin, KanZhong Yao et al. (+6 more)
End-to-end reinforcement learning (RL) has shown strong potential for legged
robot navigation, yet existing approaches commonly rely on continuously updated
robot-to-goal states from external state estimation modules, leaving part of the
navigation problem outside the learned policy. In this work, we seek to push the limits of end-to-end sim-to-real RL
navigation by investigating whether a legged
robot can internally maintain the spatial context required for long-horizon
navigation. To this end, we study goal-initialized
navigation, where the goal is provided only once at the beginning of an episode, with no subsequent external relative-goal updates. We present GUIDE, an end-to-end RL framework that jointly learns
navigation and internal directional awareness from onboard observations. GUIDE leverages multi-frequency proprioceptive history to capture egomotion and auxiliary spatial-anchor prediction to maintain task-relevant spatial states, while temporal depth observations provide local environmental geometry. The
navigation policy is trained entirely in simulation and transferred zero-shot to the real world. Across cluttered environments and structured mazes, GUIDE reliably avoids obstacles, escapes dead ends, and reaches distant goals using only onboard sensing. These results demonstrate that robust sim-to-real legged
navigation can be achieved without continuously providing external
robot-to-goal estimation, opening a promising direction for future research on more self-contained end-to-end
navigation.
Read the full abstract ▾
All authors Liang Wang, Jin Jin, KanZhong Yao, YiBin Wu, Fangqiang Ding, Jin Wang, Jun Wu, Zhe Sun, Qiuguo Zhu
Robotics
IMPACT: Learning Internal-Model Predictive Control for Forceful Robotic Manipulation
Jiawei Gao, Chaoqi Liu, Peilin Wu et al. (+2 more)
Real-world robotic manipulation tasks often involve forceful interactions with the environment, such as using tools of varying weights, transporting objects with different masses, and performing contact-rich tasks like table wiping. Previous learning-based approaches typically employ imitation learning policies that output target end-effector poses tracked by low-level impedance controllers. In these systems, forceful interactions are either implicitly realized through steady-state tracking errors or explicitly commanded using wrist force/torque or tactile sensors. However, implicit approaches generalize poorly across object weights, while explicit approaches require specialized hardware and increase system complexity. In this work, we propose IMPACT, a framework that decouples these forceful tasks into task-planning and internal-model-based predictive control. Extensive simulation and real-world experiments demonstrate that the proposed framework achieves higher success rates and improved generalization to unseen object weights, as well as better safety and energy efficiency.
Read the full abstract ▾
All authors Jiawei Gao, Chaoqi Liu, Peilin Wu, Haonan Chen, Yilun Du
Robotics
Bridging Semantics and Physical Execution: A Neuro-Symbolic Framework for Multi-Pair Robotic Assembly
Xinyi Li, Aiguo Song, Linhu Wei et al. (+1 more)
Multi-pair robotic assembly in unstructured environments faces spatial interference and contact uncertainties. Existing paradigms fail to bridge cognitive decision-making and physical execution, as they either encounter state-space explosion and knowledge bottlenecks or suffer from logical
hallucinations and topological conflicts. We propose an end-to-end neuro-symbolic framework that solves the challenge hierarchically: generating optimal subgraphs for each pair, decoupling generality from edge cases, and then resolving cross-pair interferences. Given an eye-on-hand RGB-D assembly scene, the framework extracts semantic instance identity and state while quantifying the scene for divergence calculation. For each pair, optimal subgraph is generated via LLM using barely basic actions to mitigate
hallucinations. Supportive actions for edge cases are reasoned and inserted with a lightweight discriminator. Driven by the divergence between the quantified baseline and current scene, it is easily extensible at low cost. Augmented subgraphs are topologically coordinated into global sequences while preserving internal behavioral coherence. Dynamic behavior trees
embedding atomic skills close the force-aware execution loop. Offline evaluation on 100 real-world scenes achieves 97.00% global executability, outperforming classical and state-of-the-art planners. Real-
robot deployment on a UR3 arm attains 90% success rate with 0.5 mm tolerance under strong interference, demonstrating a unified and verifiable solution for complex autonomous assembly.
Read the full abstract ▾
All authors Xinyi Li, Aiguo Song, Linhu Wei, Huijun Li
Robotics · Cross-listed from astro-ph.IM
On-sky demonstration of reinforcement learning for adaptive optics control
Jalo Nousiainen, Vincent Chambouleyron, Benoit Neichel et al. (+7 more)
Reinforcement learning (RL)-based
algorithms have recently emerged as a promising approach for adaptive optics (AO) control. In simulations and laboratory experiments, they have demonstrated robustness to real-world effects such as
photon and detector noise, misregistration, vibrations, and rapid variations in seeing conditions. However, their performance has not yet been validated on sky. We report the first on-sky demonstration of a reinforcement learning controller for adaptive optics, named Policy Optimization for AO (PO4AO). We further analyze its on-sky behavior and identify directions for improving the
algorithm and its implementation.PO4AO was implemented and deployed on the Papyrus adaptive optics system installed at the Coudé focus of the 1.52 m
telescope (T152) at the OHP. A Python-based implementation was interfaced with the existing real-time controller (DAO RTC) via shared-
memory buffers. The performance of PO4AO was compared to that of a standard integrator controller over several nights, covering a range of flux levels and atmospheric conditions. PO4AO consistently outperformed the standard integrator in all tested configurations. The controller successfully learned and compensated for vibration patterns and demonstrated strong robustness to measurement noise. Once tuned for Papyrus, PO4AO operated in a turnkey fashion, using a single set of hyperparameters across varying observing conditions and science targets. These performance gains were achieved despite a non-optimized Python implementation introducing approximately $750\,\mu\text{s}$ of additional latency, along with control jitter and occasional frame drops. When properly implemented and optimized, PO4AO constitutes a robust and high-performance turnkey controller for single-conjugate adaptive optics systems, paving the way for broader adoption of reinforcement learning strategies in on-sky AO operations.
Read the full abstract ▾
All authors Jalo Nousiainen, Vincent Chambouleyron, Benoit Neichel, Sylvain Cetre, Jean-Francois Sauvage, Angelie Alagao, Markus Kasper, Jonathan Dray, Romain Fetick, Byron Engler
Robotics
ros2probe: Non-intrusive, Kernel-selective Observability for Robot Operating System 2 Middleware
Jisang Yu, Sanghoon Lee, Yeonwoo Choi et al. (+1 more)
Robot Operating System 2 (ROS 2), the de facto standard middleware framework for
robots, runs each
robot as a graph of nodes communicating over the Data Distribution Service (DDS), a publish/subscribe substrate. Observing this inter-node communication in real time is essential to
robot development, yet it has a price. A tool can receive data only by joining the DDS domain as a subscriber that discovery has matched to the publisher, so observing folds the tool into the system it measures and perturbs it. We define this protocol-inherent perturbation as the observer's probe effect. It inflates the discovery plane, adds deserialization cost on the observer, makes the loss it reports diverge from what the subscriber actually received, and near saturation displaces the subscriber's messages. The only escape, capturing all wire traffic passively, discards ROS 2 message semantics and scales with total traffic, not what is observed. We present ros2probe, a non-intrusive observation framework that removes the probe effect. It reconstructs the full ROS 2 communication state from the domain's discovery packets at no bandwidth cost, then drives an in-kernel filter restricted to the topics the user asks for, lifting only those packets at minimal cost and observing what the real subscriber receives. Its interfaces and recordings match the standard ROS 2 tools. Across three hardware platforms (laptop, Jetson, and Raspberry Pi), two DDS implementations, and seven
robot-operation workloads, ros2probe holds the discovery graph within 0.5% of an unobserved system, whereas domain-joining tools inflate discovery up to 2.6$\times$ and drop 38.5% of the subscriber's messages at saturation while ros2probe drops none. It reports loss with a recall of 1.0, cuts observer CPU and
memory by up to 7$\times$ and 28$\times$, and stays practical on the embedded
robots where existing tools overload the system.
Read the full abstract ▾
All authors Jisang Yu, Sanghoon Lee, Yeonwoo Choi, Kyung-Joon Park
Robotics
Hand-centric Human-to-Robot Trajectory Transfer from Video Demonstrations via Object Category-agnostic Temporal Localization
Yitian Shi, Di Wen, Zhengqi Han et al. (+6 more)
Human
videos provide a scalable source of demonstrations for
robot imitation learning. However, converting them into executable and semantically aligned
robot trajectories requires determining when task-relevant interactions occur, how demonstrated human grasps should be retargeted across embodiments, and which post-contact motions should be reproduced. In this paper, we present HOCALT (Hand–Object Category–Agnostic Localization and Transfer), a hand-centric retargeting framework that unifies 3D hand motion reconstruction, temporal contact localization, and cross-embodiment trajectory generation. Given synchronized stereo
videos, HOCALT identifies task-relevant contact intervals by jointly
reasoning over reconstructed 3D hand articulation and category-agnostic object motion cues. These intervals serve as temporal anchors for initializing cross-embodiment transfer, where the demonstrated human grasps are retargeted into multi-modal
robot grasp hypotheses. Within each interval, these hypotheses are then propagated along the demonstrated hand motion, yielding executable trajectories. Finally, we augment the transferred trajectories to generate diverse variants from a single demonstration. Across various tasks, HOCALT outperforms VLM-based temporal localization baselines, achieving higher replay success than existing retargeting approaches.
Read the full abstract ▾
All authors Yitian Shi, Di Wen, Zhengqi Han, Zicheng Guo, Yu Hu, Edgar Welte, Kunyu Peng, Rainer Stiefelhagen, Rania Rayyes
Robotics
Pushing the Performance Limits in Autonomous Racing: Continuous Stability-Aware Adaptive Velocity Planning in Formula Student Driverless
Tamara Bergerhoff, Sebastian Baader, Pascal Meißner et al. (+1 more)
In autonomous racing, especially in competitions such as Formula Student Driverless, precise planning of the target velocity of a race car is crucial for competitive lap times and stable driving behavior. Especially at high speeds, Velocity Planning (VP) is a significant challenge as it has to be performed in real time, taking into account track layouts, environmental influences, mechanical tolerances, and the resulting control inaccuracies. In this paper, we present a novel approach to VP that dynamically adapts to such changing conditions. Instead of estimating the physical Tire-Road Friction Coefficient (TRFC), a continuous scaling factor is inferred indirectly from vehicle stability. This factor not only reflects the effective tire-road interaction but also captures effects of control inaccuracies. From this, we generate a continuous friction map, which serves as a robust, adaptive basis for computing the optimal target speed, accounting for both vehicle and environmental limits. Our proposed approach was evaluated on a real Formula Student race car, showing a lap time improvement of 35 % over ten laps and an average increase of 8 % compared to a non-adaptive approach.
Read the full abstract ▾
All authors Tamara Bergerhoff, Sebastian Baader, Pascal Meißner, Frank Deinzer
Robotics
Vehicle Prediction Model for Enhanced MPC Path Tracking in Formula Student Driverless
Sebastian Baader, Tamara Bergerhoff, Pascal Meißner et al. (+1 more)
Autonomous race cars, such as in Formula Student Driverless, operate close to their physical handling limits. The resulting highly nonlinear vehicle behavior increases the path tracking complexity, especially on narrow tracks. Model Predictive Control (MPC) is commonly used to address this issue, a method whose performance is closely tied to the accuracy of the underlying prediction model. This paper presents a novel, real-time capable prediction model for autonomous race cars that adjusts to changing conditions by combining information from past runs and the current driving situation. Our model is divided into three consecutive submodels: a nominal Kinematic Bicycle Model, an offline
Bayesian Linear Regression (BLR) model, and an online Sparse Gaussian Process Regression (SGPR) model. The proposed approach enables efficient integration of all available data without significantly increasing computational cost, ensuring high prediction accuracy and a quantitative uncertainty assessment right from the start of the run. Compared to existing approaches, an improvement in prediction accuracy of up to 57% was achieved. Further, we successfully demonstrated the practical applicability of the model within an MPC-based path tracking controller on a real Formula Student race car.
Read the full abstract ▾
All authors Sebastian Baader, Tamara Bergerhoff, Pascal Meißner, Frank Deinzer
Robotics
Self-Supervised Relevance Modelling in Autonomous Driving via Counterfactual Analysis
Luca Lusvarghi, Javier Gozalvez and Pablo Urbano Hidalgo
Autonomous driving relies on computationally intensive perception pipelines to continuously detect and track objects in the surrounding environment. While some objects are key to plan safe and effective maneuvers, others may not be relevant and have no impact on the autonomous vehicle's driving decisions. Focusing on relevant objects allows a more efficient usage of available computational resources, reduces processing latencies, and limits the downstream propagation of perception noise. In this work, we propose a novel self-supervised approach based on counterfactual analysis to develop a relevance model - an AI-based tool that quantifies the relevance of objects for an autonomous vehicle. To demonstrate the potential of the proposed approach, we train a relevance model on a synthetic causal
dataset generated in a selected urban scenario. Results show that the relevance model is able to accurately estimate the objects' relevance with millisecond-level latency, enabling real-time relevance estimation also in high-density scenarios. We also show that the relevance model can be used to build relevance heatmaps that offer valuable insights into the autonomous vehicle's driving policy and can be used to proactively inform perception and planning tasks. We openly release both the relevance model and the causal
dataset.
Read the full abstract ▾
Robotics
UniDexTok: A Unified Dexterous Hand Tokenizer from Real Data
Dong Fang, Youjun Wu, Yuanxin Zhong et al. (+4 more)
Dexterous hands are essential for fine-grained manipulation, but their hardware designs vary substantially across embodiments. Differences in kinematics, joint definitions, and degrees of freedom make it difficult to define a shared state representation compared with parallel grippers. As a result, dexterous-hand data remains fragmented and difficult to use for joint training. In this work, we propose the Unified Dexterous Hand Model (UDHM), which maps human and
robot hand states into a shared 22-DoF semantic interface. Based on UDHM, we introduce UniDexTok, a retargeting-free state tokenizer that learns embodiment-conditioned discrete
tokens from standardized real joint states. UniDexTok provides a unified representation for heterogeneous dexterous hands without relying on retargeting or simulation data. Compared with the recent baseline UniHM, UniDexTok reduces MPJAE from 15.63 degrees to 0.16 degrees and MPJPE from 18.51 mm to 0.18 mm, corresponding to error reductions of 98.98% and 99.03%, respectively. These results improve reconstruction from centimeter-scale to sub-millimeter accuracy. Experiments further show that data from other embodiments improves target-embodiment reconstruction accuracy, demonstrating the benefit of cross-embodiment tokenization. UniDexTok also shows strong zero-shot and few-shot reconstruction ability when new dexterous hands are introduced.
Read the full abstract ▾
All authors Dong Fang, Youjun Wu, Yuanxin Zhong, Rui Zhang, Yunlong Wang, Xiaosong Jia, Yu-Gang Jiang
Robotics
Planar-Sector LOS Guidance for Interception of Agile Targets with Lifting-Wing Quadcopters
Linkai Liu, Kun Yang, Han Zou et al. (+4 more)
Autonomous visual interception of agile aerial targets is challenging due to unpredictable target motion, limited sensing, and the strong coupling between target visibility and interceptor maneuverability. Most existing strapdown-camera interception methods preserve visibility using conic line-of-sight (LOS) constraints that keep the target near the image center. While safe, such symmetric constraints unnecessarily restrict maneuverability and can significantly reduce the usable thrust for pursuit. Motivated by the observation that aggressive FPV pilots do not maintain equal visibility margins in all image directions, this paper proposes a Planar-Sector Line-of-Sight (PS-LOS) guidance framework for autonomous interception using a lifting-wing quadcopter equipped with only a strapdown
monocular camera. PS-LOS tightly constrains lateral image error while relaxing longitudinal image error within a safe field-of-view margin, preserving visibility while releasing maneuverability for acceleration-intensive pursuit. Under the lifting-wing quadcopter model, PS-LOS provides nearly 50% more available thrust near the LOS direction than conventional conic LOS constraints. To realize LOS-only interception without direct depth measurements, a delay-compensated state-estimation framework and a nonlinear guidance-and-control architecture are developed for lifting-wing quadcopters. Extensive outdoor flight experiments demonstrate autonomous interception of agile targets exhibiting large-amplitude, high-frequency, and unpredictable motion under real wind disturbances. The proposed system achieves successful interceptions at ranges up to 138 m while maintaining continuous visual tracking throughout the engagement. The results validate PS-LOS as a visibility-preserving, maneuverability-aware guidance framework for long-range visual interception of agile aerial targets.
Read the full abstract ▾
All authors Linkai Liu, Kun Yang, Han Zou, Chen Min, Shuli Lv, Shuai Wang, Quan Quan
Robotics
Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations
Beomjun Kim, Seong Hyeon Park, Seunghoon Sim et al. (+3 more)
Robotic foundation models pre-trained on human demonstration
videos have shown promise, but a significant embodiment gap remains when the resulting policies are deployed on real
robots. A common remedy is to fine-tune these models on
robot-specific demonstrations. However,
robot data collection can be prohibitively expensive and time-consuming, which is particularly acute in dexterous manipulation, e.g., teleoperating a multi-fingered hand for even a single atomic task can take days. To address this, we introduce Dexterous Point Policy, a framework that learns dexterous manipulation policies directly from human
videos and requires no
robot demonstrations. Our core insight is that a unified 3D keypoint representation can bridge human and
robot embodiments when used for both observations and actions. Specifically, we extract 3D keypoints of task-relevant objects and human hands from raw
videos, and train an autoregressive
transformer over these keypoints. We observe that at the keypoint level, specifically the wrist and fingertips, human and
robot behaviors closely align, enabling direct policy transfer. On a suite of real-
robot tasks spanning pick-and-place and tool use, Dexterous Point Policy attains 75.0% success, whereas a state-of-the-art VLA baseline reaches only 1.0%. Furthermore, our method generalizes strongly to unseen scenarios, including multi-object environments and novel object categories.
Read the full abstract ▾
All authors Beomjun Kim, Seong Hyeon Park, Seunghoon Sim, Seungjun Moon, Sanghyeok Lee, Jinwoo Shin