Robotics · Cross-listed from cs.CV
Scene-Q: Confidence-Aware Coarse-to-Fine Querying of 3D Scenes with Selective VLM Reasoning
Juno Kim, Yesol Park, Hye-Jung Yoon et al. (+1 more)
Indoor mobile
robots require open-vocabulary scene understanding that grounds natural-
language queries in a consistent 3D map. Many existing systems ultimately rely on cosine-similarity
retrieval with contrastive image–text encoders, which is efficient but brittle when labels are near-synonymous or multiple similar instances appear. We present Scene-Q, a confidence-aware coarse-to-fine querying framework that normalizes encoder scores with temperature scaling and selectively invokes a
reasoning VLM only for low-confidence cases. High-confidence queries are answered by fast
retrieval, while ambiguous ones are reranked over a small top-K candidate set using the original multi-view images and instance bounding boxes, enabling context-aware disambiguation at low cost. Scene-Q improves open-vocabulary 3D instance
segmentation on ScanNet200 and natural-
language 3D instance
retrieval on real-world reconstructions, with the largest gains on spatial and relational queries while keeping a substantial fraction of queries on the fast path.
Read the full abstract ▾
All authors Juno Kim, Yesol Park, Hye-Jung Yoon, Byoung-Tak Zhang
Robotics · Cross-listed from cs.LG
MetaKoopman: Bayesian Meta-Learning of Koopman Operators for Modeling Structured Dynamics under Distribution Shifts
Mahmoud Selim, Sriharsha Bhat and Karl H. Johansson
Modeling and
forecasting nonlinear dynamics under distribution shifts is essential for robust decision-making in real-world systems. In this work, we propose MetaKoopman, a
Bayesian meta-learning framework for modeling nonlinear dynamics through linear latent representations. MetaKoopman learns a
Matrix Normal-Inverse Wishart (MNIW) prior over the Koopman operator, enabling closed-form
Bayesian updates conditioned on recent trajectory segments. Moreover, it provides a closed-form posterior predictive distribution over future state trajectories, capturing both epistemic and aleatoric uncertainty in the learned dynamics. We evaluate MetaKoopman on a full-scale autonomous truck and trailer system across a wide range of adverse winter scenarios, including snow, ice, and mixed-friction conditions, as well as in simulated control tasks with diverse distribution shifts. MetaKoopman consistently outperforms prior approaches in multi-step prediction accuracy, uncertainty calibration, and robustness to distributional shifts. Field experiments further demonstrate its effectiveness in dynamically feasible motion planning, particularly during evasive maneuvers and operation at the limits of traction. Project website: https://mahmoud-selim.github.io/MetaKoopman/
Read the full abstract ▾
Robotics
Reeling It In: Flexible Needle Pick Up via Thread Manipulation for Autonomous Suturing
Emma Huang, Zih-Yun Chiu, Neelay Joglekar et al. (+2 more)
Suture-needle pickup is necessary for autonomous suturing, as a needle can be unexpectedly dropped or strategically released to adjust the grasping configuration. Current methods for autonomous needle pickup typically guide a
robot to move straight toward the needle and grasp it, limited to conditions where the needle is observable and directly approachable. In addition, grasping the needle lying on tissue can lead to the
robot pinching nearby tissue or the needle jumping around due to its slippery surface, posing potential safety issues. This work proposes an autonomous framework that uses a suture thread as an assistive tool for indirect needle pickup, avoiding unnecessary tool-tissue contact and enabling pickup even when the needle is occluded or inaccessible. The framework spans the entire workflow, including thread and tissue reconstruction, safe grasp-point selection, stable thread lifting, and bimanual thread-following until securing needle grasping. The
robot policies account for visual uncertainty to maximize robustness in real-world environments. We evaluate the proposed framework on a da Vinci Research Kit under various real-world conditions. The results demonstrate robust performance even with a challenging thread configuration or a non-approachable needle, closing the gap in applying autonomous
robot policies to unstructured suturing environments.
Read the full abstract ▾
All authors Emma Huang, Zih-Yun Chiu, Neelay Joglekar, Shanglei Liu, Michael C. Yip
Robotics · Cross-listed from cs.LG
Learning Implicit Causal World Models from Multi-Agent Demonstrations
Jasorsi Ghosh
In model-based reinforcement learning, world models exist as internal simulators, but their training often conflates statistical correlations with causal mechanisms. This problem is exacerbated in multi-
agent systems where physical transitions are intertwined with strategic
agent intents, causing world models to fail under distribution shift. We introduce Implicit Causal World Models to recover environmental dynamics from offline demonstrations without requiring pre-defined causal graphs. By incorporating policy variance, we render world models discoverable via the sequential
backdoor condition. Evaluations across coordination tasks (Two-Door,
Navigation, and Giveway) demonstrate that these models provide interpretable causal representations under both full and partial observability, with model accuracy scaling directly with interventional strength.
Read the full abstract ▾
Robotics
MoMo: Dial Motion Mode in Robot Manipulation with Spatiotemporal Action Tokenization
Yuhan Hu, Hugues Thomas, Peide Huang et al. (+3 more)
To operate effectively across diverse contexts,
robots must not only perform manipulation tasks accurately but also adapt how their actions unfold to the task, object, and interaction setting. We ask whether this execution-level variation can be learned as a reusable behavioral factor shared across tasks. We present MoMo, a two-stage imitation-learning framework consisting of a spatiotemporal action tokenizer and a behavior-cloning
transformer that takes task and a continuous motion-mode condition as inputs. Across six real-
robot manipulation tasks, varying this condition produces steady, dynamic, and intermediate behaviors that human raters can distinguish and that differ in joint speed, acceleration, and end-effector approach pitch. On tasks demonstrated in only one mode, MoMo transfers the unseen requested mode while largely preserving task success. Together, these results provide evidence of compositional generalization to unseen task–mode combinations and show that motion mode can be reused across tasks to control how a manipulation skill is performed.
Read the full abstract ▾
All authors Yuhan Hu, Hugues Thomas, Peide Huang, Mouli Sivapurapu, Benoit Landry, Arto Kivila
Robotics · Cross-listed from cs.CV
HeteroPROMPT: A Real-time and Privacy-Preserving Heterogeneous Collaborative Perception Framework
Armin Maleki and Hayder Radha
Collaborative Perception (CP) improves autonomous systems' awareness of their surroundings by sharing sensor data, intermediate features, and detection results. In real-world deployments, however, collaborating vehicles often use heterogeneous sensors, perception models,
datasets, and training domains, creating feature-space shifts that degrade downstream fusion and detection. Existing approaches typically retrain fusion and detection components or introduce modality-specific feature interpreters. These methods scale poorly to newly joining
agents and often require access to proprietary metadata, raising
privacy concerns. We propose HeteroPROMPT, a real-time and
privacy-preserving framework for heterogeneous collaborative perception. HeteroPROMPT rapidly aligns each heterogeneous
agent's features with an ego-centric unified feature space through modular
prompts and lightweight learning-based tuning, while keeping
agent encoders and the collaborative fusion and detection stacks frozen. Its visual
prompt-based training and inference modulate Bird's Eye View (BEV) features across channels and spatial locations with low computational overhead. For metadata-free deployment, an autoencoder learns a compact unified representation and extracts modality cues from shared features, enabling real-time modality classification and routing to the appropriate HeteroPROMPT modules without exposing proprietary
agent information. Experiments on the OPV2V-H and V2XSet
datasets show that HeteroPROMPT improves Average Precision over state-of-the-art heterogeneous CP methods while using orders of magnitude fewer trainable parameters. This offers a scalable and practical CP solution. The proposed modality classifier also predicts the joining
agent's modality from compact features with greater than 99.99 percent accuracy during deployment. Code will be available at https://github.com/arminmaleki007/HeteroPROMPT.
Read the full abstract ▾
Robotics
Multi-Objective Compliance-Integrated Coevolution For Simulated And Real-World Deployment Of Multi-Robot Marine Autonomy
Everardo Gonzalez, Tyler M. Paine, Manuel Agraz Vallejo et al. (+3 more)
Collaborative
robots are well-suited to maritime missions that benefit from coordination, such as the exploration of unknown reef structures, inspection of subsea infrastructure, or search-and-rescue operations. These missions typically provide sparse feedback signals for measuring progress and require adherence to safety and regulatory norms, turning a mission into a multi-objective optimization problem. Coevolutionary
algorithms can process these sparse feedback signals to generate coordinated behaviors, and in some cases extend behaviors to multiple objectives. However, incorporating high-level team objectives with low-level compliance considerations on the fly to balance norm adherence with team performance remains elusive. This paper introduces a multi-objective framework that blends coevolved behaviors with compliance behaviors to achieve a balance between maximizing team progress and minimizing norm violations. The key insight is to decouple learning from compliance since operational norms are prescribed rather than discovered. We demonstrate that our framework achieves high team performance while avoiding collisions on a collaborative swimmer rescue mission with up to 8 vehicles in a hardware deployment, and 12 vehicles in simulation. The key contribution of this paper is Marine Multi-Objective Compliance-Integrated Coevolution (MMOCIC), a framework that blends team-wide optimization with established norms for real-world deployments of learning-based coordination.
Read the full abstract ▾
All authors Everardo Gonzalez, Tyler M. Paine, Manuel Agraz Vallejo, Gaurav Dixit, Michael R. Benjamin, Kagan Tumer
Robotics
Jian Zhou, Xunyi Zhao, Gengze Zhou et al. (+4 more)
Autonomous
embodied agents must sustain a long decision-making loop of perceiving, acting, verifying, and self-correcting. In trained end-to-end policies and hand-designed workflows, the interaction structure remains externally specified. We study how much of this loop a general-purpose model can direct through a minimal interface. We term this organization agentic
embodied control: the
reasoning model directly steers every action, keeping
reasoning and control aligned. We evaluate fourteen frontier
multimodal models as provided by their vendors, with
monocular RGB and primitive actions across six
navigation and
embodied question-answering
benchmarks. On 100-episode subsets, the best
agents exceed published scores and approach or surpass our human tester. Who directs the loop matters: SmartWay and O2C-Nav using the same strong models achieve substantially lower success rates on R2R-CE. The lightweight mini-swe-
agent harness performs comparably to vendor harnesses. Stronger models benefit from optional rather than imposed waypoints; adding SLAM raises R2R-CE success to 88%. Physical trials expose limitations in body awareness, execution feedback, and persistent spatial understanding.
Read the full abstract ▾
All authors Jian Zhou, Xunyi Zhao, Gengze Zhou, Zerui Li, Sihao Lin, Jiajun Liu, Qi Wu
Robotics
Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels
Xinyu Yang, Tianxing Chen, Honghao Su et al. (+38 more)
Embodied intelligence integrates learned perception and decision making with real-time computation, control, and physical interaction. Because failures can cause immediate physical or operational harm, task completion alone does not establish trustworthiness. We define trustworthy
embodied intelligence as the sustained capacity to execute specified tasks reliably under environmental and system variation while maintaining risk within acceptable bounds. We term this objective sustained safe success. Its supporting mechanisms are organized into four interdependent layers. The model layer generates task-competent action proposals with calibrated uncertainty and explicit safety preferences. The system layer realizes authorized actions dependably through integrated sensing, computation, control, hardware safeguards, fault containment, and fallback. The evidence layer substantiates bounded claims through evaluation, verification, validation, traceability, and structured assurance arguments. The deployment layer maintains claim validity through runtime monitoring, authority management, intervention, incident response, and controlled updates. Because assumptions and failures propagate across these layers, neither model capability, isolated safeguards, nor
benchmark performance alone can establish end-to-end trustworthiness. Drawing on
embodied AI, robotics, control, dependable computing, distributed systems, and autonomous driving, we further propose a non-normative hierarchy of trustworthiness levels. This hierarchy grades the strength of bounded deployment claims across task capability, safety, system assurance, operational governance, and supporting evidence, providing a basis for bounded deployment, comparative evaluation, research prioritization, and future standardization.
Read the full abstract ▾
All authors Xinyu Yang, Tianxing Chen, Honghao Su, Minxuan Wang, Chenze Yu, Zhangzheng Tu, Yue Chen, Yuxiao Huo, Lingfeng Zhang, Yan Huang, Yan Qin, Shaolong Zhu, Qiwei Liang, Hekun Tian, Shujia Liu, Guangyu Chen, Junhao Gong, Zixuan Li, Wenwei Lin, Zijian Lin, Wenxuan Zhu, Eric J Chen, Yue Yuan, Qize Yu, Jiaqi Liang, Haowen Yan, Hengfei Zhao, Weijie Wan, Zikun Xiao, Junyuan Tang, Baijun Chen, Kai-Chong Lei, Kaixuan Wang, Kailun Su, Zanxin Chen, Yao Mu, Renjing Xu, Chuqiao Lyu, Qi Xiong, Ping Luo, Wenbo Ding
Robotics
INTACT: Isomorphic Intent-to-Action Learning for Search-Free World Models
Junhan Sun, Hao Zhao and Guofeng Zhang
Forward latent world models predict how actions change a scene, but recover actions for a desired change only through expensive test-time search. We introduce INTACT (INtent-To-ACTion), an end-to-end JEPA that turns action-labeled, reward-free trajectories into a deployable intent-to-action interface. Each transition supplies physical intent $z_{t+1}-z_t$, while a future goal supplies deployment intent $\operatorname{sg}(z_g)-z_t$. The architecture is isomorphic between the local and goal motion-intent backbone-input graphs through an identical four-slot grammar and shared parameters, and between supported local and goal motion-intent families through action-law semantics induced by the same predictor rather than pointwise latent equality. INTACT also provides intact transfer from RGB evidence to action-effective latent intent coordinates and from intent families to their corresponding action-law families. Asymmetric endpoint gradients ground physical successors and fix future goals as anchors, joining representation learning and control without pointwise latent matching or globally linear dynamics. The resulting coordinates support a robust distributional action law: its conditional mean serves directly as a search-free policy, while sampling remains available for diversity or optional verification. On the four official LeWM tasks, one-epoch, zero-search models reach 85.78%, 100.00%, 97.67%, and 97.89% success. Optional local CEM centered on the Direct plan reaches 96.86% macro success using 384 instead of 9,000 candidate sequences, reducing sampling by $23.44\times$ while improving pure CEM by 16.00 points. One shared four-task encoder reaches 89.39% E5 Direct macro and improves every task over jointly trained LeWM, while predicted–expert action-family kNN tracks Direct success at $r=0.954$. Direct inference takes 2.9–5.5 ms.
Read the full abstract ▾
Robotics
$\pi\mathbf{R}^2$: Reactive Real-time Flow Policies
Sungjae Park and Shubham Tulsiani
Generalist manipulation policies increasingly take the form of action-chunking flow policies built on large pretrained backbones. Such chunks run open-loop, so the policy cannot react to sensory input arriving mid-execution, sacrificing reactivity. Replanning more often would restore it, but the perception-to-action pipeline (a large backbone plus multiple denoising steps) is too slow: this latency forbids frequent replanning and leaves committed actions stale, making such policies ill-suited for dynamic, closed-loop control. We present $\pi\mathbf{R}^2$, which makes these policies reactive and real-time while retaining large backbones, expressive multi-modal policies, and multi-action prediction. Built on the per-position noise schedule of
diffusion forcing, $\pi\mathbf{R}^2$ contributes two ideas. First, it splits conditioning into a fast channel (proprioception, fresh every tick) and an asynchronously updated slow channel (
vision-
language features), so the policy reacts to proprioception within a chunk while tolerating stale
vision. Second, a latency-adaptive flow schedule treats in-flight actions as inpainting conditioning and emits actions in one denoising step per call, letting one trained model adapt to varying hardware latency. Requiring minimal modification to existing architectures, $\pi\mathbf{R}^2$ can be finetuned from a pretrained policy: applied to GR00T-N1.7 on a real xArm6+XHand platform, it replans closed-loop roughly $4\times$ faster than the base policy (~$25$Hz on an A5000 GPU), acting on a fresh observation every $40$ms. Across simulation and real-world manipulation tasks, $\pi\mathbf{R}^2$ improves the success rate by up to $23\%$ in simulation and $30\%$ in the real world over the strongest baseline. Project page: https://pi-r2-flow.github.io/
Read the full abstract ▾
Robotics
S2A2: Audio-Visual Imitation Learning for Manipulation Tasks Using Acoustic Spatial Information
Kaneyoshi Hiratsuka, Benjamin Yen and Ryosuke Kojima
Acoustic information provides rich cues about object location, material properties, and changes caused by contact or motion. This paper introduces a new set of acoustic-aware manipulation tasks for imitation learning, in which
robots must use auditory cues to determine manipulation targets. These tasks require sound source localization and identification for active exploration in robotic manipulation. Also, we propose a
multimodal imitation learning framework, Spatial-Spectral Audio Action (S2A2), that integrates visual features with acoustic spatial and acoustic signal information for the acoustic-aware manipulation tasks. We implemented S2A2 models that integrates policies such as ACT,
Diffusion Policy, VQ-BeT, and $\pi_0$, into our framework. Simulation experiments showed that the proposed method is the most effective for tasks requiring both position and timbre. Furthermore, real-
robot experiments confirm the applicability of the proposed tasks and framework to real-world manipulation.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Pictura: Perspective-View Self-Play at Scale for Driving
Yuan Yin, Elias Ramzi, Marc Lafon et al. (+8 more)
Self-play in simulation produces robust driving policies at scale. Demonstrations of such behavior have been made using privileged vectorized observations such as exact poses and velocities, even for occluded
agents. This assumes that perception is solved and introduces a representation gap with the partial observation of a deployed
agent driving from the perspective view of
egocentric cameras. A common fix, distilling the privileged policy into a camera-input student, leaves the student imitating decisions its own view cannot justify. Instead, we establish perspective-view self-play as a practical training regime. We introduce Pictura, a GPU-accelerated multi-
agent driving simulator that renders each
agent's
egocentric view at every step, mitigating the representation gap at its source. Pictura sustains up to 500K
agent-steps/s (2M images/s) on a single H100. Using Pictura, we train Alberti by self-play with plain PPO. It is the first large-scale driving self-play policy trained directly from perspective images, without privileged observations. Training spans 50B
agent steps for ~35M km of driving. It approaches the driving performance of its privileged vectorized counterpart, and transfers zero-shot to Waymo Open Motion
Dataset layouts re-rendered in Pictura, where it outperforms privileged vectorized
agents. Project page: https://valeoai.github.io/Pictura/
Read the full abstract ▾
All authors Yuan Yin, Elias Ramzi, Marc Lafon, Valentin Charraut, Victor Bares, Yihong Xu, Éloi Zablocki, Alexandre Boulch, Thibault Buhet, Andrei Bursuc, Matthieu Cord
Robotics · Cross-listed from cs.NI
Untangling Co-Drift: Proactive Multi-Intent Failure Prediction and Root-Cause Disambiguation for Self-Driving Networks
Md. Kamrul Hossain and Walid Aljoby
The
vision of self-driving networks that monitor, reason, and act upon themselves with minimal human intervention relies on tightly coupled monitoring, analytics, and actuation functions. In this work, we treat these functions as three operational macro-intents: continuous telemetry, real-time analytics, and programmatic actuation, and formalize the health of each function as an intent that the network must continuously satisfy. A critical, yet underexplored, challenge stems from the causal coupling among these intents, where a singular fault within one macro-intent propagates as a co-drift and subsequently triggers cascading, symptomatic anomalies across the remaining intents. This ambiguity makes it exceedingly difficult for existing, reactive approaches to distinguish the true root-cause intent from symptomatic victim intents, and their reliance on threshold-crossing detection leaves insufficient time for proactive remediation. We introduce MILD, a novel framework that reformulates intent assurance from reactive drift detection to proactive failure prediction. Grounded in our three-macro-intent formulation of the self-driving control loop, MILD employs a teacher-augmented Mixture-of-Experts architecture with a hybrid objective that jointly optimizes intent failure prediction and root-cause attribution. MILD enables KPI-level diagnostics via SHAP explainability and dynamic intent failure urgency estimation via multi-horizon modeling. Our extensive evaluation of MILD across three environments of increasing realism, from a controlled statistical
benchmark, to a microservices application, to an SDN-based edge-to-cloud testbed, demonstrates that MILD achieves high failure detection rates, strong remediation lead times, and accurate intent-level root-cause disambiguation. This positions MILD as a practical enabler of closed-loop assurance in next-generation autonomous networks.
Read the full abstract ▾
Robotics
Physics-Aware End-to-End Deep Reinforcement Learning for Quadcopter Control with Actuator Dynamics
Ya-Chia Shen and Woei-Leong Chan
Unmanned aerial vehicles (
UAVs), particularly quadcopters, present unique challenges for autonomous control due to their underactuated dynamics: only four available control inputs must govern six degrees of freedom. This paper investigates a physics-aware, end-to-end deep reinforcement learning (DRL) approach that acts directly on low-level body inputs, total thrust and body torques $(T, \tau_x, \tau_y, \tau_z)$, and closes the loop through a high-fidelity Simulink environment. Our simulator integrates a 12-state rigid-body model (MATLAB Level-2 S-Function) with (i) an Action2RPM allocation based on the Moore-Penrose pseudo-inverse of a coefficient
matrix derived from thrust and drag terms, and (ii) first-order actuator dynamics for each motor (time constant $T_m = 0.076$ s), including rotor gyroscopic coupling. A shaped reward balances goal-reaching and stability using an exponential position well, attitude penalties, and quadratic velocity costs. Four DRL
algorithms, DDPG, TD3, PPO, and SAC, are evaluated in two stages: (S1) thrust-only hover and (S2) hover with pitch torque and a translated goal. Results show that SAC and TD3 achieve superior stability and exploration efficiency, while PPO is less sample-efficient. The study highlights the significance of modeling actuator lags and aerodynamic moments for stable low-level control and provides a reproducible
benchmark for quadcopter DRL.
Read the full abstract ▾
Robotics
DC-WAM: Dynamic-Centric Visual Supervision and Reasoning for World-Action Models
Haoyuan Ji, Lingxiang Fan, Shang Su et al. (+4 more)
World-Action Models (WAMs) augment
robot policies with future visual prediction, but it remains unclear what the visual modality should learn for control. While photorealistic future prediction provides dense supervision, it also incurs substantial computation and can allocate capacity to texture, illumination, and background variations that are only weakly related to action selection. Recent efficient WAM variants suggest that the main benefit of the
video branch may not lie in the rendered future itself, but in the control-relevant visual representations induced during training. In this work, we revisit future
video prediction from a dynamic-centric perspective and ask whether an existing RGB-based WAM can be redirected from appearance-dominated reconstruction toward interaction-induced visual dynamics without introducing additional modality-specific predictions or online inputs at deployment. We propose DC-WAM, a dynamic-centric WAM framework that redistributes supervision and computation in the RGB
video branch. At the supervision level, DC-WAM combines temporal-difference flow matching with trajectory-guided weighting, emphasizing dense temporal changes and localized regions where the gripper, manipulated objects, and contact areas move. At the
reasoning level, DynaRoute predicts
token-wise dynamic relevance and converts it into an
attention bias, guiding the model toward control-relevant future
tokens. Experiments in simulation and on real-world manipulation tasks show that DC-WAM consistently improves policy performance, especially under out-of-distribution perturbations in lighting, object appearance, and background texture.
Read the full abstract ▾
All authors Haoyuan Ji, Lingxiang Fan, Shang Su, Yinqiao Lu, Mengkai Shi, Jun Gao, Shuo Feng
Robotics
SAM3D-Guided Object-Centric Representation Alignment for Vision-Language-Action Models
Zonghe Liu, Shanyuan Jie, Xiaoquan Sun et al. (+4 more)
Vision-
Language-Action (VLA) models have shown strong potential for general
robot manipulation, but most existing models rely on 2D visual-
language backbones and lack fine-grained 3D understanding of target objects, especially under occlusion, pose variation, scale changes, and precise spatial interaction. We propose an object-centric 3D representation
alignment framework built upon $\pi_0$, using SAM3D as a frozen 3D teacher to provide target-object 3D priors during training. Specifically, we localize task-relevant objects with object recognition models, generate corresponding object masks, and use SAM3D to extract dense object-level 3D representations, which are aligned with intermediate visual features of $\pi_0$. This enables the policy to internalize target-object 3D information while preserving the original RGB-
language-to-action inference pipeline without requiring depth, point clouds, masks, SAM3D, or additional 3D modules at test time. Simulation experiments show consistent improvements, achieving 99.1% on LIBERO and an average length of 4.11 on CALVIN. Real-world experiments further demonstrate that our method is particularly effective in long-horizon manipulation scenarios where the
robot must focus on different target objects across multiple subtasks.
Read the full abstract ▾
All authors Zonghe Liu, Shanyuan Jie, Xiaoquan Sun, Chen Cao, Zetian Xu, Zongsheng Liu, Jiayu Chen
Robotics
HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone
Simple AI: Yuteng Wei, Jinming Ma, Jiawei Wang et al. (+14 more)
Learning deployable manipulation policies is bottlenecked by the scarcity of data that is both high-fidelity and scalable. Real-
robot teleoperation is accurate but costly to scale;
robot-free UMI capture scales readily, and current practice uses the resulting data mainly for pre-training, adding a small real-
robot "anchor" at post-training. We ask whether raising the fidelity of
robot-free UMI data, rather than shrinking the real-
robot fraction, can remove that anchor. We present HiFi-UMI, a portable UMI data-production system co-designed for trajectory accuracy, inter-gripper relative pose, synchronization, and field of view: head-mounted offline stereo-inertial SLAM, native rather than reconstructed relative pose, a shared microsecond GPIO trigger, and two wide-angle cameras per hand covering ~200 degrees. It reaches 3 mm workspace-local end-effector accuracy without external tracking infrastructure. Using this corpus, we demonstrate zero-
robot post-training: a policy post-trained solely on HiFi-UMI demonstrations deploys directly on a real
robot and matches in-domain teleoperation across three backbones spanning the
vision-
language-action and world-action-model families, with success-rate differences of -2.5, +3.1, and -0.6 percentage points on StarVLA-QwenPI, OpenPI-pi_0.5, and LingBot-VA; the strongest policy reaches 85% on a precision insertion task, even though the teleoperation baseline is collected in the evaluation scene and no HiFi-UMI trajectory is. Pre-training on 4,000 hours from the same corpus lowers action error on ten unseen tasks by 41% and, on StarVLA-QwenPI, raises real-
robot success by a further 18.1 percentage points. We open-source HiFi-UMI-2K, 2,000 hours of microsecond-synchronized, ultra-wide-FoV demonstrations, each automatically reconstructed and validated through simulation replay, as a large-scale, high-fidelity resource for the
robot-learning community.
Read the full abstract ▾
All authors Simple AI: Yuteng Wei, Jinming Ma, Jiawei Wang, Weitao Zhou, Yushen Zuo, Ke Rui, Minglei Li, Jinhao Zhang, Zhikang Pan, Xiang Wang, Haoran Jia, Huan Du, Zicheng Zeng, Jun Ma, Guiyu Qin, Di Zhang, Xiaofei Li
Robotics
Modular Robotic Catheters for Endovascular Aneurysm Repair
Alex Ranne, Jinshi Zhao, Ali Anil Demircali et al. (+5 more)
Fenestrated/Branched endovascular aneurysm repair (FEVAR/BEVAR) require surgeons to navigate catheters and guidewires into various branches of the abdominal aorta, before deploying stent grafts to alleviate pressure on the aneurysm. Previous clinical studies suggests that surgeons continue to struggle with vessel access using standard commercial instruments, prolonging the procedural time and inducing further complications. In this work, we present two contributions to solving this problem: 1) A bespoke 2-segment steerable catheter, consisting of 4 degrees of freedom to enhance dexterity. 2) An expandable, modular tendon-driven actuation platform that can accommodate for the redundancies introduced in our system. To fabricate the catheter, we capitalized on thermal fiber drawing, a technique that creates high-aspect ratio devices at scale, and processed the catheter with
laser micro-machining to soften its tip. We evaluated the system using simulations, where we investigated the catheter's bending stiffness, then its steerability with in-vitro experiments in vascular phantoms. This handheld, robotic steerable catheter system has the potential to shorten the length of future endovascular surgeries, and give clinicians the tools to resolve challenging clinical cases.
Read the full abstract ▾
All authors Alex Ranne, Jinshi Zhao, Ali Anil Demircali, Songli Moey, Ayhan Aktas, Burak Temelkuran, Nassir Navab, Ferdinando Rodriguez y Baena
Robotics
Huy Ha, C. Karen Liu and Shuran Song
An often overlooked factor of
robot manipulation performance is the embodiment of the
robot itself. Motivated by this problem, we study motion-conditioned
robot co-design, where the goal is to generate complete
robot designs that track target end-effector trajectories (from human demonstrations) while optimizing user-defined rewards. We introduce
Transformer Transformer, a
diffusion transformer trained on RoboTokens, a unified tokenization of
robot embodiments, states, and actions. The same architecture can be used across embodiment spaces (e.g., wheeled bimanual, quadrupeds,
humanoids) and use cases (embodiment generation, cross embodiment controller). Rather than overfitting to one reward function,
Transformer Transformer is a dynamics model, whose reward-agnostic state and action predictions can be converted into reward-specific value predictions. These value predictions are used to steer embodiment
diffusion towards high value
robot designs, through a procedure we call Dynamics Self-Guidance. Experiments across multiple design spaces show zero-shot optimization of unseen rewards and trajectories, improving performance and runtime over the evolutionary baseline. Finally, we fabricated an optimized ALOHA design, which reduced tracking error by over 70% compared to the original design.
Read the full abstract ▾
Robotics
Motion-Acceleration Calibration and Compensation in IMUs without External Equipment for Attitude Estimation Filters
Fabian Arzberger and Andreas Nüchter
Attitude estimation based on inertial sensing requires measurements of local angular velocities and local
gravity via gyroscopes and accelerometers. However, during the motion of a mobile system the inertial measurement unit (IMU) will be subject to additional accelerations which skews the measurement of local
gravity. This effect gets amplified the further away the IMU is from the base of the system. Many attitude estimation filters, such as "Madgwick" or "Mahony", account for this by relying more on gyroscope integration for periods of high angular velocity. However, this approach is prone to accumulate long term error especially around the
gravity vector. In this work we utilize the gyroscope measurements to compensate the additional accelerations induced by the motion of the system, i.e., centripetal- and tangential-accelerations. Additionally, we introduce a calibration method that estimates intrinsic IMU parameters such as axes misalignment, bias, scale, as well as the extrinsic base-to-IMU vector without the necessity for additional external equipment. Our evaluation in simulation as well as in the real-world shows that this method improves any attitude filter that relies on the direction of
gravity. Furthermore we demonstrate the effectivenes on highly dynamic systems, and systems that are unable to put the IMU at the center of rotation, using our real-world spherical mobile mapping system.
Read the full abstract ▾
Robotics
Tripody: An Overconstrained 3-SPR-like Parallel Robot for High-Reach Construction Tasks
Julien Kindle, Jakub Raczy, Riccardo Balbi et al. (+3 more)
Many ceiling construction tasks still rely on heavy serial manipulators that are difficult to deploy in cluttered interiors, motivating lightweight, field-ready alternatives that reach ceiling height while maintaining millimeter-level accuracy and the stiffness demanded by overhead tool loads. We introduce Tripody, a wheeled 3-DoF parallel
robot for high-reach tasks that replaces the base spherical joints of a classical 3-SPR (3 legs; S: base spherical joint; P: actuated prismatic joint; R: end-effector revolute joint) morphology with universal joints, intentionally overconstraining the mechanism; small, distributed elastic deflections absorb the resulting incompatibilities, preserving predominantly translational motion. The 33kg system extends from 1.7m to 3.4m in height, supports a continuous 32kg payload, and offers a modular end-effector interface for ceiling operations. We detail the mechanical design - including custom linear actuators and a kinematic-compatibility analysis - and a control stack for accurate positioning that combines SE(3) state estimation, forward kinematics, and task-space control. In experiments, Tripody exhibits similar in-plane stiffness to a spherical-base variant but substantially higher torsional stiffness - an increase of 67% at 1.7m, 196% at 2.6m, and 454% at 3.4m - while maintaining negligible cross-axis coupling. Closed-loop positioning with a total station converges below 0.6mm across the entire workspace; pure model extrapolation achieves a 95th-percentile error of 2.7mm (max 3.6mm). Finally, we demonstrate task-level ceiling-drilling feasibility in an open-loop study by drilling a 15-hole pattern with 4.5mm maximum relative hole-position error after rigid
alignment. These results support overconstrained, compliance-absorbing 3-SPR-like architectures as a practical path to lightweight, high- reach, millimeter-accurate construction
robots.
Read the full abstract ▾
All authors Julien Kindle, Jakub Raczy, Riccardo Balbi, Andrea Alessandretti, Cesar Cadena, Marco Hutter
Robotics
Cooperative Multi-UAV Navigation in Complex Environments via Systematic Multi-Agent Deep Reinforcement Learning
Yu Su and Nabil Aouf
Cooperative
navigation of multi-
agent UAVs in complex environments faces key challenges including local optima traps, sparse rewards, learning imbalance among
agents, and insufficient cross-scenario generalisation. This paper proposes a multi-
agent deep reinforcement learning framework that addresses these issues through coordinated exploration, demonstration exploitation, safe curriculum scheduling, and structure-aware generalisation. First, a perception mechanism combining
memory of visited states, directional novelty estimates, and penalty backpropagation enables
agents to proactively detect and escape local optima. Second, a hierarchical collaborative demonstration buffer with tiered behaviour cloning manages trajectories by degree of team collaboration and applies differential supervision to the actor network, improving demonstration utilisation under sparse collaborative signals. Third, a safety-aware dual-condition curriculum scheduling mechanism reviews mastered scenarios through back-testing and experience pre-filling during training, suppressing catastrophic forgetting while ensuring both task performance and flight safety. For generalisation, local
geometric features computed from sensor readings are abstracted into a domain parameter, through which a structure-aware gating network and mixture-of-experts mechanism condition the policy on local structural patterns rather than scenario-specific coordinates, enabling cross-scenario transfer without exposure to the target environment. The framework is further validated under mixed static-dynamic obstacle settings, showing robust adaptability to dynamic disturbances. Simulation results confirm strong performance in collaboration success rate,
navigation robustness, zero-shot cross-scenario generalisation, and dynamic environment adaptability.
Read the full abstract ▾
Robotics
Tri-Manual Visuomotor Imitation Learning of Robot Policies
James Zhao, Mingyuan Ba and Weiming Zhi
Bimanual teleoperation provides an effective way to collect
robot demonstrations, but it assumes that the operator and
robot have matching numbers of simultaneous control channels. This assumption breaks for tri-manual systems: the
robot can coordinate three arms concurrently, whereas a single operator can continuously control only two. Pairwise mode switching may therefore record otherwise independent motions sequentially, causing behaviour cloning to reproduce delays imposed by the interface rather than required by the task. We present TriManPolicy, a tri-manual imitation learning system that allows one operator to demonstrate behaviours for three arms. Its central component is Dependency-Aware Tri-Arm Scheduling (DATS). The key idea is to preserve the demonstrated arm motions while reconsidering when they occur. DATS retimes demonstrations offline by preserving local sensorimotor segments of fixed duration and repositioning them according to constraints on task order and arm usage that are reviewed by a human. The resulting data train a single synchronous policy for all three arms, while deployment requires neither the dependency graph nor the scheduler. Across six challenging tasks performed in the real world, policies trained on demonstrations retimed by DATS exhibit more efficient coordination while maintaining comparable observed task success. Offline analysis further shows that DATS changes the supervision across arms rather than merely removing idle periods. Project
videos and additional material are available at https://aus.bot/trimanpolicy/.
Read the full abstract ▾
Robotics
Shared Voxel-Map-Based Cooperative Indoor UAV Guidance with a Multi-Agent Soft Actor-Critic Controller
Thomas Hickling, Dylan Wynne, Yu Su et al. (+1 more)
This paper presents a cooperative indoor
UAV guidance framework that combines a shared voxel-map world model with a multi-
agent Soft Actor-Critic (MASAC) controller. Multiple drones fuse 360
LiDAR observations into a common world-frame occupancy map, which is converted into a compact bird's-eye-view (BEV) representation and provided to each
agent as an ego-aligned local crop. This integrate-in-world, act-in- ego design enables consistent multi-
UAV spatial fusion whilst retaining decentralised continuous control. The policy combines BEV map features, near-field obstacle observations, and compact goal and peer-state information within a centralised-training, decentralised-execution framework. In simulation, the learned controller achieves a 90.3% success rate in corridor
navigation, outperforming Astar planning, an artificial potential field controller, and a prior guidance method. To address residual sim-to-real mismatch, the simulation-trained policy is further adapted using offline imitation fine-tuning from real-world data. Real-world experiments in GNSS-denied indoor environments demonstrate stable two-
UAV cooperative operation across increasingly chal- lenging obstacle layouts. The results show that shared voxel-map representations provide an effective and scalable spatial substrate for learned cooperative indoor
UAV guidance.
Read the full abstract ▾
All authors Thomas Hickling, Dylan Wynne, Yu Su, Nabil Aouf
Robotics
When Does Legacy Data Start to Help? Emergent Transfer in Cross-Configuration Robot Learning
Tao Wang, Hudson Hou, Yingdong Hu et al. (+7 more)
Robotic hardware evolves over time, but demonstration data is often tied to a specific sensor and actuator configuration. This raises a practical and underexplored question: when does legacy data begin to benefit an upgraded
robot? We study this question on a wheeled
humanoid platform across two hardware generations, where both the camera and gripper are changed while the overall morphology remains fixed. Contrary to the common assumption that more cross-configuration data is always helpful, we observe a grokking-like transition: legacy data remains ineffective until the upgraded configuration acquires a minimum level of task competence, after which co-training gains rise sharply before diminishing near saturation. We hypothesize that this task-dependent transition is governed by a transfer threshold and characterize the resulting three-phase pattern. Across real-
robot manipulation tasks, we observe all three phases: no measurable benefit at low competence ($10.0\% \rightarrow 10.0\%$), a sharp gain after crossing the threshold ($23.3\% \rightarrow 86.7\%$ on flower insertion), and diminishing returns at high competence ($85.0\% \rightarrow 93.3\%$ on pen insertion). We provide a theoretical account based on gradient
alignment and residual policy uncertainty, and derive a phase-aware rule for deciding when to collect more new-hardware data and when to reuse legacy demonstrations. We further validate this three-phase pattern on a mobile dual-arm watering task, with results consistent with our predictions.
Read the full abstract ▾
All authors Tao Wang, Hudson Hou, Yingdong Hu, Yufeng Liu, Qinghai Li, Yingjie Jiang, Yingzhi Wang, Cheng Ma, Richard Wang, Yang Gao
Robotics
P3: Probabilistic Policy Propagation for Stable VAE-Based Robot Learning
Liyun Yan, Jianming Ma, Yang Zhang et al. (+5 more)
Variational Autoencoders are widely used to encode high-dimensional and noisy observations in robotics. However, their
stochastic latent creates a mismatch with Proximal Policy Optimization (PPO): an effective policy marginalizes over the latent distribution, whereas former implementations estimate its probability ratio and KL divergence using only one latent sample. We identify a fundamental but overlooked theoretical cause: naive single-sample approximations in
stochastic latent space induce significant variance and bias in the surrogate loss. To address this, we introduce P^3 (Probabilistic Policy Propagation), a distribution-aware optimization framework for VAE-based policies. $P^3$ couples moment-based probabilistic method for stable and efficient learning with sampling-based calibration for robust policy behavior under latent uncertainty. In our experiments, P^3 boosts data efficiency from 64.6% to >96%, reduces convergence steps by >20%. Furthermore, P^3 is evaluated on challenging
humanoid parkour tasks and shows an effective foundation for VAE-based PPO. Code is available at https://github.com/ylyem9x/P3_Open.
Read the full abstract ▾
All authors Liyun Yan, Jianming Ma, Yang Zhang, Shengcheng Fu, Zhanxiang Cao, Keqi Zhu, Yizhi Chen, Yue Gao