Robotics
SutureFormer: Learning Surgical Trajectories via Goal-conditioned Offline RL in Pixel Space
Huanrong Liu, Chunlin Tian, Tongyu Jia et al. (+8 more)
Predicting
surgical needle trajectories from endoscopic
video is critical for
robot-assisted suturing, enabling anticipatory planning, real-time guidance, and safer motion execution. Existing methods that directly learn motion distributions from visual observations tend to overlook the sequential dependency among adjacent motion steps. Moreover, sparse waypoint annotations often fail to provide sufficient supervision, further increasing the difficulty of supervised or imitation learning methods. To address these challenges, we formulate image-based needle trajectory prediction as a sequential decision-making problem, in which the needle tip is treated as an
agent that moves step by step in pixel space. This formulation naturally captures the continuity of needle motion and enables the explicit modeling of physically plausible pixel-wise state transitions over time. From this perspective, we propose SutureFormer, a goal-conditioned offline reinforcement learning framework that leverages sparse annotations to dense reward signals via cubic spline interpolation, encouraging the policy to exploit limited expert guidance while exploring plausible future motion paths. SutureFormer encodes variable-length clips using an observation encoder to capture both local spatial cues and long-range temporal dynamics, and autoregressively predicts future waypoints through actions composed of discrete directions and continuous magnitudes. To enable stable offline policy optimization from expert demonstrations, we adopt Conservative Q-Learning with Behavioral Cloning regularization. Experiments on a new kidney wound suturing
dataset containing 1,158 trajectories from 50 patients show that SutureFormer reduces Average Displacement Error by 58.6% compared with the strongest baseline, demonstrating the effectiveness of modeling needle trajectory prediction as pixel-level sequential action learning.
Read the full abstract ▾
All authors Huanrong Liu, Chunlin Tian, Tongyu Jia, Tailai Zhou, Qin Liu, Yu Gao, Yutong Ban, Yun Gu, Guy Rosman, Xin Ma, Qingbiao Li
Robotics · Cross-listed from cs.CG
Unlabeled Multi-Robot Motion Planning with Improved Separation Trade-offs
Tsuri Farhana, Omrit Filtser and Shalev Goldshtein
We study unlabeled MRMP for unit-disk
robots in a polygonal environment. Although the problem is hard in general,
polynomial-time solutions exist under appropriate separation assumptions on start and target positions. Banyassady et al.(SoCG'22) guarantee feasibility in simple polygons under start–start and target–target distances of at least $4$, and start–target distances of at least $3$, but without optimality guarantees. Solovey et al.(RSS'15) provide a near-optimal solution in general polygonal domains, under stricter conditions: start/target positions must have pairwise distance at least $4$, and at least $\sqrt{5}\approx2.236$ from obstacles. This raises the question of whether
polynomial-time
algorithms can be obtained in even more densely packed environments. In this paper we present a generalized
algorithm that achieve different tradeoffs on the
robots-separation $\rho$ and obstacles-separation $\omega$, all significantly improving upon the state of the art. Specifically, we obtain
polynomial-time constant-approximation
algorithms to minimize the total path length when (i) $\rho=2\frac{2}{3}$ and $\omega=1\frac{2}{3}$, or (ii) $\rho\approx3.291$ and $\omega\approx1.354$. These solutions are weakly-monotone; we also provide a monotone solution requiring $\omega=\approx1.614$ and $\rho=4$. We prove that monotone plans may not exist when $\omega<1.614$, and weakly-monotone plans may not exist when $\omega<1.354$. We then present tradeoffs between the separation bounds and the approximation factor, specifically achieving an (almost) optimal bound of $\rho=2$ at the cost of a linear approximation factor and requiring $\omega=2$. This applies also for the labeled variant of MRMP, in which case we show a tight bound on $\omega$. Finally, we show that without any
robots-separation assumption, obstacles-separation of at least $1.5$ may be necessary for a solution to exist.
Read the full abstract ▾
Robotics
Real-Time Optical Communication Using Event-Based Vision with Moving Transmitters
Harmeet Dhillon, Pranay Katyal, Brendan Long et al. (+3 more)
In multi-
robot systems, traditional
radio frequency (RF) communication struggles with contention and jamming. Optical communication offers a strong alternative. However, conventional frame-based cameras suffer from limited frame rates, motion blur, and reduced robustness under high dynamic range lighting. Event cameras support microsecond temporal resolution and high dynamic range, making them extremely sensitive to scene changes under fast relative motion with an optical transmitter. Leveraging these strengths, we develop a complete optical communication system capable of tracking moving transmitters and decoding messages in real time. Our system achieves over $95\%$ decoding accuracy for text transmission during motion by implementing a Geometry-Aware Unscented Kalman Filter (GA-UKF), achieving 7x faster processing speed compared to the previous state-of-the-art method, while maintaining equivalent tracking accuracy at transmitting frequencies $\geq$ 1 kHz.
Read the full abstract ▾
All authors Harmeet Dhillon, Pranay Katyal, Brendan Long, Rohan Walia, Matthew Cleaveland, Kevin Leahy
Robotics
Can LLMs Prove Robotic Path Planning Optimality? A Benchmark for Research-Level Algorithm Verification
Zhengbang Yang, Md. Tasin Tazwar, Minghan Wei et al. (+1 more)
Robotic path planning problems are often NP-hard, and practical solutions typically rely on approximation
algorithms with provable performance guarantees for general cases. While designing such
algorithms is challenging, formally proving their approximation optimality is even more demanding, which requires domain-specific
geometric insights and multi-step mathematical
reasoning over complex operational constraints. Recent Large
Language Models (LLMs) have demonstrated strong performance on mathematical
reasoning benchmarks, yet their ability to assist with research-level optimality proofs in robotic path planning remains under-explored. In this work, we introduce the first
benchmark for evaluating LLMs on approximation-ratio proofs of robotic path planning
algorithms. The
benchmark consists of 34 research-grade proof tasks spanning diverse planning problem types and complexity levels, each requiring structured
reasoning over
algorithm descriptions, problem constraints, and theoretical guarantees. Our evaluation of state-of-the-art proprietary and open-source LLMs reveals that even the strongest models struggle to produce fully valid proofs without external domain knowledge. However, providing LLMs with task-specific in-context lemmas substantially improves
reasoning quality, a factor that is more effective than generic chain-of-thought prompting or supplying the ground-truth approximation ratio as posterior knowledge. We further provide fine-grained error analysis to characterize common logical failures and
hallucinations, and demonstrate how each error type can be mitigated through targeted context augmentation.
Read the full abstract ▾
All authors Zhengbang Yang, Md. Tasin Tazwar, Minghan Wei, Zhuangdi Zhu
Robotics · Cross-listed from eess.SY
Exact and Approximate Convex Reformulation of Linear Stochastic Optimal Control with Chance Constraints
Tanmay Dokania and Yashwanth Kumar Nakka
In this paper, we present an equivalent convex optimization formulation for discrete-time
stochastic linear systems subject to linear chance constraints, alongside a tight convex relaxation for quadratic chance constraints. By lifting the state vector to encode moment information explicitly, the formulation captures linear chance constraints on states and controls across multiple time steps exactly, without conservatism, yielding strict improvements in both feasibility and optimality. For quadratic chance constraints, we derive convex approximations that are provably less conservative than existing methods. We validate the framework on minimum-snap trajectory generation for a quadrotor, demonstrating that the proposed approach remains feasible at noise levels an order of magnitude beyond the operating range of prior formulations.
Read the full abstract ▾
Robotics
A Closed-Form CLF-CBF Controller for Whole-Body Continuum Soft Robot Collision Avoidance
Kiwan Wong, Maximillian Stölzle, Wei Xiao et al. (+1 more)
Safe operation is essential for deploying
robots in human-centered 3D environments. Soft continuum manipulators provide passive safety through mechanical compliance, but still require active control to achieve reliable collision avoidance. Existing approaches, such as sampling-based planning, are often computationally expensive and lack formal safety guarantees, which limits their use for real-time whole-body avoidance. This paper presents a closed-form Control Lyapunov Function–Control Barrier Function (CLF–CBF) controller for real-time 3D obstacle avoidance in soft continuum manipulators without online optimization. By analytically
embedding safety constraints into the control input, the proposed method ensures stability and safety under the stated modeling assumptions, while avoiding feasibility issues commonly encountered in online optimization-based methods. The resulting controller is up to $10\times$ faster than standard CLF–CBF quadratic-programming approaches and up to $100\times$ faster than traditional sampling-based planners. Simulation and hardware experiments on a tendon-driven soft manipulator demonstrate accurate 3D trajectory tracking and robust obstacle avoidance in cluttered environments. These results show that the proposed framework provides a scalable and provably safe control strategy for soft
robots operating in dynamic, safety-critical settings.
Read the full abstract ▾
All authors Kiwan Wong, Maximillian Stölzle, Wei Xiao, Daniela Rus
Robotics
Speculative Policy Orchestration: A Latency-Resilient Framework for Cloud-Robotic Manipulation
Chanh Nguyen, Shutong Jin, Florian T. Pokorny et al. (+1 more)
Cloud robotics enables
robots to offload high-dimensional motion planning and
reasoning to remote servers. However, for continuous manipulation tasks requiring high-frequency control, network latency and jitter can severely destabilize the system, causing command starvation and unsafe physical execution. To address this, we propose Speculative Policy Orchestration (SPO), a latency-resilient cloud-edge framework. SPO utilizes a cloud-hosted world model to pre-compute and stream future kinematic waypoints to a local edge buffer, decoupling execution frequency from network round-trip time. To mitigate unsafe execution caused by predictive drift, the edge node employs an $\epsilon$-tube verifier that strictly bounds kinematic execution errors. The framework is coupled with an Adaptive Horizon Scaling mechanism that dynamically expands or shrinks the speculative pre-fetch depth based on real-time tracking error. We evaluate SPO on continuous RLBench manipulation tasks under emulated network delays. Results show that even when deployed with learned models of modest accuracy, SPO reduces network-induced idle time by over 60% compared to blocking remote inference. Furthermore, SPO discards approximately 60% fewer cloud predictions than static caching baselines. Ultimately, SPO enables fluid, real-time cloud-robotic control while maintaining bounded physical safety.
Read the full abstract ▾
All authors Chanh Nguyen, Shutong Jin, Florian T. Pokorny, Erik Elmroth
Robotics
SOFTMAP: Sim2Real Soft Robot Forward Modeling via Topological Mesh Alignment and Physics Prior
Ziyong Ma, Uksang Yoo, Jonathan Francis et al. (+3 more)
While soft
robot manipulators offer compelling advantages over rigid counterparts, including inherent compliance, safe human-
robot interaction, and the ability to conform to complex geometries, accurate forward modeling from low-dimensional actuation commands remains an open challenge due to nonlinear material phenomena such as hysteresis and manufacturing variability. We present SOFTMAP, a sim-to-real learning framework for real-time 3D forward modeling of tendon-actuated soft finger manipulators. SOFTMAP combines four components: (1) As-Rigid-As-Possible (ARAP)-based topological
alignment that projects simulated and real point clouds into a shared, topologically consistent vertex space; (2) a lightweight MLP forward model pretrained on simulation data to map servo commands to full 3D finger geometry; (3) a residual correction network trained on a small set of real observations to predict per-vertex displacement fields that compensate for sim-to-real discrepancies; and (4) a closed-form linear actuation calibration layer enabling real-time inference at 30 FPS. We evaluate SOFTMAP on both simulated and physical hardware, achieving state-of-the-art shape prediction accuracy with a Chamfer distance of 0.389 mm in simulation and 3.786 mm on hardware, millimeter-level fingertip trajectory tracking across multiple target paths, and a 36.5% improvement in teleoperation task success over the baseline. Our results show that SOFTMAP provides a data-efficient approach for 3D forward modeling and control of soft manipulators.
Read the full abstract ▾
All authors Ziyong Ma, Uksang Yoo, Jonathan Francis, Weiming Zhi, Jeffrey Ichnowski, Jean Oh
Robotics
VAMPO: Policy Optimization for Improving Visual Dynamics in Video Action Models
Zirui Ge, Pengxiang Ding, Baohua Yin et al. (+16 more)
Video action models are an appealing foundation for
Vision–
Language–Action systems because they can learn visual dynamics from large-scale
video data and transfer this knowledge to downstream
robot control. Yet current
diffusion-based
video predictors are trained with likelihood-surrogate objectives, which encourage globally plausible predictions without explicitly optimizing the precision-critical visual dynamics needed for manipulation. This objective mismatch often leads to subtle errors in object pose, spatial relations, and contact timing that can be amplified by downstream policies. We propose VAMPO, a post-training framework that directly improves visual dynamics in
video action models through policy optimization. Our key idea is to formulate multi-step denoising as a sequential decision process and optimize the denoising policy with rewards defined over expert visual dynamics in latent space. To make this optimization practical, we introduce an Euler Hybrid sampler that injects stochasticity only at the first denoising step, enabling tractable low-variance policy-gradient estimation while preserving the coherence of the remaining denoising trajectory. We further combine this design with GRPO and a verifiable non-
adversarial reward. Across diverse simulated and real-world manipulation tasks, VAMPO improves task-relevant visual dynamics, leading to better downstream action generation and stronger generalization. The homepage is https://vampo-
robot.github.io/VAMPO/.
Read the full abstract ▾
All authors Zirui Ge, Pengxiang Ding, Baohua Yin, Qishen Wang, Zhiyong Xie, Yemin Wang, Jinbo Wang, Hengtao Li, Runze Suo, Wenxuan Song, Han Zhao, Shangke Lyu, Zhaoxin Fan, Haoang Li, Ran Cheng, Cheng Chi, Huibin Ge, Yaozhi Luo, Donglin Wang
Robotics · Cross-listed from cs.CV
Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding
Xianjin Wu, Dingkang Liang, Tianrui Feng et al. (+5 more)
While
Multimodal Large
Language Models demonstrate impressive semantic capabilities, they often suffer from spatial blindness, struggling with fine-grained
geometric reasoning and physical dynamics. Existing solutions typically rely on explicit 3D modalities or complex
geometric scaffolding, which are limited by data scarcity and generalization challenges. In this work, we propose a paradigm shift by leveraging the implicit spatial prior within large-scale
video generation models. We posit that to synthesize temporally coherent
videos, these models inherently learn robust 3D structural priors and physical laws. We introduce VEGA-3D (
Video Extracted Generative Awareness), a plug-and-play framework that repurposes a pre-trained
video diffusion model as a Latent World Simulator. By extracting spatiotemporal features from intermediate noise levels and integrating them with semantic representations via a
token-level adaptive gated fusion mechanism, we enrich MLLMs with dense
geometric cues without explicit 3D supervision. Extensive experiments across 3D scene understanding, spatial
reasoning, and
embodied manipulation
benchmarks demonstrate that our method outperforms state-of-the-art baselines, validating that generative priors provide a scalable foundation for physical-world understanding. Code is publicly available at https://github.com/H-EmbodVis/VEGA-3D.
Read the full abstract ▾
All authors Xianjin Wu, Dingkang Liang, Tianrui Feng, Kui Xia, Yumeng Zhang, Xiaofan Li, Xiao Tan, Xiang Bai
Robotics
Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models
Bryce Grant, Xijia Zhao and Peng Wang
A fine-tuned
Vision-
Language-Action (VLA) policy can easily place the alphabet soup in the basket on command, just as it saw during fine-tuning. However, if the basket is shifted five centimeters to the left, it drops the soup off the table. We developed an interpretability pipeline to investigate what such a policy is computing. Injecting task A's activations into the action expert while the policy looks at task B's scene makes pi0.5 execute task A's motor trajectory in 99.6% of episodes ($n{=}1{,}968$). The injected program is bound to absolute workspace coordinates and ignores the visible scene, which provides a mechanistic account of the perturbation brittleness that concurrent
benchmarks report. The binding holds on a physical UR10e: directed replay redirects the commanded task in 26/26 rollouts in one direction and 18/26 in the other, with no-injection and random-direction controls at the floor, and once the source object is moved 10–15\,cm the arm grasps and releases at the recorded coordinates in 50/50 trials, picking up the wrong block when one sits there and closing on empty air when none does. Applying the same interventions to six VLAs on LIBERO, MetaWorld, SimplerEnv, and ALOHA over 420{,}000+ rollouts yields three further findings: every architecture encodes the instruction, and four of the five weight it below
vision whenever the scene alone identifies the goal; under the standard replacement write-back the SAE pooling that preserves behavior is architecture-specific (pi0.5 per-
token, SmolVLA indifferent), and goal selection and motor generation separate into VLM and expert pathways wherever the two are distinguishable. We release Action Atlas (https://action-atlas.com), an interactive platform covering all 520+ trained SAEs, 79 identified concepts, and the scored UR10e rollouts. Code: https://github.com/CWRU-AISM/action-atlas
Read the full abstract ▾
Robotics
NavTrust: Benchmarking Trustworthiness for Embodied Navigation
Huaide Jiang, Yash Chaudhary, Yuping Wang et al. (+8 more)
There are two major categories of
embodied navigation:
Vision-
Language Navigation (VLN), where
agents navigate by following natural
language instructions; and Object-Goal
Navigation (OGN), where
agents navigate to a specified target object. However, existing work primarily evaluates model performance under nominal conditions, overlooking the potential corruptions that arise in real-world settings. To address this gap, we present NavTrust, a unified
benchmark that systematically corrupts input modalities, including RGB, depth, and instructions, in realistic scenarios and evaluates their impact on
navigation performance. To our best knowledge, NavTrust is the first
benchmark that exposes
embodied navigation agents to diverse RGB-Depth corruptions and instruction variations in a unified framework. Our extensive evaluation of seven state-of-the-art approaches reveals substantial performance degradation under realistic corruptions, which highlights critical robustness gaps and provides a roadmap toward more trustworthy
embodied navigation systems. Furthermore, we systematically evaluate four distinct mitigation strategies to enhance robustness against RGB-Depth and instruction corruptions. Our base models include Uni-NaVid and ETPNav. We deployed them on a real mobile
robot and observed improved robustness to corruptions. The project website is: https://navtrust.github.io/.
Read the full abstract ▾
All authors Huaide Jiang, Yash Chaudhary, Yuping Wang, Zehao Wang, Raghav Sharma, Manan Mehta, Yang Zhou, Lichao Sun, Zhiwen Fan, Zhengzhong Tu, Jiachen Li
Robotics
OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation
Yuhang Zheng, Songen Gu, Yupeng Zheng et al. (+11 more)
Contact-rich manipulation tasks, such as wiping and assembly, require accurate perception of contact forces, friction changes, and state transitions that cannot be reliably inferred from
vision alone. Despite growing interest in visuo-tactile manipulation, progress is constrained by two persistent limitations: existing
datasets are small in scale and narrow in task coverage, and current methods treat tactile signals as passive observations rather than using them to model contact dynamics or enable closed-loop control explicitly. In this paper, we present OmniViTac, a large-scale visuo-tactile-action
dataset comprising $21{,}000+$ trajectories across $86$ tasks and $100+$ objects, organized into six physics-grounded interaction patterns. Building on this
dataset, we propose OmniVTA, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules: a self-supervised tactile encoder, a two-stream visuo-tactile world model for predicting short-horizon contact
evolution, a contact-aware fusion policy for action generation, and a 60Hz reflexive controller that corrects deviations between predicted and observed tactile signals in a closed loop. Real-
robot experiments across all six interaction categories show that OmniVTA outperforms existing methods and generalizes well to unseen objects and
geometric configurations, confirming the value of combining predictive contact modeling with high-frequency tactile feedback for contact-rich manipulation. All data, models, and code will be made publicly available on the project website at https://mrsecant.github.io/OmniVTA.
Read the full abstract ▾
All authors Yuhang Zheng, Songen Gu, Yupeng Zheng, Weize Li, Yujie Zang, Shuai Tian, Xiang Li, Ce Hao, Chen Gao, Si Liu, Haoran Li, Yilun Chen, Shuicheng Yan, Wenchao Ding
Robotics
FASTER: Rethinking Real-Time Flow VLAs
Yuxiang Lu, Zhe Liu, Xianzhe Fan et al. (+5 more)
Real-time execution is crucial for deploying
Vision-
Language-Action (VLA) models in the physical world. Existing asynchronous inference methods primarily optimize trajectory smoothness, but neglect the critical latency in reacting to environmental changes. By rethinking the notion of reaction in action chunking policies, this paper presents a systematic analysis of the factors governing reaction time. We show that reaction time follows a uniform distribution determined jointly by the Time to First Action (TTFA) and the execution horizon. Moreover, we reveal that the standard practice of applying a constant schedule in flow-based VLAs can be inefficient and forces the system to complete all sampling steps before any movement can start, forming the bottleneck in reaction latency. To overcome this issue, we propose Fast Action Sampling for ImmediaTE Reaction (FASTER). By introducing a Horizon-Aware Schedule, FASTER adaptively prioritizes near-term actions during flow sampling, compressing the denoising of the immediate reaction by tenfold (e.g., in $\pi_{0.5}$ and X-VLA) into a single step, while preserving the quality of long-horizon trajectory. Coupled with a streaming client-server pipeline, FASTER substantially reduces the effective reaction latency on real
robots, especially when deployed on consumer-grade GPUs. Real-world experiments, including a highly dynamic table tennis task, prove that FASTER unlocks substantially improved real-time responsiveness for generalist policies, enabling rapid generation of accurate and smooth trajectories.
Read the full abstract ▾
All authors Yuxiang Lu, Zhe Liu, Xianzhe Fan, Zhenya Yang, Jinghua Hou, Junyi Li, Kaixin Ding, Hengshuang Zhao
Robotics
Sparse Autoencoders Reveal Interpretable and Steerable Features in VLA Models
Aiden Swann, Lachlain McGranahan, Hugo Buurmeijer et al. (+2 more)
Vision-
Language-Action (VLA) models have emerged as a promising approach for general-purpose
robot manipulation. However, little research has mechanistically explored when and why they generalize across objects, scenes, and instructions. To probe internal representations, we train Sparse Autoencoders (SAEs) on the VLA's hidden-layer activations. SAEs learn sparse dictionaries over model activations, often revealing features that correspond to interpretable directions in the model's representation space. We identify SAE features corresponding to motion primitives and semantic concepts, including features that are general across episodes and causally steerable. We propose a metric to categorize features as general transferable primitives or episode-specific memorizations, offering a promising glimpse towards VLA generalization. We validate these findings through steering experiments on both the LIBERO simulation
benchmark and on real-world DROID hardware. We find that amplifying general and semantic features induces behaviors consistent with their meanings, whereas ablating them destroys model performance. Furthermore, we demonstrate steering as a way to control behavior in unpromptable directions. Together, these results provide mechanistic evidence that VLAs can learn reusable internal features linking perception,
language, and action across tasks and scenes. Our project page is located at https://drvla.github.io
Read the full abstract ▾
All authors Aiden Swann, Lachlain McGranahan, Hugo Buurmeijer, Monroe Kennedy III, Mac Schwager
Robotics
ADMM-Based Distributed MPC with Control Barrier Functions for Safe Multi-Robot Quadrupedal Locomotion
Yicheng Zeng, Ruturaj S. Sambhus, Basit Muhammad Imran et al. (+3 more)
This paper proposes a fully decentralized model predictive control (MPC) framework with control barrier function (CBF) constraints for safety-critical trajectory planning in multi-
robot legged systems. The incorporation of CBF constraints introduces explicit inter-
agent coupling, which prevents direct decomposition of the resulting optimal control problems. To address this challenge, we reformulate the centralized safety-critical MPC problem using a structured distributed optimization framework based on the alternating direction method of multipliers (ADMM). By introducing a novel node-edge splitting formulation with consensus constraints, the proposed approach decomposes the global problem into independent node-local and edge-local quadratic programs that can be solved in parallel using only neighbor-to-neighbor communication. This enables fully decentralized trajectory optimization with symmetric computational load across
agents while preserving safety and dynamic feasibility. The proposed framework is integrated into a hierarchical locomotion control architecture for quadrupedal
robots, combining high-level distributed trajectory planning, mid-level nonlinear MPC enforcing single rigid body dynamics, and low-level whole-body control enforcing full-order
robot dynamics. The effectiveness of the proposed approach is demonstrated through hardware experiments on two Unitree Go2 quadrupedal
robots and numerical simulations involving up to four
robots navigating uncertain environments with rough terrain and external disturbances. The results show that the proposed distributed formulation achieves performance comparable to centralized MPC while reducing the average per-cycle planning time by up to 51% in the four-
agent case, enabling efficient real-time decentralized implementation.
Read the full abstract ▾
All authors Yicheng Zeng, Ruturaj S. Sambhus, Basit Muhammad Imran, Jeeseop Kim, Vittorio Pastore, Kaveh Akbari Hamed
Robotics
Meanings and Measurements: Multi-Agent Probabilistic Grounding for Vision-Language Navigation
Swagat Padhan, Lakshya Jain, Bhavya Minesh Shah et al. (+3 more)
Robots collaborating with humans must convert natural
language goals into actionable, physically grounded decisions. For example, executing a command such as "go two meters to the right of the fridge" requires grounding semantic references, spatial relations, and metric constraints within a 3D scene. While recent
vision language models (VLMs) demonstrate strong semantic grounding capabilities, they are not explicitly designed to reason about metric constraints in physically defined spaces. In this work, we empirically demonstrate that state-of-the-art VLM-based grounding approaches struggle with complex metric-semantic
language queries. To address this limitation, we propose MAPG (Multi-
Agent Probabilistic Grounding), an agentic framework that decomposes
language queries into structured subcomponents and queries a VLM to ground each component. MAPG then probabilistically composes these grounded outputs to produce metrically consistent, actionable decisions in 3D space. We evaluate MAPG on the HM-EQA
benchmark and show consistent performance improvements over strong baselines. Furthermore, we introduce a new
benchmark, MAPG-Bench, specifically designed to evaluate metric-semantic goal grounding, addressing a gap in existing
language grounding evaluations. We also present a real-world
robot demonstration showing that MAPG transfers beyond simulation when a structured scene representation is available.
Read the full abstract ▾
All authors Swagat Padhan, Lakshya Jain, Bhavya Minesh Shah, Omkar Patil, Thao Nguyen, Nakul Gopalan
Robotics · Cross-listed from cs.CV
GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoning
Yiren Lu, Yi Du, Disheng Liu et al. (+3 more)
Effective
embodied exploration requires
agents to accumulate and retain spatial knowledge over time. However, existing scene representations, such as discrete scene graphs or static view-based snapshots, lack post-hoc re-observability. If an initial observation misses a target, the resulting
memory omission is often irrecoverable. To bridge this gap, we propose GSMem, a zero-shot
embodied exploration and
reasoning framework built upon 3D Gaussian Splatting (3DGS). By explicitly parameterizing continuous geometry and dense appearance, 3DGS serves as a persistent spatial
memory that endows the
agent with Spatial Recollection: the ability to render photorealistic novel views from optimal, previously unoccupied viewpoints. To operationalize this, GSMem employs a
retrieval mechanism that simultaneously leverages parallel object-level scene graphs and semantic-level
language fields. This complementary design robustly localizes target regions, enabling the
agent to “hallucinate” optimal views for high-fidelity
Vision-
Language Model (VLM)
reasoning. Furthermore, we introduce a hybrid exploration strategy that combines VLM-driven semantic scoring with a 3DGS-based coverage objective, balancing task-aware exploration with
geometric coverage. Extensive experiments on
embodied question answering and lifelong
navigation demonstrate the robustness and effectiveness of our framework
Read the full abstract ▾
All authors Yiren Lu, Yi Du, Disheng Liu, Yunlai Zhou, Chen Wang, Yu Yin
Robotics
Introducing M: A Modular, Modifiable Social Robot
Victor Nikhil Antony, Zhili Gong, Yoonjae Kim et al. (+1 more)
We present M, an open-source, low-cost tabletop social
robot platform aimed at lowering barriers to social robotics research by making
robots easier to reproduce, modify, and deploy in real-world settings. M combines a modular mechanical design,
multimodal sensing, and expressive yet mechanically simple actuation architecture with a ROS2-native software package that cleanly separates perception, expression control, and data management. The platform includes a simulation environment with interface equivalence to hardware to support easy sim-to-real transfer of interaction behaviors. We demonstrate extensibility through additional sensing/actuation modules and provide example interaction templates for storytelling and two-way conversational coaching. Lastly, we report real-world use of M in interaction design and week-long in-home deployments, showing how M can serve as a practical platform for longitudinal, reproducible social robotics research.
Read the full abstract ▾
All authors Victor Nikhil Antony, Zhili Gong, Yoonjae Kim, Chien-Ming Huang
Robotics · Cross-listed from cs.LG
From Inference Efficiency to Embodied Efficiency: Revisiting Efficiency Metrics for Vision-Language-Action Models
Zhuofan Li, Hongkun Yang, Zhenyang Chen et al. (+3 more)
Vision-
Language-Action (VLA) models have recently enabled
embodied agents to perform increasingly complex tasks by jointly
reasoning over visual, linguistic, and motor modalities. However, we find that the prevailing notion of “efficiency” in current VLA research, characterized by parameters, FLOPs, or
token decoding throughput, does not reflect actual performance on robotic platforms. In real-world execution, efficiency is determined by system-level
embodied behaviors such as task completion time, trajectory smoothness, cumulative joint rotation, and motion energy. Through controlled studies across model compression,
token sparsification, and action sequence compression, we make several observations that challenge common assumptions. (1) Methods that reduce computation under conventional metrics often increase end-to-end execution cost or degrade motion quality, despite maintaining task success rates. (2) System-level
embodied efficiency metrics reveal performance differences in the learned action policies that remain hidden under conventional evaluations. (3) Common adaptation methods such as in-context prompting or supervised fine-tuning show only mild and metric-specific improvements in
embodied efficiency. While these methods can reduce targeted
embodied-efficiency metrics such as jerk or action rate, the resulting gains may come with trade-offs in other metrics, such as longer completion time. Taken together, our results suggest that conventional inference efficiency metrics can overlook important aspects of
embodied execution. Incorporating
embodied efficiency provides a more complete view of policy behavior and practical performance, enabling fairer and more comprehensive comparisons of VLA models.
Read the full abstract ▾
All authors Zhuofan Li, Hongkun Yang, Zhenyang Chen, Yangxuan Chen, Yingyan Lin, Chaojian Li
Robotics
Tendon-Actuated Robots with a Tapered, Flexible Polymer Backbone: Design, Fabrication, and Modeling
Harald Minde Hansen, Nandita Gallacher, Nicholas B. Andrews et al. (+3 more)
This paper presents the design, modeling, and fabrication of 3D-printed, tendon-actuated continuum
robots featuring a flexible, tapered backbone constructed from thermoplastic polyurethane (TPU). Our scalable design incorporates an integrated electronics base housing that enables direct tendon tension control and sensing via actuators and compression load
cells. Unlike many continuum
robots that are single-purpose and costly, the proposed design prioritizes customizability, rapid assembly, and low cost while enabling high
curvature and enhanced distal compliance through
geometric tapering, thereby supporting a broad range of compliant robotic inspection and manipulation tasks. We develop a generalized forward kinetostatic model of the tapered backbone based on Cosserat rod theory using a Newtonian approach, extending existing tendon-actuated Cosserat rod formulations to explicitly account for spatially varying backbone cross-sectional geometry. The model captures the graded stiffness profile induced by the tapering and enables systematic exploration of the configuration space as a function of the
geometric design parameters. Specifically, we analyze how the backbone taper angle influences the
robot's configuration space and manipulability. The model is validated against motion capture data, achieving centimeter-level shape prediction accuracy after calibrating Young's modulus via a line search that minimizes modeling error. We further demonstrate teleoperated grasping using an endoscopic gripper routed along the continuum
robot, mounted on a 6-DoF robotic arm. Parameterized iLogic/CAD scripts are provided for rapid geometry generation and scaling. The presented framework establishes a simple, rapid, and reproducible pathway from parametric design to controlled tendon actuation for tapered, tendon-driven continuum
robots manufactured using fused deposition modeling 3D printers.
Read the full abstract ▾
All authors Harald Minde Hansen, Nandita Gallacher, Nicholas B. Andrews, Kristin Y. Pettersen, Jan Tommy Gravdahl, Mario di Castro
Robotics
Articulated-Body Dynamics Network: Dynamics-Grounded Prior for Robot Learning
Sangwoo Shin, Kunzhao Ren, Xiaobin Xiong et al. (+1 more)
Recent work in reinforcement learning has shown that incorporating structural priors for articulated
robots, such as link connectivity, into policy networks improves learning efficiency. However, dynamics properties, despite their fundamental role in determining how forces and motion propagate through the body, remain largely underexplored as an inductive bias for policy learning. To address this gap, we present the Articulated-Body Dynamics Network (ABD-Net), a novel graph neural network architecture grounded in the computational structure of forward dynamics. Specifically, we adapt the inertia propagation mechanism from the Articulated Body
Algorithm, systematically aggregating inertial quantities from child to parent links in a tree-structured manner, while replacing physical quantities with learnable parameters.
Embedding ABD-NET into the policy actor enables dynamics-informed representations that capture how actions propagate through the body, leading to efficient and robust policy learning. Through experiments with simulated
humanoid, quadruped, and hopper
robots, our approach demonstrates increased sample efficiency and generalization to dynamics shifts compared to
transformer-based and GNN baselines. We further validate the learned policy on real Unitree G1 and Go2
robots, state-of-the-art
humanoid and quadruped platforms, generating dynamic, versatile and robust locomotion behaviors through sim-to-real transfer with real-time inference.
Read the full abstract ▾
All authors Sangwoo Shin, Kunzhao Ren, Xiaobin Xiong, Josiah P. Hanna
Robotics · Cross-listed from cs.CV
DROID-SLAM in the Wild
Moyang Li, Zihan Zhu, Marc Pollefeys et al. (+1 more)
We present a robust, real-time RGB SLAM system that handles dynamic environments by leveraging differentiable Uncertainty-aware Bundle Adjustment. Traditional SLAM methods typically assume static scenes, leading to tracking failures in the presence of motion. Recent dynamic SLAM approaches attempt to address this challenge using predefined dynamic priors or uncertainty-aware mapping, but they remain limited when confronted with unknown dynamic objects or highly cluttered scenes where
geometric mapping becomes unreliable. In contrast, our method estimates per-pixel uncertainty by exploiting multi-view visual feature inconsistency, enabling robust tracking and reconstruction even in real-world environments. The proposed system achieves state-of-the-art camera poses and scene geometry in cluttered dynamic scenarios while running in real time at around 10 FPS. Code and
datasets are available at https://github.com/MoyangLi00/DROID-W.git.
Read the full abstract ▾
All authors Moyang Li, Zihan Zhu, Marc Pollefeys, Daniel Barath
Robotics
CAMO: A Conditional Neural Solver for the Multi-objective Multiple Traveling Salesman Problem
Fengxiaoxiao Li, Xiao Mao, Mingfeng Fan et al. (+4 more)
Robotic systems often require a team of
robots to collectively visit multiple targets while optimizing competing objectives, such as total travel cost and makespan. This setting can be formulated as the Multi-Objective Multiple Traveling Salesman Problem (MOMTSP). Although learning-based methods have shown strong performance on the single-
agent TSP and multi-objective TSP variants, they rarely address the combined challenges of multi-
agent coordination and multi-objective trade-offs, which introduce dual sources of complexity. To bridge this gap, we propose CAMO, a conditional neural solver for MOMTSP that generalizes across varying numbers of targets,
agents, and preference vectors, and yields high-quality approximations to the Pareto front (PF). Specifically, CAMO consists of a conditional encoder to fuse preferences into instance representations, enabling explicit control over multi-objective trade-offs, and a collaborative decoder that coordinates all
agents by alternating
agent selection and node selection to construct multi-
agent tours autoregressively. To further improve generalization, we train CAMO with a REINFORCE-based objective over a mixed distribution of problem sizes. Extensive experiments show that CAMO outperforms both neural and conventional heuristics, achieving a closer approximation of PFs. In addition, ablation results validate the contributions of CAMO's key components, and real-world tests on a mobile
robot platform demonstrate its practical applicability.
Read the full abstract ▾
All authors Fengxiaoxiao Li, Xiao Mao, Mingfeng Fan, Yifeng Zhang, Yi Li, Tanishq Duhan, Guillaume Sartoretti
Robotics
Fire as a Service: Augmenting Robot Simulators with Thermally and Visually Accurate Fire Dynamics
Anton R. Wagner, Madhan Balaji Rao, Helge Wrede et al. (+2 more)
Most existing
robot simulators prioritize rigid-body dynamics and photorealistic rendering, but largely neglect the thermally and optically complex phenomena that characterize real-world fire environments. For
robots envisioned as future firefighters, this limitation hinders both reliable capability evaluation and the generation of representative training data prior to deployment in hazardous scenarios. To address these challenges, we introduce Fire as a Service (FaaS), a novel, asynchronous co-simulation framework that augments existing
robot simulators with high-fidelity and computationally efficient fire simulations. Our pipeline enables
robots to experience accurate, multi-species thermodynamic heat transfer and visually consistent volumetric smoke without disrupting high-frequency rigid-body control loops. We demonstrate that our framework can be integrated with diverse
robot simulators to generate physically accurate fire behavior,
benchmark thermal hazards encountered by robotic platforms, and collect realistic
multimodal perceptual data. Crucially, its real-time performance supports human-in-the-loop teleoperation, enabling the successful training of reactive,
multimodal policies via Behavioral Cloning. By adding fire dynamics to
robot simulations, FaaS provides a scalable pathway toward safer, more reliable deployment of
robots in fire scenarios.
Read the full abstract ▾
All authors Anton R. Wagner, Madhan Balaji Rao, Helge Wrede, Sören Pirk, Xuesu Xiao
Robotics
ATG-MoE: Autoregressive trajectory generation with mixture-of-experts for assembly skill learning
Weihang Huang, Chaoran Zhang, Xiaoxin Deng et al. (+4 more)
Flexible manufacturing requires
robot systems that can adapt to constantly changing tasks, objects, and environments. However, traditional
robot programming is labor-intensive and inflexible, while existing learning-based assembly methods often suffer from weak positional generalization, complex multi-stage designs, and limited multi-skill integration capability. To address these issues, this paper proposes ATG-MoE, an end-to-end autoregressive trajectory generation method with mixture of experts for assembly skill learning from demonstration. The proposed method establishes a closed-loop mapping from multi-modal inputs, including RGB-D observations, natural
language instructions, and
robot proprioception to manipulation trajectories. It integrates multi-modal feature fusion for scene and task understanding, autoregressive sequence modeling for temporally coherent trajectory generation, and a mixture-of-experts architecture for unified multi-skill learning. In contrast to conventional methods that separate visual perception and control or train different skills independently, ATG-MoE directly incorporates visual information into trajectory generation and supports efficient multi-skill integration within a single model. We train and evaluate the proposed method on eight representative assembly skills from a pressure-reducing valve assembly task. Experimental results show that ATG-MoE achieves strong overall performance in simulation, with an average grasp success rate of 96.3% and an average overall success rate of 91.8%, while also demonstrating strong generalization and effective multi-skill integration. Real-world experiments further verify its practicality for multi-skill industrial assembly. The project page can be found at https://hwh23.github.io/ATG-MoE
Read the full abstract ▾
All authors Weihang Huang, Chaoran Zhang, Xiaoxin Deng, Hao Zhou, Zhaobo Xu, Shubo Cui, Long Zeng
Robotics
MERGE: Guided Vision-Language Models for Multi-Actor Event Reasoning and Grounding in Human-Robot Interaction
Joerg Deigmoeller, Nakul Agarwal, Stephan Hasler et al. (+8 more)
We introduce MERGE, a system for situational grounding of actors, objects, and events in dynamic human-
robot group interactions. Effective collaboration in such settings requires consistent situational awareness, built on persistent representations of people and objects and an episodic abstraction of events. MERGE achieves this by uniquely identifying physical instances of actors (humans or
robots) and objects and structuring them into actor-action-object relations, ensuring temporal consistency across interactions. Central to MERGE is the integration of
Vision-
Language Models (VLMs) guided with a perception pipeline: a lightweight streaming module continuously processes visual input to detect changes and selectively invokes the VLM only when necessary. This decoupled design preserves the
reasoning power and zero-shot generalization of VLMs while improving efficiency, avoiding both the high monetary cost and the latency of frame-by-frame captioning that leads to fragmented and delayed outputs. To address the absence of suitable
benchmarks for multi-actor collaboration, we introduce the GROUND
dataset, which offers fine-grained situational annotations of multi-person and human-
robot interactions. On this
dataset, our approach improves the average grounding score by a factor of 2 compared to the performance of VLM-only baselines - including GPT-4o, GPT-5 and Gemini 2.5 Flash - while also reducing run-time by a factor of 4. The code and data are available at www.github.com/HRI-EU/merge.
Read the full abstract ▾
All authors Joerg Deigmoeller, Nakul Agarwal, Stephan Hasler, Daniel Tanneberg, Anna Belardinelli, Reza Ghoddoosian, Chao Wang, Felix Ocker, Fan Zhang, Behzad Dariush, Michael Gienger