Robotics
Range-Aided SLAM Initialization Exploiting Accurate Heading Information
Isabel Lougheed and James Richard Forbes
This paper presents a novel initialization method for range-aided simultaneous localization and mapping (RA-SLAM). The general SLAM problem has a well-known separable structure where landmark and
robot positions can be solved for in a linear fashion given known
robot headings. This paper considers the case where highly accurate heading information is available, which is typical of autonomous underwater vehicle (AUV) navigation, to solve for the range transponder and
robot positions. The proposed approach consists of two steps. First, using a generalized trust region subproblem (GTRS), the positions of the transponders relative to the AUV are solved for given the nonlinear range measurements. Second, the relative transponder positions and the known heading of the AUV are used to estimate the transponder and AUV positions by solving a linear least-squares problem. These transponder and AUV position estimates, combined with the highly accurate heading information, provide a reliable initialization method for the general nonlinear RA-SLAM problem. The effectiveness of the proposed approach is tested on a real-world AUV
dataset where long baseline (LBL) range measurements are provided in concert with highly accurate heading information provided by an inertial navigation system (INS).
Read the full abstract ▾
Robotics
CAST: Collision-Aware Assembly with Construction Robots using Simultaneous Trajectory Estimation and Planning
Karthik Shaji, Chisung Kim, John D'Amato et al. (+2 more)
Multi-
robot systems have shown increasing viability in construction due to their ability to execute high-precision actions while reducing human exposure to hazardous tasks. However, these environments have high-dimensional configuration spaces and possess substantial collision-avoidance constraints, which include other
robots, assembly objects, and workspace boundaries. We utilize a single factor graph for trajectory estimation and planning that incorporates measured
robot states together with explicit collision and learned cable constraints. This supports changing workspaces and enables synchronized, high-dimensional
robot motion planning while accounting for the stiff, vibration-induced uncertainty of heavy robotic systems. We demonstrate the success of our framework on the construction of a post-and-lintel structure using one
robot arm as a timber gripper, and a second
robot as a nail-fastener.
Read the full abstract ▾
All authors Karthik Shaji, Chisung Kim, John D'Amato, Edvard Bruun, Frank Dellaert
Robotics
PredActor: Predictive Action Diffusion for Steerable Onboard Humanoid Control
Lei Ye, Haibo Gao, Yitang Li et al. (+10 more)
Diffusion models provide a flexible framework for motion generation, but turning this flexibility into closed-loop humanoid control remains challenging. Hierarchical generator-tracker systems steer motion through reference trajectories, yet these references may exceed the capabilities of the downstream tracker, leaving physical feasibility and disturbance recovery largely to a separate control module. Action-only
diffusion avoids this separation by directly generating executable actions, but provides no explicit future-state trajectory that can be steered toward test-time motion objectives. Joint state-action
diffusion offers a natural alternative, but existing controllers often rely on privileged full-body states, while learned behavior selection and test-time motion steering remain only partially integrated. We present PredActor, a predictive action
diffusion policy that unifies both steering modes in one directly executed policy using proprioception alone. Given proprioceptive history and optional task context, PredActor jointly predicts actions and an internal future-state trajectory that enables guidance: classifier-free guidance strengthens text-conditioned motion, while classifier guidance steers future states toward test-time objectives. Only actions are executed, requiring neither a motion-reference tracker nor privileged full-body states. In simulation, PredActor reaches 44 of 45 destination targets and achieves a text
retrieval score of 0.539 versus 0.424 for conditional action
diffusion, with similar disturbance survival. Rolling denoising and computation-preserving runtime optimizations reduce the complete callback to 16.790 ms median and 19.383 ms p95 on a Jetson Orin NX, within the 20 ms control period. Deployed on a Unitree G1, PredActor demonstrates text-conditioned motion, disturbance response, joystick control, and semantic interpolation in simulation and hardware.
Read the full abstract ▾
All authors Lei Ye, Haibo Gao, Yitang Li, Peng Xu, Zetong Jing, Junhan Sun, Fanrong Dong, Ziqi Han, Xue Wang, Jianhua Sun, Cewu Lu, Hao Zhao, Liang Ding
Robotics
Minimum Time Trajectories for a Car-Like Mobile Robot Moving with Rigid Wheels Under Non-Sliding Constraints
Joseph Ben-Asher, Elon Rimon and Leeor Ravina
This paper studies the minimum time trajectoriesvof a car-like mobile
robot navigating in an obstacle free environment. The
robot, with forward and backward speeds, is controlled by bounded front-wheels acceleration and limited front-wheels steering rate. The paper extends previous results which solved this problem for the kinematic car-like
robot. However, the kinematic model assumes pure rolling at the wheels ground contacts. This assumption requires non-sliding constraints for the front and rear wheels that can only be handled by the
robot dynamics. This paper formulates the non-sliding constraints based on the
robot dynamics then augments the kinematic model time-optimal path primitives with three new path primitives associated with the non-sliding constraints. The three non-sliding path primitives together with the kinematic model twelve path primitives form the car-like
robot time optimal trajectories. Approximate analytic solutions for the non-sliding path primitives are also provided. Examples study the time-optimal path primitives along representative maneuvers, illustrating how the non-sliding constraints influence the time optimal trajectories of the car-like
robot.
Read the full abstract ▾
Robotics
Uranus: Building the Next-Generation Simulation Infrastructure for Embodied AI
Wenkang Qin, Yukun Zhou, Noah Shen et al. (+5 more)
Scalable simulation is essential for
robot data generation, policy training, evaluation, and safe iteration, yet real-world interaction is costly and conventional simulators require labor-intensive construction. We present Uranus, a data-driven
robot simulator built around a joint-trajectory-conditioned autoregressive
diffusion model. Uranus offers three key capabilities: (1) streaming, open-ended rollout, which receives future joint-position trajectories online and autoregressively generates one latent frame per step, corresponding to four RGB frames, without a fixed horizon; (2) low-latency generation, achieving 24 FPS after inference optimization; and (3) scalable, extensible
robot control, providing a unified interface for synchronized multi-view generation across diverse
robot embodiments and camera configurations. We conduct comprehensive quantitative and qualitative evaluations on both in-distribution and out-of-distribution data, providing an objective assessment of Uranus and clearly identifying its current limitations. We release the code and model weights to empower the community with practical tools and insights.
Read the full abstract ▾
All authors Wenkang Qin, Yukun Zhou, Noah Shen, Jisong Cai, Dongxiao Mao, Baicheng Li, Yue Zhang, Wei Sui
Robotics · Cross-listed from cs.CV
Toward a foundation model for forest point clouds
Yuanwen Yue, Stefano Puliti, Damien Robert et al. (+7 more)
Forest inventories increasingly rely on artificial intelligence (AI) models to derive forest attributes from large-scale 3D point clouds. Current models are typically specialized to a single task, sensor, and forest type, making adaptation expensive in terms of annotations, computation, and expertise. We ask whether a single pretrained model can instead learn transferable representations across diverse forest inventory settings. Inspired by recent developments in
language modelling and computer
vision, we take a step toward a foundation model (FM) for 3D forestry. Using LitePT as backbone, we first establish a strong supervised baseline that sets a new state of the art on forest semantic and instance segmentation, tree species classification, and age regression
benchmarks. We then curate a large-scale unlabelled corpus spanning airborne,
UAV, and mobile
laser scanning across diverse forest ecosystems, and pretrain the same backbone using self-supervised learning. We systematically evaluate representation learning strategies by comparing training from scratch, supervised pretraining, and self-supervised pretraining across four representative forestry tasks, under varying annotation budgets. Compared with training from scratch, self-supervised pretraining accelerates model convergence and consistently improves performance when annotations are scarce. Compared with task-specific supervised pretraining, self-supervised pretraining yields more transferable representations across downstream forestry tasks. These findings identify the practical regime in which pretrained representations are most valuable and suggest that instance discrimination, rather than forest semantics, is the main remaining obstacle to a general-purpose 3D forest foundation model. Code and models are available at: https://github.com/prs-eth/ForPT.
Read the full abstract ▾
All authors Yuanwen Yue, Stefano Puliti, Damien Robert, Atakan Topaloğlu, Binbin Xiang, Maciej Wielgosz, Jan Dirk Wegner, Rasmus Astrup, Christian Rupprecht, Konrad Schindler
Robotics
H2RBench: A Real-to-Sim Benchmark for Evaluating Human-to-Robot Transfer
Chuyang Xiao, Haotian Zhan, Sriram Krishna et al. (+4 more)
Learning
robot manipulation policies from human
video demonstrations constitutes a promising avenue for scalable
robot learning. However, comparing different human-to-
robot (H2R) transfer methods remains challenging, as existing approaches are evaluated under different settings, including differing task suites, scene layouts, object instances, and amounts of
robot supervision. To address this challenge, we present H2RBench, a Real2Sim
benchmark for evaluating H2R transfer methods. H2RBench provides a standardized protocol built on real human
video demonstrations and simulated
robot demonstrations, and includes four manipulation tasks spanning diverse interaction requirements. We evaluate multiple representative H2R transfer methods, each adopting a different strategy for bridging the embodiment gap. Using H2RBench, we systematically characterize how each method scales with the amount of human demonstrations, revealing that methods differ substantially in their ability to leverage additional human data. We further show that simulation performance is broadly predictive of real-world
robot performance, with an overall Pearson correlation of r = 0.89, Spearman correlation of h̊o} = 0.85 and Mean Maximum Rank Violation (MMRV) of 0.06 across method-task configurations. These results establish H2RBench as a practical and scalable
benchmark for comparative H2R evaluation prior to real-world deployment.
Read the full abstract ▾
All authors Chuyang Xiao, Haotian Zhan, Sriram Krishna, Peilin Meng, Muhammad Zubair Irshad, Sergey Zakharov, David Held
Robotics
A Switched Adaptive Control Framework for Aerial Manipulators Under Dynamic Transitions
Rishabh Dev Yadav, Saksham Gupta, Amitabh Sharma et al. (+4 more)
Aerial manipulators represent the forefront of aerial robotics. Although potentially capable of complex interaction tasks, controlling aerial manipulators throughout the dynamic transitions occurring during task execution presents significant challenges. Abrupt or discontinuous changes in system dynamics generated by the transitions suggest the use of a switched approach, yet the available aerial manipulation methods are not designed for coping with switched regimes. In addition, most available methods fall short in coping with the tight couplings between the aerial vehicle and the manipulator, as well as in coping with the state-dependent uncertainties arising from the difficulty in modeling such couplings. We propose a switched-based adaptive control framework for aerial manipulators not relying on a priori knowledge of the vehicle-manipulator couplings and of state-dependent uncertainties. To guarantee stable manipulation despite changes in system dynamics, the framework provides a class of switching signals characterizing those transition phases for which the system is guaranteed to remain stable. Comparative experiments further validate the effectiveness of the proposed switched-based framework over the state of the art.
Read the full abstract ▾
All authors Rishabh Dev Yadav, Saksham Gupta, Amitabh Sharma, Sarthak Mishra, Wei Pan, Spandan Roy, Simone Baldi
Robotics
D-JEPA: A Decision-Aligned Latent World Model
Shuaijun Liu, Chengyu Wu, Qifu Wen et al. (+5 more)
Latent world models predict the consequences of actions, but accurate prediction does not guarantee that latent distance reflects which candidate will execute successfully. We identify a decision-local prediction gap: among the few futures competing for execution, a candidate predicted closer to the goal can produce a worse realized outcome than an available alternative. We introduce D-JEPA, a decision-aligned latent world model that learns decision-relevant relations among candidate futures from executed outcomes. A bounded, permutation-equivariant operator jointly reasons over goal-relative predictive features and ordinal evidence, refining pretrained predictive geometry where action choices are most consequential. Restricted predictor adaptation and a shared ordinal interface extend this
alignment across complementary predictive geometries. D-JEPA further realizes the learned decision structure in JEPA-compatible future representations, enabling deployment through native latent-distance planning. Evaluations across latent control, manipulation, pretrained action-producing models, physical
robots and autonomous driving demonstrate improved action selection, including 87.89% success on PushT, a 15.04-point average gain on RoboTwin, and a 17-point gain on physical
robot tasks. These results establish decision-relevant relational structure as a direct bridge between predictive world modeling and effective control.
Read the full abstract ▾
All authors Shuaijun Liu, Chengyu Wu, Qifu Wen, Feiyang You, Chenglong Zhang, Shuyang Hao, Xi Lin, Ningxin Su
Robotics
Beyond Visual Quality: A Study of Test-Time Planning with World Action Models
Jianhao Yuan, Yu Yuan, Benjamin Ramtoula et al. (+5 more)
World action models generate actions together with visual predictions of their consequences. These paired outputs create the potential for planning by sampling multiple actions from one state, comparing their imagined outcomes, and choosing the action with the most promising predicted outcome. However, how to use imagined futures to guide action selection remains unclear. We examine this planning potential empirically. First, we estimate an oracle upper bound on selection by choosing the sampled candidate whose realised outcome is best. In a controlled same-state analysis, this choice raises success from 68.9% under uniform random selection to 79.2%. We then test selectors based on visual quality, physical consistency, and task progression as controlled interventions. Some tested selectors yield higher observed success, but the gains are uneven and the matched selectors leave much of the measured opportunity unrecovered. To investigate this gap, we examine whether sampled actions lead to different outcomes, whether these differences are visible in the predictions, and whether a score recognises them. Counterfactual branching from the same states shows that selection opportunity is concentrated in relatively few decisions in the initial candidate sets. Action spread and outcome coverage need not increase together. In a further evaluation across trajectory phases with complete action execution, the tested scores again recover little of the available improvement despite a small gain from learned value. These findings distinguish producing consequential action choices from recognising them in generated futures, motivating the evaluation of WAM predictions through their usefulness for decisions rather than visual quality alone.
Read the full abstract ▾
All authors Jianhao Yuan, Yu Yuan, Benjamin Ramtoula, Lukas Vierling, Paul Newman, Lars Kunze, Philip Torr, Daniele De Martini
Robotics
LLM-based Conversational AI Knowledge Assistant for MyBuddy Humanoid Robot
Hanxiao Chen
Humanoid
robots are increasingly being popular and developed for human-centered applications, yet their ability to provide intelligent conversations and natural interactive knowledge assistance remains constrained by traditional rule-based dialogue systems, pre-defined responses and limited knowledge repositories. Large
language models (LLMs) have emerged as a powerful foundation for enabling natural, adaptive, and context-aware Human-
Robot Interaction (HRI), which provides a significant opportunity to address such limitations by enabling
robots to understand natural
speech language, reason over complicated queries, maintain high-quality conversational context, and generate knowledge-rich responses. In this work, we originally present and implement an LLM-based versatile Conversational AI Knowledge Assistant for the Raspberry-Pi-powered 13-Axis MyBuddy humanoid
robot, which integrates LLM-driven
language understanding and AI
reasoning with real-time
speech recognition, knowledge
retrieval via extensible access of internet engines (e.g., Wikipedia, arXiv), flexible dialogue management, and natural
speech synthesis to enable much more intelligent multi-turn continuous conversations and advanced emotional-support Human-
Robot Interaction.
Read the full abstract ▾
Robotics
SPARSER: Sparse Variable Projection by Exploiting Separable Structure in Robotic Perception
Nikolas R. Sanderson, Andrew Fishberg, Haoyu Han et al. (+5 more)
Robotic perception often requires solving large nonlinear least-squares (NLS) problems. While sparsity has been widely exploited to scale solvers, a complementary and underused structure is separability: some variables, such as visual landmarks, appear linearly in the residuals and admit a closed-form solution once the remaining variables, such as poses, are fixed. Variable projection (VarPro) exploits this structure by analytically eliminating the linear variables, yielding a reduced problem with favorable computational properties. However, its use in robotic perception has been limited by gauge
symmetries, such as invariance to global translations and rotations, which introduce challenges for standard VarPro methods. We present SPARSER (Sparsity Preserving Analytic Reduction for Separable Robotic Perception), a VarPro framework for gauge-symmetric problems that jointly exploits separability and sparsity. Our method constructs a
matrix-free Schur complement operator for efficient evaluation of reduced costs, gradients, and Hessian-vector products, enabling integration with iterative NLS solvers. We characterize the applicable problem class, identify common cases admitting further analytical simplifications, and show that IRLS-based robust costs preserve most of the exploitable structure. Across synthetic and real SLAM, SNL, and SfM
benchmarks, SPARSER is on average $5\times$–$7\times$ faster than state-of-the-art baselines on CPU and GPU, with gains exceeding $40\times$ on individual
datasets. On outlier-corrupted multi-
robot SLAM data, the robust variant is $2\times$–$16\times$ faster than a state-of-the-art GNC solver. We release open-source C++ code and all
datasets.
Read the full abstract ▾
All authors Nikolas R. Sanderson, Andrew Fishberg, Haoyu Han, Heng Yang, Jonathan P. How, Hanumant Singh, Michael Everett, Alan Papalia
Robotics
AC-DC: Adaptive Communication for Scalable Dynamic Average Consensus in Multi-Robot Ergodic Search
Robin Inho Kee, Begum Cannataro and Vasileios Tzoumas
We study scalable peer-to-peer dynamic average consensus (DC) for multi-
robot systems under finite-range, finite-rate, and interference-constrained communication. We introduce Adaptive Communication for Dynamic Average Consensus (AC-DC), which jointly adapts Who communicates with whom, When, and over What parts of the consensus state, using local inputs and successfully received neighbor information. Each
robot's consensus state estimates the current average of the
robots' local inputs. AC-DC updates these estimates as local inputs change and averages the values exchanged between
robot pairs. In AC-DC,
robot pairs update without waiting for every
robot to complete a communication round, and the selected-state messages carry consensus state coordinates independent of team size for a fixed state representation. We apply AC-DC to dynamic-priority multi-
robot ergodic search: one consensus stream estimates team visitation for motion coordination, while the other fuses regional measurement information to update uncertainty maps and search targets. Across twelve settings with up to 80
robots and 20 paired trials per setting, AC-DC has the lowest mean (i) normalized covariance-trace area under the curve (AUC) and (ii) attempted modeled communication payload among the compared decentralized methods. Averaged across settings, AC-DC achieves paired AUC reductions of 27.5% relative to state-of-the-art baselines, with 8.7x less communication traffic. As the number of
robots increases, we observe that AC-DC's communication payload approaches that of the ideal centralized baseline (one ground compute-station communicating directly with all
robots): with 120
robots in a fixed 600 x 600 m scaling test, AC-DC uses 19.3 MB versus 19.2 MB for the ideal centralized baseline, while remaining peer-to-peer.
Read the full abstract ▾
Robotics
Think Like a World Model, Act Like a VLA: Distilling World-Model Representations into Compact Robot Policies
Trung Dao, Sankalp Yamsani, Jaden Park et al. (+2 more)
Vision-
Language-Action (VLA) models map observations to actions with no objective that accounts for how the world responds, so their robustness is bounded primarily by data coverage. World models carry precisely that missing objective and are better grounded for it, yet rolling the future forward costs seconds per decision and rules them out of the control loop. We show the two can be separated. What a world model knows about physical scenes lives in its internal features; generating the future is merely the objective that produced them, so the grounding can be inherited while the generative machinery is left behind. We add one feature-
alignment term to ordinary VLA training: a frozen world model is run over the training frames once and cached, and the student learns to agree with that cache. No teacher is loaded during training, the projector is discarded after it, and the deployed policy is identical to the undistilled baseline, running in 32ms and 1.86GB on a consumer RTX5090, so every gain is attributable to the representation rather than to added capacity or test-time compute. A 0.8B student reaches 97.9% on LIBERO, improves from 48.2% to 50.5% on RoboCasa-GR1 humanoid manipulation, and the same objective carries over to real hardware, on both a single-arm and a bimanual platform. The gain survives changes of student scale, backbone,
alignment layer, and teacher, indicating a broad representational prior rather than a fragile
alignment between two particular networks. Project page: https://thaw-vla.trung-dt.com/.
Read the full abstract ▾
All authors Trung Dao, Sankalp Yamsani, Jaden Park, Joohyung Kim, Yong Jae Lee
Robotics
Touch2Robot: Robot Touch in the Human Demonstration Loop
Shengcheng Luo, Xiaoyang Cheng, Hong Ying et al. (+6 more)
Human demonstrations offer a scalable way to collect manipulation data, but their contacts may be unstable or infeasible when transferred to a
robot hand. Collecting demonstrations directly on the target
robot avoids this mismatch but substantially increases the cost of data collection. To address this trade-off, we present Touch2
Robot, a framework that lets humans collect demonstrations while seeing how the target
robot hand would contact the object. We capture human hand motion, tactile-glove measurements, and object motion during human manipulation. These recordings guide object-specific RL policies to reproduce the demonstrated object motion while favoring contacts consistent with the recorded human touch. We distill the learned behaviors into a unified real-time retargeter that maps incoming human observations and object geometry to
robot hand configurations. During collection, the predicted
robot configuration is synchronized with the tracked object pose in simulation to reconstruct
robot-object contacts, which are visualized to help the demonstrator adapt subsequent interactions to the target hand. Across four real-world tasks, Touch2
Robot improves average real-
robot replay completion from 37.9% to 72.1% over visual-only feedback, while reducing the collection time per replay-successful demonstration from 58.6s to 18.2s. Reconstructed target-hand contacts achieve 44.2% F1 against real-
robot tactile measurements, and policies trained on Touch2
Robot demonstrations improve downstream
Diffusion Policy performance by 29.1 percentage points over visual-only feedback. These results show that bringing
robot touch into the human demonstration loop improves both the quality and efficiency of scalable dexterous data collection. Project webpage: https://Touch2
Robot.github.io/.
Read the full abstract ▾
All authors Shengcheng Luo, Xiaoyang Cheng, Hong Ying, Xiaoying Zhou, Jiaming Jiang, Haoran Guo, Wanlin Li, Ziyuan Jiao, Chenxi Xiao
Robotics
Feasibility Distance Fields for Heterogeneous Constraints in Robot Configuration Space
Xijing Cui, Huayan Pu, Jun Luo et al. (+1 more)
Robot manipulators are monitored by constraint-specific indicators whose units and gradient scales are not comparable, so they do not provide a common measure of the configuration-space motion remaining before violation. We define the feasibility distance field (FDF) as the distance, under a fixed positive-definite joint-space metric, to the union of infeasible configuration sets. Classical distance-to-set theory gives 1-Lipschitz continuity, almost-everywhere differentiability, and unit dual-gradient norm wherever the nearest projection is unique. The robotics contribution is an admissibility analysis showing when practical constraints define non-empty closed sets. We derive admissible formulations for external and self-collision, joint limits, dexterity, Cartesian and task-projected compliance, joint torque under payload, and dynamic manipulability. Since every field uses the same metric, heterogeneous constraints compose by a pointwise minimum, conditioned constraints retain a fixed distance space, and multi-
robot constraints produce block-sparse gradients that identify which
robots must react. We generate projection-based labels and train neural approximations with a distance loss and an Eikonal penalty. Simulations on a UR5e and a dual-arm
cell evaluate seven fields using value, projection, sign, gradient, composition, and moving-obstacle diagnostics. Across 8,000 configurations, the largest feasible-side secant ratio is 0.920, mean learned gradient norms range from 0.994 to 0.998, and projection residuals range from 0.011 to 0.034 rad. Across 24 random obstacle paths, the external and composed collision fields achieve 90.4% and 91.6% success within 3 cm, with sign-error rates below 2%. The results support a common configuration-space margin and identify approximation errors near medial axes and sparsely sampled boundaries.
Read the full abstract ▾
All authors Xijing Cui, Huayan Pu, Jun Luo, Gang Wang
Robotics
From Semantic Decisions to Feasible Trajectories: Self-Evolving LLM-Guided Optimal Control for Narrow-Space Parking
Zhengbao Yao, Yuanfu Luo and Kehan Xue
Autonomous parking in nonconvex and narrow environments remains challenging. Although optimal-control methods can explicitly enforce vehicle dynamics and collision constraints, nonconvexity compromises solver robustness and can cause failures. Large
language models (LLMs) exhibit strong semantic
reasoning capabilities, but directly generating dense trajectories makes it difficult to guarantee physical feasibility. We introduce SE-LLM-OCP, a unified framework in which LLMs make high-level discrete maneuver decisions, while an optimal-control module enforces low-level vehicle dynamics and collision constraints. Online, the LLM proposes sparse maneuver plans, decomposing the parking task into a sequence of short-horizon trajectory-optimization problems. A low-level solver then sequentially solves optimal-control problems. If the solver fails, the LLM aggregates failure evidence from the solver and validation stages to guide replanning. Offline, SE-LLM-OCP automatically evolves a structured decision-making knowledge base from scratch, driven by accumulated online failures. We validate our proposed framework in simulation on a car-like vehicle model and on a differential-drive
robot. Our experimental results show that SE-LLM-OCP enables safer autonomous parking in narrow scenarios and demonstrates transfer of the same maneuver representation to a different kinematic platform.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Relationally Grounded Latent World Models for Autonomous Driving
Fabian Schmidt, Markus Enzweiler and Abhinav Valada
Latent world models learn predictive representations for autonomous driving, but the relational semantics these states preserve often remain implicit. We investigate whether traffic scene graphs can serve as privileged semantic supervision for latent world representations. Building on LAW, we construct actor-centric scene graphs from nuScenes 3D annotations, encode their serialized relational structure using a frozen text
embedding model, and align the visual latent representations with this semantic target during training. We remove the supervision branch at inference, so it requires neither scene graphs nor 3D annotations and adds no test-time computation. On nuScenes, our method reduces average trajectory L2 error from 0.661 to 0.622 (5.9%) and collision rate from 0.456 to 0.217 (52.4%) relative to our retrained LAW baseline. It also outperforms an unstructured caption-style semantic target, supporting the benefit of explicit relational structure for latent world-model representation learning.
Read the full abstract ▾
Robotics
Learning tactile perception from high-bandwidth single-point sensing
Joseph Rigal, Emmanuel Virot and Caroline Pascal
Tactile sensing is increasingly being incorporated into learning-based robotic manipulation, yet many existing approaches rely on spatially distributed sensors. Here we introduce {SpectRobot}, a framework that transforms single-point tactile signals into compact time-frequency spectrograms. These spectrograms encode high-bandwidth tactile histories as fixed-size image-like representations. They can be processed by standard
vision encoders and integrated into learning pipelines originally developed for
vision, while preserving temporal and frequency information unavailable to conventional cameras. Rather than increasing spatial density through arrays of tactile elements, SpectRobot exploits the rich dynamics contained in sparse, high-bandwidth single-point measurements. In our implementation, the sensors are mounted away from the contact surface while remaining mechanically coupled to it, reducing direct exposure to wear and potentially improving robustness in harsh environments and for long-term deployment on dexterous
robots. Our experiments demonstrate that: (1) a
robot can exploit single-point vibration signals to solve a visually occluded manipulation task; (2) temporal history strongly influences policy performance, while sensing bandwidth controls the spectral information available, with measurements extending to 100~kHz; and (3) the same representation can be used across different tactile sensing technologies mediated by acceleration, force, or strain. We further show that capabilities previously associated with research-grade instrumentation can be accessed using readily available, off-the-shelf hardware. We believe that broader access to high-bandwidth tactile sensing could facilitate the integration of contact dynamics into embodied learning systems and, for some tasks, offer an alternative or complement to increasing the spatial density of tactile sensing.
Read the full abstract ▾
Robotics
What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior
Débora Oliveira Makowski, Samiran Gode, Abhijeet Nayak et al. (+4 more)
Modern
Vision-
Language Navigation (VLN) models rely mostly on pre-trained large
Vision-
Language Models (VLMs) to predict navigation actions. While this fusion of
language instructions and visual observations allows
multimodal reasoning, it obscures how information is routed across modalities or what mechanisms drive navigation decisions. Thus, it remains unclear whether VLN models ground their predictions in relevant semantic cues or can track task progress. In this work, we study the interpretability and steerability of VLN models. We use intervention-based metrics that measure how visual observations, instructions, and visual
memory causally influence navigation decisions. Our results show that these navigation policies are sensitive to all input modalities and do not depend on a single one. We further show that these
agents encode navigation progress and retain semantic structure from their VLM backbones, enabling concept-level steering through internal activations. Finally, we extract activation vectors for abstract behaviors to transfer them zero-shot to out-of-distribution real-world scenarios, improving performance without additional fine-tuning.
Read the full abstract ▾
All authors Débora Oliveira Makowski, Samiran Gode, Abhijeet Nayak, Marco Hutter, Cordelia Schmid, Lukas Rosenberger Schmid, Wolfram Burgard
Robotics
ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation
Bowei Li, Yuner Zhang and Changliu Liu
Adapting visuomotor policies to new manipulation tasks often requires substantial manual engineering or teleoperated data collection. Simulation can provide task-specific data at scale, but constructing the scene, designing expert behavior, and configuring data generation still require significant per-task effort. We present ARSTAG, an agentic Real2Sim2Real system that turns a single RGB image and a natural-
language instruction directly into
robot policy-learning data. A hierarchy of
language agents constructs a task-scoped simulation scene, generates
robot-feasible demonstrations, and expands the training distribution through task-consistent randomization, while a coordinator
agent manages cross-stage feedback and recovery. Across seven manipulation tasks spanning grasping, placement, and stacking, the ARSTAG-generated demonstrations enable sim-to-real transfer of three visuomotor policy architectures to a dual-arm
robot, with pi0.5 achieving an average real-world success rate of 74.6%. Ablations show that task-consistent randomization substantially improves robustness, and policy performance increases with generated
dataset size. Project webpage: https://boweili666.github.io/ARSTAG/.
Read the full abstract ▾
Robotics
Smoothness as a Constraint for Stable Humanoid Locomotion
Utsav Panchal, Denis Kleyko, Unal Artan et al. (+1 more)
Embodied AI systems, particularly humanoid
robots deployed in real world scenarios require whole-body control policies that are both task-responsive and physically smooth. However, smoothness is not uniform across the body: lower body must remain sufficiently reactive, while the upper body must be tightly regulated to preserve stability. Existing reinforcement learning approaches typically impose smoothness through auxiliary terms in the reward function, which compete with task objectives, treating the body as uniform and provide no direct control over the physical quantities responsible for smooth behavior. We introduce DeCap (Decoupled Constraint-aware policy), a constrained reinforcement learning
algorithm that decouples whole-body smoothness into separate upper- and lower-body constraint groups, each formulates smoothness as explicit constraints on physical motion limits. To improve constraint satisfaction near feasibility boundaries, DeCap incorporates a bounded barrier penalty that activates proactively as limits are approached and remains bounded at the constraint limit. On real-world humanoid whole-body control task, DeCap reduces upper-body action rate by 2.50x and acceleration by 2.18x relative to reward-based smoothness policies, while also improving lower-body smoothness and reducing transient motion. We demonstrate that a fixed set of smoothness constraints transfers across diverse terrains, alleviating the need of extensive reward tuning.
Read the full abstract ▾
All authors Utsav Panchal, Denis Kleyko, Unal Artan, Amy Loutfi
Robotics
MIRA: Real-Time Full-Duplex Human-Robot Interaction for Embodied Companions
Lijian Lin, Ye Zhu, Fan Zhang et al. (+5 more)
% !TEX root = ../main.tex Real-time embodied companion interaction requires a
robot to infer user intent from streaming
speech, generate timely responses, and execute expressive, interruptible motions. Existing systems typically decouple dialogue orchestration from gesture synthesis, relying on offline motion generation from complete audio. This separation leaves open how a deployed
robot can dynamically synchronize response content, prosodic timing, and physical safety under incremental inputs and uncertain turn boundaries. We present MIRA, a unified framework for real-time full-duplex embodied companion interaction. Given streaming user
speech, dialogue history, and vocal affect, MIRA predicts both the response text and an explicit embodiment cue that routes the response to the appropriate physical behavior. Discrete social behaviors (\eg listening and greeting) are mapped to validated
robot trajectories, while speaking responses are accompanied by streaming, generative co-
speech motion. For co-
speech motion generation, we propose ROSCO, a prefix-conditioned
diffusion model for streaming audio-to-joint motion generation. We further design RHPC, an inference scheme that maintains a sufficiently long temporal context for motion prediction while bounding physical commitment to a short, interruptible prefix. At the interaction level, we design CORTEX, a dual-timescale interaction policy that combines low-latency barge-in preemption and streaming response generation with deliberative turn decisions, backed by a
robot-side execution layer that enforces physical safety constraints during execution. MIRA is deployed on an Astribot S1 humanoid
robot. Quantitative evaluations demonstrate competitive audio-motion
alignment relative to state-of-the-art motion-generation baselines, while real-
robot deployment measurements characterize streaming responsiveness and interruption handling.
Read the full abstract ▾
All authors Lijian Lin, Ye Zhu, Fan Zhang, Yunfei Liu, Baofeng Li, Xianwen Zeng, Jianan Wang, Yu Li
Robotics
Estimation and Control of Tensegrity Manipulator Kinematics based on Strut Inclination Angles
Tufail Ahmad Bhat and Shuhei Ikemoto
Unlike conventional rigid-link
robots defined by discrete joints, continuum
robots pose a fundamental challenge for expressing their complex continuous bending configurations for closed-loop control. Several modelling approaches have been proposed for conventional continuum
robots, but tensegrity-based continuum
robots remain largely open. Moreover, many of these approaches assume a continuous elastic backbone and are therefore not directly applicable to tensegrity manipulators, whose bodies are networks of rigid struts and tensioned cables. This work presents a reduced-order model for shape and posture control of a tensegrity-based continuum manipulator. The manipulator is modelled as a serially connected parallel-link mechanism. The proposed method is formulated as an optimization problem that uses
geometric constraints of the tensegrity structure together with information from the Inertial Measurement Unit (IMU) sensors embedded in the strut elements. To the best of our knowledge, this work presents the first experimental demonstration of a real-time IMU-based shape estimation method on a full-scale tensegrity manipulator and demonstrates posture control using a simple Proportional-Integral (PI) controller. The results show that the proposed method can estimate the shape of both single-module tensegrity structures and multi-module tensegrity manipulators from arbitrary static configurations and achieve desired postures.
Read the full abstract ▾
Robotics
Bridge3D: Enabling Vision-Language-Action Models to See and Act in 3D
Haoxuan Li, Sixu Yan, Lianghui Zhu et al. (+3 more)
Vision-
Language-Action (VLA) models have demonstrated remarkable generalization in robotic manipulation via large-scale
multimodal pretraining. However, VLA models are mainly trained on 2D-centric observations, which inherently constrains their capacity for precise spatial manipulation. Previous methods enhance 3D awareness by introducing implicit spatial priors, but still lack explicit geometry guidance. In this paper, we propose Bridge3D that integrates both implicit and explicit 3D geometry guidance into pre-trained 2D VLA models, enabling them to ”see” and ”act” in 3D. Bridge3D introduces two strategies: 1) Implicit Fusion, which enriches visual
tokens with features from 3D foundation models to improve ”seeing” in 3D; 2) Explicit Conditioning, which integrates action denoising with an explicit 3D semantic field to achieve ”acting” in 3D. Furthermore, we utilize the proposed layer-wise linear probing to improve learning efficiency. Experiments show that Bridge3D achieves superior performance against state-of-the-art methods. On the RoboTwin 2.0
benchmark, Bridge3D exceeds $\pi_0$ by 14.0 percentage points, while in real-world experiments, it outperforms Spatial Forcing by 11.7 percentage points. These results demonstrate Bridge3D's strong capabilities in high-precision and spatial-sensitive manipulation tasks.
Read the full abstract ▾
All authors Haoxuan Li, Sixu Yan, Lianghui Zhu, Xuanlai Tang, Shikang Wang, Xinggang Wang
Robotics
InsertAnything: Generalizable Contact-Rich Precision Insertion from Simulation to Reality
Zhenghua Ma, Xinpan Meng, Zeyu Liu et al. (+4 more)
Contact-rich precision insertion is a key manipulation skill in robotic assembly. Tight clearances make insertion more sensitive to
alignment errors and prone to collisions and jamming, while variations in geometry and clearance across parts further complicate policy reuse. We present a reinforcement learning framework that trains insertion policies entirely in simulation for direct deployment without real-world demonstrations or policy fine-tuning. By combining target poses with compact three-dimensional fingertip force feedback, the policy learns to search for
alignment and correct its motion despite errors in the estimated hole position. A decoupled gated reward coordinates
alignment and insertion. Force-signal smoothing and state-independent standard deviations stabilize the learning process. The resulting policies perform real-world insertion across multiple hole geometries with a minimum nominal clearance of 0.02 mm and improve success while reducing peak contact forces under hole-position errors. Cross-clearance and cross-geometry evaluations further confirm policy generalization. The system achieved the first perfect score of 20/20 on ManipulationNet's peg-in-hole
benchmark under its Human-in-the-Loop protocol, with fully autonomous insertion motions. A single policy trained only on a simulated hexagonal insertion task achieved an overall success rate of 95.0% across eight unseen real-world insertion tasks. These results show that learning entirely in simulation can yield precision insertion skills that can be deployed directly and reused across real-world tasks. The project website (https://mzhsoul.github.io/InsertAnything/) provides open-source simulation and real-
robot experiment scripts, assets, and trained checkpoints.
Read the full abstract ▾
All authors Zhenghua Ma, Xinpan Meng, Zeyu Liu, Muyuan Ma, Hengdi Zhang, Houcheng Li, Long Cheng
Robotics
TACIT: Tactile Contact Supervision for Spatial Attention in Dexterous Manipulation
Yanhou Lai, Fucai Zhu, Ruiqiang Wang et al. (+1 more)
Visuomotor policies trained from a few demonstrations may reproduce demonstrated trajectories without reliably following changes in object position. Existing approaches with explicit
attention typically obtain spatial priors from human annotation or visual models. We introduce TACIT (tactile contact informs
attention), which uses measured tactile contacts from teleoperated demonstrations to supervise spatial
attention without additional point annotation. Gaussian targets over preceding camera point clouds supervise an
attention head whose pooled output conditions a visuotactile
diffusion policy. Targets are used only during training; tactile observations remain inputs at inference. In the primary real-
robot benchmark, with ten demonstrations per task and five demonstrated placement regions, TACIT achieves 66.7% success on ball placement and 73.3% on peg insertion, compared with 10.0% and 20.0% for input-matched 3D visuotactile fusion and 20.0% and 43.3% for
vision-only DP3. TACIT enters the 150 mm palm-to-object approach region within 12 seconds in all 30 trials per task; all remaining failures occur after arrival. Across three training seeds on real ball and simulated peg, TACIT outperforms input-matched fusion and an architecture-matched control without explicit
attention supervision, supporting the contribution of supervision beyond branch capacity. Pre-contact and contact-time supervision show no consistent ordering. These results demonstrate that measured tactile contact provides effective spatial supervision for approach behavior from few demonstrations within the evaluated workspace.
Read the full abstract ▾
All authors Yanhou Lai, Fucai Zhu, Ruiqiang Wang, Koichi Hashimoto
Robotics
A Monolithic Force-Proprioception Soft Acutuator Enabled by Single-Material 3D printing
Nan Huang, Lele Liu, Junfeng Lu et al. (+3 more)
Pneumatic proprioceptive actuators integrate actuation and sensing for soft
robots that attract interest due to functional potential. Existing approaches often suffer from assembly errors or stress concentrations caused by heterogeneous materials. In this work, we propose the Monolithic Force-Proprioception Soft (MFPS) design and fabrication method that integrates an Asymmetric Origami Bending (AOB) chamber and a Force-Proprioception Soft (FPS) sensor with single material through one-step Fused Deposition Modeling (FDM) fabrication. Based on the resistance response to strain of conductive thermoplastic polyurethane (TPU), we design and analyze the structure of the FPS sensor, and conduct parametric analysis on the sensing characteristics. The FDM fabrication parameters of the MFPS actuator are analyzed, followed by actuator fabrication and characterization of the actuation and proprioception performance. Experimental results show that the MFPS actuator achieves a bending angle of 40{\deg}, an output force of 12.5 N, and a resistance change of 26.9% as the applied external force increased from 0 to 45 N. A two-finger force-proprioception gripper is developed based on the MFPS actuator. The grasping and force-proprioception capabilities are experimentally validated, proving that the MFPS design method provides a new approach for the development of self-sensing actuators.
Read the full abstract ▾
All authors Nan Huang, Lele Liu, Junfeng Lu, Yipan Zhu, Jiansheng Dai, Sicong Liu
Robotics
ScaleMPA: Rethinking Scalable RRT* Acceleration With a Grid-Native Representation
Zilong Wang, Yuzhou Chen, Xinyue He et al. (+2 more)
Real-time motion planning remains challenging in large and high-dimensional environments. Prior acceleration of RRT* follows tree-centric state organization, which reduces per-query cost but preserves superlinear end-to-end complexity and limits parallelism through structural dependencies. This paper presents ScaleMPA, a motion-planning accelerator that rethinks RRT* with a grid-native representation. By replacing hierarchical traversal with direct grid-based access, ScaleMPA reduces the planner critical path and exposes fine-grained parallelism. To make this reformulation practical under sparse high-dimensional planning, ScaleMPA further proposes a multi-resolution grid search engine and a hash-grid
memory system. Implemented in 28 nm CMOS, ScaleMPA achieves millisecond-level planning latency and delivers 4.7$\times$–44.4$\times$ speedup over state-of-the-art motion-planning accelerators.
Read the full abstract ▾
All authors Zilong Wang, Yuzhou Chen, Xinyue He, Chen Zhang, Guanghui He
Robotics · Cross-listed from cs.CV
Do LiDAR Language Models Really Understand Spatio-temporal Relationships?
Runyi Yang, Murat Akkoyun, Di Wen et al. (+8 more)
Recent 4D
LiDAR language models aim to reason about objects and their evolving spatial relationships. Yet, in our evaluation, always selecting the same option nearly matches the multiple-choice accuracy of two B4DL-derived configurations. We introduce
LiDAR-Hallu, a geometry-referenced
benchmark and diagnostic protocol with 10,000 questions across 150 nuScenes scenes. It covers object existence, ego-relative position, distance ordering, relative motion, and temporal localization, with explicit rules for selecting objects, comparing times, and determining reference answers. Our protocol combines fixed-answer and candidate-content controls, cross-scene pairs with identical
prompts but opposite reference answers, and relation-specific recall. Analysis of 100,000 recorded responses reveals failures hidden by aggregate accuracy. Candidate duration alone makes temporal answers predictable without observing
LiDAR. On paired questions, the models frequently give the same answer to scenes requiring opposite answers. Relation-specific analysis further shows that both configurations miss every positive lateral-motion case across all tested conditions. Temporal-shuffle contrastive decoding provides little net improvement, as repairs are largely offset by new errors and the main failures persist. These results show that evaluating spatio-temporal
reasoning requires testing whether models distinguish the queried physical relationships, rather than relying on individual-answer accuracy alone. The source code, checkpoints, and data are released at https://github.com/Awesome4D/4DMLLM_
Hallucination_Bench.
Read the full abstract ▾
All authors Runyi Yang, Murat Akkoyun, Di Wen, Ruiping Liu, Yufan Chen, Junwei Zheng, Xiaoye Wang, Kailun Yang, Danda Pani Paudel, Luc Van Gool, Kunyu Peng