Robotics
CF-JEPA: Improving Robustness of JEPA World Models via Controllability Factorization
Morgan Byrd, Robert Wright and Sehoon Ha
Controlling an
agent with
vision requires being able to separate useful information from irrelevant background information. JEPA-style latent world models seem like a natural approach for this, as they do not perform pixel-level reconstruction; however, they are still sensitive to these distractor signals and experience latent collapse. In this work, we introduce Controllability Factorized JEPA (CF-JEPA), a JEPA-style world model which splits the latent space into controllable and uncontrollable subspaces. This factorization allows us to capture all the distractor information into the uncontrollable region, while we use the control-relevant latent information for our task. With this, we show comparable performance across 2D and 3D control tasks under nominal conditions and improved performance under distracted conditions, where CF-JEPA is the only model that does not experience latent collapse. We also validate our model under distracted conditions for a simulated
robot task, highlighting the practical application of such a scheme.
Read the full abstract ▾
Robotics
Toward Humanoid Robots in Construction: A Teleoperation Feasibility Study
Parastoo Ali Pour, David R. Martin, Chang Min Hur et al. (+6 more)
We present a teleoperation system that enables a single operator to perform construction tasks on a Unitree G1 humanoid, combining extended reality (XR) based upper body control with pedal-based locomotion to enable simultaneous manipulation and locomotion. Motivated by persistent labor shortages, hazardous working conditions, and challenges in humanoid autonomy, we investigate teleoperation as a practical near-term approach for reducing physical strain on workers while generating high quality demonstration data. We evaluate the system on two representative construction tasks drawn from O*NET occupational database, and report task success and completion time relative to a manual baseline. The system achieved 100% success on tool transport and 80% success on surface painting, with teleoperation requiring substantially more time compared to manual execution.
Read the full abstract ▾
All authors Parastoo Ali Pour, David R. Martin, Chang Min Hur, Bo Zhang, Tommy Zhou, Brandon Thomas Lichter, Shane Stanfield, Pramod Khargonekar, Mohammad Abdullah Al Faruque
Robotics · Cross-listed from cs.LG
Learning Transferable Skills using Goal-Conditioned Bisimulation
Mohammad Amin Abbasfar, Farbod Azimmohseni and Mohammad Hossein Rohban
Unsupervised skill discovery has emerged as a promising approach for leveraging reward-free
datasets to pretrain general-purpose policies. However, current skill discovery methods either require access to expert data or exhibit limited generalization, failing to transfer effectively to previously unseen layouts. A key challenge is to learn representations that capture the temporal structure of the environment while remaining robust to variations across layouts. To address this issue, we present an objective for learning action-aware temporal representations that satisfy the functional equivariance property while preserving the local temporal structure of the environment. Building upon this
embedding, we further propose unsupervised skill discovery using bisimulation, which learns transferable skills by conditioning the behavior of skills exclusively on the subset of state features that directly affect their execution. This enforces invariant behavior across different layouts, enabling skills to transfer effectively to other configurations. Finally, through comprehensive empirical evaluations, we show that skills learned in a given environment can be effectively applied to solve downstream tasks in various environment layouts, demonstrating strong out-of-distribution generalization.
Read the full abstract ▾
Robotics
TacDyn-WAM: Learning Implicit Tactile Dynamics in a Heterogeneous Visuo-Tactile World Action Model
Enyi Wang, Mingxin Wang, Quan Shi et al. (+12 more)
World action models improve robotic manipulation by conditioning actions on predicted futures, yet existing tactile variants largely inherit
video-generation pipelines that reconstruct future tactile observations through iterative denoising. Such prediction can become unreliable under deployment drift: small changes in contact position or force may substantially alter tactile pixels even when the underlying contact
evolution remains predictable. We introduce TacDyn-WAM, a heterogeneous visuo-tactile world action model that predicts implicit tactile dynamics rather than reconstructing future tactile observations. It learns TacRep, a dynamics-aware tactile target space trained through masked spatio-temporal prediction on tactile clips and regularized by relational structure
distillation. A visual expert and an Implicit Tactile Dynamics Expert predict future visual and tactile representations in separate target spaces while interacting through joint
attention; the tactile expert predicts future representations and their changes at multiple horizons in a single forward pass, and a read-only tactile
memory supplies the current tactile state. On UniVTAC, TacDyn-WAM achieves an average success rate of 81.5% using only the provided demonstrations, reaching state-of-the-art-level performance and remaining competitive with models pretrained on large-scale visuo-tactile trajectories. Ablations confirm the benefits of both tactile pathways and TacRep over pixel-reconstruction and static alternatives. On five real-
robot tasks, TacDyn-WAM reaches 71.0% average success, and modest-scale pretraining raises it to 85.0%, further validating our method.
Read the full abstract ▾
All authors Enyi Wang, Mingxin Wang, Quan Shi, Hetian Guo, Hongyu Wang, Xi Wang, Bin Qian, Yupeng Zheng, Wenxuan Song, Houde Liu, Yong Xu, Cheng Chi, Wenchao Ding, Yilun Chen, Yan Wang
Robotics · Cross-listed from cs.AI
Spatial Strategies, Not Actions: Vector-Quantized Geodesics as Tools for LLM-Driven Agents
Gabriel Turinici
Large
language model (LLM) based
agents are often criticized for lacking spatial understanding and mainly exploiting statistical text patterns. We investigate their spatial comprehension through an architecture combining geometrical tools with a LLM serving as a high-level orchestrator in grid-world environments. The
agent first collects geodesic trajectories, which are then vector-quantized to extract a representative subset. Offline, the LLM associates a natural
language description of the underlying behavioral patterns to each selected trajectory, making it a tool. Online, the LLM chooses the appropriate tool conditioned on the current state and goal. Low-level control is handled by primitive actions that execute the trajectory associated with the tool. From an agentic AI perspective, this approach separates learning into two levels: tool discovery is handled through unsupervised quantization of trajectories, while
reasoning and decision-making are handled by the LLM. We test the approach in a partially observable dynamic 2D grid environment with an open
vision-
language model (Qwen3.6-35B-A3B). Pairing the geometry-derived tool library with an
agent-centered zoom tool and a collision detection tool lets a fast, non-
reasoning configuration match the goal-reaching rate of a much more costly chain-of-thought version, while cutting the cost of a decision from minutes to seconds.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation
Egor Cherepanov, Nikita Kachaev, Aleksandr I. Panov et al. (+1 more)
Vision-
language-action policies often see only one or a few recent frames, which makes it difficult to evaluate how they use information that disappears during a task. We introduce MIKASA-Robo-VLA, a
benchmark of 90
language-conditioned manipulation tasks. All but 10 hide the cue an action depends on. Those 10 are reactive controls. MIKASA-Robo, the suite it rebuilds, has 32 tasks and uses
language only in a representative VLA subset. Here every task provides an instruction, while
memory-dependent tasks hide a task-relevant cue and reactive controls keep it available. For 70 tasks, environment phase timings specify an information gap, and for 28 of them the gap exceeds the 16-frame window of the widest fixed-context VLA we survey. The gap counts only the interval the cue is provably absent, not the full duration a policy must retain it, so every
memory-dependent task still requires
memory by construction, including the ones whose measured gap is short. We release 22,500 oracle trajectories across 10
memory types in RLDS and LeRobotDataset v3. A reference $\pi_{0.5}$ baseline with current images and proprioception, but no observation history or explicit
memory module, is fine-tuned on 14 tasks and achieves 0.211 $\pm$ 0.044 mean task success. Its lower success on the evaluated Long-split tasks is confounded by open-loop chunking and the
memory types represented in that subset. Project page: https://mikasarobo.github.io/
Read the full abstract ▾
All authors Egor Cherepanov, Nikita Kachaev, Aleksandr I. Panov, Alexey K. Kovalev
Robotics
When Reasoning Helps Action: Monitoring and Steering Chain-of-Thought in Vision-Language-Action Policies
Sathwik Karnik, Joseph JR. Lee, Aryaman Gupta et al. (+1 more)
Reasoning-enabled VLA policies expose chain-of-thought (CoT) traces that appear to explain and guide their actions, creating a potential interface for runtime safety through
reasoning monitoring and correction. In this work, we define and operationalize two evaluation axes for assessing when this interface can improve embodied behavior: correctability, which measures whether unreliable
reasoning can be detected and improved during generation, and actionability, which measures whether
reasoning corrections produce behaviorally meaningful changes in the intended direction. To enable correctability, we introduce
Token-level Reward for Utility-Steered Chain-of-Thought (TRUST), an offline-trained value model that predicts eventual
reasoning correctness from partial prefixes and uses these estimates to monitor and selectively steer
reasoning generation in frozen VLA policies. On the Alpamayo 1.5 driving VLA, TRUST monitors correctness with 88.9% accuracy and improves
reasoning correctness from 75.9% to 90.0%. On a baseline-defined challenging subset in AlpaSim, TRUST reduces collision rate by 30.4% and maximum trajectory error by 11.5% relative to the unsteered policy, outperforming a compute-matched Best-of-4 baseline. On the DeepThinkVLA manipulation VLA, TRUST improves the correctness of grasp-state claims from 69.3% to 90.2% and action-choice claims from 68.8% to 85.9%, yet closed-loop task performance on LIBERO-Plus remains largely unchanged. Empirical analysis reveals intent-consistent behavioral effects in Alpamayo 1.5 but limited effects in DeepThinkVLA, helping interpret these different task-level outcomes. Together, our results show that gains in
reasoning correctness do not automatically imply gains in embodied performance, motivating evaluation of correctability and actionability when using CoT as a runtime safety interface.
Read the full abstract ▾
All authors Sathwik Karnik, Joseph JR. Lee, Aryaman Gupta, Somil Bansal
Robotics
Chuyao Fu, Xiaowei Chi, Yuhan Rui et al. (+14 more)
A common approach to world-model simulation for
vision-
language-action (VLA) systems is to predict future RGB observations and then re-encode them into policy inputs, introducing an indirect interface between simulation and downstream policy execution. We instead investigate whether world dynamics can be modeled in a compact, policy-oriented state derived from VLM visual
tokens. A key challenge is that raw VLM visual
tokens are high-dimensional, making efficient and accurate autoregressive dynamics modeling challenging. To address this, we introduce
Token-World, an action-conditioned world model that compresses VLM features into a compact
token state, learns future dynamics in this reduced space, and maps predicted states back to the original policy-facing representation for downstream use. Across manipulation
benchmarks,
Token-World improves open-loop feature fidelity and policy-action consistency over recent world-model simulators, with slower degradation over long rollout horizons. In closed-loop evaluation, its simulated policy performance correlates more strongly with reference policy performance than Ctrl-World ($r=0.794$ vs.\ $0.583$), while requiring lower simulation latency. Ablations further show that compact-representation design and dimensionality substantially affect future-state prediction. Code will be available at https://chuyaofu.github.io/
Token-World/.
Read the full abstract ▾
All authors Chuyao Fu, Xiaowei Chi, Yuhan Rui, Yu-kai Wang, Zezhong Qian, Xiaojie Zhang, Yunfan Lou, Kevin Zhang, Kuangzhi Ge, Chak Wing Mak, Zhiyang Chen, Athena Zhuoming Zhong, Hongyang Chen, Haoran Li, Yike Guo, Sirui Han, Shanghang Zhang
Robotics
Does Continual Imitation Learning Remain Grounded? A Language-Perturbed Benchmark for Robotic Task Retention
Siddeshwar Raghavan, Ziqin Yuan, Fengqing Zhu et al. (+1 more)
Continual imitation learning evaluates whether a
robot can learn new knowledge without forgetting previously learned skills. However, retaining task performance does not ensure the behavior remains grounded in
language because policies may rely on scene cues, object associations, or memorized task structure. We introduce a
benchmark protocol to study how
language-guided behavior changes as robotic policies learn successive tasks. We construct meaning-preserving and meaning-changing instruction variants for the Goal, Spatial, Object, and Long suites of LIBERO. Policy experiments focus on LIBERO-Goal, evaluating Original and Paraphrase instructions after each continual-learning stage. We compare representative continual imitation learning methods under their original assumptions while separating task competence from
language sensitivity. The proposed diagnostics complement standard learning and forgetting metrics by measuring semantic robustness, goal adaptation, and
language sensitivity. Results show that strong continual-learning performance does not always translate to reliable
language grounding, and our diagnostics help determine whether retained skills remain correctly guided by their instructions. Additional materials are available at https://sites.google.com/view/stillgrounded
Read the full abstract ▾
All authors Siddeshwar Raghavan, Ziqin Yuan, Fengqing Zhu, Byung-Cheol Min
Robotics · Cross-listed from eess.SY
Admissibility-Preserving Control for Multi-Input Systems with Joint Capacity Constraints
Saurabh Kumar, Lohitvel Gopikannan, Shashi Ranjan Kumar et al. (+1 more)
This paper addresses the control of multi-input strict-feedback nonlinear systems subject to a joint capacity constraint, in which the admissible input set is a coupled subset of the individual actuator limits. Unlike existing constraint-handling methods that enforce actuator bounds channel by channel and may unnecessarily suppress admissible control directions, we develop an Anisotropic Joint-Admissibility-Preserving Input Realization (AJ-APIR) framework that explicitly exploits the geometry of the joint constraint. The proposed realization constructs a state-dependent gain
matrix whose spectral decomposition separates the commanded input into normal and tangential directions relative to the constraint boundary. The normal component is attenuated as the boundary is approached, while the tangential component is preserved, which allows the admissible control effort to be redistributed without loss of tracking authority. Integrated with a backstepping controller, the AJ-APIR framework guarantees forward invariance of the joint admissible set for all time. We establish exponential convergence of the tracking error to zero together with uniform boundedness of all closed-loop signals, and characterize the resulting command-demand behavior under the joint constraint. Simulation results for a representative second-order, two-input nonlinear system subject to a power-budget constraint demonstrate the efficacy of the proposed method to enforce the joint input constraint.
Read the full abstract ▾
All authors Saurabh Kumar, Lohitvel Gopikannan, Shashi Ranjan Kumar, Abhinav Sinha
Robotics
Same Scene, Different Task: Skill Alignment for Compositional Generalization in VLAs
Taegeun Yang, Youngju Na, Yoonki Cho et al. (+1 more)
Vision-
language-action (VLA) models often struggle to generalize to skill combinations absent from their fine-tuning demonstrations, even when every constituent skill has been demonstrated. We focus on a
vision shortcut as one failure mode: during fine-tuning, visual observations can serve as a proxy for the instruction, so a policy may execute a demonstrated combination associated with similar observations rather than the instructed combination. This motivates training with counterfactual pairs formed by holding a demonstration observation fixed while changing the instruction to specify an undemonstrated combination. These pairs, however, lack corresponding demonstrated action targets. Crucially, the currently required skill has already been demonstrated, but actions from those executions cannot serve as direct targets because the same skill can require different actions across observations. We propose CRAFT, which transfers supervision from demonstrated executions of the required skill to counterfactual pairs using skill representations that can be reused across executions of the same skill. Across three VLA models and two simulation
benchmarks, CRAFT improves success on undemonstrated combinations while maintaining high success on demonstrated ones; it also improves compositional generalization on a real
robot. Project website: https://taegeunyang.github.io/craft/
Read the full abstract ▾
All authors Taegeun Yang, Youngju Na, Yoonki Cho, Sung-Eui Yoon
Robotics
ScaffoldM3C: A Multimodal Sequential Monte Carlo Framework for Generative Stable Construction Planning
Gadiel Sznaier Camps, Chengyang He, Guillaume Sartoretti et al. (+2 more)
Autonomously constructing physically realizable 3D structures remains a significant challenge due to combinatorial action spaces, interchangeable components, equifinal assembly sequences, and strict stability requirements during construction. State-of-the-art methods fine-tune large
language models for text-based generative construction. However, these approaches do not allow for
Multimodal (text, image, sketch) conditioning, overlook the practical role of scaffolding for stabilizing intermediate structures, and suffer from slow inference speeds. Therefore, we formulate construction as a probabilistic next-block generation task with multiple potential assembly actions and multiple potential task conditioning modalities. Concurrently, we explicitly consider the utility of scaffolding by introducing an auxiliary scaffold block
token. We present Scaffold
Multimodal Monte Carlo (ScaffoldM3C), a
multimodal, lightweight, auto-regressive model for stable block-based construction, that proposes a set of next-step candidate blocks. Leveraging these candidates, we utilize Sequential Monte Carlo (SMC) to maintain a population of possible assembly sequences, allowing us to consider multiple, potentially different, assembly directions simultaneously. We train our
multimodal architecture by extending the StableText2Brick
dataset to contain image conditioning
prompts and scaffold-stabilized build sequences. ScaffoldM3C is 4x smaller than competing baselines, yielding a 5x to 20x speedup during inference, while achieving comparable construction quality to state-of-the-art methods and higher overall stability. We demonstrate the effectiveness of our approach through simulations and real-world
robot assembly demonstrations.
Read the full abstract ▾
All authors Gadiel Sznaier Camps, Chengyang He, Guillaume Sartoretti, Eduardo Montijano, Mac Schwager
Robotics
Towards a General Humanoid Loco-Manipulation Model via Egocentric Whole-Body Human Data Pretraining
Chongyang Xu, Zhao Wu, Jin Chen et al. (+9 more)
Humanoid whole-body manipulation has advanced rapidly, enabling policies to coordinate locomotion, posture, bimanual interaction, and dexterous hand movements. Meanwhile, egocentric human
videos provide diverse examples of everyday interactions across objects and scenes, offering scalable supervision without
robot operation. However, existing supervision from these
videos provides limited coverage of whole-body movement and coordination with hand-object interaction, while obtaining such supervision through humanoid teleoperation is also costly and difficult to scale. We therefore explore how human experience can support scalable learning of humanoid loco-manipulation. To support this study, we introduce HumanVerse-500, a 500-hour
dataset of diverse human loco-manipulation behaviors in open-world environments, collected with a lightweight wearable system that synchronizes egocentric
video with body and hand motion. Building on this
dataset, we develop $\lambda_0$, a whole-body humanoid
vision-
language-action policy, through three-stage training that first learns interaction from diverse egocentric
datasets, then coordinates body and hand motion using HumanVerse-500, and finally adapts the policy to downstream tasks and
robot embodiments. Across these stages, $\lambda_0$ learns a shared representation space for human experience transfer, while domain-specific interfaces handle differences between human and
robot states and actions. We evaluate $\lambda_0$ on SIMPLE and 4 real-world loco-manipulation tasks, achieving state-of-the-art performance, and further analyze its scaling behavior, generalization, and training-stage contributions to understand how human data support downstream whole-body humanoid control. We will release our code, models, and data to support further research.
Read the full abstract ▾
All authors Chongyang Xu, Zhao Wu, Jin Chen, Yiming Jiang, Jinhui Ye, Yuming Jiang, Shifeng Zhang, Ziliang Feng, Mu Xu, Yilun Chen, Li Lu, Steven C.H. Hoi
Robotics
Whole-Body Aerial Grasping and Lifting via Partial Visual Observations
Jiaye Jin, Rui Jin, Xinhang Xu et al. (+6 more)
Aerial grasp-and-lift tasks require whole-body coordination across approach, acquisition, and lifting under partial target observations. Early approach failures can limit exposure to later task stages during training, while changing visibility complicates
alignment and closure timing during execution. We present a recurrent teacher-student framework that learns a single policy in simulation to jointly command flight, arm motion, and gripper closure without an explicit task-phase input. A privileged teacher learns through reinforcement learning with a critical-state curriculum that exposes acquisition and lifting states before connecting them to normal approach trajectories. Its behavior is distilled into a recurrent visual student that replaces privileged target states with dual-view point clouds and proprioception, integrating observation history for closed-loop control. A dedicated closure objective supervises closure timing from sustained model-defined readiness sequences. Training and primary evaluation use a simulated acquisition-and-payload model with condition-triggered latching, virtual attachment, and wrench-based payload loading for short-distance lifting. Across 8,996 completed simulation episodes under this model, the frozen student achieves full-task success rates of 99.97%, 97.14%, and 95.84% under nominal, physics/control-randomized, and additional camera-randomized conditions, respectively. The nominal latch-count-weighted mean of per-seed 90th-percentile
alignment errors at acquisition is 8.12 mm.
Read the full abstract ▾
All authors Jiaye Jin, Rui Jin, Xinhang Xu, Haotian Jin, Ruiyang Liu, Yi Wang, Jiayan Zhao, Kun Cao, Lihua Xie
Robotics
DeepJEPA: Scaling World Models from Within
Zijian Jin, Yunbei Zhang, Yuanzhe Liu et al. (+5 more)
World-model planners typically scale outward by rolling farther, sampling more trajectories, or optimizing longer, while assigning the same computation to every imagined transition. We show that making every transition uniformly deeper wastes computation and can degrade planning because useful refinement is concentrated at a small set of decision-critical events. We introduce DeepJEPA, a weight-tied joint-
embedding predictive world model that treats transition depth as an inner test-time scaling axis and learns when another recurrent update is worth computing for each candidate and rollout step. Across five visual-control settings, DeepJEPA improves or matches the strongest fixed-depth planner while averaging only 1.00-1.26 updates per transition. Its additional computation concentrates at contact onset and sustained object interaction, where latent corrections can change which candidates enter the planner's elite set and which action is selected. Representation probes further show that improved planning does not require uniformly better object-state decodability. DeepJEPA therefore reframes world-model scaling as a problem of allocating internal computation where it can change the planner's decision: think deeper at decision-critical transitions instead of making every rollout uniformly deeper or longer.
Read the full abstract ▾
All authors Zijian Jin, Yunbei Zhang, Yuanzhe Liu, Ming Liu, Baian Chen, Weirui Ye, Shilong Liu, Marco Pavone
Robotics
DexPolicy: Scheduled Exploration for Trajectory-Guided Dexterous Manipulation
Haoyu Wang, Siyuan Qian, Yanjun Li et al. (+4 more)
Reinforcement learning (RL) for dexterous manipulation must discover finger-object contacts and then control the object precisely; the action noise that serves the first goal can interfere with the second. In trajectory-guided settings such as ViViDex, where RL refine hand-object trajectories from human
video, our baseline PPO runs end near their initial action noise after 5M steps, motivating explicit control of exploration scale. DexPolicy makes that scale an explicit function of training steps, annealing from broad to narrow exploration while holding loss, architecture, reward, and optimizer settings fixed. We study three policy-optimization settings: PPO, critic-free GRPO continuation, and a flow-parameterized PPO variant (FPO). Across five YCB objects and three training seeds, mean deterministic Target success rises from 49.4% to 68.1% (FPO), 14.1% to 45.4% (GRPO), and 32.0% to 35.7% (PPO). On a RealMan RM75 arm with an Inspire/RH56 hand, 360 trials over three objects raise mean Target success from 25.0% to 85.0% (FPO), 10.0% to 63.3% (GRPO), and 8.3% to 43.3% (PPO), with one trained model per object-method condition. PPO component screening favors noise control over the tested optimizer contraction; the selected PPO schedule yields higher mean Target success than linear decay with the same endpoints on three tested objects. Training return, deterministic Target success, and tolerance to execution noise dissociate; schedules should therefore be judged by terminal task success under the intended execution conditions, per task and policy-optimization setting. Code: https://github.com/AIGeeksGroup/DexPolicy. Website: https://aigeeksgroup.github.io/DexPolicy.
Read the full abstract ▾
All authors Haoyu Wang, Siyuan Qian, Yanjun Li, Zeyu Zhang, Yandong Guo, Boxin Shi, Hao Tang
Robotics
IndoorBEV: A Lightweight Real-Time LiDAR BEV Perception System for Indoor Mobile Robots
Haichuan Li
Efficient indoor
LiDAR perception is challenging because mobile
robots must understand cluttered three-dimensional environments under strict latency and
memory constraints. Existing point-based and voxel-based methods often incur substantial computational overhead, whereas conventional bird's-eye-view (BEV) representations improve efficiency at the cost of discarding vertical
geometric information. We present IndoorBEV, a lightweight
LiDAR perception framework that mitigates this tradeoff through a height-aware BEV representation and geometry-conditioned feature fusion. IndoorBEV summarizes the vertical point distribution in each BEV
cell using statistical height features and multi-frequency height encoding, allowing informative three-dimensional cues to be processed efficiently by two-dimensional convolutions. A lightweight encoder then integrates complementary
geometric features with multi-scale local representations and compact global scene context. Decoupled dense prediction heads jointly produce semantic BEV maps and oriented object bounding boxes. IndoorBEV contains only 0.6M parameters and requires 2.3 MB of model storage. On an NVIDIA AGX Orin, it uses 21.52 MB of GPU
memory per inference and achieves a mean latency of 169.6 ms under a 200 ms perception deadline, with a deadline miss ratio of 1.8%. Evaluations on simulated scenes, real-world
robot scans, and an open-source indoor point-cloud
dataset demonstrate a favorable tradeoff among perception accuracy, latency, and
memory consumption. These results indicate that explicitly encoding vertical geometry within a compact BEV representation provides an effective approach to resource-efficient indoor
LiDAR perception.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents
Jiahao Zhang, Yeying Fan, Moitreya Chatterjee et al. (+5 more)
The task of 3D assembly requires translating an understanding of parts and their relationships into precise spatial arrangements. Can pretrained general-purpose
agents assemble objects through visual interaction without additional assembly-specific fine-tuning? To investigate this question, we introduce AssemblyWorld, an interactive 3D environment in which
agents inspect rendered views and manipulate supplied rigid parts, guided by images or assembly manuals when available.
Agents perceive part geometry through 2D views rather than direct access to mesh vertices or faces, while their resulting assemblies are evaluated geometrically. Building on this environment, we construct AssemblyWorldBench, comprising 100 assembly tasks across 80 objects spanning furniture, industrial assembly, and fracture reassembly. Evaluating eight
agent systems reveals substantial differences in their capabilities. The strongest system achieves 80.9% part accuracy but 59.4% complete-assembly success. The evaluated open-source systems lag substantially behind their stronger closed-source peers in both execution reliability and assembly accuracy. Analyses of visual references, interaction trajectories, and failures show how
agents revise assemblies while leaving residual positioning errors. AssemblyWorld provides a common setting for both assessing the capabilities of interactive assembly
agents and characterizing the gap between approximate structure recovery and precise reconstruction.
Read the full abstract ▾
All authors Jiahao Zhang, Yeying Fan, Moitreya Chatterjee, Suhas Lohit, Bernhard Egger, Tim K. Marks, Anoop Cherian, Stephen Gould
Robotics
Ego4WAM: What Matters When Scaling Egocentric Human Data for Robot Learning?
Zhihao Sun, Liu Liu, Xinjiang Wang et al. (+6 more)
Egocentric human data provides a scalable source of experience for
robot learning, but varies substantially in human-
robot alignment, behavioral coverage, and available supervision. Existing work shows favorable scaling with increasing human data, but it remains unclear which data properties drive downstream
robot gains and how to use such data throughout the training pipeline. We present a systematic study of egocentric human data with different
alignment and supervision under a unified world-action model framework. With the model backbone fixed, we disentangle the effects of human-
robot alignment, data duration and task diversity, action supervision, and data usage strategies. We find that aligned human demonstrations substantially improve out-of-distribution generalization and reduce target-task
robot data requirements; data duration and task diversity affect downstream capabilities differently; and
video-only supervision remains effective without action labels, providing a strong foundation for subsequent
video-action training. We validate these findings through closed-loop policy evaluation on both real
robots and RoboDojo. Rather than treating data duration as the sole scaling axis, Ego4WAM shows how
alignment, task diversity, available supervision, and usage strategy jointly shape the value of egocentric human data for
robot learning.
Read the full abstract ▾
All authors Zhihao Sun, Liu Liu, Xinjiang Wang, Haoyi Jiang, Wei Feng, Huiqiang Zhang, Xiaosong Jia, Zhizhong Su, Zuxuan Wu
Robotics
DynaHarness: A Dynamic Physical Harness for Self-Evolving Robot Agents
Haoyuan Deng, Jiebin Liu, Tengxiao Zhang et al. (+3 more)
Pretrained
robot policies provide useful action priors, but long-horizon manipulation still requires coordination between semantic
reasoning and physical execution. Semantic
reasoning operates at a coarser timescale than physical interaction, while episode-level failures provide limited guidance on which system component should be revised. We propose DynaHarness, a dynamic physical harness that couples semantic
reasoning with physical governance through a shared execution contract and turns failure evidence into validated capability revisions. To be more specific, the slow
brain proposes capabilities and symbolic arguments, while the fast
brain grounds and monitors commands, refuses unresolved actions, substitutes capabilities, and requests replans when needed. The physical execution contract bounds each accepted command and records execution evidence across analytic skills, recovery skills, and the frozen VLA. Failure attribution localizes faults in these records and directs targeted revisions of reusable capabilities or execution mechanisms. Paired regression checks govern admission or rejection, closing the self-
evolution loop. On LIBERO-Pro, DynaHarness achieves 75.2% on 800 newly sampled initial states, compared with 17.5% for the frozen policy. With the same capability library, full dynamic execution reaches 74.0% versus 63.9% under nominal one-step replanning. This demonstrates the value of DynaHarness as a dynamic physical harness that governs how existing capabilities are grounded, monitored, and coordinated during execution. Our project page is at https://denghaoyuan123.github.io/Dynaharness_page/.
Read the full abstract ▾
All authors Haoyuan Deng, Jiebin Liu, Tengxiao Zhang, Langning Yan, Hongye Cao, Ziwei Wang
Robotics
GPU-Accelerated Path-Dependent Marginal Information Gain for Autonomous Exploration
João Félix Mendes, Rodrigo Ventura and Meysam Basiri
Autonomous exploration demands that
robots continuously evaluate candidate viewpoints based on their expected information gain and execution cost. Sampling-based planners estimate this gain by volumetric raycasting and, due to its computational cost, evaluate candidates under an assumption of mutual independence, ignoring the overlap between viewpoints along the same path. This work presents a GPU-accelerated method for computing path-dependent marginal information gain, where instead of storing and merging the observed unknown voxels along each candidate path, previous observations are represented using depth buffers. Candidate rays are projected into the depth buffers of their ancestors to identify observation overlap and exclude regions expected to be observed. The planning tree is evaluated in depth order to maintain the dependency between viewpoints and their optimized yaws, while candidate nodes and rays at each level are processed in parallel on the GPU. The proposed method stays within 5-10% of the exact marginal gain computed using voxel hash maps, with speed-ups of up to 118x on a desktop GPU and 28x on an NVIDIA Jetson Orin NX. The method was integrated into two sampling-based exploration planners and evaluated in three simulation environments, where marginal gain reduced the time to 95% coverage in five of the six evaluated planner-environment combinations. Real-world experiments also showed a 30% reduction in the time to 95% coverage, as well as earlier exploration termination times.
Read the full abstract ▾
Robotics · Cross-listed from cs.AI
Belief-Aware Multi-Agent Path Finding under Map Uncertainty
Viraj Parimi, Shao-Hung Chan, Han Zhang et al. (+2 more)
Multi-
Agent Path Finding (MAPF) aims to find collision-free paths for multiple
agents in a shared environment. Classical MAPF assumes that all static obstacles are known in advance, but real-world environments can change unexpectedly due to fallen objects, spills, or other local disturbances. When such changes are spatially correlated, an observation can inform traversability estimates beyond the observed location. Prior approaches address uncertainty in traversability through contingent plans or replanning based on direct observations, but do not leverage this spatial dependence to infer the traversability of nearby unobserved locations. As a result, they cannot use one observation to anticipate nearby unobserved obstacles that may cause costly rerouting later. We focus on Belief-Aware MAPF, where map discrepancies are fixed during execution but initially unknown, and observations can be informative beyond the observed location. We propose Multi-
Agent Gaussian belief Inference for Coordination (MAGIC), a framework that updates a shared belief about traversability online based on
agents' observations. MAGIC uses a Gaussian Markov Random Field and Gaussian Belief Propagation to approximately infer traversability and construct detour-aware costs for standard MAPF planners. Our experiments on MAPF
benchmarks show that MAGIC reduces the executed sum of costs compared to existing approaches on 96.3% of instances, across several planner families and teams of up to 800
agents, demonstrating its applicability to large-scale MAPF problems.
Read the full abstract ▾
All authors Viraj Parimi, Shao-Hung Chan, Han Zhang, Jingkai Chen, Brian Williams
Robotics
STARS: From Spatiotemporal Dynamics to Social Representations in Human-Robot Interaction
Nathan Tsoi, Michael J. Munje, Tejas Oberoi et al. (+5 more)
Robot navigation in dynamic, human-centered environments requires socially-compliant decisions grounded in robust scene understanding. Recent
Vision-
Language Models (VLMs) exhibit promising capabilities such as object recognition, common-sense
reasoning, and contextual understanding, capabilities that align with the nuanced requirements of social
robot navigation. However, it remains unclear whether VLMs can accurately understand complex social navigation scenes (e.g., inferring the spatial-temporal relations among
agents and human intentions), which is essential for safe and socially compliant
robot navigation. While some recent works have explored the use of VLMs in social
robot navigation, no existing work systematically evaluates their ability to meet these necessary conditions. In this paper, we introduce the Social Navigation Scene Understanding
Benchmark (SocialNav-SUB), a Visual Question Answering (VQA)
dataset and
benchmark designed to evaluate VLMs for scene understanding in real-world social
robot navigation scenarios. SocialNav-SUB provides a unified framework for evaluating VLMs against human and rule-based baselines across VQA tasks requiring spatial, spatiotemporal, and social
reasoning in social
robot navigation. Through experiments with state-of-the-art VLMs, we find that while the best-performing VLM achieves an encouraging probability of agreeing with human answers, it still underperforms simpler rule-based approach and human consensus baselines, indicating critical gaps in social scene understanding of current VLMs. Our
benchmark sets the stage for further research on foundation models for social
robot navigation, offering a framework to explore how VLMs can be tailored to meet real-world social
robot navigation needs. An overview of this paper along with the code and data can be found at https://larg.github.io/
stars.
Read the full abstract ▾
All authors Nathan Tsoi, Michael J. Munje, Tejas Oberoi, Rishab Maheshwari, Pengen Zheng, Tanush Chauhan, Peter Stone, Joydeep Biswas
Robotics · Cross-listed from cs.CV
StreamRig: Exploiting Intra-Rig Geometry for Streaming Multi-Camera Odometry
Yufei Wei, Shuhao Ye, Qi Wang et al. (+4 more)
Mobile
robots and vehicles carry synchronized multi-camera rigs, yet many streaming 3D foundation models are designed for monocular input, leaving efficient use of rig geometry a challenge. We present StreamRig, a freeze-and-stream framework that builds causal streaming odometry for calibrated rigs on a frozen multi-view 3D foundation model. The frozen front-end jointly perceives the synchronized views using rig calibration. A Rig-Resampler compresses their features, a CausalBridge applies causal
attention with a key-value cache, and a lightweight head regresses rig poses. A periodic re-anchoring protocol supports stable pose estimation over long sequences. Only these modules are trained, 74.6M parameters in total, with relative poses as the sole supervision. Our two-stage training strategy combines group relocalization pretraining with causal rig training to transfer the
geometric priors of the frozen front-end and the
alignment ability of the pretrained modules to streaming odometry. We evaluate on NCLT, TartanGround, KITTI-360, and our self-collected humanoid-
robot dataset ZJH, where training uses only simulation and real-world evaluation is zero-shot. Across all four
datasets, StreamRig achieves lower translation and rotation drift than the evaluated non-oracle monocular streaming and rig-aware offline models, while maintaining low inference cost. Ablations and controlled camera-count experiments identify the sources of these gains. We further examine how longer training windows affect inference over longer horizons. Code has been released at https://github.com/WeiYuFei0217/StreamRig.
Read the full abstract ▾
All authors Yufei Wei, Shuhao Ye, Qi Wang, Xin Zheng, Qing Huang, Rong Xiong, Yue Wang
Robotics
Centralized Multi-UAV Exploration and 3D Reconstruction Using Single-UAV Planners
João Félix Mendes, Meysam Basiri and Rodrigo Ventura
Extending single Unmanned Aerial Vehicles (
UAVs) exploration methods to multi-
UAV teams can improve coverage speed and robustness, but introduces challenges such as consistent mapping, safe navigation, and deployment strategy. In this work, we present a centralized multi-
UAV exploration framework that enables the use of existing single-
UAV sampling-based planners in a multi-
UAV setting. The proposed architecture allows multiple
UAVs to collaboratively explore unknown environments using a shared global Truncated Signed Distance Field (TSDF) map and centralized planning. Building on the voxblox library, we adapt its mapping pipeline to support real-time fusion of depth measurements from multiple
UAVs into a common TSDF representation. In addition, inter-
UAV collision avoidance and
robot self-filtering mechanisms are integrated into the system to ensure safe navigation and prevent reconstruction of other
UAVs as static obstacles. The framework is evaluated in simulation using four sampling-based exploration planners - RH-NBVP, KRH-NBVP, AEP, and KAEP - whose core sampling logic is preserved, with only system-level adaptations for multi-
UAV operation. Experiments are conducted across multiple environments and under two deployment configurations: Joint Start (JS), where
UAVs are initialized in close proximity, and Separated Start (SS), where
UAVs are initialized in distinct locations. Results show that SS deployments consistently achieve faster exploration and improved coverage across all planners, highlighting the importance of the deployment strategy in multi-
UAV exploration performance.
Read the full abstract ▾
Robotics
Non-Invasive Inspection of Water Canals Using Dronar
Michael Zielinski, Zhizhan Wang, Benjamin Dymond et al. (+2 more)
Open concrete canals play a vital role in water transportation, serving as primary water infrastructure for millions of people across the Phoenix, Arizona, metro area. Over time, the concrete canals can experience a range of issues, including canal lining deformation, cracked concrete, and sediment buildup on the canal floor. Identifying such critical issues is a resource-intensive process, which currently happens only during four-year dry-up cycles. This prevents the maintenance crew from prioritizing operations on the most affected canal segments. To address this issue, the research team has developed and verified an easily deployable and non-invasive method to inspect canal beds without draining the water. This inspection system integrates affordable, off-the-shelf drone and sonar technology (termed dronar). This dronar system includes a consumer-grade sonar system integrated into an unmanned surface vehicle (USV) that carries the sonar transducer just under the surface of the canal water. This paper presents a proof-of-concept demonstration of the dronar system across three field tests on the Arizona Canal in Phoenix. DownScan depth profiles from the sedimented canal segment were consistently shallower than profiles from the same segment after cleaning, with offsets of up to 15 cm observed along the track. Repeated runs over the clean segment produced closely overlapping DownScan depth profiles, confirming that the dronar yields repeatable measurements across the natural variation of the canal bed. These results establish the dronar as a viable proof-of-concept tool for non-invasive canal bed inspection.
Read the full abstract ▾
All authors Michael Zielinski, Zhizhan Wang, Benjamin Dymond, Reza Razavian, Zhongwang Dou
Robotics
Social-WM: Safety-Aware Latent World Models for Robot Social Navigation
Zhihao Zheng and Mooi Choo Chuah
Safe social navigation requires a
robot to anticipate not only the future consequences of its actions, but also whether a nominal action can actually be executed under surrounding physical and social constraints. We present Social-WM, an efficient latent world-model planning framework trained from egocentric RGB
video sequences. Our key observation is that social-navigation experience contains a systematic discrepancy between the nominal action and the realizable action: a nominal forward action may be fully executed in free space, but needs to be constrained when heading towards a pedestrian or obstacle. Social-WM learns these safety-relevant consequences directly through action-conditioned future prediction, where the target is the actual observed future following each command. We further introduce a realizable inverse-dynamics objective that associates observed latent transitions with the action actually realized rather than the nominal one. At deployment, candidate actions are imagined through the latent world model, and the inverse dynamics model estimates their realizability; nominal–realizable discrepancy then provides a safety signal before execution. The learned dynamics and realizability model remain goal-independent and support both position- and image-goal navigation. On Social-HM3D, Social-WM achieves 63.77% success while reducing human collisions to 21.67%, and maintains strong performance under zero-shot transfer to Social-MP3D, without explicit pedestrian tracking, privileged human state, or online reinforcement learning.
Read the full abstract ▾
Robotics
PrefPI: Preference-Guided Steering into Out-of-Distribution Behaviors
Seungeun Rho, Wontaek Kim, Danfei Xu et al. (+1 more)
We present PrefPI (Preference-Guided Policy Iteration), an iterative framework for steering pretrained generative
robot policies using only relative preferences over self-generated trajectories. Unlike prior preference-learning methods that primarily sharpen modes already represented by the policy, we study steering beyond the initial effective support, where desired behaviors are rarely or never observed under the initial policy. Our key idea is to formulate preference learning as preference-conditioned generative modeling: preferred trajectories define a conditional distribution, whose density ratio with the broader behavior prior provides an implicit preference signal amplified by classifier-free guidance (CFG). Repeating this preference-conditioned modeling and guidance step yields a form of preference-guided policy iteration, turning incremental improvements toward previously inaccessible behaviors. Across
diffusion policies and the PI0.5 flow- matching VLA in simulation and the real world, PrefPI produces substantial behavioral shifts with limited feedback. In particular, PrefPI increases object transport height from 10.7 cm to 19.8 cm on real hardware with only 150 preference-labeled trajectories.
Read the full abstract ▾
All authors Seungeun Rho, Wontaek Kim, Danfei Xu, Sehoon Ha
Robotics
Rethinking Legibility in Social Robot Hallway Navigation: Impact of Intent Representation and Human Distraction
Pranav Goyal, Andrew Stratton and Christoforos Mavrogiannis
We focus on legible
robot motion generation in social navigation settings. Legibility in human-
robot interaction (HRI) is often described as the property of
robot motion that enables an observer to confidently infer the
robot's intent. While mature frameworks exist for generating legible motion in front of static observers, social
robot navigation presents a new challenge: the
robot must clearly convey its intent while ensuring human safety in dynamic pedestrian environments where human
attention is often divided. With the goal of enabling
robots to generate legible motion in dynamic and constrained spaces, we investigate how the choice of representation and the level of human
attention shape navigation performance and human impressions. Focusing on the ubiquitous and demanding scenario of hallway navigation, we conduct two controlled user studies involving alternative legibility formulations implemented within a shared model predictive control framework. Study 1 (N = 45) investigates the role of intent representation, showing that passing-side legibility, particularly when adaptively updated, leads to smoother human motion and is perceived as more competent and less mentally and physically demanding than destination-based and non-legible baselines. Study 2 (N = 45) examines the effect of pedestrian
attention, demonstrating that legible motion allows for smooth human motion even under distraction, even if this is not consistently reflected in subjective ratings. Together, these findings suggest that effective legible motion in social
robot navigation benefits from interaction-level intent representations that support coordination, with some effects persisting even when human
attention is divided. Code is available at https://github.com/fluentrobotics/Legible_MPPI.
Read the full abstract ▾
Robotics
Dream4ACT: A Shared Visual Action Interface for Multi-Embodiment Video-Action Modeling
Xiangyu Zhu, Jin Xu, Yue Guo et al. (+6 more)
Video generation models (VGMs) offer strong spatiotemporal priors for embodied observation–action modeling. However, joint-space action vectors lack explicit image-space structure and vary in dimensionality and semantics across embodiments, making it challenging to directly leverage the rich spatiotemporal priors of VGMs. End-effector visualizations provide an alternative but do not specify the full articulated configuration needed for
robot execution. We present Dream4ACT, a world model built for joint
video-action modeling across embodiments. To unify action representations across embodiments, we introduce a shared visual action interface, called action views, which render target joint configurations from four prescribed virtual cameras using URDF-based forward kinematics. This shared visual representation preserves embodiment-specific articulated geometry while allowing observation and action sequences to share a
video autoencoder and
diffusion transformer. Through masked flow-matching, our model supports forward dynamics, inverse dynamics, and joint observation–action generation within a single jointly trained model by varying which future sequences are corrupted. To recover executable action sequences from predicted action views, we propose a training-free, URDF-constrained multiview recovery mechanism, without a learned embodiment-specific decoder. Dream4ACT achieves an average success rate of 88.98% on RoboTwin~2.0 and an overall score of 65.66 on TriWorldBench, supporting effective closed-loop manipulation and competitive action-conditioned multiview prediction through the visual action interface.
Read the full abstract ▾
All authors Xiangyu Zhu, Jin Xu, Yue Guo, Xin Wu, Yifan Sun, Xiancong Ren, Jianxin Sun, Yong Dai, Xiaozhu Ju