Robotics
Memory Without Curation: Learning Multi-Stream Latent Memory for Robot Policies
Yilin Wu, Rutav Shah, Shuaiyi Huang et al. (+5 more)
It is widely appreciated that robotic policies require long-term
memory to perform complex tasks. However, directly conditioning policies on all past observations leaves policy learning susceptible to learning spurious shortcuts, and introduces large latency at policy deployment. As such, today's
memory-augmented policies curate what to remember, either by retaining only recent frames, selected keyframes, or textual summaries, or by learning latent representations of the visual stream alone. In this work, we propose an alternative approach IMPRINT, a policy-agnostic latent
memory framework that augments
vision-
language-action (VLA) models with multi-stream
memory without explicitly constraining what the policy should remember. IMPRINT compresses different streams into a latent
memory that is updated incrementally at inference, enabling low latency. To avoid shortcut learning, IMPRINT adopts a two-stage training procedure, first supervising
memory to reconstruct multiple streams (observations, actions, proprioception) of past information, as well as estimate current task progress, followed by end-to-end finetuning, jointly with the policy, on action prediction. Across both simulation and real-world tasks demanding
memory, IMPRINT improves average success over the base VLA by at least 26%, outperforming popular alternatives, and maintaining lowest inference latency among
memory-augmented VLAs. Through careful ablations, we demonstrate the utility of incorporating diverse information streams into robotic
memory, validating the advantages of foregoing history curation. Project website: https://yilin-wu98.github.io/imprint_website/.
Read the full abstract ▾
All authors Yilin Wu, Rutav Shah, Shuaiyi Huang, Sumedh A. Sontakke, Homanga Bharadhwaj, Andrea Bajcsy, Max Simchowitz, Guanya Shi
Robotics
OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting
Solvin Sigurdson, Prasanna Sriganesh, Juan Borbon et al. (+3 more)
Retargeting human demonstrations into dynamically feasible dexterous hand trajectories requires resolving complex contact physics, yet current methods either omit physical constraints or rely on computationally expensive sampling-based optimization in simulation. Reinforcement learning has previously been used to achieve planning and control through contact dynamics; however, standard policies require object-specific training and fail to generalize across arbitrary geometries. This work reframes physics-based retargeting as a residual learning problem, where we demonstrate that zero-shot generalization across objects relies on structured state representations. Specifically, formulating the observation space to include a localized contact-error signal guiding each
robot link to its target coordinate on the object surface enables a single, unified residual policy to successfully grasp novel, unseen object geometries. Rolling out this policy in simulation on a kinematic reference trajectory yields a physics-based retargeter, OATH. Operating at 0.4 seconds of compute per second of reference, OATH achieves 2.4x faster-than-real-time performance and a 60-350x reduction in computational cost compared to sampling-based methods. The system recovers 58% of held-out and 54% of unseen-object DexYCB trajectories while maintaining a sub-2 cm tracking error.
Read the full abstract ▾
All authors Solvin Sigurdson, Prasanna Sriganesh, Juan Borbon, Joel Burdick, Dingqi Zhang, Jiuguang Wang
Robotics
DexFLEX: Contact-Aware Foundation Controller for Command-Guided Dexterity
Weikang Wan, Zeyu Lai, Jianyu Hou et al. (+4 more)
Dexterous
robot hands can receive useful motion intent from teleoperation or learned policies, but successful execution also depends on contact decisions that these commands rarely specify. We introduce DexFLEX, a contact-aware foundation controller that turns upstream fingertip-motion drafts into contact-consistent joint commands. Instead of treating a draft as a trajectory to copy, DexFLEX treats it as evidence about intent: from the current tactile-proprioceptive state, it proposes feasible short-horizon motion chunks, predicts their contact consequences, and selects the candidate that best follows the command while preserving future contact stability. At inference time, a simple Draft-Dream-Select procedure combines pure-prior proposals for recovery with draft-seeded proposals for responsiveness, then decodes the selected motion into executable joint-space targets. Across degraded-command simulation, real-world shared control, and visuomotor policy learning, the same trained controller improves robustness without retraining, increasing real-world teleoperation success from 29.2% to 78.3% and reaching 46.7% policy-learning success, 3.5 times direct joint-action prediction, and 1.56 times prior-only correction.
Videos are available at dex-flex.github.io.
Read the full abstract ▾
All authors Weikang Wan, Zeyu Lai, Jianyu Hou, Yuchen Zhou, Jiawei Fu, Henrik I Christensen, Hao Su
Robotics · Cross-listed from cs.CV
Learning Transferable World-Action Models from Task-Paired Human Videos
Jeongeun Park, Taekyung Kim, Dongyoon Han et al. (+1 more)
World-action models inherit broad visual and physical priors from
video pre-training. However, standard within-
video next-frame prediction allows them to rely on appearance shared between the context and future, rather than learning interactions that transfer across scenes and embodiments. We introduce WATCH, a cross-
video pre-training that learns from task-paired human demonstrations collected across diverse environments. Conditioned on the
video and actions of one demonstration, the model jointly predicts another demonstration's actions and future frames from its current observation. Predicting one demonstration from another encourages the model to learn task-relevant interactions that transfer across scenes and demonstrators. Our pre-training shows that these capabilities can be learned from readily scalable human-human pairs alone, without requiring matched human-
robot demonstrations. Analyses before
robot post-training show improved motion
alignment across embodiments and better action and
video prediction on unseen
robots, with representations closer to those learned from
robot pairs. After
robot post-training, WATCH improves MimicDroid in-context success from 40.9% to 68.3% with Cosmos Policy and from 59.8% to 77.7% with Cosmos3, with the largest gains on the unseen levels. Gains persist under context-free post-training across
robot embodiments. WATCH outperforms within-
video pre-training and co-training baselines, supporting cross-
video prediction as a scalable approach to learning transferable world-action representations.
Read the full abstract ▾
All authors Jeongeun Park, Taekyung Kim, Dongyoon Han, Sangdoo Yun
Robotics · Cross-listed from cs.LG
Continuous-Time Cloth Dynamics with Physics-Informed Neural ODEs
Yingdong Ru, Lipeng Zhuang, Zhuo He et al. (+1 more)
We present ContinuousCloth, a continuous-time, physics-informed neural solver for deformable cloth dynamics that enforces a spring–mass ODE within a neural trajectory model. Unlike classical time-stepping simulators that return trajectories only on a discrete integration grid, ContinuousCloth supports querying at arbitrary timestamps and computing displacement, velocity, and acceleration via automatic differentiation. We evaluate ContinuousCloth along two dimensions. In a physics-only setting, ContinuousCloth achieves trajectory accuracy comparable to high-precision solvers across six diverse boundary-condition scenarios while reducing runtime by up to 17.4 times. When fused with sparse and noisy observations, the spring–mass ODE acts as a strong inductive bias for downstream tasks: interpolation and parameter estimation. That is, ContinuousCloth reduces interpolation error by 7.3 times over a data-only neural baseline in simulation and 2.4 times on real-world cloth sequences, and achieves 2.6 times lower parameter error than a differentiable-solver baseline under high-noise conditions, demonstrating consistent robustness under realistic measurement imperfections.
Read the full abstract ▾
All authors Yingdong Ru, Lipeng Zhuang, Zhuo He, Gerardo Aragon Camarasa
Robotics
CoPlanVLM: Coordinated Path Planning for a Robot Team using Vision Language Models
David R. Martin, Jared Paquet, Mohammad Abdullah Al Faruque et al. (+4 more)
We present CoPlanVLM, which uses a
Vision Language Model (VLM) to translate a single natural-
language instruction and image into coordinated paths for a team of two
robots. While VLMs offer a flexible, natural-
language interface for commanding
robot teams, their weak spatial
reasoning has greatly limited VLM-based path planning. CoPlanVLM improves the VLM's spatial
reasoning and path planning abilities through three components: (1) an image overlay marking both free space and obstacles, (2) task classification that selects the most appropriate low-level planner and
reasoning prompt for each instruction, and (3) task-specific chain-of-thought
reasoning that
prompts the VLM to identify key objects, allocate roles, and verify routes are collision-free before committing to a plan. Together, these components enable the team to complete complex coordinated maneuvers requiring both
robots and divide large tasks between
robots to increase efficiency. CoPlanVLM also supports detailed maneuvers involving specific routes, loops, and coverage of large areas, as well as tasks requiring directional
reasoning such as north/south or left/right. We validate CoPlanVLM in both Gazebo simulation and on physical Turtlebot 4 hardware. CoPlanVLM attains an average success rate of 84.4% compared to 50.0% for the closest baseline. On trials where both methods succeed, CoPlanVLM's paths are 27% shorter on average. Project code is available at https://github.com/AICPS/CoPlanVLM.
Read the full abstract ▾
All authors David R. Martin, Jared Paquet, Mohammad Abdullah Al Faruque, Pramod Khargonekar, Kyle Volle, J. Humberto Ramos, Scott Nivison
Robotics
PILOT: Guiding Long-Horizon World-Model Planning for Robot Manipulation with Visual Subgoals and Policy Priors
Yang Liu, Hanchen Cui and Karthik Desingh
Latent world-model planning has shown promising results in goal-conditioned robotic manipulation, but remains challenging for long-horizon planning due to limited temporal guidance and inefficient action search. A distant visual goal provides little guidance for preparatory actions, while uninformed action sampling makes local optimization inefficient. We propose PILOT (Policy-Initialized Local Optimization with Temporal subgoals), which uses a
video model to generate intermediate visual subgoals and a learned policy to initialize the cross-
entropy method (CEM). CEM refines candidate action sequences toward each subgoal using an action-conditioned world model in a receding-horizon control loop. In our primary real-
robot evaluation across four UR5e manipulation tasks, PILOT achieves 68.8% success, compared with 55.0% without policy initialization, 13.8% for policy-only execution, and 0.0% for direct final-goal planning. Simulation results show that these gains persist over longer horizons: PILOT exceeds direct execution of the same policy by 30 percentage points on long Push-T tasks and by 22 percentage points on LIBERO-10. Policy initialization also accelerates local optimization: on Push-T, CEM matches the cost reached by generic initialization after 10 iterations in only 4 iterations and reduces the final planning cost by 24.1% under the same sampling budget. These results show that combining temporal subgoals with policy-guided action search improves both long-horizon task execution and local planning efficiency.
Videos are available on the webpage: https://pilot-planner.github.io/.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
Learning to See through Action: Predictive Curiosity for Robots
Stelios Zarifis, Panagiotis Filntisis, George Retsinas et al. (+1 more)
Humans and animals do not learn
vision from a fixed set of data: they move, interact, and learn from the visual consequences of their actions. Motivated by this developmental view, we study whether
robots can improve visual learning by actively collecting experience through curiosity-driven interaction. We introduce DySCOVER, a self-supervised framework for active
robot visual learning. A
robot observes RGB frames, learns a visual autoencoder that encodes and reconstructs them, and trains a recurrent world model to predict the next visual latent state conditioned on the current state and action. The
agent's policy is trained only with intrinsic rewards: reconstruction error encourages visually novel observations, while the world model's error encourages transitions whose consequences are not yet predictable. This prediction error biases the policy toward more intricate trajectories that enrich the
agent's set of experiences with observations beyond those encouraged by visual novelty alone. Our experiments conducted with robotic arms as the embodied
agents show that adding predictive curiosity improves the
vision model over using visual curiosity alone. Across variations in initial conditions, experience capacity, embodiment, and scene content, adding predictive curiosity consistently improved visual learning. These results show that predictive curiosity can improve active visual learning beyond reconstruction-based curiosity alone.
Read the full abstract ▾
All authors Stelios Zarifis, Panagiotis Filntisis, George Retsinas, Petros Maragos
Robotics · Cross-listed from cs.CV
RoboFake: Are Deepfake Detectors Ready for Humanoid Face Robots?
Hao Chen and Junnan Xu
Humanoid face
robots can now move their lips in sync with
speech and show detailed facial expressions. Give such a
robot the face of a real person and drive it with generated
speech, and you get a face forgery that happens in front of the camera: the camera records a real object, and no pixel is edited. We call this a RoboFake. In this paper we ask what current deepfake detectors do when they see a
robot face. We test six public detectors trained on FaceForensics++ on
video of a physical lip-syncing face
robot, on a synthesized
video of the same
robot, and on rendered
robot faces. All thresholds are fixed on FaceForensics++, and every result is compared with real human faces that went through the same processing as the
robot videos. At a threshold that flags 5% of real faces, three detectors flag almost every physical
robot frame, while two flag fewer than 7%. A matched human control and masking show that the response comes from the face, not from the processing or the background. However, it does not track synthesis: with only the face visible, three of the six detectors give the physical
robot a higher fake score than its synthesized
video. The response is also fragile, and light JPEG compression is enough to push most
robot frames below the threshold for some detectors. Our results suggest that current detectors flag today's
robots because their faces do not look human, not because they recognize a forgery. We release our evaluation protocol and code, and state three testable hypotheses about realistic, interactive RoboFakes.
Read the full abstract ▾
Robotics
Akule: Fast and Scalable Multi-Robot Motion Planning via Sparse Interaction Diffusion
Manav Vora, Gokul Puthumanaillam, Aditya Penumarti et al. (+2 more)
Diffusion models provide expressive trajectory priors for multi-
robot motion planning (MRMP), but scaling them to large teams requires coordinating individual motions without expensive joint computation. We introduce Akule, a sparse interaction
diffusion framework that augments a frozen single-
robot prior with three learned modules: a reusable pairwise correction model $R_{\phi}$, a contextual selector $U_{\psi}$ that determines which corrections to evaluate, and a composer $G_{\omega}$ that combines their effects through context-dependent signed weights. This decomposition separates which interactions merit computation from how their effects should be combined into coordinated denoising updates. We train $R_{\phi}$ using matched pairwise trajectory optimizations, then learn $G_{\omega}$ and $U_{\psi}$ from physically optimized joint trajectories without requiring a prescribed ground-truth interaction graph. The resulting joint proposals are passed to a downstream search-based planner for validation and repair. Across diverse MRMP
benchmarks, Akule achieves competitive planning success, complete-planner speedups of up to $25\times$, zero-shot population transfer, and successful planning with up to $256$
robots. Ablations further demonstrate that learned selection and composition reduce pre-repair conflicts and downstream repair effort relative to heuristic selection and uniform summation.
Read the full abstract ▾
All authors Manav Vora, Gokul Puthumanaillam, Aditya Penumarti, Jane Shin, Melkior Ornik
Robotics
From LLM Plans to Authorized Motion: Execution-Bound Authorization for Excavators
Mehdi Heydari Shahna, Seihun Kim, Soyi Jung et al. (+3 more)
An LLM-directed excavator can execute mechanically feasible motion whose destination or authority no longer matches the operator's approval. We propose execution-bound authorization for the approval–dispatch–execution interval. A one-use lease binds the action, generated motion, mission contract and admission tolerance; atomic dispatch rechecks current authority and measured joint drift. Periodic supervision detects revocation and requests a quintic stop with a carry-pitch endpoint, selected through sampled joint, stroke, flow and reference-acceleration checks. The framework connects exact motion identity to dispatch integrity and load-aware interruption. Evaluation combines a 672-case interface suite, four loaded digging cases in synchronous simulation, unloaded MuJoCo/Pinocchio transfers and two command-level LLM planners. The gateway admitted all 120 benign interface cases and produced zero unauthorized dispatches in the attack suite. Loaded dispatch-window tests produced zero unauthorized deposition. During simulated dump-phase revocation, maximum closed-site deposition decreased from 1.23 to 0.33\,m$^3$; loaded-slew stops used less than 75% of nominal swing torque. These results demonstrate motion-bound authorization and quantify the retention and stopping behavior of supervised excavation skills.
Read the full abstract ▾
All authors Mehdi Heydari Shahna, Seihun Kim, Soyi Jung, Soohyun Park, Jouni Mattila, Joongheon Kim
Robotics
Composable Dynamics for Cooperative Humanoids: A RoboCompiler Extension
Mehdi Heydari Shahna and Jouni Mattila
Physically connected humanoids need consistent dynamics for internal actuator loops, inter-
robot forces, and changing ground supports. We extend RoboCompiler with member-wise dynamics assembly, compiling reduced dynamics from rigid bodies, moving actuator parts, and reflected rotor inertias. Compiled body Jacobians and analytic constraint
curvature yield an explicit Coriolis factor satisfying the power identity for floating-base quasi-velocities. A two-level reduction retains reusable
robot models while composing their supports and hand connections; local inertial updates have rank at most six, and prescribed-acceleration constraint wrenches are affine in actuator efforts. These interfaces support actuator-aware tracking, coupled force allocation, payload updates, and support-mode switching for five hand-linked humanoids. Across 264 feasible states in eight humanoid configurations, normalised acceleration discrepancies remain within $1.6\!\times\!10^{-10}$ against a hard-constraint reference and $1.4\!\times\!10^{-10}$ against a compiler-independent reference; MuJoCo simulations achieve 0.73~mm root-mean-square (RMS) pelvis error in the ring dance and complete side steps through 20 support-mode changes.
Read the full abstract ▾
Robotics
Mapping Tire-Road Friction Coefficients Along Driven Trajectories
Risto Ojala
Information of the tire-road friction coefficient is critical for intelligent vehicle systems, as friction establishes the limits at which the vehicle can be manoeuvred without loss of control. While there is extensive literature on online friction estimation, no previous studies have explored post-processing and mapping friction coefficients along a driven trajectory based on recorded sensor data. Mapped friction information could be shared by connected vehicles, enabling proactive measures in low-friction areas. This paper presents an offline post-processing method, FrictionMap, for mapping friction coefficients from recorded driving data. The method operates on commonly available on-board vehicle sensor information, and requires only a basic vehicle parameter set and a slip threshold as prior knowledge. FrictionMap runs an extended Kalman filter combined with a Rauch-Tung-Striebel smoother to estimate vehicle dynamics. Data points where the friction limit is observable are identified by filtering points based on tire slip. A Gaussian process is fitted to these points to acquire a continuous estimate of the friction coefficient along the trajectory. FrictionMap is validated via a simulation study, and shown to outperform commonly used online friction estimation techniques. The code implementation and the simulation
benchmark are released as open-source after paper acceptance to enable further research.
Read the full abstract ▾
Robotics
ClothCode: Generative Modeling for Garment State Estimation and Dynamics Prediction
Yingdong Ru, Lipeng Zhuang, Shiyu Fan et al. (+3 more)
We introduce ClothCode, a unified generative framework for garment state estimation and action-conditioned dynamics prediction in a shared discrete configuration space. A material-aligned tokenizer encodes garment configuration as residual vector-quantized patch
tokens, while task-specific Masked Generative
Transformers predict these
tokens through parallel confidence-guided refinement. Across three garment templates, ClothCode reduces mean displacement error by 26.6% for state estimation and 37.0% for dynamics prediction relative to the strongest baselines. Dynamics prediction requires only 16.7 ms per window, compared with 556.2 ms for
diffusion. Integrated into the same model-predictive controller, ClothCode improves real-world folding and flattening success from 56.7% to 78.9%. Further results and
videos are available at: https://anonymous.4open.science/r/ClothCode-DE32
Read the full abstract ▾
All authors Yingdong Ru, Lipeng Zhuang, Shiyu Fan, Zhuo He, Florent P. Audonnet, Gerardo Aragon Camarasa
Robotics
Observation-Referenced Tissue Deformation Forecasting with Tool-Driven and Residual Dynamics
Tom Olesch, Nan Xiao, Xiongye Xiao et al. (+2 more)
Forecasting tissue deformation from future instrument motion could support predictive planning and increasingly autonomous surgical systems. We study this problem using a scene-specific differentiable surface model fitted to an observed tool-tissue interaction window. Rather than forecasting absolute simulated positions, we introduce an observation-referenced formulation that anchors each tissue point to its last valid observation and adds only the deformation induced by the known, preplanned future tool trajectory. This reduces sensitivity to reconstruction, initialization, and persistent simulation offsets while preserving the predicted deformation response. The physical model uses differentiable, mesh-free position-based dynamics with scene-specific rigidity, attachment, region scale, and grasp coupling. We further introduce an optional observation-referenced residual model combining graph-spectral spatial modes with periodic harmonic dynamics for structured motion not explained by the tool input. We evaluate the framework retrospectively on 12 surgical manipulation clips from three
datasets. Observation referencing provides the largest gain, with a median within-clip error ratio of $0.598$ relative to the corresponding absolute simulation forecast. The optional periodic residual further improves forecasting when the remaining motion exhibits periodic structure. These results support observation-referenced deformation prediction as a practical building block for future surgical world models, digital twins, and generative simulators that must predict how tissue evolves under instrument actions.
Read the full abstract ▾
All authors Tom Olesch, Nan Xiao, Xiongye Xiao, Jonathan C. DeLong, Fei Liu
Robotics
Disentangling Task-Relevant Latent Dynamics for Reward Modeling
Dayou Li, Jionghao Wang, Peihao Wang et al. (+9 more)
Reward design remains a major bottleneck for reinforcement learning in robotic manipulation. Handcrafted dense rewards can compensate for the limited guidance provided by sparse rewards, but require substantial task-specific reward engineering. Learning rewards from
videos provides a scalable, data-driven alternative to manual reward design. However, existing
video-based rewards are often computed in entangled visual spaces where task-relevant state changes are confounded with extraneous visual variation. As a result, the learned reward may respond to visual distractors instead of actual task execution. To address this limitation, we propose DiTaR (Disentangling Task-Relevant Latent Dynamics for Reward Modeling), a framework for learning dense rewards from heterogeneous
videos. DiTaR learns a continuous task-relevant representation from multi-frame
video latents through an asymmetric two-stage procedure that suppresses task-irrelevant visual variation. In the task-relevant latent space, we train a history- and
language-conditioned rectified-flow model to capture task dynamics and derive a directional dense reward from temporal changes in dynamics compatibility. Together, these components enable reward modeling directly over task-relevant dynamics rather than generic visual variation. DiTaR substantially improves policy learning on MetaWorld-v3 and ManiSkill over
video-based reward baselines, and transfers to real-world offline RL through reward relabeling. Reward analyses further show that DiTaR provides more reliable rewards.
Read the full abstract ▾
All authors Dayou Li, Jionghao Wang, Peihao Wang, Jiuzhou Lei, Hao Wang, Jichen Chai, Hezhen Hu, Minghui Zheng, Zhangyang Wang, Xin Li, Wenping Wang, Zhiwen Fan
Robotics
Axiom-0: Hierarchical Self-Refinement for Vision-Language-Action Policies
Renbin Shu, Jing Yang, Xiangyuan You et al. (+7 more)
Reliable manipulation requires policies that generalize across tasks, layouts, and object interactions.
Vision-
language-action (VLA) models combine pretrained
multimodal understanding with learned control, but their performance remains limited on unfamiliar task compositions. We present Axiom-0, a VLA policy that addresses both the quality of its representations and their use in action generation through hierarchical self-refinement. Intra-VLM Representation Refinement (IVRR) strengthens selected intermediate features through consistency supervision with learned feature projections during training. VLM-Action Interface Refinement (VAIR) recursively updates the resulting conditioning before action generation. Joint optimization of the supervised VLM layers, VAIR, and the action generator connects the two components. On RoboCasa365, Axiom-0 achieves 62.32% overall success in our evaluation, exceeding the highest published leaderboard score (Paimon-0, 58.1%) by 4.22 percentage points and the Xiaomi-Robotics-1 reference by 4.92 points. Under the evaluated profiling setup, mean action-chunk generation latency increases by 1.7% over Xiaomi-Robotics-1. These results demonstrate improved task completion with modest inference overhead.
Read the full abstract ▾
All authors Renbin Shu, Jing Yang, Xiangyuan You, Guolin Qiu, Yufei Xia, Yiming Zhu, Nanjun Dai, Fei Zhang, Feng Wu, Keke Tang
Robotics
Refine Where It Matters: Contact-Adaptive Meshing for Deformable Object Manipulation
Gabriel Johnson, Tom Olesch and Fei Liu
Deformable-object manipulation requires accurate modeling of highly localized contact while maintaining simulation speeds suitable for interactive use. Uniformly fine tetrahedral meshes can resolve these interactions, but they introduce unnecessary degrees of freedom throughout the object. Coarse meshes are more efficient, yet they may under-resolve contact-induced deformation. We present a contact-adaptive mesh refinement framework that introduces spatial resolution only where it is most useful. The method uses
geometric, elastic, and contact quantities already available from the deformable-body solver to score candidate edges. A fully GPU-parallel procedure then selects a conflict-free subset and updates both mesh
topology and physical state directly within the simulation loop. We evaluate the approach on real deformable-object manipulation sequences. Adaptive refinement reduces simulation-to-real error, improves agreement with a fine-mesh reference, and remains substantially more efficient than uniformly fine simulation. These results show that local refinement can improve physical fidelity without incurring the cost of a globally fine discretization. The framework provides a practical basis for contact-rich simulation in interactive manipulation, sim-to-real modeling, and online physical digital twins.
Read the full abstract ▾
Robotics
Make {\pi}0.5 Learn to Call Astra for Reliable Robot Manipulation
Jianli Sun, Chengxiang Li and Yonglin Tian
GPT-6 Astra provides robust embodied
reasoning, but frequent external queries increase inference time and
token costs. We equip a frozen $\pi_{0.5}$ policy with a lightweight CALL head that learns to predict when assistance is necessary. CALL reuses features from the standard policy forward pass and operates locally without consulting external models. Astra is activated solely for critical recovery events and returns control to $\pi_{0.5}$ upon stabilization. Evaluated across 40 LIBERO tasks (200 episodes), our system achieves a 97.5% success rate, outperforming the 95.5% baseline of pure $\pi_{0.5}$. In the Object, Goal, and Spatial suites, Astra executes only 5.49% of environment steps, consuming approximately 5,027
tokens per test episode. This selective intervention preserves the fast ordinary execution of $\pi_{0.5}$ while seamlessly integrating Astra's
reasoning on demand. Our project page is available at https://kykgeek.github.io/Pi05_CALL_Astra/.
Read the full abstract ▾
Robotics
Splat2Act: From Captured Scenes to Personalized VLA Manipulation
Xiaomeng Ye, Krishna Jaganathan, Gyubeum Lim et al. (+3 more)
A user who brings a generalist manipulation policy into their home or office wants it to pick up their objects from their shelves, counters, and tables. Yet even a strong generalist like $\pi_{0.5}$ achieves 0% pickup success in our captured deployment scenes without adaptation, and collecting real demonstrations in every user's environment is prohibitively expensive. We present Splat2Act, an automated real-to-sim-to-real pipeline that personalizes a pretrained
vision-
language-action (VLA) policy to a specific scene and object set using only a handheld phone scan and images of the user's objects. Unlike prior real-to-sim work that reconstructs individual tabletops, Splat2Act operates on building-scale indoor environments: it builds a photorealistic digital twin with 3D Gaussian Splatting and collision geometry, automatically discovers tabletop and non-tabletop receptacles, constructs feasible pickup tasks, and generates successful demonstrations with a privileged simulation oracle, with only scene and object capture required from a human. Applied to 101 scanned campus environments with a median navigable area of 104.87 m$^2$ and 7 receptacles, the pipeline yields 852 receptacles and over 68,000 pickup demonstrations. We fine-tune $\pi_{0.5}$ in two stages: Stage 1 adapts to hundreds of public object assets, drawn either from the deployment scene alone or from all 101 scenes; Stage 2 personalizes to the user's own objects on the deployment scene's receptacles. Our strongest personalized policy reaches 62.5% aggregate pickup success in the target scene. Deployed on a real Stretch 3, the same policy achieves 63.9% success on a 12-task subset, closely tracking its 66.7% simulated success on those tasks. These results suggest that automated real-to-sim-to-real training can personalize VLA manipulation policies using only simulation-oracle data.
Read the full abstract ▾
All authors Xiaomeng Ye, Krishna Jaganathan, Gyubeum Lim, Gunjan Chhablani, Muhammad Zubair Irshad, Zsolt Kira
Robotics
Noise Space Learning for VLA Adaptation
Junjie Lu, Chuheng Zhang, Yucen Wang et al. (+4 more)
Pretrained
vision-
language-action (VLA) models encode broad manipulation priors, yet adapting them to specific tasks requires balancing limited
robot interaction, computational resources, and safety constraints. Noise space learning (NSL) adapts flow-matching-based VLAs by learning a context-conditioned distribution over noise while keeping the pretrained VLA fixed. Building on prior noise-steering methods, we formulate NSL as an interface for VLA adaptation that accommodates different learning
algorithms and instantiate it with PPO, SAC, and AWR. Theoretically, we provide an information-theoretic analysis of how NSL enables efficient adaptation. Empirically, we study these NSL
algorithms on LIBERO, RoboTwin, and four real-
robot manipulation tasks under interaction, computation, and execution constraints. Across these settings, we develop a suite of effective NSL variants and derive practical recommendations for choosing among them under different resource and deployment regimes.
Read the full abstract ▾
All authors Junjie Lu, Chuheng Zhang, Yucen Wang, Yuhua Jiang, Xinyao Qin, Min Xu, Li Zhao
Robotics
Chronos Policy: Learning Robot Policies on Physical Time
Changhao Wang, Jaden Clark and Shuran Song
Robot policies often convert sensing and control into fixed-rate observation-action sequences, leaving physical time implicit in sequence position. This assumption breaks when observation rates or delays vary, or when inference latency changes. We introduce Chronos Policy, which makes physical time explicit by conditioning on observation capture times and desired action execution times. We study two ways to incorporate time into a
diffusion transformer: adding timestamp
embeddings to
token features, or encoding physical time in positional encoding through time-aware rotary positional encodings (time-aware RoPE). We find that time-aware RoPE is more robust, allowing
attention to depend directly on physical time while keeping
token content unchanged. During training, we temporally randomize which observations are sampled and when the action chunk begins, exposing the policy to a wider range of sensing and execution schedules that may occur during deployments. This also enables learning from asynchronous, multi-rate data without forcing all measurements onto a fixed temporal grid. Across simulation and real-world experiments, Chronos Policy improves performance on dynamic tasks, where observation-to-action timing is critical, and on multisensory tasks, where the relative timing between observations matters. On standard quasi-static tasks, it preserves nominal performance while improving robustness to changes in sensing and inference timing.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Beyond the Snowflakes: Disentangling Weather Effects from Confounding Meta-Factors in LiDAR Object Detection
Yiming Yang and Jeremy P. Bos
Light Detection and Ranging (
LiDAR) is critical for autonomous perception, yet its performance is known to degrade in adverse weather. While existing literature models snow as atmospheric noise that degrades performance, our granular evaluation on the WADS-3D and CADC
datasets reveals a counter-intuitive phenomenon: detection accuracy consistently improves as snow intensity increases. Through ablation study and meta-information analysis, we reconcile this by confirming that while falling snow acts as noise, its negative impact is outweighed by a dominant, concurrent factor: significantly increased object point count. We hypothesize that this object point count boost acts as a
geometric enhancement, driven by snow accumulation increasing vehicle surface area and a shift from specular to diffuse reflectance. Our findings demonstrate that evaluating perception systems using broad weather-severity bins without accounting for correlated object-level data can lead to misleading conclusions about detector robustness. We conclude that accurate evaluation requires decoupling atmospheric noise from surface accumulation effects and outline future research to validate these physical models. The WADS-3D
dataset is available at: https://digitalcommons.mtu.edu/wads-3d/.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Tracking the Invisible: Self-Supervised Wheel Track Detection with Thermal-RGB Fusion
Yiming Yang and Jeremy P. Bos
Accurate detection of drivable area is essential for autonomous driving and advanced driver assistance systems. While state-of-the-art methods focus on general traversability estimation, they prove inadequate for snow-covered roads where following existing wheel tracks offers crucial advantages: improved traction, predictable vehicle dynamics, and verified path stability. To address this, we propose a low-cost, self-supervised approach detecting safety-critical wheel tracks using synchronized RGB-I (RGB + Long-Wave Infrared) imagery. Our method automatically generates training labels by projecting future wheel trajectories from recorded GNSS data. A neural network is designed to identify wheel tracks with the 4-channel RGB-I images from challenging, white-out road surfaces. Experimental results demonstrate the effectiveness of our approach in visually monotonous and featureless adverse environment, offering a robust and cost-effective alternative for winter autonomy. The newly introduced ThermalTrack
dataset is available at: https://digitalcommons.mtu.edu/thermaltrack/.
Read the full abstract ▾
Robotics
BT-Agent: Deploying Onboard Robot Agents with Behavior Trees and Multi-Fidelity Evaluation
Fabio Hübel, Adam Lilja, Yue Yin et al. (+5 more)
Deploying
language-model
agents onboard mobile
robots requires task
reasoning and responsive execution within limited compute and
memory. We propose BT-
Agent, an onboard architecture combining LLM-driven task decisions with predefined behavior-tree (BT) skills. These skills enforce execution conditions and return structured failure feedback. We demonstrate fully onboard operation with a 9B model on a quadruped across five task categories. Intervention tests show that local checks stop ongoing navigation without waiting for another model decision, while the
agent remains active to retry or report constraints. To compare
agent designs efficiently, we introduce a 161-task multi-fidelity
benchmark, evaluated in text on all tasks and in Habitat on 39. Text supports 10.9$\times$ higher evaluation throughput, while simulation reveals how perception and motion failures change task outcomes. BT-
Agent's largest success gains over the strongest Act/ReAct baseline occur at 4B and 9B: approximately 10 and 5 percentage points. Ablations identify structured tool calls and informative failure feedback as the most useful components.
Read the full abstract ▾
All authors Fabio Hübel, Adam Lilja, Yue Yin, Siming He, Claire Tomlin, Shankar Sastry, Jonas Frey, Marco Pavone