Robotics · Cross-listed from cs.LG
Beyond Policy Support: Interaction Constrained Offline Reinforcement Learning for Autonomous Driving
Mahmoud Selim, Cristina Cipriani and Karl Henrik Johansson
Offline reinforcement learning enables reward-driven policy improvement from fixed
datasets without requiring online exploration, making it particularly attractive in safety-critical domains. A central challenge, however, is distribution shift: policy optimization may favor actions that are weakly supported by the offline data, rendering value estimates unreliable. Existing approaches primarily control this shift in the policy's own action space. In interactive environments such as autonomous driving, this can be insufficient: a candidate ego trajectory may remain well supported under the marginal behavior distribution while being poorly supported jointly with the surrounding-
agent behavior observed in the logged interaction. We refer to this degradation in interaction support as interaction distribution shift (IDS), and introduce Interaction-Constrained Drive Policy (ICDP), an offline reinforcement learning framework that explicitly controls interaction-level distribution shift. Starting from the joint data distribution over ego and surrounding-
agent futures, we show that joint-support degradation decomposes exactly into an ego-support component and a residual interaction-support component. We recover the latter through contrastive density-ratio estimation, isolating interaction compatibility without explicit joint-density modeling, surrounding-
agent prediction, or rollouts in reactive simulators or learned world models during policy optimization. Closed-loop evaluations on nuPlan, Interplan and real-world truck experiments show that ICDP suppresses high-value yet interaction-unsupported trajectory selections and improves performance in interaction-critical driving scenarios. Project webpage: https://mahmoud-selim.github.io/ICDP/
Read the full abstract ▾
Robotics
{\Delta}WAM: Distilling Action Tangent Fields into World Action Models
Ke Wu, Hanwen Huang, Bo Gu et al. (+6 more)
World Action Models (WAM) improve
robot policies by augmenting sparse action supervision with dense future prediction. However, much of the predictable future is dominated by appearance and scene persistence rather than action-dependent dynamics. We observe that several recent WAM designs, including optical flow, motion-centric representations, and latent actions, can be understood from a common perspective in which world supervision becomes more efficient as it contains a higher proportion of action-relevant variation. Based on this insight, we introduce Action Tangent Fields, which reformulate world supervision through a local Taylor expansion of how actions induce changes in future dynamics. We represent future dynamics in Residual-VAE space, where the future latent remains recoverable from the current latent and its residual, and use a strong action-conditioned world model (ACWM) to probe the local correspondence between action variations and residual-world variations. This local first-order structure is distilled into the WAM to guide its denoising supervision toward dynamics that are more tightly coupled to action, rather than merely predictable from appearance. Across LIBERO-Plus, RoboTwin, and RoboTwin2.0-Plus, our method consistently improves robustness to lighting, background, camera, layout, and other environmental perturbations. Despite using no large-scale embodied pretraining, it achieves stronger robustness under several distribution shifts than pretrained policies. We further distill multi-step VideoDiT denoising into a single step for efficient inference. Our results suggest that effective WAM supervision should remain information-rich while concentrating its predictive capacity on the directions along which actions change the future.
Read the full abstract ▾
All authors Ke Wu, Hanwen Huang, Bo Gu, Kaizhao Zhang, Xiangting Meng, Yupeng Zheng, Zijun Xu, Jieru Zhao, Wenchao Ding
Robotics
YUBI-STAG: Contact and Semantic-Rich Alignment for VLAs via Automated Video-Language Grounding
Masatoshi Tateno, Takehiko Ohkawa, Yueh-Hua Wu et al. (+4 more)
Vision-
Language-Action (VLA) models acquire broad manipulation capabilities via large-scale pretraining, yet eliciting them through
language requires fine-grained
alignment between instructions and physical interactions. Existing
robot demonstrations typically provide only coarse task descriptions, omitting how actions are executed, including which gripper acts, which object is contacted, and how it is grasped and moved. We introduce YUBI-STAG, a framework for Spatio-Temporal Annotation and Grounding that automatically enriches manipulation demonstrations with interaction-rich semantics to align pretrained VLAs with fine-grained manipulation
language. Combining contact-object segmentation with
vision-
language models, YUBI-STAG annotates object identities, attributes and states, per-gripper actions, bimanual coordination, and spatially grounded interactions. To address YUBI-STAG's reliance on localized sequences and multi-stage VLM inference, we distill it into YUBI-VLM. YUBI-VLM directly recovers action structure and annotations from raw, unsegmented
video in few inference calls and operates from wrist views alone. We evaluate both frameworks on YUBI-STAG-Bench across temporal, semantic, and spatial grounding tasks. YUBI-VLM retains much of YUBI-STAG's annotation accuracy with fewer inference calls and shorter runtime while generalizing to unseen manipulations. Finally, post-training VLA policies on these annotations aligns them with fine-grained
language and contact-aware structure. Bimanual experiments demonstrate improved performance and instruction following, including control over object identity, acting gripper, target location, and spatial relations absent from original labels.
Read the full abstract ▾
All authors Masatoshi Tateno, Takehiko Ohkawa, Yueh-Hua Wu, Hanlong Li, Tatsuya Matsushima, Yoichi Sato, Kei Ota
Robotics
RoboPace: Contact-Aware Time-Optimal Retiming for Action-Chunk Policies
Mimo Shirasaka, Takehiko Ohkawa, Takuya Okubo et al. (+3 more)
Robot manipulation data collection has been shifting from teleoperation toward
robot-free demonstrations, through interfaces such as the Universal Manipulation Interface (UMI) or directly from human hands.
Vision-
Language-Action (VLA) policies trained on such data inherit the demonstrator's timing. Yet human timing does not directly transfer to
robots: compliant hands tolerate fast contact, whereas
robots may overshoot due to actuator and tracking limitations; conversely,
robots can move faster in free space. This motivates a unified approach that reconciles execution speed with contact safety. We present RoboPace, an online retiming layer that preserves the policy's
geometric path while adapting its timing, respecting the target
robot's kinematic and dynamic constraints. It adapts execution speed based on predicted contact, jointly accounting for contact-dependent speed limits and the
robot's motion constraints. The method requires no policy retraining and operates in real time. Across three contact-rich tasks on a dual-arm
robot, faster uniform execution and physical-limit-only retiming largely fail. RoboPace instead achieves higher overall success than slow uniform execution while completing four of five commands in approximately half the time, retaining the reliability of slow execution without its time cost.
Read the full abstract ▾
All authors Mimo Shirasaka, Takehiko Ohkawa, Takuya Okubo, Nicola Scianca, Tatsuya Matsushima, Kei Ota
Robotics
Do Better Visual Representations Always Lead to Better End-to-End Autonomous Driving?
Zihao Zhang, Haochen Tian, Tianyu Li et al. (+5 more)
Visual foundation models (VFMs) are increasingly integrated into end-to-end autonomous driving for their powerful representations, yet it remains unclear when these representations improve driving performance. To investigate this question, we introduce ViRA, a planner-agnostic visual representation
alignment framework that keeps the planner architecture and inference cost unchanged. Our study reveals three findings: (1) VFM-guided visual representations consistently improve driving performance across diverse end-to-end planners, with gains extending to zero-shot closed-loop evaluation. (2) The choice of VFM target matters for planning performance, and
alignment to a different VFM can further benefit planners with pre-trained VFM encoders. (3) Auxiliary perception supervision reduces sensitivity to VFM target selection, narrowing the EPDMS spread across five targets from 2.7 to 0.5 points and potentially compensating for less effective VFM targets. Guided by these findings, we develop ViRA-
Diffusion, a
diffusion-based planner trained without auxiliary perception supervision, which achieves 92.3 EPDMS on NAVSIM v2 navtest, outperforming recent methods in our comparison by at least 1.9 points. The results motivate jointly considering target selection and planner supervision when integrating VFMs into end-to-end autonomous driving. The results and demo are available at https://github.com/OpenDriveLab/ViRA.
Read the full abstract ▾
All authors Zihao Zhang, Haochen Tian, Tianyu Li, Changhui Jing, Jingliang He, Naisheng Ye, Ziyuan Pu, Zhenjie Yang
Robotics
NAViLoss: An Underwater Navigation-Aware Dual-Residual Objective for Physics-Consistent Learning
Arup Kumar Sahoo and Itzik Klein
Autonomous underwater vehicles (AUVs) commonly rely on inertial navigation systems (INS) aided by Doppler velocity logs (DVLs) for reliable underwater navigation. Accurate DVL velocity estimation is therefore essential for successful operation. Recent learning-based methods have demonstrated improved DVL velocity estimation, particularly under degraded measurement conditions. However, their training objectives typically rely on conventional regression losses that are highly sensitive to large residuals and corrupted observations. Additionally, they do not explicitly account for the physical consistency and measurement uncertainty associated with the underlying sensing process. To address these limitations, this paper introduces navigation-aware loss (NAViLoss), a robust and uncertainty-aware objective function for learning-based AUV velocity estimation. NAViLoss jointly penalizes the velocity-estimation residual in the navigation-state domain and the beam-consistency residual in the DVL measurement domain. Its bounded formulation limits the influence of large residuals, while an adaptive mechanism regulates the uncertainty in beam geometry. Furthermore, NAViLoss is integrated with a DeepONet architecture to form a novel NAVi-DeepONet model for seamless estimation of an underwater vehicle's velocity. Lastly, our model is evaluated using approximately 10,000m of semi-synthetic AUV experimental data collected during multiple real-world sea trials. Experimental results demonstrate a 44% improvement in velocity-estimation accuracy compared with conventional and learning-based baselines. These results demonstrate the effectiveness of navigation-aware and uncertainty-adaptive loss design for robust learning-based underwater velocity estimation.
Read the full abstract ▾
Robotics · Cross-listed from cs.AI
MeshSIPP: Efficient Lattice Planning in Dynamic Environment
Marat Agranovskiy and Konstantin Yakovlev
Autonomous navigation in dynamic environments requires computing spatiotemporal trajectories that satisfy non-holonomic motion constraints. When the trajectories of the moving obstacles are predictable or known, a promising approach is to rely on the combination of state
lattices constructed from precomputed feasible motion primitives and Safe Interval Path Planning – a search-based
algorithm with strong theoretical guarantees. While this approach yields feasible paths, the rich primitive sets needed for smooth navigation induce a large branching factor, which becomes costly when coupled with time-dependent obstacle intervals. To this end, we present MeshSIPP, an efficient planner that removes the computational bottleneck by exploiting the fact that many primitives sweep the same regions and can therefore be validated together. MeshSIPP propagates primitives as spatial bundles, screens them with lightweight bounding-interval checks, and defers the expensive exact departure-time search until a primitive reaches its terminal state. A time-aware
pruning rule additionally discards redundant space-time branches early in the search. We prove that the resulting search is complete and optimal. Extensive experiments over more than 6,000
benchmark instances and real-time ROS~2 simulations show that MeshSIPP achieves up to a 3$\times$ speedup over state-of-the-art spatiotemporal planners.
Read the full abstract ▾
Robotics
Design optimization of tendon-driven robots considering tendon wrapping and shortcut
Yuta Sahara, Kento Kawaharazuka and Kei Okada
This study proposes a multi-objective optimization method for tendon-driven arm design, addressing the inherent trade-off between joint torque and arm thickness through tendon wrapping and shortcut effects. We formulate the problem to simultaneously maximize torque and minimize thickness, solved using NSGA-II. The
algorithm optimizes tendon routing, pulley configurations, and attachment points. Results reveal Pareto-optimal designs with effective moment arm expansion via strategic shortcuts, providing practical guidelines for compact high-torque arms. The optimized configurations demonstrate non-trivial patterns beyond conventional design intuition, offering new insights for engineering efficient robotic systems.
Read the full abstract ▾
Robotics
Fast and Robust Teach-and-Repeat Navigation Using MixVPR Visual Place Recognition*
Václav Truhlařík, Tomáš Pivoňka and Libor Přeučil
Teach-and-repeat navigation systems employing advanced visual place recognition techniques for localization exhibit key attributes for long-term mobile
robot navigation, such as the ability to operate in unstructured and dynamic environments. However, existing solutions based on deep-learning techniques are computationally demanding, limiting their applicability. This work introduces a novel and efficient teach-and-repeat system built on the modern visual place recognition method MixVPR. Real-world testing demonstrated its ability to operate both indoors and outdoors, achieving robustness and navigation precision comparable to other state-of-the-art systems. In addition, its lower hardware requirements make it suitable for a wide range of robotic platforms and practical applications.
Read the full abstract ▾
Robotics · Cross-listed from cs.AI
Learning Situation-Conditioned Thinking Policies for Long-Term LLM Agents
Hong Su
Long-running autonomous
agents must reuse accumulated
reasoning experience without allowing explicit historical
memory and LLM context to grow indefinitely. However, existing
memory mechanisms mainly retrieve, summarize, or compress past content and do not directly learn when particular kinds of thinking should be activated or discover new thinking knowledge from temporally dispersed experiences. This paper proposes a situation-conditioned thinking
memory framework that transforms historical
reasoning experience into a lightweight policy for predicting what should be thought about in the current situation, while leaving detailed
reasoning to a large
language model. Situations may represent temporal or spatiotemporal
evolution rather than only current states. Temporary experiences are also periodically analyzed across multiple independent episodes to identify repeated long-range regularities, which are consolidated into new thinking knowledge and further internalized by the lightweight policy. Experiments show that the learned policy achieves 1.000 F1 on temporal-rule generalization, improves DeepSeek
reasoning F1 from 0.789 to 0.868, reduces online processing time from 0.3636 ms to 0.0382 ms per query at 30,000 historical situations, and reaches 1.000 relation-discovery F1 and future-thinking accuracy after sufficient repeated cross-experience evidence.
Read the full abstract ▾
Robotics
Adaptive Code Generation for Controlling Robots
Justus Flerlage, Thorsten Wittkopp, Alexander Acker et al. (+1 more)
Deploying
robots as Complex Adaptive Systems (CAS) in unknown and dynamic environments necessitates a transition from rigid command libraries toward intention-based autonomy, as natural
language represents the only medium capable of articulating complex goals beyond the capacity of finite instruction sets. While Large
Language Models (LLMs) offer a path toward natural
language goal description, their integration introduces significant challenges: the formalization gap between imprecise intentions and executable actions, the taxonomy gap induced by unpredictable environments, and the challenge of maintaining temporal state and progress awareness. This work introduces an architectural framework that enables robotic control by leveraging generative AI. The system follows a dual-AI design: an LLM translates high-level intentions into executable program code restricted to a formal robotic library and constrained by verifiable syntax, while a
Vision-
Language Model (VLM) provides semantic grounding via a
distillation process. To ensure robustness, the framework incorporates environment-driven replanning triggers based on
geometric and semantic thresholds, complemented by continuous runtime monitoring and an adaptive planning loop. Benchmarked across frontier models, our framework architecture demonstrates that grounding generative AI in a reactive, constrained loop enables robust fulfillment of complex intentions in dynamic and unknown environments.
Read the full abstract ▾
All authors Justus Flerlage, Thorsten Wittkopp, Alexander Acker, Odej Kao
Robotics
Point It, Strike It: Direction-Conditioned Dynamic Manipulation of Deformable Linear Objects
Yi Yang, Xiang Fei, Lehong Wang et al. (+9 more)
Goal-conditioned dynamic manipulation of deformable linear objects has mainly specified goals as positions for a rope tip to reach. Many tasks, however, depend on how the tip arrives. We therefore study single-swing rope striking with goals that specify the tip's 3D position and arrival direction, across the workspace and on different ropes. This is challenging because rope dynamics are hard to model, no demonstrations exist, distinct swings reach the same goal with different reliability, and the sim-to-real gap extends beyond the rope. To address these challenges, we extend the state-of-the-art DLO simulator DeformX with GPU acceleration, a stable Cosserat rod solver, and a cross-flow aerodynamic model, yielding DeformX2.0, which is more than $20{,}000\times$ faster. We then propose TRACE (Trace-rooted Adaptive Cross-
Entropy), which generates striking data by warm-starting each new target from the stored swing whose tip path passes closest to it. Its cost penalizes rope bending and abrupt tip motion to favor repeatable swings. A conditional flow-matching policy trained on this data reaches 92.1% accuracy in simulation. Finally, we propose RECAP (Residual Calibration Policy), which fits the simulator's rope and rig parameters to a few calibration swings and adapts actions with a correction policy trained in simulation. On a real
robot, across three ropes, RECAP raises success within 5cm from 72% to 87% for position goals, and within 10cm and 10{\deg} from 50% to 79% for goals that also specify the arrival direction.
Read the full abstract ▾
All authors Yi Yang, Xiang Fei, Lehong Wang, Zilin Dai, Ruogu Li, Jiting Cai, Liyao Chang, Xinyi Yang, Henry Kou, Ruijie Fu, Lu Li, Howie Choset
Robotics
Targeted Modality Dropout for Real-Robot Manipulation Robust to Intermittent Vision Loss
Genki Shikada, Kazuki Osamura, Masaru Ide et al. (+2 more)
Imitation learning policies that integrate multiple sensory modalities are prone to overreliance on a dominant modality, such as
vision, during training, which can disrupt policy execution when that modality is lost at inference time. In this paper, we introduce Targeted Modality Dropout (TMD), in which the dependence on each modality is estimated using
attention and the most dominant modality is selectively dropped. This is combined with
entropy regularization over the dependence distribution. Through real-
robot evaluation using a bimanual manipulator, we show that under
vision loss the success rate of the baseline policy drops substantially, whereas TMD sustains task execution. In contrast, a conventional dropout that selects the dropped modality at random, without the
entropy regularization, fails on many tasks even without
vision loss.
Read the full abstract ▾
All authors Genki Shikada, Kazuki Osamura, Masaru Ide, Tetsuya Ogata, Kanata Suzuki
Robotics
MagCilia: A Compact Magnetociliary Tactile Sensor with 3D Force Sensing for Robotic Contact Perception and Grasping Feedback
Yu Feng, Hao Wu, Haotian Guo et al. (+7 more)
Robotic grasping and surface exploration benefit from simultaneous measurement of normal and tangential forces and from surface information obtained through contact. Here, we present a compact magnetociliary tactile sensor (MagCilia) that combines a flexible
magnetic-cilia structure with a Hall sensor for 3D force sensing. Quasi-static finite element analysis is used to investigate structural deformation and
magnetic responses under multidirectional loading. To reconstruct forces from the coupled
magnetic channels, we propose causal history fusion regression (CHFR), which combines current
magnetic-field measurements with their recent changes. Five-fold cross-validation grouped by calibration record yields root-mean-square errors of 0.40, 0.57, and 0.69 N for Fx, Fy, and Fz, respectively, with corresponding coefficients of determination of 0.93, 0.90, and 0.92. Robotic experiments demonstrate tangential-force-guided gripper adjustment and multi-axis load monitoring under external perturbations. Frequency-domain features of the reconstructed forces distinguish six surface categories with 99.39% accuracy in three-fold cross-validation grouped by acquisition session. An online robotic demonstration additionally identifies all six tested surfaces. These results demonstrate 3D force reconstruction, grasping feedback, and surface recognition using a single compact tactile unit.
Read the full abstract ▾
All authors Yu Feng, Hao Wu, Haotian Guo, Haoming Liu, William Su, Jingxiang Guo, Jiankun Li, Masayoshi Tomizuka, Wen Jung Li, Jianshu Zhou
Robotics · Cross-listed from cs.CV
WAPR: A Foundation Model for Wide-Angle Refinement in Unseen Object Pose Estimation
Yulin Wang, Mengting Hu, Hongli Li et al. (+2 more)
Real-world applications require 6D pose estimation to be accurate, fast, and scalable to unseen objects. This paper introduces WAPR, a zero-shot wide-angle pose refinement model that refines candidate poses with rotational deviations up to 90 degrees. With as few as 12 candidate poses per detected object instance, WAPR supports fast inference within 1 s per frame and reaches a pose-estimation throughput of up to 25 detected object instances per second. To support wide-angle training for rotationally symmetric objects, WAPR uses rotational
symmetry priors to canonicalize
symmetry-equivalent pose targets before loss computation. We further construct SA6D, a large-scale 6D training
dataset with such priors. SA6D obtains KASAL-assisted rotational
symmetry priors for 944 GSO scans and expands them through geometry and texture augmentation into about 50K augmented object instances and about 2M rendered RGB-D images. In addition, an angle-balanced loss stabilizes learning across different angular ranges by reducing the influence of uninformative large-error cases. Experiments on seven BOP core
datasets show that WAPR achieves state-of-the-art performance in unseen-object 6D pose localization and detection under both fast and unconstrained inference settings. Project page: https://github.com/WangYuLin-SEU/WAPR.
Read the full abstract ▾
All authors Yulin Wang, Mengting Hu, Hongli Li, Jianghao Zhou, Chen Luo
Robotics
Contact-Aware Imitation Learning Through Contact Factorization
Jiho Hong, Daeun Song, Sanghyun Kim et al. (+1 more)
Generalizable contact-rich manipulation requires
robots to preserve intended task behavior while adapting its physical realization to changing contact conditions. However, interaction forces can vary substantially with small changes in surface geometry, orientation, and friction, making policies trained directly on raw force measurements difficult to transfer beyond demonstrated conditions. We introduce FACE, a contact-factorized imitation learning framework that separates intended task behavior from environment-dependent contact factors. Our representation expresses interaction forces in normalized, contact-relative coordinates, while a learned contact-normal
estimator and an online friction
estimator infer the local contact normal and effective friction scale. Together, these
estimators enable force observations to be encoded and policy outputs to be decoded into physical motion and force commands during execution. In this way, FACE adapts execution to current contact conditions while preserving the intended task behavior, without updating the policy parameters. We evaluate FACE on real-
robot contact-rich manipulation under unseen variations in surface properties and geometry, demonstrating robust generalization across contact conditions through controlled comparisons with variants that adapt prior approaches to our setting.
Videos and additional materials can be found on the project page: https://rcilab.khu.ac.kr/face.
Read the full abstract ▾
All authors Jiho Hong, Daeun Song, Sanghyun Kim, Mingyo Seo
Robotics
Not All Uncertainty Matters: Simulation-in-the-Loop Fast-Slow Reasoning for Decision-Critical Autonomous Driving System
Jiayi Chen, Shuai Wang, Guangxu Zhu et al. (+3 more)
Large
vision-
language models (VLMs) provide powerful open-world perception and
reasoning for autonomous driving, but their high computational cost and inference latency make continuous cloud-side use impractical. This motivates fast–slow collaboration, where efficient onboard modules handle real-time perception and control while cloud models provide high-level
reasoning only when needed. The key challenge is deciding when cloud
reasoning should influence time-critical driving decisions. Existing methods often rely on perception uncertainty, heuristic triggers, or resource-driven policies, without assessing whether resolving an uncertainty will improve planning. We propose SIGMA, a simulation-in-the-loop framework for task-oriented fast–slow collaboration. SIGMA embeds the planner into uncertainty assessment and evaluates how plausible scene realizations under semantic and
geometric uncertainty affect feasible trajectories and planning cost. Based on these outcomes, it estimates the expected reduction in planning cost from resolving uncertainty. We further introduce expected planning gain (EPG), a decision-level metric for cloud invocation, cloud-guidance integration, and request prioritization under deadline and resource constraints. Experiments in CARLA show that SIGMA reduces unnecessary cloud interactions while improving planning, efficiency, and navigation success in static and dynamic obstacle scenarios. Compared with fixed-period collaboration, SIGMA reduces unnecessary cloud interactions by 50%, improves navigation success by more than 6%, and cuts finish time by up to 26.2% in dynamic scenarios.
Read the full abstract ▾
All authors Jiayi Chen, Shuai Wang, Guangxu Zhu, Derrick Wing Kwan Ng, Chengzhong Xu, Kaibin Huang
Robotics · Cross-listed from cs.CV
ActiveLang: Active Open-Vocabulary 3D Mapping with Semantic-Uncertainty-Guided Exploration
Liyan Chen, Hairong Yin, Huangying Zhan et al. (+3 more)
As
robots increasingly assist humans with diverse tasks, they need both
geometric and semantic understanding of their surroundings. Moreover,
robots often operate in unfamiliar environments and take on new tasks without knowing the relevant concepts ahead of time. This motivates
language-annotated 3D maps that support open-vocabulary scene understanding and human-
robot interaction. We introduce ActiveLang, an autonomous system for active open-vocabulary 3D mapping with semantic-uncertainty-guided exploration. ActiveLang performs online
language-feature adaptation on a compact dual-Gaussian representation to jointly reconstruct scene geometry, appearance, and open-vocabulary semantics with modest
memory overhead. Its planner efficiently selects informative viewpoints, enabling effective mapping with fewer observations and lower computational cost. Experiments on Replica and ScanNet++ demonstrate substantial improvements in 2D and 3D open-vocabulary segmentation over both online and offline baselines, highlighting that actively exploring scenes builds
language-annotated 3D maps more efficiently.
Read the full abstract ▾
All authors Liyan Chen, Hairong Yin, Huangying Zhan, Yi Xu, Raymond A. Yeh, Philippos Mordohai
Robotics · Cross-listed from cs.CV
TERRA: Learning Transportable Latent Actions through Temporal Effect Representation and Relational Alignment
Tianxingjian Ding, Mubarak Shah and Yu Tian
Latent actions supervise
robot policies with action-like codes inferred from visual transitions, and their usefulness hinges on two questions: what a code keeps from a transition, and whether it still means the same thing when reused in a different initial state. The first is a tension in time: an endpoint difference discards how motion unfolds, while the full sequence admits nuisance variation. The second is left open by reconstruction, which only ever observes a latent together with the state it came from. We argue that both questions can be answered in the same place. TERRA (Temporal Effect Representation and Relational
Alignment) describes a transition by a compact temporal effect, its net feature change together with a low-order within-window dynamics component, and learns a continuous latent from this effect. The same effect space then serves as the reference for reuse: Effect-Anchored Transport (EAT) decodes a latent in other initial states and anchors the resulting effect to the one observed at its source, so that the latent is shaped by what it does across contexts rather than only by the transition it came from. With frozen linear readers, TERRA predicts actions more accurately than UniVLA and a LAPA-style baseline, degrades more slowly under visual distractors, and keeps transported transitions faithful to the donor action as the recipient context moves farther away; a same-budget control shows that these gains come largely from EAT. At matched pretraining scale, the complete system reaches 93.4% average success on LIBERO, compared with 91.8% for UniVLA.
Read the full abstract ▾
Robotics
Sparse Feature Policy Unlearning Mitigates State Hallucination in Vision-Language-Action Models
Jiho Lee, Jeongeun Park, Heayoun Choi et al. (+2 more)
Vision-
Language-Action (VLA) models have shown strong generalization in robotic manipulation by leveraging rich representations from pretrained
vision-
language models. However, their deployment in real-world environments remains limited by recurring unreliable behaviors. In this work, we study state
hallucination, a recurring failure pattern in which a VLA continues acting as if an unrealized
robot-object state had been achieved. Our analyses find that state
hallucination coincides with weakened
attention to task-relevant visual regions, and a mechanistic interpretation via sparse autoencoders reveals that
hallucination-associated sparse features are activated when these failures occur. Based on this analysis, we propose SOUL (Sparse feature pOlicy UnLearning), which selectively unlearns policy knowledge associated with state
hallucination behaviors, where sparse features identified from
hallucination failures and successful behaviors serve as explicit forgetting and retention targets, respectively. Experiments across VLA architectures in simulated and real-world environments show that our method substantially reduces hallucinated failures and improves task success without substantially compromising the existing manipulation capabilities. These results suggest that interpretable feature analysis provides a practical basis for selectively modifying undesirable knowledge in
robot policies.
Read the full abstract ▾
All authors Jiho Lee, Jeongeun Park, Heayoun Choi, Taekyung Kim, Eunwoo Kim
Robotics
SiGNgapore - An Interactive Dataset for Sign-based Visual Navigation
Nicky Zimmerman, Joel Loo, Zishuo Wang et al. (+1 more)
Robots navigating open-world, human-centered environments must look beyond prebuilt high-fidelity maps and leverage navigational aids embedded in their surroundings, particularly, signs. To advance sign-guided visual navigation, we introduce SiGNgapore, an interactive,
multimodal dataset designed to
benchmark sign-guided multi-step navigational decision-making. SiGNgapore covers diverse public spaces in Singapore, including shopping malls, public transportation hubs, and hospitals, featuring multi-floor, multi-building layouts and indoor-outdoor transitions. It consists of 56 long-horizon navigation missions and 456 scenes, each containing a navigational sign and its surrounding environment. We provide synchronized RGB, depth, IMU, and odometry data for each scene. We additionally provide the layout map and the GPS-aligned scene graph for selected mission sites. Beyond navigational decision-making, SiGNgapore supports a variety of spatial-semantic
reasoning tasks, including navigational sign parsing and visual sign grounding, with the ultimate aim of driving progress in mapless
robot navigation.
Read the full abstract ▾
All authors Nicky Zimmerman, Joel Loo, Zishuo Wang, David Hsu
Robotics
Precise SE(3) End-Effector Tracking in Whole-Body Humanoid Control
Joohwan Seo, Xiaofeng Guo, Jinkun Cao et al. (+4 more)
Precise end-effector tracking during humanoid whole-body motion is challenging due to floating-base oscillations,
gravity, dynamic coupling, and locomotion-induced disturbances. We propose ResGAC, a whole-body humanoid controller for precise end-effector pose tracking that combines
geometric admittance control (GAC) with residual reinforcement learning. GAC provides structured $\SE$ task-space feedback and generates nominal arm joint-position targets, while residual RL compensates for unmodeled dynamics and coordinates locomotion and balance in the shared joint-position action space. The left-invariant
geometric formulation allows the same GAC law to be used across manipulation reference frames. This enables the use of a ground-attached heading frame that preserves planar locomotion while removing pelvis roll, pitch, and heave from the manipulation reference, thereby reducing reference-induced end-effector motion during locomotion. ResGAC is validated on a real Unitree G1 humanoid. Across four standing end-effector tracking
benchmarks, ResGAC consistently outperforms representative baselines, including SONIC, achieving lower translational and rotational errors. Real-world experiments further demonstrate reduced propagation of pelvis motion to the desired end-effector pose using the proposed ground-attached heading frame. ResGAC achieves $90\%$ success in a standing peg-in-hole task compared with $50\%$ for SONIC, and accurate world-frame $\SE$ end-effector pose tracking during lower-body motion. Experimental
videos are included in the supplementary material and are also available on the project website: https://resgac.github.io/ResGAC-website/.
Read the full abstract ▾
All authors Joohwan Seo, Xiaofeng Guo, Jinkun Cao, Roberto Horowitz, Rocky Duan, Guanya Shi, Koushil Sreenath
Robotics
TMT: Runtime Backdoor Detection for Vision-Language-Action Policies on Unseen Tasks
Zirun Zhou, Jingfeng Zhang, HaoChuan Xu et al. (+4 more)
Backdoored
vision-
language-action (VLA) policies can preserve benign task performance while producing malicious actions when a trigger appears. Detecting such activation is difficult because malicious behavior can comprise individually plausible actions, while unfamiliar tasks introduce legitimate changes in observations and behavior. We introduce TMT, a runtime backdoor detector based on
Token Manifold and latent Transition modeling. Trained on benign rollouts, its two branches assess input-
token structure and prediction errors in adjacent-layer latent dynamics. A suspicious rollout identified by the
token manifold branch, once confirmed through latent deviations, guides transition selection for subsequent monitoring. We further explore policy purification through self-
distillation: a frozen copy of the backdoored policy provides benign-input actions to supervise a student on paired benign and triggered observations, without requiring a separate clean reference policy. For evaluation, we adapt traditional backdoor detectors and repurpose anomaly and failure detection methods as VLA backdoor detectors. In a post-hoc comparison with ten baselines, TMT achieves state-of-the-art backdoor detection performance on unseen tasks across three VLA backdoor attacks. Our project page is available at https://zzr42.github.io/tmt/.
Read the full abstract ▾
All authors Zirun Zhou, Jingfeng Zhang, HaoChuan Xu, Xizhe Zhang, Elliott Wen, Jing Sun, Hong Jia
Robotics · Cross-listed from cs.LG
DSReg: Provably Recovering Individual World Latents without Reconstruction
Yujia Zheng, David Klindt, Randall Balestriero et al. (+1 more)
Methods that recover individual latent variables of the world, from nonlinear ICA to dictionary learning and causal representation learning, anchor the latents to observations through reconstruction, auxiliary supervision, or distributional asymmetries such as non-Gaussianity. Methods without these anchors, including joint-
embedding predictive architectures (JEPAs), identify the latent state only up to a linear transformation, so individual latents remain mixed. We close this gap: individual world latents can be provably recovered with no reconstruction, no decoder, and no labels. The key condition is Structural Diversity: different latents leave distinct dependency footprints on observations, just as no two snowflakes are alike. Building on the linear identifiability that LeJEPA provides, we prove that under Structural Diversity, DSReg (Dependency-Sparsity Regularization) recovers individual world latents up to signed permutation, without reconstruction or a decoder. It applies post hoc to any linearly identified representation, reusing trained checkpoints at no loss over joint training, and establishes the first fully identifiable JEPA that recovers every world latent. Moreover, as a condition on dependency footprints, Structural Diversity is strictly weaker than all structural conditions of prior identifiable latent variable models. Across synthetic regimes, world model probes, learned visual encoders, and external renderers, DSReg preserves dense prediction while improving individual-latent recovery and downstream use with scales.
Read the full abstract ▾
All authors Yujia Zheng, David Klindt, Randall Balestriero, Bernhard Schölkopf
Robotics · Cross-listed from cs.CV
RLHND: Video Foundation Models as Physically Grounded Hand Trackers for Robot Learning
Seungjun Moon, Subin Jeon, Sangwoo Kim et al. (+2 more)
Recently, approaches that leverage human
video datasets for
robot policy training have become increasingly prevalent. However, most existing hand trackers regress pose from cropped frames with limited priors on hand motion and object interaction, resulting in inaccurate and physically inconsistent estimates. Moreover, the lack of physical cues, e.g., contact and force, limits the use of human
videos for
robot policy training. To this end, we propose RLHND, a
video foundation model-based hand tracking model that jointly estimates hand pose and realistic tactile information from monocular egocentric
videos. RLHND turns the pre-trained Cosmos 3
video diffusion backbone into a deterministic clip-level feature extractor via clean-latent conditioning, carrying its learned priors on hand motion and hand-object interaction into tracking. For pose estimation, RLHND (i) predicts hand poses with anatomically plausible joint angles and (ii) enables optional conditioning on the shape parameter to maintain consistent hand shape within the same
video and even across
videos recorded by the same actor. For tactile estimation, a separate tactile expert stream, trained with the pose stream frozen, predicts dense contact and force over the hand surface. We further adopt LBS-based feature spreading to enable vertex-wise feature extraction without costly per-vertex
attention. RLHND achieves state-of-the-art performance across various
benchmark datasets for pose estimation, while also achieving state-of-the-art performance in contact and force estimation. Moreover, we demonstrate the utility of RLHND for
robot learning through retargeting results and real-world
robot experiments. The code will be publicly available at https://seungjun-moon.github.io/rlhnd/.
Read the full abstract ▾
All authors Seungjun Moon, Subin Jeon, Sangwoo Kim, Hanbyul Joo, Jinwoo Shin
Robotics
RobotAPO: Adversarial Physics Preference Optimization for Robotic Manipulation Video Generation
Kerui Li, Zhe Jing, Chenyi Huang et al. (+4 more)
Robotic manipulation
videos are increasingly used as visual plans for embodied
agents, but optimizing purely for visual plausibility often fails to capture the fragile physical
manifold of real-world interactions. Even minor physics-violating errors at the interaction boundary, such as interpenetration or premature object motion, can completely invalidate the inferred timing and pose needed for downstream execution. Because standard supervised fine-tuning lacks the direct pressure to penalize these localized failures, we introduce AgiBot-PhysPref. This rigorously curated 10,000-sample preference
dataset isolates condition-matched physics violations, turning the generator's own failure distribution into a foundational signal for physical consistency. Building upon this, we propose RobotAPO, an
adversarial physics preference optimization framework operating in the continuous flow-matching denoising space. To prevent the policy from merely memorizing static curated failures, RobotAPO employs a lightweight
adversarial counterfactual proposer that learns a condition-dependent, physical-failure-biased direction in denoising space. This encourages the model to explore and better respect the physical interaction boundary, all while maintaining a pure
prompt-and-reference inference interface without requiring external structural conditioning. Comprehensive evaluations demonstrate that explicitly correcting these localized physics violations improves downstream
robot execution from generated
videos. On held-out AgiBot conditions, RobotAPO outperforms the strongest controlled internal baseline in physical consistency by 6.8% hard score and 10.0% soft score. Crucially, in real-
robot replay, it translates these physical-consistency gains into a 37.4% relative improvement in task success over the strongest controlled internal baseline.
Read the full abstract ▾
All authors Kerui Li, Zhe Jing, Chenyi Huang, Xiaofeng Wang, Zheng Zhu, Haoming Cui, Huaibo Huang
Robotics
TempoBridge: Language-Guided Tempo Control for Vision-Language-Action Policies
Yeonseo Lee, Hyosup Shin, Guebin Hwang et al. (+1 more)
Vision-
Language-Action (VLA) models are effective at understanding what task to perform, but provide limited control over how it should be executed, such as moving quickly or slowly. We introduce TempoBridge, a lightweight framework that uses frozen VLA representations to modulate actions according to tempo cues in the instruction at each task phase, without additional tempo-conditioned
robot demonstrations or tempo-specific base-policy fine-tuning. TempoBridge extracts tempo cues from contextual VLM representations, aligns them with task progress through a causal phase router, and modulates nominal motion commands during execution. Across LIBERO tasks, TempoBridge improves Tempo Success Rate from 52.6% to 89.7% under canonical tempo instructions while retaining high task success. It also preserves near-baseline performance when no tempo cue is present and generalizes to unseen tempo expressions without additional training. Experiments on a physical
robot further demonstrate
language-conditioned tempo modulation in real-world manipulation.
Read the full abstract ▾
All authors Yeonseo Lee, Hyosup Shin, Guebin Hwang, Sungho Jo
Robotics · Cross-listed from cs.CV
Event-Aligned Visual Action Reasoning for World Action Models
Xiaomeng Yang, Yushu Wu, Yi Gao et al. (+4 more)
World-Action Models (WAMs) utilize future visual prediction as an intermediate
reasoning process to guide action generation. However, existing WAMs typically structure visual imagination according to predefined temporal intervals, without explicitly accounting for the different roles of task-critical interactions and connecting transitions. We argue that effective visual foresight should align directly with task-relevant interactions and their corresponding
reasoning demands. To this end, we introduce an event-aligned visual action
reasoning framework that organizes visual-action prediction around interaction events. Through event-aligned visual-action supervision, WAM learns to generate event-aligned visual context in each imagined rollout, placing greater emphasis on critical state changes that inform action generation. This shapes the visual
reasoning granularity according to the underlying interaction dynamics, with detailed
reasoning around task-critical events and coarser progression through connecting transitions. Furthermore, we introduce an execution validity head that identifies the valid portion of each predicted action sequence, avoiding redundant actions during chunked inference. Experiments demonstrate a 10.26 percentage point improvement in DOMINO success rate over baseline and competitive performance on RoboTwin 2.0. It also transfers from DOMINO Level 1 to Levels 2 and 3 without target-level adaptation.
Read the full abstract ▾
All authors Xiaomeng Yang, Yushu Wu, Yi Gao, Yuhao Lei, Xuan Zhang, Pu Zhao, Yanzhi Wang
Robotics
IVG-UAV: An Intelligent Voice-Guided UAV System for Autonomous Ripe Fruit Harvesting with Vision-Based Classification and Adaptive Path Planning
Dinh Trung Duong
In tropical regions, their agricultural sectors remain highly dependent on manual labor for fruit harvesting. On large-scale farms, this dependency often results in significant labor cost and logistic complexities. This project presents the development and simulation of a voice-controlled Unmanned Aerial Vehicle (
UAV) system designed to automate harvesting tasks in extensive plantations. The proposed system integrates
speech recognition using Whisper [1] and LLM, computer
vision-based ripeness classification, and adaptive path planning within a unified framework. The entire system is modeled and validated in a Gazebo simulation environment, allowing performance evaluation under controlled agricultural scenarios
Read the full abstract ▾
Robotics
Immiscible Diffusion Policy: Preserving Multimodal Robot Actions through Label-Free Noise Assignment
Xiao Zhang, Yuxin Chen, Zhixuan Liang et al. (+5 more)
When
diffusion policies were first introduced, they were expected to recover multi-modal action distributions. However, we find this expectation does not always hold, as
diffusion policies often collapse to a single modality even when we guarantee the balance of
dataset modalities and exact within-batch
symmetry. Our analysis indicates that independent action-noise pairing contributes to this failure by increasing mixing and crossing among
diffusion paths, which can produce averaged denoising responses and suppress modality-specific behavior. This issue is especially severe in
robot planning, where action spaces are dense and low-dimensional, significantly increasing such mixing and crossing. To alleviate this problem, we propose Immiscible
Diffusion Policy, a label-free training-time add-on to
diffusion policy that uses action-noise assignment to preserve relatively distinct noise-to-action routes without modifying the policy architecture or inference procedure. Across five simulated and two real-world humanoid manipulation tasks spanning state, RGB, and point-cloud observations, our method significantly improves the policy's preservation of action modalities while maintaining strong task performance. It increases the proportion of the non-dominant modality by 6.0x-14.6x across three two-modality tasks and recovers demonstrated modalities that are entirely absent from vanilla policy rollouts on both four-modality tasks. These results demonstrate that Immiscible
Diffusion Policy provides a simple yet robust approach to preserving action multi-modality in general
robot learning tasks.
Read the full abstract ▾
All authors Xiao Zhang, Yuxin Chen, Zhixuan Liang, Guojian Zhan, Chenran Li, Chenfeng Xu, Masayoshi Tomizuka, Yiheng Li