Robotics
Surface Constraint Policy for Learning Surface-Constrained and Dynamically Feasible Robot Skills
Shuai Ke, Jiexin Zhang, Huan Zhao et al. (+4 more)
Diffusion-based imitation learning methods have driven rapid progress in
robot dexterous manipulation tasks. However, they have limitations when applied to tasks that involve complex free-form surface constraints because of their lack of explicit surface geometry constraint modeling and the dynamic feasibility issue, resulting in
stochastic action generation that fails to achieve reliable surface
alignment and maintain stable contact. To address these limitations, we propose a novel surface constraint policy (SCP) for generating
robot actions that satisfy free-form surface constraints on the basis of human demonstrations and real-time visual observations. First, the surface geometry constraint is encoded using a two-dimensional weighted Gaussian kernel function that is derived from demonstrations. Building on the encoded surface geometry constraints, the
diffusion-based policy is used to infer task-level action intentions from
multimodal sensory inputs, including visual observations and
robot state feedback. These intentions are further transformed into surface-constrained dynamic movement primitives (DMPs) through a similarity-based action mapping method, thereby enabling smooth and compliant motion execution. The SCP achieves generation of structured surface
geometric intent and dynamically admissible actions. The proposed method is validated on multiple surface manipulation tasks and compared with existing techniques. The experimental results demonstrate superior task success rates and contact stability under surface constraints.
Read the full abstract ▾
All authors Shuai Ke, Jiexin Zhang, Huan Zhao, Zhiao Wei, Yikun Guo, Jie Pan, Han Ding
Robotics
AR Forcing: Towards Long-Horizon Robot Navigation World Model
Yifei Yang, Zehua Fan, Huan Li et al. (+9 more)
The
diffusion based
robot navigation world models are typically trained using parallel supervision, while autoregressive inference is employed during path planning. This results in a distribution shift between training and inference, which destabilizes the performance over long-horizon prediction. We propose AR Forcing, an autoregressive training strategy, which integrates the standard
diffusion loss into the autoregressive training loop. At each step, the model uses its own predictions to update the context and optimize the single step noise prediction objective, thereby explicitly exposing the model to the inference state distribution during training. Our method does not require additional discriminators or distribution-matching losses, retains the original
diffusion framework and sampler, and is easy to integrate. Experiments on multi-domain
navigation datasets (RECON, SCAND, HuRoN, TartanDrive) show that compared with strong baselines, AR Forcing improved the consistency of generated images during long-horizon
navigation and the accuracy of predicted trajectories, enhancing robustness of the model in complex known and unknown environments. We will release the code soon.
Read the full abstract ▾
All authors Yifei Yang, Zehua Fan, Huan Li, Aoqi Wang, Lida Huang, Haibao Yu, Haiyan Liu, Xuanyao Mao, Jason Bao, Liang Xu, Bingchuan Sun, Yan Wang
Robotics
DeMaVLA: A Vision-Language-Action Foundation Model for Generalizable Deformable Manipulation
Taiyi Su, Jian Zhu, Tianjian Wang et al. (+9 more)
Real-world household
robots require
Vision-
Language-Action (VLA) foundation models that can acquire reusable manipulation skills across diverse objects, task conditions, and household environments. Deformable-object folding is a representative challenge, requiring
robots to handle clothing items from random initial states across varying categories, geometries, materials, and scenes. However, existing VLA systems commonly train separate policies for different object categories, while naively mixed multi-task training often suffers from task interference and degraded performance. To move beyond category-specific folding policies, we introduce DeMaVLA, a VLA foundation model for generalizable Deformable Manipulation. DeMaVLA adopts a VLM backbone with an action expert and formulates continuous action generation using flow matching. To improve efficiency, the action expert is constructed by
pruning every other
transformer layer while preserving layer-wise
alignment with the VLM backbone, reducing training and inference cost. DeMaVLA is first pre-trained on approximately 5,000 hours of selected real-world dual-arm demonstrations to acquire general manipulation priors. It is then post-trained on mixed folding data that aggregates self-collected demonstrations and corrective trajectories from real-
robot failures across multiple folding tasks through a human-in-the-loop Data Aggregation~(DAgger) pipeline. Experiments show that DeMaVLA achieves competitive performance on RoboTwin 2.0 and strong real-world results on our household folding
benchmark. These results highlight the value of scalable real-world data, efficient action generation, and corrective learning for general-purpose VLA policies in deformable-object manipulation.
Read the full abstract ▾
All authors Taiyi Su, Jian Zhu, Tianjian Wang, Youzhang He, Zitai Huang, Jianjun Zhang, Chong Ma, Hanyang Wang, Tianjiao Zhang, Munan Yin, Weihao Ding, Yi Xu
Robotics
Before Parc Fermé: RL-Time Pruning for Efficient Embodied LLMs in Autonomous Driving
Luca Benfenati, Ali Azimi, Matteo Risso et al. (+3 more)
Embodied Large
Language Models (LLMs) are increasingly used as
reasoning modules in robotic control pipelines to improve human-
robot interaction, but their
memory and generation latency make real-time deployment difficult.
Pruning can reduce these costs, but for controllers that undergo multiple pre- and post-training phases, the crucial question is not only how much to prune, but when
pruning should occur. In this work, we propose Before Parc Fermé (BPF), a
pruning strategy performed during RL that compresses
embodied LLM controllers while they are still being optimized for closed-loop behavior. This allows
pruning decisions to account for the task-specific supervision and closed-loop feedback that shape the final controller. We propose two variants: BPF-RL, which performs iterative
pruning during RL by removing part of the model at predefined training intervals, and BPF-SFT/RL, which first prunes part of the model structure during SFT and then further compresses it during RL using the same iterative strategy as BPF-RL until the target
pruning ratio is reached. We evaluate BPF on RobotxR1, an LLM-based autonomous-driving control pipeline, using an established LLM
pruning framework (LLM-Pruner), and compare it against post-training
pruning, post-training
pruning with RL recovery, SFT-stage
pruning, and smaller dense models from the same family. Our results show that BPF provides the best task-performance vs.
memory and throughput trade-off among the considered
pruning strategies. When compressing the larger RobotxR1 models, BPF-SFT/RL achieves a $1.69\times$ better size-end-to-end performance trade-off than directly selecting a smaller dense model from the same family, measured as removed parameters per lost percentage point of control adaptability. On the Jetson AGX Orin mounted on the target robotic platform, the compact models improve decode throughput by up to $27\%$.
Read the full abstract ▾
All authors Luca Benfenati, Ali Azimi, Matteo Risso, Fabio Carapellese, Daniele Jahier Pagliari, Alessio Burrello
Robotics
HARP-VLA: Human-Robot Aligned Representation Learning for Vision-Language-Action Model
Xiang Zhu, Puzhen Yuan, Yichen Liu et al. (+1 more)
Learning generalizable
vision-
language-action (VLA) models from large-scale human
videos is promising but challenging due to cross-embodiment discrepancies in both visual observations and executable actions. While latent action models reduce the action execution gap by learning action abstractions, they still rely on visual features. Thus, misaligned human and
robot visual representations can lead to inconsistencies in policy inputs and induce domain-dependent latent actions, hindering effective co-training with human
videos.To address this, we propose HARP, a human-
robot aligned representation learning framework for more effective VLA pretraining from human
videos. Specifically, HARP uses paired human-
robot demonstrations as cross-embodiment bridges and abundant unpaired human and
robot videos as a scalable dynamics supervision data source. It trains a
robot-adapted visual encoder and a latent action model with manipulation-centric auxiliary cues and a source-relative pair-discriminative
alignment loss, which adapts
robot representations toward human semantics while preserving pair-level discrimination. The learned aligned
vision encoder and latent action model provide a unified
vision and action representation for VLA-style policy learning, where human and
robot videos provide
vision-
language-to-latent-action supervision and a lightweight
robot action head grounds latent actions into executable commands. Experiments on feature visualization, simulation, and real-world manipulation show improved human-
robot alignment and downstream policy performance, achieving 4.481 average length on CALVIN ABC$\rightarrow$D and a 7.1 percentage-point improvement in real-world success rate over the strongest evaluated baseline. Code,
videos, and demos are available at https://puzhenyuan.github.io/HARP-VLA-website/.
Read the full abstract ▾
All authors Xiang Zhu, Puzhen Yuan, Yichen Liu, Jianyu Chen
Robotics
Simulation of collision avoidance behavior in crowd movement by data-driven approach
Xuanwen Liang and Eric Wai Ming Lee
Crowd movement simulation is essential for pedestrian safety management and facility layout optimization. Data-driven models enhance trajectory prediction accuracy under Euclidean metrics, yet they suffer from excessively high collision rates, especially in bidirectional and multidirectional flows. In this paper, we establish a novel data-driven crowd simulation model that incorporates the pedestrian collision mechanism into the loss function to reduce collisions. A new lateral-acceleration-based collision loss function and a Voronoi-based motion feature extraction approach are proposed. The model is based on a Generative
Adversarial Network (GAN) architecture and is termed CPGAN (Collision-Penalized GAN). We evaluate CPGAN in bidirectional flow scenarios, which involve frequent collision avoidance behaviors. Results show that the proposed lateral-acceleration-based collision loss significantly reduces opposite-direction pedestrian collision rates to levels comparable with controlled experiments. CPGAN effectively simulates bidirectional flow, reproducing lane formation and N-t curves. The research outcomes can provide inspiration for integrating pedestrian dynamics mechanisms into loss functions in data-driven crowd simulation.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Probing Collision Grounding in Vision-Language Models for Safe Human-Robot Collaboration
Jun Wang, Xiaohao Xu and Xiaonan Huang
Safe human–
robot collaboration requires more than visual description: a monitor must determine whether the
robot body is safely separated, already colliding with the scene or a person, or about to collide. We call this capability collision grounding: binding visual observations to
robot body geometry, camera viewpoint, scene layout, human proximity, and temporal motion in order to infer present and imminent contact. We introduce TouchSafeBench, a physics-grounded
benchmark for evaluating collision grounding in
vision-
language models (VLMs). Built in Habitat~3.0, TouchSafeBench contains 2,940 simulated indoor co-presence episodes across social
navigation and social rearrangement, with synchronized multi-view RGB-D observations, top-down trajectory maps, calibrated camera metadata, and simulator-derived contact labels. We study two deployment-facing tasks: classifying the current safety state and warning about imminent collision before contact. Across three frontier or robotics-oriented VLMs and nine visual representations, current models remain far from reliable: the best average Macro-F1 stays below 50%, explicit depth is not automatically transformed into
robot-body collision evidence, and
robot–scene contact is consistently harder than human-contact risk. TouchSafeBench reveals a central limitation of
embodied VLMs: visual fluency does not imply physical accountability. Reliable
robot safety monitors will need representations that explicitly bind viewpoint,
robot morphology, metric geometry, and future collision. The
benchmark is available at https://jun25336-bit.github.io/TouchSafeBench/ .
Read the full abstract ▾
Robotics
TARIC: Memory-Augmented Traversability-Aware Outdoor VLN under Interrupted Semantic Cues
Tianle Zeng, Hanjing Ye, Jianwei Peng et al. (+3 more)
Outdoor
vision-
language navigation (VLN) in long-range, open-world environments is frequently disrupted by semantic-cue interruptions, where informative goal cues become sparse, occluded, or leave the field of view. Once such cues disappear,
agents enter a cue-free phase and often degrade into backtracking, oscillatory headings, or aimless exploration. While
memory-based methods attempt to bridge these gaps, they often fail under traversability-driven detours: the remembered cue direction may be infeasible, forcing detours that prolong cue-free phases and gradually render
robot-centric cues stale and implicit histories blurred. This makes traversability a stability condition for maintaining goal-directed guidance, rather than merely a local safety concern. We propose a unified outdoor VLN framework that survives semantic-cue interruptions by maintaining traversability-consistent executable guidance throughout prolonged cue-free phases. Specifically, our method extracts semantic bearings from visibility-gated goal or exploration cues and grounds them into executable headings using a real-time near-field traversability profile, providing goal-consistent feasible guidance beyond reject-only safety filtering. To prevent guidance degradation during detours, we lift intermittent 2D evidence into a world-aligned 3D cue
memory with an uncertainty-aware readout mechanism, ensuring guidance remains continuously reachable and stable as the
robot moves. We evaluate the framework on quadrupedal and wheeled platforms over 600–1000 m routes. Our method improves simulation success rate by over 10 percentage points over the strongest baseline and achieves a real-world success rate of 40%, compared to 17.5% for the strongest baseline, with substantially higher robustness during prolonged cue-free intervals.
Read the full abstract ▾
All authors Tianle Zeng, Hanjing Ye, Jianwei Peng, Jingwen Yu, Hanxuan Chen, Hong Zhang
Robotics
Don't Fool Me Twice: Adapting to Adversity in the Wild with Experience-Driven Reasoning
Navin Sriram Ravie, Andrew Jong, Krrish Jain et al. (+4 more)
In robotics, dangers and adversity modes are often embodiment-specific and relative to each
agent. A frontier of autonomous mobile robotics is to enable
agents to operate effectively in the wild in unseen unstructured environments. A significant challenge in unseen unstructured environments is that it may not be possible to predict all the dangers to the specific
robot. Although recent work has used large foundation
vision-
language models (VLMs) to preemptively predict an exhaustive list of common-sense dangers, it remains difficult to capture possible interaction and embodiment-dependent adversities. We propose a continual learning framework for a mobile
embodied agent to learn online from disturbances and attribute anomalous behaviours to causes through semantics, enabling better prediction and planning of the world in the future. Our framework, "Don't Fool Me Twice", first observes disturbances and describes their effects on the
robot; this description is augmented with visual context to query a VLM to predict possible causes; the local disturbance is characterized using kernel regression, which allows for efficient, few-shot modeling of transient anomalies. We leverage semantic voxel-centric modeling to estimate epistemic uncertainty, enabling richer downstream recovery by treating interaction-driven disturbances as learnable spatial behaviors. We present four hypotheses and validate them in simulation and on hardware across embodiments and adversity modes.
Read the full abstract ▾
All authors Navin Sriram Ravie, Andrew Jong, Krrish Jain, John Liu, Omar Alama, Bijo Sebastian, Sebastian Scherer
Robotics · Cross-listed from cs.CV
NTR: Neural Token Reconstruction for Scene Token Bottleneck in End-to-End Driving
Jiahui Li, Jiawei Sun, Zixiang Ren et al. (+7 more)
Recent perception-free end-to-end (E2E) autonomous driving methods bypass explicit perception outputs by compressing dense image patch
tokens into compact scene
tokens for downstream trajectory generation and scoring. While these scene
tokens form a compact visual bottleneck for the planner, they receive supervision solely from the planning objective, providing limited constraints on the encoded visual information. To address this limitation, we introduce Neural
Token Reconstruction (NTR), a representation learning framework to directly constrain the compact scene-
token bottleneck in perception-free driving. NTR introduces a self-
distillation masked latent reconstruction objective that reconstructs masked patch-level latent features using only compact scene
tokens as reconstruction
memory. This forces reconstruction gradients to pass exclusively through the scene-
token bottleneck, encouraging scene
tokens to preserve richer and less redundant visual representations for planning. We further introduce semantic priors derived from foundation-model annotations as a weak semantic interface biasing reconstruction targets toward driving-related structures without introducing explicit perception heads. All auxiliary reconstruction components are removed at inference time, leaving the deployed planner unchanged. NTR achieves state-of-the-art performance on three public autonomous driving
benchmarks, including 8.0461 RFS on Waymo E2E and 94.1 PDMS / 90.9 EPDMS on NavSim1&2. The learned scene
tokens exhibit lower pairwise redundancy and higher effective rank, indicating that effective bottleneck supervision improves both compact visual representation learning and planning performance.
Read the full abstract ▾
All authors Jiahui Li, Jiawei Sun, Zixiang Ren, Ming Liu, Jiamin Shi, Ruiteng Zhao, Zhiyang Liu, Liying Liu, Zuoguan Wang, Kaidi Yang
Robotics
Building Generalization Into Behavior Generation Via Adaptive Compositions of Regularities
Aravind Battaje, Malte Bernhard, Vito Mengers et al. (+1 more)
Generalization in robotics requires prior knowledge about how the world is structured, yet this structure changes from one situation to the next. This paper investigates the proposition that generalization arises from adaptively composing regularities – predictable relationships within the
robot-environment system – into situation-appropriate structures for behavior generation. We examine this proposition by analyzing the mechanism in AICON (Active InterCONnect), a framework representing regularities as interacting processes in a differentiable network, where sensory feedback realizes composition and gradient descent generates behavior. To isolate adaptive composition as the key mechanism, we study a simple simulated problem in which all relevant regularities can be identified. We expose the resulting model to a wide range of novel conditions not considered during design, and we find that it generates context-appropriate behavior in all but one case, where encoded regularities are provably insufficient. Ablations reveal that the network automatically modulates which regularities influence behavior based on their informativeness. These results suggest that adaptive composition of regularities constitutes a powerful inductive bias for building generalization into behavior generation.
Read the full abstract ▾
All authors Aravind Battaje, Malte Bernhard, Vito Mengers, Oliver Brock
Robotics
Seeing Fast and Slow: Bimodal 3D Scene Graphs for Open-set Tasks
Marcel Bartholomeus Prasetyo, Shrutika Vishal Thengane, A Manicka Praveen et al. (+2 more)
Open-set task execution can significantly benefit from seamlessly switching between coarse and fine scene representations depending on the context and the evolving information as the
robot explores the environment. For example, it is often sufficient to start with a coarse scene representation initially and only employ a finer, more granular scene representation when the
robot encounters regions which are likely to contain the task relevant objects. Hence, in this work, we propose BiMoSG, a bimodal 3D scene graph generation approach for open-set tasks. BiMoSG employs a "fast" mode by default to efficiently generate a coarse 3D scene graph and can switch to a "slow" mode for generating a finer open vocabulary 3D scene graph of task relevant objects. We demonstrate that our proposed 3D scene graph generation approach is significantly faster than the open-source state-of-the-art approaches. This allows us to integrate the scene graph generation process with task execution for real-time deployment.
Read the full abstract ▾
All authors Marcel Bartholomeus Prasetyo, Shrutika Vishal Thengane, A Manicka Praveen, Yi Loo, Malika Meghjani
Robotics
Can Aerial VLA Models Cooperate? Evaluating Closed-Loop Air-Ground Coordination with CARLA-Air
Tianle Zeng, Yanci Wen, Xueang Yu et al. (+1 more)
Recent aerial
vision-
language-action (VLA) models show promising single-
UAV capabilities, such as tracking moving objects and navigating to
language-specified landmarks. However, it remains unclear whether these capabilities can transfer to air-ground cooperation, where a
UAV and a UGV must act jointly in a shared, closed-loop physical world. We study this question with CARLA-Air, a single-process air-ground evaluation environment that unifies CARLA and AirSim inside one Unreal Engine runtime. By sharing the same world state, physics tick, and sensing pipeline, CARLA-Air enables physically consistent
UAV–UGV interaction and precise measurement of simulation-timestamp
alignment and effective coordination latency. Using CARLA-Air, we evaluate representative aerial VLA and planning baselines on two complementary diagnostic tasks: moving-platform landing and occlusion-recovery escort. The results show that current aerial VLA models can often track or follow a ground partner, but struggle to convert this single-
agent competence into stable cooperative behavior. State prompting provides limited benefit, and naive bidirectional interaction fails to consistently improve performance and can amplify errors for most baselines. These findings suggest that, under the tested text-based cue interfaces, zero-shot cooperative air-ground VLA requires three components beyond the current paradigm: explicit partner-state grounding, low-latency action coordination, and team-level objective
alignment. Our code is available at https://github.com/louiszengCN/CarlaAir.
Read the full abstract ▾
All authors Tianle Zeng, Yanci Wen, Xueang Yu, Hong Zhang
Robotics
A study on a Real-Time VR-Based Teleoperation Framework for Manipulator in Dynamic Environment
InGyu Choi, GeonYeong Go, SunWoo Ahn et al. (+2 more)
Robot teleoperation enables safe, non-contact task execution in hazardous environments where direct human access is difficult, and its application has expanded with recent VR technologies. Many VR teleoperation studies, however, have primarily served as data-collection tools for
robot imitation learning, so they often do not explicitly address dynamic obstacles, workspace changes, or collision risks during operation. For real deployment aimed at operator safety, teleoperation must react to dynamic situations with low latency and remain robust to mistakes made by inexperienced operators. This paper presents a VR teleoperation framework that supports real-time manipulation while handling collisions with both static and moving obstacles. The framework integrates GPU-accelerated inverse kinematics and trajectory optimization within a VR interface to generate feasible joint commands at each control cycle under
robot constraints. Experiments with a 7-DoF manipulator demonstrate stable online behavior and collision-aware motion generation across three scenarios: obstacle-free, static-obstacle, and moving-obstacle environments. The results indicate that the proposed approach generates motion consistent with the operator's command while producing safe detours when obstacles interfere with the commanded path.
Read the full abstract ▾
All authors InGyu Choi, GeonYeong Go, SunWoo Ahn, HyoJae Kang, Min-Sung Kang
Robotics
RDGen: Demonstration Generation for High-Quality Robot Learning via Reinforcement Learning
Zijian Zhu, Menglin Zou, Zhuang Li et al. (+2 more)
Vision-
Language-Action (VLA) models have emerged as a promising paradigm for general-purpose
robot control. However, their performance remains fundamentally constrained by the availability of high-quality
robot trajectory data. In current
robot learning practice, such data are primarily collected through human teleoperation, which is labor-intensive, costly, and difficult to scale. In this paper, we propose RDGen, a sim-to-real reinforcement learning framework for generating high-quality
robot demonstrations. Rather than employing reinforcement learning solely as the final control policy, RDGen leverages trained RL policies as a structured trajectory generator. The system consists of a VLM-based task parser that identifies task-relevant objects, a Grounding DINO-based object localizer, and an RL policy transferred from simulation to the real
robot. Successful rollouts are then harvested as clean, high-quality demonstrations for downstream VLA training, while the simulation stage further provides a scalable source of additional trajectories at little marginal cost. Experiments on a pick-and-place task demonstrate that the transferred RL policy achieves a high task success rate. Compared with human teleoperation, RDGen produces significantly smoother trajectories and yields superior downstream VLA performance. These results indicate that RL-generated demonstrations can serve as more reliable and consistent supervisory signals for
robot policy learning.
Read the full abstract ▾
All authors Zijian Zhu, Menglin Zou, Zhuang Li, Yaojie Tu, Xinhai Sun
Robotics
Enhancing Human-Likeness in Reinforcement Learning Agents via Hierarchical Macro Action Quantization
M. Shaheer Luqman, Usman Nizamani, Fawad Javed Fateh et al. (+4 more)
Human-like
agents are a long-standing goal of artificial intelligence. Despite strong performance, most reinforcement learning (RL)
agents remain reward-driven and often exhibit behaviors that differ from humans, limiting interpretability and reliability. In this work, we introduce a novel human-like RL framework that predicts action sequences closely aligned with human behaviors while maximizing rewards. Specifically, we encode human demonstrations into macro actions using a hierarchical macro action quantization approach (HiMAQ) consisting of two successive levels of vector quantization. The lower quantization level maps input actions to fine-grained subaction clusters, while the higher quantization level aggregates these subaction clusters into action clusters. Extensive evaluations on the D4RL
benchmarks show that our hierarchical approach outperforms the non-hierarchical baseline (MAQ), achieving higher human-likeness scores and better success rates than previous RL
agents. The improvements generalize across integrations with various RL
algorithms, namely IQL, SAC, and RLPD.
Read the full abstract ▾
All authors M. Shaheer Luqman, Usman Nizamani, Fawad Javed Fateh, Ali Shah Ali, Murad Popattia, Quoc-Huy Tran, M. Zeeshan Zia
Robotics
Trajectory Planning for Non-Communicating Mobile Robots using Inverse Optimal Control
Nina Majer, Yannick Epple, Xin Ye et al. (+2 more)
To enable an efficient interaction of non-communicating mobile
robots in collision avoidance scenarios, we present a novel combined trajectory planning and prediction
algorithm. Inverse optimal control is used to estimate unknown goal states of all
robots based on observed past trajectories. Each
robot also takes the perspective of other
robots in considering self-prediction and solves a joint prediction problem using the estimated goal states. The resulting predictions are then considered for planning. Simulation results of scenarios with 2-8
robots show that the median of the durations until all vehicles reach their goals is 9.8 % faster compared to planning with constant acceleration based estimated goal states. Moreover, the proposed approach never leads to the solver being unable to find a solution to the planning or prediction problem.
Read the full abstract ▾
All authors Nina Majer, Yannick Epple, Xin Ye, Stefan Schwab, Sören Hohmann
Robotics
Wall-OSS-0.5 Technical Report
Ryan Yu, Pushi Zhang, Starrick Liu et al. (+24 more)
Large-scale
Vision-
Language-Action (VLA) pretraining is increasingly adopted as the foundation for
robot policies, yet the evidence for pretrained VLAs is almost invariably reported after task-specific fine-tuning. This leaves a foundational question unanswered: does VLA pretraining itself yield executable
robot behavior, or does it merely furnish a better initialization for downstream policy learning? We present Wall-OSS-0.5, an open-source 4B VLA built upon a 3B VLM backbone augmented with action-generation components, designed so that pretrained robotic capability is directly measurable on physical hardware. The model is pretrained across more than 20 embodiments, processing over one million
robot trajectories per epoch alongside a grounded
multimodal corpus. We adopt a gradient-bridged co-training recipe in which three objectives play distinct and complementary roles: discrete action prediction routes strong VLM-native gradients into the backbone,
multimodal prediction preserves grounded
vision-
language understanding, and continuous flow matching serves as the deployment-time action interface. Before task-specific fine-tuning, the pretrained checkpoint achieves non-trivial zero-shot real-
robot behavior, completing several tasks, including a held-out deformable manipulation task, at high task progress on a 17-task suite. After fine-tuning, the same checkpoint serves as a stronger adaptation prior, reaching 60.5% average task progress on 15 real-
robot tasks and outperforming \pi_0.5 by 17.5%.
Multimodal evaluations further confirm that action training does not erode grounded
vision-
language competence: the model preserves broad
vision-
language ability while strengthening
embodied grounding. Together, these results reposition VLA pretraining from an initialization strategy to a directly testable, already useful source of
robot capability.
Read the full abstract ▾
All authors Ryan Yu, Pushi Zhang, Starrick Liu, Brae Liu, Miracle Kang, Shalfun Li, Lights Shi, Ellie Ma, Ping Yang, Chris Pan, Jerry Chen, Dongxiu Liu, Rain Sun, Miles Guo, Byron Zhang, Hugo Zhou, Zach Xu, Vincent Chen, Harrison Huang, James Wang, Dance Kuzi, Andy Zhai, Hang Su, Roy Gan, Lucy Liang, Hao Wang, Qian Wang
Robotics
High-Load-Density Electro-Permanent Magnetic Foot with Controllable Adhesion for Quadruped Wall-Climbing Robots
An Li, Bo Tao, I-Ming Chen et al. (+1 more)
To enable reliable climbing locomotion of quadruped
robots on
ferromagnetic surfaces, this paper presents a high-load-density electro-permanent
magnetic foot with controllable adhesion, featuring force-feedback circular Halbach-net electro-permanent magnet (CHN-EPM) adhesion units and a magnetization control system. Due to its three-dimensional
magnetic circuit structure and flux-concentration effect, the CHN-EPM enables a distributed parallel
magnetic flux path with enhanced flux utilization, resulting in reduced sensitivity to air-gap variations and allowing effective adhesion to be maintained even under partial contact conditions. The proposed CHN-EPM generates a maximum adhesion force exceeding 1000 N with a load-to-weight ratio over 200:1. A magnetization driver and a two-stage pulse current control strategy are developed to regulate the excitation current amplitude and duration, enabling accurate and reliable magnetization. By incorporating a flexible pressure sensor for contact force feedback, the system can effectively monitor attachment and detachment states, ensuring robust adhesion switching under uncertain contact conditions. The proposed system is integrated into a commercial quadruped
robot (Unitree GO2), demonstrating high-load adhesion on ceiling and vertical-wall surfaces and stable locomotion on painted, perforated, and curved
ferromagnetic surfaces.
Read the full abstract ▾
All authors An Li, Bo Tao, I-Ming Chen, Han Ding
Robotics
Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring
Seongheon Park, Wendi Li, Changdae Oh et al. (+4 more)
Vision-
Language-Action (VLA) models enable
robots to follow natural
language instructions and generalize across diverse tasks, but they remain vulnerable to execution failures that compromise reliability in real-world deployment. Detecting such failures during execution is therefore critical for the robust deployment of
embodied systems. Existing failure detection methods either rely on expensive action resampling or external models, while alternatives propagate trajectory-level labels uniformly across every timestep, obscuring localized failure signals. In this paper, we propose Hide-and-Seek, a framework that formulates VLA failure detection as a coarsely supervised learning problem. By combining inter-trajectory and intra-trajectory contrastive objectives, Hide-and-Seek localizes failure-indicative actions and induces temporally structured failure signals from trajectory-level supervision alone, without any step-level annotation. We evaluate Hide-and-Seek on LIBERO, VLABench, and a real-world robotic platform across three representative VLA policies: OpenVLA, $\pi_0$, and $\pi_{0.5}$.Our method achieves state-of-the-art multi-task failure detection performance with a practical accuracy–timeliness trade-off under conformal prediction, and generalizes well to both seen and unseen tasks.
Read the full abstract ▾
All authors Seongheon Park, Wendi Li, Changdae Oh, Samuel Yeh, Zsolt Kira, Michael Hagenow, Sharon Li
Robotics
Feat2Go: Visual Feature-Grounded Value Estimation for Embodied Reinforcement Learning
Junyang Shu, Zhiwei Lin, Bingqing Wei et al. (+1 more)
Reinforcement learning is a promising approach for improving the capabilities of
vision-
language-action (VLA) models while avoiding the heavy data requirements of imitation learning. However, its effectiveness for VLA models is often constrained by sparse supervision and the difficulty of designing informative reward signals for long-horizon manipulation. In this work, we present Feat2Go, a fine-grained value estimation framework for
embodied reinforcement learning. Specifically, Feat2Go first derives a continuous progress target from a pretrained visual world model by measuring patch-level similarity to subgoal states and partitioning episodes into semantic stages with trend-based clustering. We then train an
embodied value model to predict this structural progress from the current observation and task instruction, and use the predicted value to reshape terminal rewards during policy optimization. The proposed framework is compatible with existing VLA policy reinforcement learning pipelines, including PPO and GRPO, and does not rely on manual reward engineering. Extensive experiments on ManiSkill3 and RoboTwin 2.0 demonstrate that Feat2Go consistently improves the performance of existing VLA models under both single-arm and bimanual manipulation settings. More specifically, on ManiSkill3, Feat2Go improves OpenVLAOFT from 17.5% to 82.9% average out-of-distribution success while retaining 96.9% in-distribution performance. On RoboTwin 2.0, Feat2Go achieves an average success rate of 88.8% in domain-randomized task settings, outperforming prior reinforcement learning methods.
Read the full abstract ▾
All authors Junyang Shu, Zhiwei Lin, Bingqing Wei, Yongtao Wang
Robotics
Two Degree-of-Freedom Vibratory Transport in a Grasp
C. L. Yako, Shenli Yuan and Kenneth Salisbury
In this paper, we use asymmetric vibrations to demonstrate two degree-of-freedom (DoF) in-hand manipulation of grasped parts. The asymmetric vibrations are achieved through closed-loop position control of a moving surface, which applies a periodic stick-slip waveform to the part to be manipulated. We show analytically how two vibratory waveform parameters, the sticking acceleration and the slipping acceleration, affect average part velocity when moving against
gravity. The theoretical trends are then validated using an experimental setup where the squeeze force is controlled and part motion is recorded by a high-resolution encoder. We also develop a 2-DoF vibratory surface capable of translation in one direction and rotation about the surface normal. Using two of these 2-DoF surfaces in a parallel jaw gripper configuration, we bidirectionally translate and rotate a variety of grasped parts, as well as demonstrate that the same waveform trends for translation also persist for in-plane rotation.
Read the full abstract ▾
Robotics
Closed-Loop Object-Informed Control for Non-Prehensile Robot Manipulation
Nikola Raicevic, Hyomuk Kim, Shahid Mulla et al. (+5 more)
Non-prehensile
robot manipulation is challenging due to discontinuous, long-horizon interactions between the
robot and the objects it manipulates. Sampling-based model predictive control methods are effective with discontinuous contact but face challenges with finding promising trajectories in long-horizon planning. We propose a closed-loop object-informed (CLOI) method that splits the problem into object-level planning to find long-horizon object poses that lead the object to its goal, and
robot-level planning to select
robot actions that follow those poses. We use model predictive path integral (MPPI) control to solve the subproblems and couple their solutions through consensus on the object poses using the alternating direction method of multipliers (ADMM). The object plan is revised toward
robot-realizable object trajectories, while the
robot plan is aligned with the object poses the task requires. In planar pushing tasks with obstacles using an xArm6 manipulator, CLOI increases the success rate by 35% in simulation and 43% on hardware, compared to standard MPPI given the same computational budget.
Read the full abstract ▾
All authors Nikola Raicevic, Hyomuk Kim, Shahid Mulla, Hyungjun Doh, Bharath Raam Radhakrishnan, Chenbin Yu, Ki Myung Brian Lee, Nikolay Atanasov
Robotics
SSR: Scaling Surefooted and Symmetric Humanoid Traversal to the Open World
Ruiqi Yu, Yiwen Wang, Yuan Hao et al. (+2 more)
Extending
humanoid traversal to the open world is key to practical deployment in human environments, but remains challenging. The
robot must use
vision to ensure safe and reliable foot placement on heterogeneous terrain under highly dynamic motion, while producing coordinated, natural whole-body behaviors. We propose SSR, an efficient end-to-end framework for
egocentric vision-based
humanoid traversal that jointly learns these capabilities. SSR introduces imagined foothold guidance, which learns to model forthcoming swing-foot contacts and evaluates their support to guide pre-touchdown swings toward stable regions, reducing edge slips. It further employs equivariant latent-space
symmetry augmentation to efficiently induce bilateral coordination under high-dimensional visual observations, and uses terrain-specific multi-discriminator motion priors to encourage human-like behavior across scenes. Extensive experiments show that SSR achieves safe, stable, and high-quality locomotion on diverse real-world terrains, including stairs with varied structures and extreme challenges such as wide gaps and high platforms, while enabling reliable long-horizon traversal in open outdoor environments.
Read the full abstract ▾
All authors Ruiqi Yu, Yiwen Wang, Yuan Hao, Jun WU, Qiuguo Zhu
Robotics · Cross-listed from cs.CV
DisPlace: Discriminative Place Projections for Multi-Reference Visual Place Recognition
Dhyey Manish Rajani, Michael Milford and Tobias Fischer
A key challenge in Visual Place Recognition (VPR) is matching query images against reference maps captured under diverse environmental conditions and viewpoints. While multiple reference traversals improve robustness, existing fusion strategies either aggregate references uniformly or rely on heuristic selection, without distinguishing descriptor variations that preserve stable place identity from those caused by changing conditions or viewpoints. In this paper, we propose DisPlace, a multi-reference VPR framework that fuses multiple reference descriptors into a single compact and discriminative place representation. DisPlace formulates descriptor fusion as a generalized
eigenvalue problem that maximizes between-place separability while suppressing within-place variation across references, rather than preserving overall descriptor variance. Unlike existing multi-reference fusion methods, DisPlace exploits variation across reference traversals to identify which linear combinations of descriptor dimensions preserve place identity and which capture condition- or viewpoint-specific variation. We evaluate DisPlace on Oxford RobotCar, Nordland, Pittsburgh30k, and Google Landmarks v2 across six state-of-the-art VPR descriptors. DisPlace outperforms seven multi-reference baselines in 49 out of 54 appearance-varying conditions, consistently improves descriptor-level fusion performance under viewpoint and unstructured settings, and requires less storage during inference than all compared fusion methods.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidance
Sungha Kim, Gawon Lee, Jusuk Lee et al. (+3 more)
Maximum
entropy reinforcement learning (MaxEnt-RL) enables robust exploration, yet practical implementations often restrict policies to simple Gaussians. While recent approaches incorporate expressive generative policies via importance-weighted supervised learning, they are prone to importance weight collapse, which limits their scalability in high-dimensional action spaces. Our key insight is to mitigate this limitation by localizing the sampling region, avoiding the weight degeneracy induced by importance sampling over the entire action space. To instantiate this insight, we introduce FLAG (Flow policy with Latent-Augmented Guidance). FLAG augments the state space with a flow latent variable and optimizes a provably consistent proxy MaxEnt-RL objective. We empirically demonstrate that FLAG enables expressive policy optimization with limited importance samples and scales to high-dimensional control tasks. Furthermore, FLAG achieves state-of-the-art performance across challenging
benchmarks. Our project webpage: https://flag-rl.github.io/
Read the full abstract ▾
All authors Sungha Kim, Gawon Lee, Jusuk Lee, Jonghae Park, H. Jin Kim, Daesol Cho
Robotics
GSAM: A Generalizable and Safe Robotic Framework for Articulated Object Manipulation
Beichen Shao, Mengying Xie, Heng Su et al. (+5 more)
Articulated object manipulation is a unique challenge for service
robots. Existing methods employ end-to-end policy learning, visionmotion planning, and large-
language/visual-
language model (LLM/VLM), but often overlook the diversity of articulated objects and the complexity of interactions between end-effector and handle, leading to limited generalization and destructive collisions. To address this, we propose GSAM, a generalizable and safe robotic framework for articulated object manipulation. Specifically, a
vision-based perceiver generates the kinematic parameters. Considering that pre-trained markers in perceiver yield raw estimations that may deviate from commonsense, we present a f ine-tuned VLM-based refiner, using chain-of-thought (COT) commonsense
reasoning to refine perception. To prevent destructive collisions, we design an interaction constraint function generator, integrating articulated object, interaction pose, and obstacle avoidance knowledge into a base. LLM then functionalize these constraints and apply them to trajectory and posture planning. A kinematic-aware manipulation planner verifies reachability for trajectory and posture. Experiments on 50 hinge tasks across 5 object categories and 50 randomly initialized end-effectorhandle configurations show that GSAM reduces standard deviation by 3.1% and improves manipulation success rate by 36.0% compared to the best baseline, respectively demonstrating the superior object generalization and interaction safety of GSAM in practical scenarios.
Read the full abstract ▾
All authors Beichen Shao, Mengying Xie, Heng Su, Wanyi Zhang, Mingyan Li, Yan Ding, Fausto Giunchiglia, Chao Chen
Robotics
Geometry-Aware Control Barrier Functions for Collision Avoidance via Bernstein Polynomial Approximations
Siwon Jo, Yanze Zhang, Yupeng Yang et al. (+1 more)
Safe
navigation often relies on well-defined conditions based on the shape of
robots and obstacles, and can be challenging when they have irregular geometries. While Control Barrier Functions (CBFs) offer an efficient mechanism to enforce safe set forward invariance, common shape surrogates (e.g., spheres or super-ellipsoids) are either overly conservative in unstructured scenes or require many local primitives, which inflates constraint counts and degrades real-time performance. In this paper, we introduce a novel geometry-aware Control Barrier Function (CBF) based on Bernstein-
Polynomial Signed Distance Fields (BP-SDFs). It provides a unified way to represent the obstacles and
robots, so as to represent the barrier function with a unified minimum distance. Benefiting from the differentiability of the Bernstein
polynomials, one can easily enforce the control constraints in a closed loop. We validate the method's efficiency and performance to guarantee safety in single-
robot navigation and heterogeneous multi-
robot collision avoidance via simulations under different environments.
Read the full abstract ▾
All authors Siwon Jo, Yanze Zhang, Yupeng Yang, Wenhao Luo
Robotics
Primitive Subspaces Mediate Few-Shot Transfer in VLAs
Anya Singh, Cabrel Happi, Jai Relan et al. (+2 more)
Deploying
vision-
language-action (VLA) policies in industrial environments requires the ability to teach new tasks at low cost, a property current VLAs lack, since each new task requires fine-tuning. We investigate whether primitive-aware training produces a transferable artifact: a learned library of sub-skills that can be composed at inference time, conditioned on a small number of demonstrations, to perform tasks the policy was never trained on. We train two VLA architectures with different inductive biases, OpenVLA and $\pi_{0.5}$, on the REASSEMBLE contact-rich assembly
dataset under matched LoRA fine-tuning recipes and locked hyperparameters, varying training between flat trajectories and primitive-segmented episodes with primitive-specific
language prompts. We hold out 6 object-task combinations from training and evaluate few-shot transfer: models receive $m \in \{0, 1, 3, 5, 10\}$ demonstrations of a held-out task and attempt execution without weight updates. We replicate across three training seeds and validate on a second
dataset (LIBERO-Long). Primitive-trained models reach 78% of fine-tuned upper-bound performance with only m=3 demonstrations, while flat-trained models require m=10 demonstrations to reach the same level – a $3\times$ sample efficiency gap that replicates across seeds, architectures, and
datasets. To establish causation, we ablate the primitive-decodable subspace of hidden states and show few-shot transfer degrades by 32 percentage points while ablating a random subspace of equal dimensionality has no effect, indicating primitive representations are causally necessary rather than incidentally correlated with transfer. We identify and correct a methodological pitfall in evaluating chunked policies: family-wise inflation of single-step action-range gates produces order-of-magnitude higher false-failure rates against ground-truth human demonstrations.
Read the full abstract ▾
All authors Anya Singh, Cabrel Happi, Jai Relan, Varun Nair, Vidyut Baradwaj
Robotics · Cross-listed from cs.CV
WristCompass: Kinematic Coupling as a Learnable Visual Concept for Ego-Camera Orientation
Varun Nair, Vidyut Baradwaj, Jiahang He et al. (+3 more)
Recovering ego-camera orientation from manipulation
video is a prerequisite for disentangling hand motion from camera motion, a key step in imitation learning from
egocentric demonstrations. The obvious approach, inferring orientation from scene geometry, fails when hands occlude the frame: VGGT, a 1B-parameter scene reconstruction model, scores worse than a constant predictor on the TACO
benchmark. We identify an alternative visual concept that is present precisely when scene geometry is absent: kinematic coupling dynamics, the structured physical relationship between wrist motion and camera orientation imposed by the arm-shoulder-head chain. We find that this concept is compact (4D inter-wrist features outperform 126D full hand keypoints), temporal (requiring a GRU over short windows rather than per-frame
retrieval), and physically grounded (transferring zero-shot across
datasets because it is rooted in anatomy rather than scene appearance). Trained only on tabletop manipulation, WristCompass transfers zero-shot to Epic Kitchens cooking
video, achieving 14.3$^\circ$ median geodesic error and approaching the performance of a 1B-parameter scene model at 200K GRU parameters.
Read the full abstract ▾
All authors Varun Nair, Vidyut Baradwaj, Jiahang He, Anya Singh, Jai Relan, Cabrel Happi