Robotics · Cross-listed from cs.HC
Refuses in Chat, Complies with Arms: LLM Refusal Does Not Survive Robot Control
Edward Sun, Christopher Leet, Sravanthi Machcha et al. (+4 more)
Frontier
language models reliably refuse to explain how to cause physical harm. We ask whether this refusal survives when the same models control a
robot. We introduce RoboHarm, a physical AI safety
benchmark of harmful
robot instructions, and test seven frontier models on it. We hold each model fixed and vary only the interface, from text to Code as Policies to
vision agents that command end-effector poses or joint angles. In a MuJoCo simulation twin of a bimanual YAM arm
robot, refusal falls from 73% of text requests to 18-25% once the model controls the
robot, even when it is told what the objects are. On the physical
robot, six models with end-effector control refused 18% of 612 trials against 19% in sim, and three with joint control refused 41% of 252 against 31%. GPT-5.6 Sol refused every text request and none of its trials on the real
robot. Case studies of the
robot transcripts show that models miss the hazard, name it without inferring its consequences, or act despite recognizing it, and their refusals often hinge on uncertainty about what an object is rather than on the harm of the action. Safety
alignment learned in text thus fails to generalize to action, leaving current models with a poorly understood safety gap in the physical world.
Read the full abstract ▾
All authors Edward Sun, Christopher Leet, Sravanthi Machcha, Achyutha Menon, Tzu Kit Chan, Yuchen Cui, Jay Chooi
Robotics
Autonomous Droplet Navigation via Model-Based Reinforcement Learning: Zero-Shot Transfer and Emergent Dynamics
Rajneesh Anand and Mayuresh V. Kothare
Self-driving laboratories (SDLs) are transforming chemical and materials discovery through closed-loop automation, yet automated infrastructure for physical manipulation of soft, deformable matter remains beyond current robotic platforms. A critical instance is autonomous droplet transport on an open surface, where contact-angle hysteresis, capillary pinning, and surface heterogeneity produce partially observable dynamics that pose significant challenges for classical model-based controllers. We introduce the first robotic platform for closed-loop autonomous
liquid droplet navigation on an open, unconfined surface using model-based reinforcement learning. A two-axis tilting board coated with a thin silicone oil film drives the droplet, while an overhead camera provides real-time feedback. A learned policy was trained on just 50 to 150 physical episodes depending on
geometric complexity, without simulation or analytical models. Beyond performance alone, the platform demonstrates three capabilities of interest to the SDL community: it robustly transfers zero-shot to unseen geometries; it autonomously discovers an oscillatory depinning strategy to free the droplet when it sticks; and it completes its full training pipeline in under 90 minutes. These results extend reinforcement-learning manipulation from rigid microrobots to deformable soft-matter systems for next-generation SDLs.
Read the full abstract ▾
Robotics
ACG-WAM: World-Action Modeling via Action-Conditioned Geometric Latent Prediction
Jiangtao Liu, Zishang Xiang, Yage He et al. (+4 more)
World action models jointly learn visual predictionand
robot actions, providing a way to use observations ofscene
evolution for policy learning. Their
video and actionlosses, however, provide no explicit target for the geometricconsequences of a demonstrated action sequence. Moreover,visual features taken after temporal
attention can contain futureobservations, making them unsuitable as the sole current visualinput to an auxiliary predictor. We introduce ACG-WAMand its auxiliary objective, the Action-Conditioned GeometricJoint-
Embedding Predictive Architecture (ACG-JEPA), whichpredicts
geometric features at several horizons from the currentobservation and intervening actions, using the future slot of afrozen VGGT encoding of each current and future image pairas the target. We apply this supervision from the head and wristcameras to a shared visual
embedding before temporal mixing,and remove the teacher and auxiliary modules at inference.On 50 RoboTwin 2.0 tasks, ACG-WAM achieves 93.46%success in clean scenes, with the best randomized success(92.68%) and mean across both settings (93.07%) among thecompared methods; across three tasks on a real
robot, itachieves 85.00% success and 91.67% partial completion score,exceeding Motus by 10.00 and 9.17 percentage points, respec-tively. Code:https://github.com/RoboOpus/ACG-WAM.Website:https://RoboOpus.github.io/ACG-WAM.
Read the full abstract ▾
All authors Jiangtao Liu, Zishang Xiang, Yage He, Lingguo Cui, Baihai Zhang, Runqi Chai, Senchun Chai
Robotics
Learning Modular Policy for Multi-Floor Object Navigation:A Factorized Framework for Diagnostic Study
Shichao Zhai, Shuhao Ye, Rong Xiong et al. (+1 more)
Object-goal navigation (ObjectNav) in multi-floor scenarios presents a challenge due to sparse rewards caused by long-horizon decision-making. In this paper, we propose a diagnostic study based on a modular framework with an effective learnable policy to analyze failure factors in multi-floor scenarios. To achieve an effective policy for diagnosis, we design the hierarchical factorization policy that deconstructs a single global policy into an intra-floor exploration policy and an inter-floor switching policy. To providing an effective initialization for Reinforcement Learning (RL), the lightweight intra-floor policy is learned by distilling the exploration logic of Visual
Language Models (VLMs). Under idealized assumptions, we show that the factorized policy is theoretically equivalent to a single global policy at the policy-representation level. Experiment results indicate that perception performance and stair climbing stability are the primary bottlenecks in multi-floor navigation.
Read the full abstract ▾
All authors Shichao Zhai, Shuhao Ye, Rong Xiong, Yue Wang
Robotics
ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception
Ruoxuan Feng, Yutong Chen, Ruihua Song et al. (+4 more)
Humans inherently understand the physical world through an active process. When sensory evidence is insufficient to infer physical properties, we naturally interact with the environment by deciding what information is missing, how to acquire it, and when sufficient evidence has been obtained. In stark contrast, existing multi-sensory
robot systems mainly integrate sensory inputs rather than actively acquiring missing evidence through interactions. In this work, we introduce ROMA, an LLM-based system for Real-World Object-Centric Multi-Sensory Active Perception. ROMA integrates
vision, audio, tactile, and force sensing into a
reasoning-interaction-feedback loop. The model identifies missing evidence and determines the target objects, interactions, and modalities, while a physical interface executes the selected interactions and collects the multi-sensory feedback. To support this capability, we construct ROMI-2K, a large-scale real-world multi-sensory object interaction
dataset covering nearly 2,000 objects and 6 atomic interactions with synchronized sensory feedback. Building on these data, we develop a two-stage training framework that aligns sensory modalities and equips the LLM to assess evidence sufficiency, select informative interactions, and reason over the multi-sensory feedback. We further characterize active perception as perception chains, where acquired evidence guides subsequent interactions and
reasoning, and establish ROMA Bench to evaluate single-attribute, long-horizon multi-attribute, and intent-driven active perception. Experiments show that ROMA can actively acquire missing evidence and solve complex, long-chain multi-sensory perception tasks that existing methods struggle to handle, laying a strong perceptual foundation for active multi-sensory embodied
agents.
Read the full abstract ▾
All authors Ruoxuan Feng, Yutong Chen, Ruihua Song, Huan Yang, Zhongyuan Wang, Guocai Yao, Di Hu
Robotics
Physical Twins: Accelerating and Enabling Robot Learning with Phantom Platforms
Elizabeth Peiros, Derek Chen, Lucas Yager et al. (+4 more)
Improvements in human-
robot physical interaction (pHRI) can have major implications for physical therapy, search and rescue, and telemedicine. However, a major challenge concerns human constraints and safety in human-
robot physical experiments. Concerns about human studies also include repeatability, scalability, and participant diversity. To conduct such experiments, an IRB and willing human participants are required. In this work, we present an improved phantom device, a physical twin, that enables real-world RL-type testing for physically interactive
algorithms. The new device not only replicates the ball-and-socket motion of the shoulder but also renders scapular and protraction/retraction motions. The experiments showcase the device's ability to render multiple 3D joint limits and demonstrate a Franka Panda arm sensing limits and interacting with the device as an arm for ADLs (activities of daily living) and pHRI tasks.
Read the full abstract ▾
All authors Elizabeth Peiros, Derek Chen, Lucas Yager, Edward Lee, Kevin Wang, Zoe Samuels, Michael C. Yip
Robotics
SWAP: Stepwise Action Policy Routing for Vision-Language-Action Models
Mousumi Das, Aditeya Prajapati, Abrar Anwar et al. (+1 more)
Robot manipulation systems using
Vision-
Language-Action (VLA) model backbones typically use just one VLA for task execution. However, individual VLAs do not perform well across different task states and environments. We introduce a framework for dynamically composing multiple VLA policies during execution: StepWise Action Policy Routing (SWAP). SWAP formulates policy routing as an offline reinforcement learning problem, learning a routing critic that selects the most appropriate policy at each decision step given the current observation. SWAP enables
robots to select new policies to execute online rather than committing to a single policy for the duration of an episode. We evaluate SWAP on both real-world DROID manipulation tasks and LIBERO simulation experiments. SWAP improves over fixed-policy execution and routing baselines, giving absolute improvements in real-world task success up to 33% while reducing successful trajectory
robot action step length by 28.3%.
Read the full abstract ▾
All authors Mousumi Das, Aditeya Prajapati, Abrar Anwar, Jesse Thomason
Robotics
ExStereo: Lifting 2D Vision-Language-Action Models to 3D with Explicit Stereo Representations
I-Chun Arthur Liu, Jason Chen, Gaurav S. Sukhatme et al. (+1 more)
Three-dimensional perception is critical for robotic manipulation, particularly for high-precision tasks, as recovering metric depth and precise 3D object positions from monocular RGB observations is inherently ill-posed. However, many
Vision-
Language-Action (VLA) models rely solely on RGB observations for perception. Leveraging recent advances in foundation models for stereo matching, we introduce ExStereo, a stereo module that augments pre-trained 2D VLAs with 3D perception. ExStereo reconstructs scene geometry from stereo image pairs and renders multi-view observations as an explicit stereo representation for stereo feature extraction. The action
tokens from the action expert selectively attend to the resulting stereo
tokens through our proposed action-stereo cross-
attention mechanism, enabling the policy to generate
robot actions conditioned on 3D scene information. To learn robust 3D representations, we introduce a mid-training stage before task-specific post-training, using a self-supervised learning objective on large-scale stereo data. We validate our approach by fine-tuning two publicly available VLAs, $\pi_{0.5}$ and SmolVLA, and evaluate them in simulation and on a real-world bimanual PiPER platform. Across both settings, VLAs fine-tuned with ExStereo consistently outperform baselines, demonstrating the effectiveness of stereo perception for robotic manipulation. Our project website is at: https://exstereo-vla.github.io/ExStereo/.
Read the full abstract ▾
All authors I-Chun Arthur Liu, Jason Chen, Gaurav S. Sukhatme, Daniel Seita
Robotics
Distributed Cascade Force Control of Soft-Tactile-Based Multi-robot System for Object Transportation
Duy Anh Nguyen, Nhat Minh Dinh Le, Nhan Huu Nguyen et al. (+3 more)
In this paper, we present a distributed cascade force control system (DCFC) for multiple
robots with the aim of pushing a rigid object towards a desired moving target without their inter-
robot communication. These mobile
robots are equipped with 360-degree
vision-based soft tactile sensors utilized to determine contact location and resultant impact force. By investigating the dynamics of moving rigid objects on the flat, we proposed a distributed cascade control. The inner loop control incorporates contact force and positioning, ensuring the
robots' pushing contact and application of the desired force to the object. The outer loop control coordinates the
robots to push the object in a desired direction without inter-
robot communication, regardless of the unknown object mass. The stability and convergence of the system are verified using the Lyapunov stability theory. We also conducted simulation and real-word experiments to validate the performance of the proposed control method, and the experimental results showcase the successful coordination of multiple
robots in pushing an object towards a moving desired direction.
Read the full abstract ▾
All authors Duy Anh Nguyen, Nhat Minh Dinh Le, Nhan Huu Nguyen, Pham Duy Hung, Van Anh Ho, Trung Dung Ngo
Robotics
Learning Task and Motion Plans from Real Demonstrations with Hybrid Flow Matching
Zuleika Redondo Garcia, Andreu Matoses Gimenez and Javier Alonso-Mora
Long-horizon mobile manipulation requires a task plan and the motion that executes it. Generative planners trained on demonstration produce both in one pass, requiring neither a symbolic domain nor search. To date, however, they have relied on thousands of scripted demonstrations of fixed-base arms and executed open loop. This paper presents a hybrid flow matching planner: a single network generates the symbolic plan with masked discrete flow matching and the motion trajectory with continuous flow matching. Unlike prior generative planners, we aim to learn from a much smaller set of demonstrations and to execute the plan in closed loop. Two properties of the data compensate for the small
dataset. A demonstration resumed from any of its intermediate actions is itself a demonstration, which multiplies the training samples and enables replanning after every action. Objects of the same kind are interchangeable, which turns demonstrations of one goal into demonstrations of every permuted goal. Our base implementation produces valid plans on 68% of held-out scenes; a training and generation scheme for the discrete plan raises this to 76%, and replanning after every action raises the task completion rate from 40% to 53% in a kinematic simulation. The planner matches the task completion rate of motion-only flow matching policies while additionally providing the symbolic plan, and it outperforms previous hybrid
diffusion formulations on both task completion and plan validity. We validate the planner on a real mobile manipulator.
Videos and project page: https://andreumatoses.github.io/research/hybrid-flow-planning
Read the full abstract ▾
Robotics
Flow Policies as Actions of Skill-Level World Models: Learned and Symbolic Abstractions for Long-Horizon Planning
Andreu Matoses Gimenez, Andrei-Carlo Papuc, Chris Pek et al. (+1 more)
Latent world models enable
robots to plan by predicting the consequences of actions. Planning long tasks with control-rate actions requires many prediction steps, which enlarges the search space and accumulates error. Skill-level actions shorten these sequences, but a symbolic skill vocabulary requires domain knowledge and labeled demonstrations. We construct skill-level actions from the inputs of a flow-matching policy trained on demonstrations segmented into complete skills. The policy maps a noise seed and an observation, optionally with a code or label, to a complete skill execution, so one execution is one world-model transition. On this mechanism we propose four action abstractions with increasing task knowledge: a compressed seed, two discrete codes learned from the demonstrations, and a symbolic label. We evaluate them with a common world-model training procedure and planning framework on simulated block rearrangement tasks that require up to 14 sequential skills. The symbolic label succeeds in over 90% of the tasks that require up to six skills and degrades beyond. Without any label, an object-centric learned code matches it on single-skill tasks and retains half to three quarters of its success on tasks of two to five skills. Ablations attribute much of the label's advantage to its planner knowing which actions are applicable, rather than to the label itself. Beyond six skills the search, not the world model, limits success. Project page: https://andreumatoses.github.io/research/flow-skill-wm
Read the full abstract ▾
All authors Andreu Matoses Gimenez, Andrei-Carlo Papuc, Chris Pek, Javier Alonso-Mora
Robotics
PatternDex: Learning Interaction Patterns to Guide Reinforcement Learning of Bimanual Dexterous Manipulation of Articulated Objects
David Minkwan Kim, Runfa Blark Li, Beckham Po-Ju Lee et al. (+2 more)
In this paper, we develop a method that enables bimanual dexterous hands to manipulate articulated objects with a high success rate without suffering from an embodiment gap. We observe that the correlation between hand motions and object motions is dictated by the object rather than the hands and can be learned from human-object demonstrations. Based on this observation, we propose PatternDex, a method that learns this correlation and represents it as a
token sequence, which we call an interaction pattern. From this pattern, PatternDex estimates the wrist motions and contact points that fit the target
robot, and then trains a reinforcement learning policy that exploits these estimates as guidance. Since the guidance fits the target embodiment, the policy explores only the actions that the target
robot can execute and thus achieves high success rates. PatternDex also requires only simple fine-tuning to train a new
robot, since it can reuse the learned interaction pattern. We evaluate PatternDex with bimanual dexterous hands on human demonstrations from the ARCTIC
dataset. PatternDex achieves, on average, a 92.8% success rate with Allegro hands, while the state-of-the-art baseline achieves 52.2%. Also, it achieves success rates above 70% with three other
robot hands after fine-tuning alone. Furthermore, we verify that the learned policy transfers well to a real-world task of opening a microwave.
Videos and additional results are available at https://patterndex.github.io/PatternDex/
Read the full abstract ▾
All authors David Minkwan Kim, Runfa Blark Li, Beckham Po-Ju Lee, Nikolay Atanasov, Truong Nguyen
Robotics
DNPush: Distributed Nonparallel Pushing Force Control of Multi-Robot Systems for Object Transportation
Duy Anh Nguyen, Pham Duy Hung and Trung Dung Ngo
This letter proposes DNPush, a Distributed Nonparallel Pushing force controller for cooperative object transport without inter-
robot communication. Each
robot uses local force measurements, a preassigned force weight, and a locally expressed desired transport direction, without requiring an object model, contact locations, a CoM estimate, or moment arm magnitudes online. DNPush regulates nonparallel force magnitudes from local force direction misalignments through a reciprocal-sine law. Under the stated contact conditions, Lyapunov analysis with LaSalle's invariance principle proves convergence to a unique rotational
equilibrium, where perpendicular force contributions cancel and the net torque from normal forces is zero. Object velocity converges to the desired transport direction, and suitable contact regions exist for convex polygons. Simulations isolate the reciprocal-sine scaling mechanism, five-
robot coordination, and active rotational damping. Hardware experiments with soft tactile differential-drive
robots achieve a velocity direction error of 0.61 deg +/- 0.79 deg over five trials with a fixed transport direction and demonstrate object transport through waypoints with three
robots.
Read the full abstract ▾
Robotics
Lie Rarely, Lie Big: Stealthy Insider Attacks on LLM Robot Teams
Sribalaji C. Anand and George J. Pappas
When a team of
robots delegates planning and mutual trust to LLM
agents, a single compromised
robot can corrupt the shared outcome. We study this threat in a grounded task: a multi-
robot survey in which measurements can be verified against the physical world, but every verification costs budget that would otherwise advance the mission. We treat the compromised
robot as a stealthy adversary in the system-theoretic sense: it is limited not by an energy bound but by the team's own detectors. We then derive two bounds. First, the probability that the adversary's reports are verified is bounded below in terms of the degrees in the communication graph and the verification budget. Second, the map error caused by any stealthy adversary is bounded above by the value of a linear program over the adversary's bias distributions; its solution is an exchange rate between stealth budget and damage: below a critical verification level the worst stealthy attack tells rare, full-magnitude lies on the records least likely to be verified, and above it the better purchase is small biases hidden in the noise. In experiments where the honest
robots are LLM
agents, both bounds hold at the budget the attack actually spent. The experiments also show that which records an LLM
robot re-checks is unbiased, but how much it re-checks is unpredictable.
Read the full abstract ▾
Robotics
Robot Learning with Visual Predicted Force
Haonan Chen, Feiyang Wu, Yuxiang Ma et al. (+11 more)
Force-aware manipulation typically relies on specialized force or tactile sensors. We show that force-aware manipulation can instead be achieved through visual force prediction from the deformation of a compliant Fin Ray gripper. Our approach trains two models. First, we train a visual force
estimator on calibration data and use it to annotate task demonstrations with force estimates. Second, we train an action–force proposal policy on these force-augmented demonstrations to jointly generate candidate
robot actions and their associated forces. At test time, we sample candidate actions and the forces they are expected to produce, then execute the action whose predicted force is closest to a target from the demonstrations. We evaluate our approach on berry picking, empty-can grasping, in-hand reorientation, and plug insertion. Our results show that visual force prediction can guide inference-time action selection for contact-rich manipulation without requiring force or tactile sensors at deployment.
Read the full abstract ▾
All authors Haonan Chen, Feiyang Wu, Yuxiang Ma, Mustafa Mete, Pengfei Ye, Junxuan Shen, Cheng Zhu, Aurora Ruggeri, Kelvin Cheung, Jiayuan Mao, Edward Adelson, Jiajun Wu, Robert D. Howe, Yilun Du
Robotics · Cross-listed from cs.CV
Probabilistic Pedestrian Forecasts from a Handheld Phone: World-Frame Heat Maps, Visual-Inertial Height Drift, and Evaluation without Ground Truth
Danial Safaei
A pedestrian with a phone could be shown where nearby people will be in the next few seconds, if the forecast stays on the ground while the phone moves, is calibrated, and needs only a monocular camera and visual-inertial odometry (VIO). We build and evaluate such a system. People are detected, lifted onto the floor by ray-plane intersection, tracked in a
gravity-aligned metric frame, and forecast as per-step probability maps by a small U-Net trained with a negative log-likelihood (NLL) loss on bird's-eye (SDD) and first-person (EgoTraj-Bench) trajectories. On handheld ADVIO recordings, vertical VIO drift and the user's own changes of level silently rescale monocular ground positions (by 87% within 90 s on one clip; on another, all tracks are lost for the last 31% of the clip); keeping the camera's height above the floor constant under a low-pass-filtered altitude avoids this, though it lags on escalators. On the SDD and EgoTraj-Bench test splits, the final forecaster lowers the NLL at 4.8 s by 1.51 and 1.37 nats relative to a fitted constant-velocity Gaussian. Lacking ground truth for people in handheld
video, we score forecasts against the tracker's own later raw measurements. In an internally pre-registered evaluation on seven held-out clips, the final forecaster's NLL is lower than the
benchmark-fitted baseline's at 1.2, 2.4 and 4.8 s (by 0.17, 0.23 and 0.47 nats; 95% intervals over people exclude zero), but by less than half as much as on the development clips. Exploratory analyses cut both ways: resampling clips instead of people widens the intervals to include zero at 1.2 and 2.4 s, and once both forecasters are recalibrated on the development clips the network is significantly better only at 1.2 s; but two clips run with ADVIO's reference poses favour the network much more when re-run with the phone's own poses. We discuss what such self-consistency scores can and cannot show.
Read the full abstract ▾
Robotics
Emergent Underactuation in Fully Actuated Multirotors by Minimizing a Homogeneous Allocation Cost
Antonio Franchi and Barbara Bazzana
Full actuation is commonly valued for independent position and attitude tracking. This work establishes a complementary principle: when only a position trajectory is prescribed, minimizing a positively homogeneous allocation cost selects underactuated-like behavior on a fully actuated multirotor. This cost class includes norms and power laws, including standard minimum-norm pseudoinverse allocation. We use fiberwise infimization to transport the actuator-output cost to wrench space, yielding a homogeneous infimal allocation cost. Restriction to unit zero-moment forces defines a directional landscape whose minimizing directions are independent of force magnitude. We show that aligning any fixed minimizing direction with the required world-frame force minimizes instantaneous and integrated zero-moment allocation costs. The resulting body force remains in a fixed one-dimensional subspace without transverse components. We derive an exact optimality-gap decomposition quantifying the additional cost of every other attitude trajectory and characterizing the minimum-cost class. When a disturbance or prescribed-motion change modifies the required-force direction, a fully actuated platform can immediately generate the modified force using transverse authority and reorient toward the new minimum-cost family. A structurally underactuated platform must reorient before generating that force. Thus, homogeneous allocation-cost minimization explains why optimal nominal motion can be underactuated-like, while full actuation remains valuable for disturbance rejection and transient force generation.
Read the full abstract ▾
Robotics
Test-Time Training as Residual Memory for Robot Policies
Haoxuan Wang, Gengyu Zhang, Ramana Rao Kompella et al. (+2 more)
Memory is essential for long-horizon robotic manipulation, where successful actions may depend on past events that are no longer recoverable from the current observation. As episodes grow longer, however, retaining the full history becomes increasingly costly, creating a fundamental scalability challenge for
memory-augmented policies. Existing approaches address this challenge by either storing selected past observations in a bounded
memory bank or compressing interaction history into a fixed-size parametric state through Test-Time Training (TTT). Yet these formulations do not explicitly distinguish between historical information that can already be recovered from the policy's current context and information that must persist beyond it. We introduce TTT-RM, which repurposes TTT as Residual
Memory, using fast weights not to generically compress history but to complement a bounded
memory bank by preserving task-relevant historical information that cannot be recovered from the policy's current context. Concretely, TTT-RM learns a history decoder that reconstructs historical representations from the current observation and retrieved
memory. The resulting reconstruction residual captures what this context fails to explain and serves as the learning target for TTT. The TTT slow weights are optimized against this residual target so that online fast-weight updates learn to encode complementary historical information over time. The fast-weight state is then queried to produce a residual
memory representation that conditions action generation. Extensive experiments on
memory-intensive simulation
benchmarks and real-world tasks show that TTT-RM consistently improves across multiple
memory designs, outperforms diverse baselines, and supports sustained execution on a three-minute, eight-stage stowing task.
Read the full abstract ▾
All authors Haoxuan Wang, Gengyu Zhang, Ramana Rao Kompella, Gaowen Liu, Yan Yan
Robotics
RoboIRS: Inference-Time Internal Representation Steering for Generalist Robot Policies
Jiuzhou Lei, Chang Liu, Dayou Li et al. (+5 more)
Vision-
language-action (VLA) and world-action models (WAMs) often degrade under out-of-distribution task variations despite retaining partial task capability. To recover such capability, we propose RoboIRS, an inference-time internal representation steering method that uses successful and failed rollouts to train linear classifiers, select outcome-relevant intervention locations, and derive task-specific steering directions without updating policy parameters. On 15 simulation tasks with a frozen $\pi 0.5$ policy, RoboIRS improves the average success rate from 44.4% to 66.2%, outperforming alternative inference-time intervention baselines while adding little inference time. We further validate RoboIRS on real-
robot manipulation using the same $\pi 0.5$ policy and demonstrate its applicability to a world-action model Cosmos Policy, where the average success rate improves from 35.4% to 55.4%. These results show that directly steering internal
robot-policy representations can improve the performance of
robot policies at inference time. Project website is available at https://rollingoat.github.io/roboirs/.
Read the full abstract ▾
All authors Jiuzhou Lei, Chang Liu, Dayou Li, Zhiyuan Zhang, Xiao Liang, Yu She, Zhiwen Fan, Minghui Zheng
Robotics · Cross-listed from eess.SY
Differentiable Ternary Temporal Logic Semantics with Polynomial Surrogate Networks
Ryan Matheu, Sai Sandeep Damera, John S. Baras et al. (+1 more)
Temporal logic provides designers a formal
reasoning tool for specifying complex spatio-temporal behaviors and tasks for autonomous systems. Many techniques exist for control synthesis according to temporal logic specifications, spanning a large variety of systems, including robotics. However, the vast majority of implementations are limited to Boolean temporal logics. Boolean temporal logic does not have an innate mechanism for quantifying abstention, rather verdicts are either definitively True or False. Recent work demonstrates that ternary logic is a capable formalism for
reasoning about robotic behaviors, with the natural ability to classify uncertainty with the Unknown literal. Ternary temporal logic is still in its infancy, with work limited to synthesis for linear systems and monitoring. This work proposes a probabilistic relaxation of ternary temporal logic that allows for value and gradient computation through the same network while retaining exact verdicts on actual data. The relaxed network can be used as an objective in gradient-based control synthesis for both offline and online robotic control applications. We demonstrate the utility of our framework with two robotic manipulator tasks, one offline and one online, involving intermediate checking and sequencing of subtasks to demonstrate the benefit of our approach in terms of correct and timely execution.
Read the full abstract ▾
All authors Ryan Matheu, Sai Sandeep Damera, John S. Baras, Calin Belta
Robotics · Cross-listed from cs.AI
PermVLA: Factorization Order as a Regularizer for VLA Learning
Yanqiao Chen, Yuhan Rui, Dongsheng Hou et al. (+3 more)
Vision-
language-action (VLA) policies commonly learn action chunks through a fixed left-to-right (LTR) factorization, although the same expert trajectory distribution admits many valid chain-rule factorizations. We identify factorization order as an overlooked regularization choice and introduce causally anchored permutation (CAP), which samples action reveal orders with a tunable chronological prefix. Its auxiliary objective trains one shared policy to predict actions from different known subsets of the same expert chunk, while deployment retains deterministic LTR control. We call this conditional-set augmentation: it creates multiple conditional prediction problems from one expert chunk without adding demonstrations. This discourages reliance on the single chronological prefix used by ordinary teacher forcing. Controlled experiments show that CAP consistently outperforms standard LTR training on LIBERO and LIBERO-Plus, with the same advantage appearing in cross-
dataset CALVIN evaluation. A diagnostic that measures the expected squared difference between a chunk's joint log likelihood under two reveal orders verifies that CAP training internalizes agreement across reveal orders. These findings position sampled subset-conditioned auxiliary objectives as a general recipe for constructing VLA regularizers, illustrated by an extension to
diffusion action generators.
Read the full abstract ▾
All authors Yanqiao Chen, Yuhan Rui, Dongsheng Hou, Zijie Nie, Yutong Wan, Qi Hao
Robotics
PerturBot: Breaking Shortcut Priors in Vision-Language-Action Models with Perturbative Training
Mingyu Liu, Chonghao Sima, Tianjian Feng et al. (+4 more)
A
vision–
language–action (VLA) policy can complete complex tasks while ignoring the evidence that should determine its actions. An object held near the wrist camera can displace the instructed target.
Language and action show the same pattern: a familiar noun can trigger the operation it was paired with in training even after the verb changes, and a gripper that closed on nothing may lift anyway. We call these dependencies modality shortcuts: regularities in successful demonstrations make visual, lexical, or motor cues sufficient to predict expert actions without the task evidence needed for the underlying decision. More demonstrations of the same kind can raise task success while leaving these shortcuts intact. We propose Perturbot which makes task-relevant evidence easier to use and shortcuts insufficient on their own: it applies task-preserving wrist-view perturbations, enriches instructions with decision-relevant captions, and adds random and failed trajectory segments relabeled with the behavior they contain. It complements scaling by changing what is scaled, and leaves inference unchanged. Moreover, we propose GroundingFscore, an offline score that diagnoses how severely a policy relies on modality shortcuts. Task success rate shows whether a policy improves, while GroundingFscore reveals whether the policy scales healthily, relying on task evidence rather than shortcuts. Together, Perturbot and GroundingFscore provide a training-and-evaluation framework for disentangling VLA decisions from shortcut priors while preserving responsiveness to task-relevant evidence.
Read the full abstract ▾
All authors Mingyu Liu, Chonghao Sima, Tianjian Feng, Hanqing Wang, Cong Chen, Hao Chen, Chunhua Shen
Robotics
An End-to-End Framework for Modelling Pneumatic Soft Robots Based on Differentiable Finite Element Methods
Shaohong Zhong, Yao Yao, Perla Maiolino et al. (+1 more)
Soft
robots present significant modelling challenges due to their non-linearity, complex dynamics and potentially intricate geometries. These difficulties in accurate system identification and dynamics modelling limit their applications in precise robotics tasks. Prior modelling approaches typically suffer from trade-offs in accuracy, computational efficiency, or speed. The differentiable finite element method (FEM) offers a promising balance between these desiderata for soft
robot modelling, and enables the use of gradients for efficient calibration and trajectory optimisation. In this paper, we propose an end-to-end differentiable FEM-based framework designed to streamline modelling and system identification for pneumatic soft
robots, enabling the generation of reliable models for downstream tasks. The framework automates the conversion of computer-aided designs into voxelised tetrahedral viscoelastic FEM meshes that accurately represent the
robot's geometry and actuation mechanism. By integrating easily acquired point cloud data with differentiable FEM, the framework achieves precise identification of material and dynamic actuation parameters using a minimal experimental setup. Additionally, the differentiable nature of the model facilitates trajectory optimisation by leveraging gradients from
robot dynamics and contact interactions. We validate the framework by modelling a complex bellow-shaped pneumatic soft
robot and demonstrate its efficacy in real-world motion planning tasks. Experimental results indicate high modelling accuracy, with a maximum positional error of less than 3 mm, and successful application in tasks such as path following and grasping.
Read the full abstract ▾
All authors Shaohong Zhong, Yao Yao, Perla Maiolino, Ingmar Posner
Robotics
Exploiting Hierarchical Controller Structure in Contextual Parameter Learning for Humanoid Loco-Manipulation
Sebastian Hirt, Lukas Theiner, Jan Peters et al. (+1 more)
Hierarchical control architectures are widely used to decompose complex control problems into interacting control levels and are particularly important in robotics, where planning, whole-body motion, and lower-level control must be coordinated across different levels of abstraction and time scales. Their overall closed-loop performance, however, depends strongly on parameters distributed across the hierarchy, such that tuning controllers on different levels independently may neglect relevant cross-layer interactions. We propose a contextual
Bayesian optimization framework for joint parameter learning in hierarchical control systems. Rather than modeling closed-loop performance only as a scalar black-box function, we retain separate observations of task performance, realization quality, and control effort. A correlated multi-output Gaussian process models these performance components, while their known aggregation into the overall closed-loop objective is evaluated analytically. The formulation exploits three complementary consequences of hierarchical control: informative performance quantities exposed by the hierarchy, coupling between parameters of different controller levels, and variations of these relations with operating conditions. We evaluate the approach for humanoid loco-manipulation, jointly tuning a centroidal predictive controller and a whole-body controller for physical box pushing under varying box mass. The proposed method achieves the lowest mean empirical regret during both training and adaptation among the considered baselines.
Read the full abstract ▾
All authors Sebastian Hirt, Lukas Theiner, Jan Peters, Rolf Findeisen
Robotics
ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies
Zhe Tao, Feiran Wang, Gaowen Liu et al. (+2 more)
Robots need to anticipate how their actions will change the world, since manipulation success hinges on the resulting contacts and object motions. However, existing
Vision-
Language-Action (VLA) policies that predict future observations from shared features leave the forecast decoupled from the actions the policy will actually execute, and impose no physical constraints on how the scene may evolve. We introduce ForeAct3D, a framework for policy-grounded future world modeling within VLA policies. Learnable
geometric queries decode depth, semantic segmentation, and camera pose from the policy representation into current and future semantic 3D scene states, and the future queries are conditioned on the policy-generated action chunk to ground the forecast in the planned interaction. A physical-consistency closure relates the two states through background staticity and instance-level rigidity, and anchors the wrist-camera pose to end-effector kinematics. These objectives shape the shared representation used for action generation during training, and no future prediction is required at inference. Without
robot pretraining, ForeAct3D achieves 98.3% average success on LIBERO and an average task length of 3.73 on CALVIN, outperforming its base policy on every suite. Ablations show that semantic 3D supervision, physical consistency, and action conditioning each improve manipulation performance, and that action conditioning substantially improves future object localization. Real-world experiments on spatial placement, object insertion, and sequential manipulation further raise average success from 6.7% to 37.8% over the base policy. The project page and code are available at https://github.com/anthonytao80-crypto/ForeAct3D.
Read the full abstract ▾
All authors Zhe Tao, Feiran Wang, Gaowen Liu, Ramana Rao Kompella, Yan Yan
Robotics
RiskFly: Frustum-Aligned Spatio-Temporal Risk Fields for One-Stage Agile Flight in Dynamic Clutter
Luxia Ai, Haopeng Chen, Yuchao Mei et al. (+2 more)
Agile flight in unknown, cluttered, and dynamic environments requires a planner that knows where and when danger will appear, not only that a trajectory is dangerous. One-stage learning-based planners trained with differentiable privileged costs are fast and expert-free, but the only signal reaching their encoder is a scalar trajectory cost with no spatial or temporal structure, so avoidance degrades into late reactive maneuvers. We present RiskFly, a one-stage planner that predicts risk in the same space in which it acts. A dual-stream observation pairs a short depth sequence with a frustum-aligned inverted spherical range-map sequence, whose angular
cells match the end-state proposals one to one. An auxiliary head regresses a frustum-aligned spatio-temporal risk field, supervised by a privileged closest-point-of-approach (CPA) target. This self-predicted field is queried differentiably along the instantiated quintic trajectory at its own arrival times, and also enters the training objective, so representation supervision and planning gradients meet in a single space. Privileged signals are discarded at deployment, and the planner runs map-free from onboard depth and proprioception. Extensive simulation and zero-shot real-world flights on resource-constrained platforms show higher success rates at comparable end-to-end latency.
Read the full abstract ▾
All authors Luxia Ai, Haopeng Chen, Yuchao Mei, Guohao Zhang, Wenbing Tao