Robotics
Physical SLAM: Physics-aware Simultaneous Localisation and Mapping
Guangming Wang, Zhishuo Liu, Haibing Wu et al. (+4 more)
To reconstruct a predictive 4D representation of the physical world from monocular
video captured by a moving camera, we jointly reconstruct dynamic Gaussian radiance fields and infer their underlying physical dynamics. This paper proposes Physical SLAM, a physics-aware method for constructing 4D Gaussian fields of deformable objects and predicting their future state when subjected to external actions, given a sequence of RGB-D images. Our method separates the scene into a static background Gaussian field for camera localization and a dynamic object Gaussian field driven by a spring–mass physical model of elastic deformation, inertia, and damping. To integrate visual observations with physical dynamics, we introduce virtual observation forces that gradually pull simulated surface nodes toward observed depth points, avoiding abrupt replacement of simulated node positions. Meanwhile, the physical parameters of the spring–mass model are updated from simulation errors aggregated over the full reconstruction observation window, enabling the reconstructed dynamics to align with observations. To couple appearance, geometry, and physics, we further design a physics-aware Gaussian transformation and surface
alignment strategy, where dynamic Gaussians are transformed along the deforming physical surface of the spring–mass model and constrained to remain aligned with the surface during rendering-based optimization. The resulting representation supports observed-frame rendering and action-conditioned simulation, extending SLAM maps to describe how deformable objects respond to manipulation. On the PhysTwin
benchmark, our method improves reconstruction and prediction PSNR over PhysTwin by 10.4% and 11.0%, respectively. On the TUM RGB-D
benchmark, it reduces average camera tracking error by 20.6%, improves SSIM by 3.6%, and reduces LPIPS by 8.3% compared with 4DGS-SLAM.
Read the full abstract ▾
All authors Guangming Wang, Zhishuo Liu, Haibing Wu, Wanli Ma, Yixiong Jing, Olaf Wysocki, Brian Sheil
Robotics
Value-Preserving Action Quantization
Max Muchen Sun, Jueun Kwon, Benjamin Riviere et al. (+1 more)
Search-based policy learning, such as Monte Carlo tree search (MCTS), offers the flexibility to trade runtime computation for data efficiency and adaptation to novel environments. For
robot learning, however, applying search requires discretizing the continuous action space and associated dynamics, typically through action quantization. While reconstruction-based quantization is sufficient for representing the continuous action space alone, policy learning additionally requires the quantized actions to preserve their effects on future states and values. We show that such requirement is not necessarily captured by action reconstruction error. Instead, we prove that the error in the value function resulting from action quantization is tightly bounded by the reconstruction error of state reachability under the quantized actions. Based on these results, we propose value-preserving action quantization (VAQ), which quantizes actions by preserving state reachability. We further combine VAQ with MCTS (VAQ-MCTS) as a value-preserving, search-based policy learning framework for behavioral cloning. Using a wide range of object pushing tasks as a controlled
benchmark, we evaluate VAQ-MCTS against state-of-the-art behavioral cloning methods. VAQ-MCTS matches or exceeds their performance under standard settings, while substantially improving data efficiency and robustness to heterogeneous demonstrations, while enabling zero-shot adaptation to novel environments. Hardware experiments further demonstrate the effectiveness of VAQ-MCTS, completing all 30 real-world object pushing trials with only 10 minutes of demonstration data.
Read the full abstract ▾
All authors Max Muchen Sun, Jueun Kwon, Benjamin Riviere, Todd Murphey
Robotics
LiDAR-based Humanoid Navigation in Dynamic, Non-Convex Environments via Spatio-Temporal Safe Corridors
Shubham S. Kumbhar and Panagiotis Artemiadis
Navigating humanoid
robots through dynamic, non-convex environments requires tightly coupling onboard perception with footstep planning and dynamic locomotion control. Existing methods frequently discard complex free space through simplified convex obstacle bounds and fail to account for real-time dynamic occupancy predictions. This paper presents a real-time framework for humanoid navigation that directly constructs spatio-temporal safe corridors from
LiDAR-based particle representations and couples them with Linear Inverted Pendulum (LIP)-based planning and Whole Body Control (WBC) to enable navigation around and within moving non-convex structures without restrictive
geometric assumptions. Simulations on the Digit humanoid yield a 96% success rate in navigating around and inside moving non-convex obstacles in a 24 by 24 m^2 arena. Ablation studies confirm that the proposed iterative polytope generation significantly outperforms single-iteration baselines in narrow corridors, while evaluations demonstrate robustness to clutter, sensor noise, control delays, and force perturbations.
Read the full abstract ▾
Robotics
DexNinja: Learning Robust Dexterous Cutting Policy with a Real-to-Sim-to-Real Data Engine
Haozhe Lou, Runyi Yang, Wanqi Zhong et al. (+10 more)
Cutting a piece of food looks effortless in human hands. For a
robot, however, every cut is a high-contact interaction with a deformable, elasto-plastic object whose behavior changes with pose, contact mode, and cutting stage. Collecting real-world data is expensive and wasteful: each trial typically contaminates or damages the food, so obtaining scalable real world data for dexterous cutting is expensive and difficult. Simulation is an appealing alternative, but a large sim-to-real gap exists in both physics and perception. This gap is especially severe for
topology-changing interactions like cutting, where small errors in contact, friction, or material and elasticity responses quickly compound. We propose DexNinja, a real2sim2real framework that turns a handful of real demonstrations into a dense and realistic training distribution. DexNinja (i) reconstructs object instances from real captures, (ii) randomizes physically meaningful parameters under category-level constraints, and (iii) produces augmented dexterous manipulation episodes using a custom differentiable simulator that couples
robot dynamics, tactile contact, and deformable cutting. We evaluate DexNinja on a dexterous food-slicing task and show that (1) with tens of real demonstrations and hundreds of augmented simulator data, the success rate could reach more than 60%. (2) the augmented data consistently improves success rates when only using real data with more than 20% via sim-to-real, and (3) the resulting policies generalize to out-of-distribution objects with varying shapes and sizes.
Read the full abstract ▾
All authors Haozhe Lou, Runyi Yang, Wanqi Zhong, Caiyun Liu, Yicheng Liu, Wenhao Liu, Wenlong Ma, Jiawei Xia, Kanzhi Wu, Danda Pani Paudel, Luc Van Gool, Hang Zhao, Yiming Li
Robotics
Generative Flow on Symbolic Abstractions for Embodiment Capability Discovery
Youwei Yu and Lantao Liu
Achieving complex, long-horizon
robot behavior remains challenging because successful execution depends on skills grounded in a
robot's physical affordances. Existing training-task distributions are often handcrafted or sampled from large
language models (LLMs), relying on semantic plausibility rather than embodiment-specific feasibility. This mismatch can produce curricula that are too difficult, redundant, or blind to non-obvious but learnable behaviors. We propose Generative Flow on Symbolic Abstractions (FOSA), a framework for discovering embodiment-specific capabilities by learning a grounded task generator. FOSA parameterizes an abstract policy as a Generative Flow Network that constructs symbolic task skeletons. Each skeleton is compiled into a simulation environment, where low-level policy-learning performance provides the reward signal for the abstract policy. Across heterogeneous
robot platforms, FOSA discovers diverse and executable tasks near the
robot's empirical capability frontier. After distilling the discovered task policies into a single
language-conditioned policy, FOSA improves performance on unseen complex and long-horizon tasks. The code and demonstration
videos are available at https://www.youwei-yu.com/fosa.
Read the full abstract ▾
Robotics
Force-Modulated Dexterity: Unified Compliance for Contact-Rich Manipulation
Ying Yuan, David Held, Jitendra Malik et al. (+1 more)
Teleoperation is widely used to collect high-quality demonstrations for
robot learning. Dexterous manipulation, however, requires coordinated control of motion and contact forces. Operators can specify motion, but struggle to reproduce the subtle force adjustments of human manipulation without feeling the
robot's contacts. We introduce Force-Modulated Dexterity (FOMO-Dex), a unified compliance formulation for contact-rich manipulation. FOMO-Dex expresses nominal actions in coordinates defined by task geometry and contact structure. It applies force-driven motion corrections along selected directions while leaving the remaining command components unchanged. Across card picking, paper cutting, screwdriving, and nut-and-bolt assembly, FOMO-Dex reduces excessive contact forces compared with stiff control. A user study shows that operators collect more successful demonstration data per hour after brief practice. For policy learning, imitating the corrected actions improves card-picking success over imitating raw operator commands. Policies trained on FOMO-Dex demonstrations also achieve more rotation in nut-and-bolt manipulation and screwdriving than those trained on demonstrations collected with the stiff controller.
Videos are shown on https://yingyuan0414.github.io/fomo-dex/ .
Read the full abstract ▾
All authors Ying Yuan, David Held, Jitendra Malik, Haozhi Qi
Robotics
D-World: An Explicit World-Language-Action Model with Recursive Self-Improvement under Dual Control
Xuan Wen and Baichao Wang
Frontier large
language models (LLMs) now reason about space well enough to program
robots, yet the world models behind
robot policies are implicit, held in network weights or predicted
video, and cannot be inspected or corrected directly. We present D-World, an explicit world-
language-action model built on the Sense, Plan and Act decomposition. A perception
agent writes the scene as an executable digital twin using generated
vision and geometry code. A planning
agent searches for actions in simulation and, after each executed action chunk, revises the twin's object states and named physical parameters against what the
robot measured. A parameter revision is accepted only if replaying the latest chunk does not increase any measured residual beyond its tolerance, and every chunk is checked in the updated twin before it runs. Structural mismatches return to perception for reconstruction. We call this measurement-grounded cycle of correction and revision recursive self-improvement (RSI), in which the LLM stays frozen while the world it acts in improves. We model the choice of actions that both advance the task and expose model error as a dual control problem and show that the cost of incorporating evidence into the twin sets the threshold at which probing pays, so cheap local revision counteracts the turn-off of cautious controllers. Each episode runs a real-to-sim-to-real loop, and its recorded twins can be refitted and re-rendered offline into higher-fidelity environments for post-training. Because perception and planning communicate only through the twin and
language, either can be replaced by a learned model, and since inference dominates the loop, the loop speeds up with faster models. On a dual-arm
robot, towel and toaster tasks show local revisions, such as one reducing the bread's rendered-depth residual from 22.4 to 1.5 mm, that let execution continue without reconstruction.
Read the full abstract ▾
Robotics
Interactive Navigation Beyond Reachability: Benchmark and Agentic Framework
Delun Li, Xueyao Wei, Rui Chen et al. (+4 more)
Existing embodied navigation tasks commonly assume that traversability and object states remain fixed during an episode. Yet reaching a goal may require opening a door to access a room and then opening a container to reveal a hidden target. We formulate interactive navigation beyond reachability, unifying channel-induced changes in spatial connectivity, container-induced changes in target visibility, and their sequential dependencies. We introduce MultINav-Bench, a 3K-episode ObjectGoal
benchmark in MolmoSpaces spanning channel, container, and mixed tasks, with interaction-necessity annotations covering required and unnecessary episodes. Its data generation pipeline automates task construction across multiple task types. The evaluation protocol separately measures navigation success, interaction effectiveness, and cost. We further propose MultINav-
Agent, a three-stage agentic framework built on a persistent, state-conditioned interaction graph. Semantic interactive perception grounds entities and interaction states; a
vision-
language model selects task-relevant actions based on expected changes in connectivity and target observability; and post-interaction verification updates the graph to guide subsequent navigation, exploration, or interaction. Experiments on MultINav-Bench show that MultINav-
Agent consistently outperforms adapted baselines across channel, container, and mixed tasks.
Read the full abstract ▾
All authors Delun Li, Xueyao Wei, Rui Chen, Yifei Wang, Junzhi Yu, Wenjie Song, Xingyu Chen
Robotics
Declarative Reactive Robot Behaviors from Imperative Language Models
Vincent Berenz, Tasha Kim and Bernhard Schölkopf
Large
Language Models (LLMs) are predominantly trained on imperative programming
languages, and existing work on LLM-based
robot programming reflects this: it focuses on goal-directed tasks expressed as plans. However, many robotic applications require not tasks but behaviors–open-ended, continuous patterns of sensory-motor engagement with no termination condition (e.g., greeting visitors, patrolling an area). Such behaviors are better served by declarative and reactive programming, on which LLMs have virtually no training. We investigate whether this gap can be bridged. Using Playful, a reactive scripting
language, we show that an LLM-based coding
agent can generate correct declarative and reactive programs for robotic behaviors. These programs are produced from under-specified natural
language descriptions, given suitable context and a set of reusable programming guidelines. The guidelines encode design principles that redirect the
agent from imperative defaults toward proper reactive patterns. Developed once by an expert, they generalize across
robots and behaviors. Starting from only the generic
robot operating system API, the
agent jointly designs both the imperative component (Python modules for sensing, acting, and evaluating) and the declarative component (the reactive script that composes them into behavior logic). The declarative nature of reactive programming supports both incremental refinement through conversational turns and one-shot generation of complete behaviors.
Read the full abstract ▾
Robotics
Task-Space Motion-Force Nonlinear Model Predictive Control for Precise and Compliant Control under Uncertainties
Quang-Nam Nguyen and Ludovic Righetti
In Nonlinear Model Predictive Control (NMPC), the control law is implicitly determined by the solution of an Optimal Control Problem (OCP). Consequently, OCP design is often tailored to specific platforms and tasks, demanding extensive expert knowledge and tuning effort. To address this, we propose a unified NMPC framework that directly plans and controls the
robot joints subject to task-space motion and force requirements and is capable of seamlessly generalizing across multiple robotic platforms and diverse tasks. Furthermore, optimization-based methods often treat interactions as rigid contacts, making the torque solution sensitive to the modeling errors of the
robot dynamics and the environment. We study the intrinsic task-space impedance of our proposed controller, its partial contraction for differential stability and differential passivity under unmodeled interactions. As a result, our framework enables closing the force-feedback loop in the high-level planning layer to achieve precise motion-force tracking performance, addressing the long-standing challenge of force-feedback NMPC. Finally, we validate our framework on physical hardware, demonstrating hybrid motion-force control on a redundant manipulator and compliant locomotion on a quadruped, while preserving core NMPC advantages such as hard constraint satisfaction, optimal redundancy exploitation, and real-time planning under unforeseen interactions.
Read the full abstract ▾
Robotics · Cross-listed from eess.SY
Multi-Agent Deferred-Decision Trajectory Optimization with Anytime-Valid Target Retention for Moving Candidates
Farbod Siahkali, Mengxue Hou and Vijay Gupta
We study multi-
agent terminal interception when one of several moving candidates is the unknown target and both target identity and candidate terminal locations are uncertain. We develop a receding-horizon deferred-decision trajectory optimization (DDTO) controller that combines conformal terminal-location prediction, sequential candidate rejection, capacity-constrained assignment, and candidate-contingent planning. Under exchangeability conditions for terminal and identity residuals, with predictable querying, we establish fixed-time terminal-location coverage and anytime-valid target retention. At the final replan, when one control step remains, we bound the joint probability that a tightened-set DDTO plan passes nonlinear rollout verification but capture fails. The closed-loop controller applies fallback guidance to any
agent without a verified DDTO plan and, across 100 paired scenarios, captures 79% of targets under the strict control-effort budget and 85% when that budget is relaxed.
Read the full abstract ▾
Robotics
MOSAIC: Mode- and Support-Aware Inference-Time Control of Robot Diffusion Models
Sabrina Bodmer, Rahel Rickenbach, René Zurbrügg et al. (+2 more)
Generative policies such as
diffusion models provide expressive data-driven priors for
robot behavior, yet the situations a
robot faces at execution time rarely match its training data and impose constraints and objectives the policy never learned. Existing inference-time guidance methods either rely on gradients that can push samples away from the learned prior, or on sampling procedures that can be inefficient, collapse modes, or average them into out-of-distribution predictions. We introduce MOSAIC, a mode- and support-aware framework for gradient-free inference-time control of robotic
diffusion models. MOSAIC treats the initial noise and all
stochastic reverse-step perturbations as a joint input to the pretrained denoising process and adapts a distribution over this input using importance weights computed from fully denoised outputs. Physical feasibility and consistency with the learned support are balanced using adaptive penalties, while mode-wise importance-weighted updates preserve modes and avoid destructive averaging. The same formulation can jointly steer multiple independently trained
diffusion models through shared coupling costs, enabling coordinated behavior without requiring a joint model. We evaluate MOSAIC on navigation, dexterous grasping, motion planning with unseen geometry, and collaborative box manipulation with independently trained
robot priors. Across these settings, MOSAIC adapts and composes frozen
diffusion priors under new deployment constraints without retraining.
Read the full abstract ▾
All authors Sabrina Bodmer, Rahel Rickenbach, René Zurbrügg, Melanie N. Zeilinger, Colin Jones
Robotics
Joint-Feasibility Margin Feedback for Bounded-Input Second-Order Control Barrier Functions
Siyuan Yang and Shaoming He
Multiple second-order control barrier function constraints can exhaust their common control authority under bounded acceleration, even when the current state remains feasible. We define a signed local joint-feasibility margin whose sign exactly identifies whether the current barrier half-spaces intersect the admissible input set. For unit-scaled distance constraints, it has acceleration units and a support-function dual that identifies weighted constraint conflicts. We predict the margin one sample ahead for nine task-directed candidates, selecting either the least-deviating candidate that attains a desired margin or the candidate with the largest predicted margin before solving the same hard HOCBF quadratic program. A controlled one-step case shows that, at the state predicted under the hard-QP command, two constraints remain individually feasible but become jointly infeasible, whereas the margin-feedback controller retains a positive next margin. Separately, across 20 paired initial conditions, the margin-feedback controller retains the hard-QP command in a low-pressure task and acts at positive margin in a critical task before the hard-QP problem becomes infeasible; its trajectories remain feasible at that time and complete the task. Source code is publicly available. https://github.com/xirhxq/cbf/tree/robio2026-preventive-regulation
Read the full abstract ▾
Robotics
UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation
Lei Yao, Qiao Feng, Mutian Tong et al. (+3 more)
Latent Action Models (LAMs) can capture manipulation priors from action-free
videos, but discrete latents usually impose a granularity bottleneck, limiting their ability to represent fine-grained action. Recent continuous alternatives mitigate this, but they are typically used as proxies for world modeling and require a further planning step to produce executable actions. We thus introduce UniLAP, a unified continuous latent-action policy that models the latent space via flow matching and bypasses reliance on future frames during inference. By distilling a LAM teacher with a flow-adaptive objective, UniLAP inherits the teacher's action representation while retaining the
multimodal distribution of feasible behaviors. To decouple latent representations from task-irrelevant appearance, we extract a label-free motion target to isolate relevant kinematic cues from spurious visual dynamics. With the established embodiment-agnostic latent
manifold, UniLAP can be easily adapted to novel platforms. Pretrained on multi-source
videos at a compact 1.1B parameter scale, UniLAP outperforms state-of-the-art latent-action and
Vision-
Language-Action (VLA) baselines across hand pose forecasting, single-arm and humanoid manipulation, and real-world deployment.
Read the full abstract ▾
All authors Lei Yao, Qiao Feng, Mutian Tong, Qihao Yun, Lap-Pui Chau, Lingjie Liu
Robotics · Cross-listed from cs.CV
Proactive Physical AI through Continuous Observation and Timely Intervention
Zhenhua Yang, Jiafeng Liang, Xingyu Zhou et al. (+26 more)
Physical AI is commonly formulated around an explicitly specified task: given a goal or instruction, a system perceives, reasons, and acts. In many real-world settings, however, important tasks emerge from continuously changing world states before a person explicitly formulates them. We study this complementary problem as Proactive Physical AI, where an AI system continuously observes the physical world, identifies situations that warrant intervention, and initiates appropriate actions at the right time. We develop ProAgent, an end-to-end system built around two components: a Jev-style proactive timing model (dubbed StreamingJev) that determines whether and when intervention is needed, and a Response Model that generates responses for people or invokes available device capabilities for physical execution. To support learning at scale, we develop an automated data pipeline that converts
videos into supervision for proactive timing. Experiments demonstrate strong proactive performance under both explicit objectives and standing roles, combining more accurate intervention timing with fewer false triggers than the source-backbone baseline. Real-
robot evaluations further demonstrate ProAgent's ability to identify emerging needs and translate them into appropriate physical actions.
Read the full abstract ▾
All authors Zhenhua Yang, Jiafeng Liang, Xingyu Zhou, Yi Jiang, Xiayu Cao, Zui Chen, Luqi Wang, Haiyang Sun, Yufei Xiang, Chang Li, Rongchuan Mu, Bowen Chi, Zihe Tong, Mingyu Li, Huijie Ma, Yiqiao Yang, Baoqi Ren, Zhiming Ma, Zhihao Zhu, Jiahao Ji, Zhuoyao Wang, Qianyu Wang, Xiangyu Liu, Wang Zixiang, Shunliang Zhang, Ming Liu, Xiaocheng Feng, Duyu Tang, Bing Qin
Robotics · Cross-listed from cs.MA
DiTSG: Context-Aware Air Traffic Scene Generation with Multi-Agent Diffusion Transformers
Seokbin Yoon, Kyungmin Kim, Taejung Kwon et al. (+2 more)
Understanding how many aircraft, and which traffic configurations, can be safely and efficiently managed by air traffic controllers within a given airspace is fundamental to air traffic management. Air traffic controllers continuously modify aircraft trajectories through interventions such as speed advisories and radar vectoring, and the patterns of these interventions are difficult to capture effectively using existing air traffic simulators. To address this limitation, we propose
Diffusion Transformers for Traffic Scene Generation (DiTSG), which learns to generate air traffic scenes from recorded multi-aircraft trajectories. DiTSG employs stacked multi-
agent inverted
Transformer layers as the denoising backbone of a
diffusion transformer, jointly generates the trajectories of multiple aircraft within a traffic scene, and conditions the generation on
agent-specific required times of arrival. The resulting trajectories generated by DiTSG preserve realistic inter-aircraft interactions observed in actual air traffic operations. Experiments show that the DiTSG generates diverse traffic scenes that reproduce the statistical properties of recorded traffic while preserving realistic inter-aircraft interactions.
Read the full abstract ▾
All authors Seokbin Yoon, Kyungmin Kim, Taejung Kwon, Keumjin Lee, Peng Wei
Robotics · Cross-listed from cs.CV
ConDenseDrive: Knowledge-Dense, Token-Sparse Visual Encoder for End-to-End Autonomous Driving
Shengzhen Li, Yongkang Li, Hao Gao et al. (+5 more)
End-to-end driving requires rich spatiotemporal representations to support scene understanding and trajectory planning under strict latency constraints. However, most systems initialize their visual encoders from generic image pretraining and densely process all patch
tokens for each camera view and input frame, limiting both representation quality and inference efficiency. To address these challenges, we present ConDenseDrive, a knowledge-dense and
token-sparse visual encoder for end-to-end autonomous driving. Our three-stage framework first makes the encoder knowledge-dense by distilling complementary visual, semantic, and
geometric knowledge from three
vision foundation models. It then adapts the distilled encoder to multi-frame end-to-end driving and integrates spatiotemporal context into register
tokens for planning. Finally, it makes the encoder
token-sparse by applying register-guided
token compression, reducing redundant patch computation and yielding a 2.0$\times$ speedup while improving planning performance. ConDenseDrive achieves 94.5 PDMS on NAVSIM v1 and 90.8 EPDMS on NAVSIM v2, and reaches 37.4 HD-Score in zero-shot evaluation on HUGSIM. Extensive experiments and qualitative analyses demonstrate that visual encoder design provides an effective route to reliable and efficient end-to-end driving.
Read the full abstract ▾
All authors Shengzhen Li, Yongkang Li, Hao Gao, Chenxi Wu, Yajing Sun, Lu Hou, Wenyu Liu, Xinggang Wang
Robotics · Cross-listed from cs.AI
Observe, Share, or Predict? Disentangling Partner Information in Embodied Multi-Agent Collaboration
Adrián Szlatincsán, Filippo Ziliotto, Bruno Lepri et al. (+3 more)
LLMs are used as high-level controllers for embodied
agents, but it remains unclear whether they support collaborative
reasoning, such as anticipating a collaborator's actions in real-time teamwork without dialogue. We ask if observing the partner's state, exchanging object knowledge, and predicting the partner's next action, shape task outcomes and coordination when
agents can share percepts but never intentions, plans, or task-level messages, and whether the two
agents occupy symmetric roles. Using the Habitat-based PARTNR
benchmark, we introduce a Decoupled Partial Observability regime that enforces strictly individual perception, and we augment decentralized zero-shot ReAct planners with Other
Agent's State (OAS) observation, object Knowledge Exchange (KE) between co-located
agents, and Other
Agent's Next Action (OANA) prediction, evaluated on the full 2 x 2 x 2 grid with four planners on 1,000 validation tasks. OAS observation improves task quality, reducing extraneous effort for every planner, whereas KE primarily improves execution efficiency, cutting simulation steps by up to 25% with little change in success. The two are largely redundant for strong planners but complementary for the weakest, for which KE without the OAS is the worst configuration. OANA prediction lowers the success rate in 15 of 16 model x context pairs while raising explicit partner mentions to 55-100%, and its accuracy gains lie mainly in the coarse action type, partly because
agents recognize the action the partner is already performing. The
robot retrieves objects sooner and completes its tasks earlier than the simulated human. Shared context reduces total task time by shortening the remaining duration that the human works alone. Also, coordination benefits arise from responding to shared context, especially the OAS, rather than explicit anticipation, yielding shallow, planner-dependent partner modeling.
Read the full abstract ▾
All authors Adrián Szlatincsán, Filippo Ziliotto, Bruno Lepri, Luciano Serafini, Massimiliano Luca, Tommaso Campari
Robotics · Cross-listed from cs.AI
Acceptable Misalignment
Maud van Lier, Ludovica Marinucci, Vieri Giuliano Santucci et al. (+4 more)
There is a specific challenge designers face in developing autonomous
robots capable of open-ended learning. This challenge is how one can develop such systems in an ethically sound way when there are many unknowns during design time about their manner of deployment, what they can and will learn, and how these systems will act on the basis of what they have learned. One response to this challenge is to align, as far as possible, the activities and goals of autonomous
robots with human values, raising the question of how to make this
alignment possible, a question known as the value
alignment problem. We argue that the value
alignment problem faces a number of challenges that show that full
alignment is both theoretically and practically impossible, meaning that misalignment incidents will keep occurring. This undermines current
alignment approaches, as the attempt to maximize
alignment cannot properly deal with the possibility that the risk of misalignment that remains after
alignment methods have been implemented might still be unacceptable to society. To overcome this difficulty, we argue that a shift in perspective is required from a focus on the maximization of
alignment to the conditions under which misalignment incidents become acceptable. In this paper, we explore what this shift in perspective entails. We introduce the concept of acceptable misalignment, formulate normative requirements for
alignment mechanisms aimed at bringing misalignment incidents to an acceptable level, and we conclude by presenting a new methodological framework that consists of
alignment mechanisms that can meet these requirements.
Read the full abstract ▾
All authors Maud van Lier, Ludovica Marinucci, Vieri Giuliano Santucci, Raja Chatila, Stéphane Doncieux, Émiland Garrabé, Richard Duro
Robotics
Human-Like Redundancy Resolution for Real-Time Bimanual Humanoid Teleoperation
Chao Tan, Ming Shin Cindy Sung, Shuxian Liang et al. (+3 more)
Real-time bimanual humanoid teleoperation requires accurate wrist tracking while keeping arm configurations predictable. Because a 7-DoF arm admits multiple elbow configurations for the same wrist pose, wrist-only tracking can cause elbow drift or abrupt configuration changes. Fixed anthropomorphic constraints can stabilize elbow posture, but they can also block the redundant adjustments needed near singular configurations. We propose a human-like redundancy-resolution framework that combines adaptive elbow guidance with singularity-robust wrist tracking. Shoulder-wrist
geometric retargeting generates
robot-scale wrist targets. An adaptive elbow-offset criterion uses the offset direction to guide posture and scales this guidance by the offset magnitude, so the redundant degree of freedom is progressively released as the human arm straightens. A hierarchical velocity solver prioritizes wrist tracking and projects the adaptive elbow guidance into the wrist-task null space. In a fixed-wrist-target ablation, adaptive guidance reduces mean elbow-direction error by 54.4% compared with removing this guidance. In a workspace-boundary stress test, singularity-robust tracking reduces peak raw joint-velocity demand by 97.9% compared with the undamped pseudoinverse. Real-world bimanual tasks and cross-platform retargeting show applicability across
robot morphologies.
Read the full abstract ▾
All authors Chao Tan, Ming Shin Cindy Sung, Shuxian Liang, Chaoqun Wan, Zhen Huang, Xian-Sheng Hua
Robotics
CopyQuad: Quadrupedal Skill Transfer from a Single Side-View Video Without a Depth Model
Mohamad H. Danesh, Raina Hu, Mengxue Xu et al. (+1 more)
Video-to-
robot pipelines for quadrupedal imitation insert a learned lifting stage, either a monocular depth model or a per-clip 3D reconstruction, to recover the one axis a side-view camera cannot see. We present CopyQuad, which omits that stage and pins the missing axis to the
robot's own stance width instead. The setting is side-view footage of sagittally dominated motion, where limb motion is almost entirely in the image plane. A frozen 2D keypoint detector and a fixed analytic retargeting then turn a single uncalibrated clip into a
robot reference whose foot targets are reachable by construction, so no per-clip trajectory optimization is needed. One phase-indexed reward structure, whose only per-behavior freedom is three scalars from a shared sweep, learns a dog walk, a horse walk, a horse trot and two non-periodic gestures, all deployed zero-shot on hardware. We then measure what omitting the axis costs. In a training-free rigidity test, a
video depth model varies the distance between the animal's hips by $40$ to $107\%$ of its own mean and a single-image metric model by $22$ to $58\%$, though that distance cannot move. Motion capture puts the true lateral paw travel of a dog walk at $4.5$ cm peak to peak, a mean deviation near $1.4$ cm against the $3.5$ cm of lateral foot error our policies already leave, so the discarded axis sits below the pipeline's resolution. Depth repaired to feasibility never beats the constant, and the unrepaired estimates put the
robot on the ground.
Read the full abstract ▾
All authors Mohamad H. Danesh, Raina Hu, Mengxue Xu, Hsiu-Chin Lin
Robotics
Belief Space Planning for Risk Signal Temporal Logic via Mixed-Integer Convex Programming
Shaohang Han, Joris Verhagen, Kaier Liang et al. (+1 more)
Signal Temporal Logic (STL) specifies
robot tasks with spatio-temporal requirements, and its robustness semantics allow planning by trajectory optimization. In practice,
robots estimate their state from noisy measurements, and when the information gained from a measurement depends on the state, the
robot can actively reduce its uncertainty by planning in the belief space. The resulting problem is, however, generally non-convex, for which gradient-based methods are neither complete nor globally optimal. In this paper, we consider Risk STL (RiSTL) planning in Gaussian belief space and show that an information-form parameterization of the belief, together with a convex relaxation of the belief dynamics, yields a mixed-integer convex program (MICP). We prove that the MICP is sound and complete, and that the relaxation preserves the globally optimal input trajectories. We validate the effectiveness of our method in numerical simulations.
Read the full abstract ▾
All authors Shaohang Han, Joris Verhagen, Kaier Liang, Jana Tumova
Robotics
FEM-Validated Real-Time Modelling of Soft Tactile Deformation
Silvia Terrile, Kaichen Shi, Taejun Kwon et al. (+5 more)
Accurate modelling of soft tactile sensors remains difficult because finite element methods (FEM) are computa- tionally expensive, while real-time simulators often lack fidelity for hyperelastic deformation. Large physical
datasets are also impractical to collect because repeated testing can damage fragile sensors. We present a data-efficient framework that combines limited real-world validation, high-fidelity FEM, and graph neural networks (GNNs) for accurate real-time prediction of two-dimensional tactile deformation in a planar soft tactile structure. The key idea is to decouple validation from training: 82 camera-tracked contact configurations are used only to tune and validate an ANSYS FEM model, which then generates 768 synthetic contact scenarios for GNN training. We evaluate both forward prediction from marker deformation to contact localisation and inverse inference from contact configuration to marker deformation. The validated FEM achieves sub-0.5 mm error in more than 90% of validation cases, while the learned GNN surrogate reaches a mean absolute error of 0.32 mm with millisecond-scale inference, compared with minute-scale FEM simulation. Models trained entirely on synthetic FEM data retain sub-millimetre accuracy on real measurements from the 2D tactile structure, demonstrating strong sim-to-real transfer within the tested planar setting. We further integrate the learned model into a real-time visualisation pipeline, enabling rapid deformation prediction for virtual prototyping and simulator-based design exploration without extensive physical testing.
Read the full abstract ▾
All authors Silvia Terrile, Kaichen Shi, Taejun Kwon, Jiseok Lee, Nathan Lepora, Yeejin Lee, Saekwang Nam, Efi Psomopoulou
Robotics
What Limits Cross-Task Generalization in World Action Models?
Chen Zhao, Zipeng Ye, Yihan Lin et al. (+2 more)
Cross-task generalization remains difficult for world action models (WAMs), even when their predicted futures depict plausible manipulation. This gap can be attributed directly: we make this gap directly testable by decoupling a WAM into a world model and an inverse-dynamics model (IDM), then holding the IDM fixed while replacing the source of future images. We introduce Align Once, Evaluate Everywhere (AoE), a camera-space inverse-dynamics model pretrained on large-scale
robot data, which provides this recovery interface for different future sources across tasks and environments. Through it, we find that actions recovered from recorded futures of unseen tasks execute almost as well as replayed demonstrations, whereas futures predicted by world models collapse across
benchmarks. The failures are not generic: they concentrate on the gripper and interaction geometry while generation capacity is spent on task-irrelevant background. This diagnosis suggests that supervision should target exactly these regions. We therefore introduce the
vision-to-action (VITA) loss, which passes generated futures through the frozen AoE and penalizes discrepancies in the recovered actions, redirecting generation toward action-relevant detail. Across two world models and three manipulation
benchmarks, this simple change raises the execution success of generated futures from near zero to 47.0%. The same loss also transfers to WAMs and yields consistent gains in native control.
Read the full abstract ▾
All authors Chen Zhao, Zipeng Ye, Yihan Lin, Yuchen Xie, Jing Zhang
Robotics
Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?
Zihan You, Zehua Fan, Wenqi Lyu et al. (+19 more)
Recent
Vision-
Language-Action (VLA) models increasingly use natural-
language instructions to condition autonomous driving behavior. Yet existing closed-loop
benchmarks primarily measure route completion, safety, and rule compliance, leaving a fundamental question unresolved: does
language actually determine which feasible behavior the vehicle executes? A trajectory may be safe and successful while still violating or even ignoring the passenger's instruction. To address this, we introduce \ltd, a
benchmark designed to directly test this capability. Its key evaluation unit, the paired counterfactual unit (PCU), holds the initial driving scene fixed while changing only the instruction. This design enables us to distinguish isolated instruction success from counterfactual paired success across four instruction-following scopes. To test whether this distinction persists across evaluation regimes, we instantiate
Language2Drive in complementary closed-loop CARLA simulation, reconstructed real-world scenes, and open-loop driving
benchmarks, including our released CARLA-
Language2Drive simulator with purpose-built maps. Evaluating representative driving VLAs across these settings reveals a striking mismatch between strong driving competence and weak
language control: 93.2% of successful rollouts do not have a successful counterpart under the alternative feasible instruction. This failure is dominated by one-sided completion, where a model repeatedly executes one preferred behavior but fails when the same scene requires the counterfactual alternative. These results expose two fundamental limitations of current evaluation: isolated success and driving scores, which are widely used by conventional driving
benchmarks, can substantially overstate
language-following capability, while current driving VLAs themselves lack reliable instruction following under matched counterfactual tests.
Read the full abstract ▾
All authors Zihan You, Zehua Fan, Wenqi Lyu, Aoqi Wang, Ni Wang, Yifei Yang, Linge Zhao, Huan Li, Sining Ang, Yuguang Yang, Jueyu Chen, Xiangyu Li, Qianshan Wei, Jieyuan Pei, Yuxin Wang, Zhiwen Chen, Jiaqi Li, Wenxuan Song, Yanyuan Qiao, Benjin Zhu, Haoang Li, Yan Wang
Robotics · Cross-listed from cs.LG
DR-APO: Decoupled-Representation Advantage-Guided Policy Optimization from Continuous Control to Embodied Navigation
Runhua Zhang, Yingjie Niu and Chao Li
Policy representations in reinforcement learning have expanded beyond Gaussian policies to
diffusion and flow-matching based generative policies. However, existing RL methods remain representation-specific, lacking a unified optimization interface for emerging generative policies. In particular, extending likelihood-based objectives to multi-step generative policies couples optimization to their internal generation process, where accumulated likelihood ratios can weaken the update signal. Meanwhile, generative-policy-specific methods often rely on multi-candidate generation and critic-based selection, improving overall return while providing limited improvement to the policy itself. We introduce DR-APO (Decoupled-Representation Advantage-Guided Policy Optimization), which decouples policy improvement from policy representation through advantage-guided target improvement and representation-compatible direct action fitting, providing a shared optimization principle for Gaussian,
diffusion, and flow policies. Across five MuJoCo-v5 tasks, DR-APO remains competitive with SAC for Gaussian policies and achieves the best aggregate policy-only performance for both
diffusion and flow-matching policies. NavDP post-training improves the Success Rate (SR) from 70.33% to 89.55% in simulation and from 25% to 65% in real-world navigation. These results demonstrate a unified policy-improvement principle across Gaussian,
diffusion, and flow policies, from continuous-control learning to embodied-policy post-training.
Read the full abstract ▾
Robotics
Design and Analysis of a Nonlinear Parallel Mechanism Coupling for Dynamic Humanoid Legs
Rohit Kumar, Heiner Peters, Felix Wiebe et al. (+5 more)
Humanoids increasingly demand lightweight leg mechanisms for agile, dynamic, and energy-efficient locomotion. Reducing distal inertia through proximal actuation has thus become a common design goal. Yet, compact transmissions that preserve torque capability and efficient force transmission remain hard to achieve. This paper presents a novel series-parallel leg design combining a knee parallel mechanism, an ankle parallel mechanism, and a nonlinear knee-ankle coupling where all actuators placed closer to the hip. As the knee flexes, the coupling passively assists ankle dorsiflexion for foot clearance and, as it extends it assists plantarflexion for push-off, redistributing ankle load through the knee transmission. We derive the coupling's nonlinear transmission models and validate them in simulation and on hardware. The design shows that the shared transmission lets both joints draw on each other's actuator capacity by easing the knee's peak torque demand under load and the ankle's peak speed demand during fast motion. %Together With the leg's proximal actuator placement, the leg achieves an effective reflected mass of 0.37kg at the foot for a 6kg leg, demonstrating very low distal leg inertia for dynamic humanoid
robots.
Read the full abstract ▾
All authors Rohit Kumar, Heiner Peters, Felix Wiebe, Franek Stark, Janne Janzen, Dennis Mronga, Shubham Vyas, Frank Kirchner
Robotics · Cross-listed from cs.LG
DEFlow: Deep Equilibrium Flow Maximum-Entropy Reinforcement Learning
Ruslan Ageev, Huaze Tang, Runqing Miao et al. (+2 more)
Expressive flow policies can represent multiple high-value actions, but sampling them with few integration steps while retaining tractable action densities remains challenging. We introduce DEFlow, a deep
equilibrium flow policy that replaces explicit Euler sampling with implicit Euler integration and solves all intermediate actions jointly as an
equilibrium system. We combine implicit differentiation with a discrete change-of-variables formula for the final-action density, enabling off-policy training within soft actor-critic without backpropagating through solver iterations. DEFlow achieves an average relative improvement of approximately 10% in mean successful coverage over the strongest baseline across four Multi-Goal docking variants. It also achieves the highest mean time-averaged return on four of five manipulation tasks, while remaining broadly competitive across five MuJoCo tasks. These results support implicit flow policies for tasks requiring both diversity and precision, at the cost of additional nonlinear and linear solves. Primary
Read the full abstract ▾
All authors Ruslan Ageev, Huaze Tang, Runqing Miao, Jiarong Li, Wenbo Ding
Robotics · Cross-listed from cs.CV
VLA-GAUGE: Systematically Evaluating Generalization and Robustness in Vision-Language-Action Models
Jun Xiao, Rui Ye, Dong Cao et al. (+3 more)
Vision-
Language-Action models (VLAs) have achieved high success rates on familiar manipulation tasks, yet it remains unclear whether such scores reflect genuine zero-shot generalization and perturbation robustness. Existing evaluation frameworks provide limited capability-level diagnosis and combinatorial perturbation coverage, often without connecting diagnosis to targeted model refinement. We introduce VLA-GAUGE, a diagnostic framework that unifies both evaluations under a shared executable task representation. It combines (1) paired in-distribution and out-of-distribution tasks across six capability suites, (2) single-factor sensitivity analysis with combinatorial perturbation screening and evaluation, and (3) diagnosis-driven refinement through targeted data augmentation and model fine-tuning. We evaluate 12 VLAs spanning different model sizes and action-modeling paradigms under a unified protocol. We find that (1) The mean Zero-Shot Score on VLA-GAUGE is only 22.3%, with SimpleVLA-RL scoring 8.0% and ranking second to last among the evaluated models. (2) Under three representative combinatorial perturbation configurations, the mean R-Score is 9.3. (3) Diagnosis-driven fine-tuning improves $\pi_{0.5}$'s zero-shot generalization and robustness under a selected perturbation combination on held-out scenarios by 72.4 and 31.8 scoring points, respectively; the latter exceeds the 10.9-point gain from LIBERO-Plus single-factor fine-tuning despite fewer successful training trajectories. These results show that high reported success rates on familiar manipulation tasks do not guarantee strong zero-shot generalization to unseen tasks or robustness to combinatorial perturbations, while diagnostic evaluation provides actionable signals for targeted model improvement.
Read the full abstract ▾
All authors Jun Xiao, Rui Ye, Dong Cao, Jinyu Gu, Ying Wang, Zhiliang Zhu
Robotics · Cross-listed from cs.AI
G$^2$WAM: World Action Models for Efficient and Geometry Grounded Image-Goal Navigation
Ke Gui, Chenghao Shi, Ziyue Wang et al. (+3 more)
World models capture environment dynamics and support counterfactual
reasoning for planning. In image-goal navigation, where the target often lies beyond the current field of view, the
agent must not only model environmental dynamics but also maintain a persistent spatial
memory of the scene. Existing approaches either introduce spatial context by fine-tuning pretrained models on the target task, which underexploits their spatial priors, or rely on external 3D reconstruction, which is computationally expensive for real-world deployment. In this paper, we propose the Geometry Grounded World Action Model (G$^2$WAM) for image-goal navigation. G$^2$WAM encodes spatial cues through a learned scene-coordinate localization module that anchors the current, goal, and history frames in a common reference frame, yielding a stable and consistent spatial context that enables efficient navigation without explicit 3D reconstruction. Following the world-action-model paradigm, G$^2$WAM jointly predicts future states and actions, eliminating the need for costly cross-
entropy method (CEM) style planning. In addition, G$^2$WAM operates entirely in latent space, avoiding the computational burden of pixel-level reconstruction and substantially reducing inference cost. Across four trajectory prediction
benchmarks, G$^2$WAM reduces average ATE and RPE by 40.1% and 35.2%, respectively. In closed-loop navigation experiments, it improves the success rate from 54.0% to 63.0% and the success weighted by path length (SPL) from 36.4% to 59.2%, while maintaining high efficiency.
Read the full abstract ▾
All authors Ke Gui, Chenghao Shi, Ziyue Wang, Longwen Chen, Xieyuanli Chen, Huimin Lu