Robotics
Tac2Pix: Image-Space Visuo-Tactile Fusion for Dexterous Manipulation
Shengcheng Luo, Kefei Wu, Xiaoying Zhou et al. (+3 more)
Vision and touch are naturally complementary for dexterous manipulation. Yet they are represented in fundamentally different forms: visual observations are dense and spatially structured, whereas tactile measurements are sparse, heterogeneous, and
robot-centric. This mismatch makes it challenging to establish explicit spatial correspondence between the two modalities, particularly when they are fused in latent space. However, each tactile measurement is associated with a known location on the
robot's kinematic chain, offering a localization prior that could bridge this representational gap. This raises a natural question: can this localization prior be used to construct a shared representation that explicitly aligns touch with
vision? We introduce Tac2Pix, an image-space visuo-tactile fusion framework that projects sparse tactile measurements onto the RGB image plane using forward kinematics and camera calibration, then renders them as force-aware saliency maps. By expressing touch in the same pixel space as
vision, Tac2Pix directly leverages pretrained 2D visual representations. We further employ zero-initialized input adaptation to preserve the original visual pathway at the start of policy learning. To assess whether this geometry-based interface yields consistent policy gains, we evaluate Tac2Pix on three simulated and three real-world dexterous manipulation tasks. In simulation, Tac2Pix consistently improves visuo-tactile policy learning across tasks, policy architectures, and visual backbones. In real-world experiments, it improves average success under visual occlusion by 26.7 percentage points over the strongest latent visuo-tactile fusion baseline, with gains persisting under both random and physical occlusions. Together, these results support image space as an effective interface for fusing sparse touch with pretrained visual representations.
Read the full abstract ▾
All authors Shengcheng Luo, Kefei Wu, Xiaoying Zhou, Wanlin Li, Ziyuan Jiao, Chenxi Xiao
Robotics
Guided Discovery of New Behaviors using Diffusion Policies
Dian Yu, Sebastian Sanokowski and Majid Khadiv
Diffusion models have become a powerful tool for generative modeling in robotics, with
diffusion policies excelling at modeling
multimodal action-trajectory distributions. However, when demonstrations are limited, standard sampling often reproduces dominant behaviors while neglecting valid but rare modes, limiting the discovery of novel solutions. Existing approaches, such as guidance methods or combining reinforcement learning with
diffusion, either push samples into infeasible regions or struggle to escape local minima, failing to systematically uncover diverse behaviors. To address these challenges, we propose a framework that combines Feynman-Kac correctors with a novel guiding potential that systematically guides
diffusion policy samples towards promising yet underrepresented samples. These trajectories are refined using sampling-based trajectory optimization and reincorporated into the training set to retrain the
diffusion policy. Our method effectively mines and repairs novel trajectories, enabling the systematic discovery of diverse and executable behaviors. We demonstrate the effectiveness of our framework across a range of manipulation environments, consistently discovering new behaviors.
Read the full abstract ▾
Robotics
Safe, Fluent and Acceptable Motion Generation and Execution for Human–Robot Interaction in Manufacturing Environments
Thibaut Lopez, Olivier Aycard, Pierre-Brice Wieber et al. (+2 more)
Robots operating in human environments must not only ensure physical safety but also exhibit behaviors that are understandable, fluent, and acceptable to human partners. This paper investigates motion generation strategies that combine safety guarantees with interaction quality considerations, such as motion smoothness and human comfort. While the design of
robots capable of ensuring safety in shared human-
robot environments has enabled closer and more advanced forms of interaction, these new proximity-based tasks require moving beyond purely technical considerations. In particular,
robot behavior must also be addressed from psycho-cognitive and social perspectives. In this context, we argue for the relevance of integrating social-aware motion control into robotic systems. First, we identify the motion parameters that influence human perception and operator experience. Then, we implement a Model Predictive Control (MPC) framework that generates four distinct socially-informed
robot behaviors. Finally, we conduct a user study to evaluate and validate these behaviors and assess their social impact on non-expert participants. The results demonstrate that variations in
robot behavior significantly affect the perceived social acceptability of the system. These findings highlight the importance of incorporating human-centered considerations into motion generation strategies for
robots operating in shared environments.
Read the full abstract ▾
All authors Thibaut Lopez, Olivier Aycard, Pierre-Brice Wieber, Mohamed Boua, Christine Jeoffrion
Robotics · Cross-listed from cs.NI
Systems-Level Planning and Coordination of Truck-Drone Collaborative Delivery Networks
Didem Cicek and Burak Kantarci
Urban last-mile parcel delivery increasingly relies on heterogeneous fleets whose performance depends on timely coordination, reliable communication, and scalable control. Truck-drone collaboration has emerged as a networked cyber-physical delivery paradigm that combines the payload capacity and range efficiency of trucks with the agility of drones in congested or access-limited urban environments. This paper proposes a layered planning and coordination framework that structures truck-drone collaborative delivery (TDCD) from a systems and control perspective. The framework consists of five interrelated layers: spatial-demand
alignment, collaborative delivery configuration, resource and workflow orchestration, performance evaluation, and scalability analysis, providing a unified view of coordination, control, and system-level performance in networked delivery operations. The proposed framework is evaluated using a realistic urban last-mile delivery scenario derived from the 2021 Amazon Last Mile Routing Research Challenge
dataset. The case study demonstrates how coordinated truck-drone operation, enabled by structured task orchestration and inter-
agent synchronization, improves end-to-end system efficiency under operational constraints. Results show a 42.4% reduction in total delivery time and a 44.2% reduction in energy consumption compared to a conventional truck-only delivery model. The scalability analysis further highlights how coordination gains persist as system size increases, and shows the importance of efficient control and communication in heterogeneous delivery networks.
Read the full abstract ▾
Robotics
Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation
Yunfan Lou, Yifan Ye, Yankai Fu et al. (+7 more)
World action models inherit the predictive capability of world models, enabling action generation to be guided by anticipated future observations. However, they rely primarily on
vision and often fail in contact-rich manipulation, where critical cues arise from physical interaction. In this paper, we propose Dream-Tac, a unified Tactile-World Action Model that jointly models actions, future visual observations, and tactile dynamics. Specifically, Dream-Tac introduces (i) contact-gated visuotactile fusion to selectively integrate tactile signals and (ii) a contact-aware
attention bias to better regulate cross-modal interactions during manipulation. To support real-time deployment, we further design a dual-level acceleration strategy, reformulating the contact-aware bias to preserve the fused
attention path during training and introducing cache-based
diffusion acceleration at inference, achieving up to 2.9$\times$ faster training and 1.8$\times$ faster inference. Across six contact-rich manipulation tasks, Dream-Tac improves action accuracy by 31.7% on average, demonstrating the effectiveness of unified visuotactile world modeling.Code is available at https://github.com/LYFCLOUDFAN/Dream-Tac.
Read the full abstract ▾
All authors Yunfan Lou, Yifan Ye, Yankai Fu, Jun Cen, Xiaowei Chi, Yaoxu Lyu, Peidong Jia, Sirui Han, Zhihe Lu, Shanghang Zhang
Robotics
IR-SIM: A Lightweight Declarative Simulator for Navigation Learning and Benchmarking
Ruihua Han, Shuai Wang, Chengyang Li et al. (+8 more)
Developing
navigation policies requires simulation scenarios that support repeatable training and evaluation. Despite the availability of numerous simulators, constructing diverse
navigation scenarios often involves writing simulator-specific code, using complex graphical interfaces, performing substantial manual configuration, or relying on resource-intensive computing platforms, making scenarios difficult to reproduce and reuse across experiments. To this end, we develop the Intelligent
Robot Simulator (IR-SIM), a lightweight declarative simulator implemented as a Python library to support
navigation learning and benchmarking through rapid construction of reusable and diverse scenarios and efficient execution on accessible platforms. IR-SIM represents scenarios as human-readable YAML configurations, in which necessary objects, behaviors, sensors, maps, and environment parameters can be flexibly specified and composed. This declarative representation makes IR-SIM friendly to large
language model (LLM)-powered
agents, enabling them to compose and modify executable
navigation scenario files through the provided
agent skills, instead of writing a large amount of code that is difficult to reuse. Despite its lightweight implementation, IR-SIM provides the essential components for
navigation simulation, while retaining interfaces to high-fidelity simulators for downstream validation. Experiments demonstrate that IR-SIM runs up to several hundred times faster on the evaluated CPU platform, while
agent skills reduce mean LLM-based scenario construction time by more than $50\%$ across both evaluated models. The experiments further show ability to support reproducible benchmarking and RL-based
navigation policy learning.
Read the full abstract ▾
All authors Ruihua Han, Shuai Wang, Chengyang Li, Rui Gao, Xinyi Wang, Zhe Liu, Guoliang Li, Yupu Lu, Qi Hao, Jia Pan, Hengshuang Zhao
Robotics
Real-Time and Accurate Collision-Free Teleoperation via Differentiable Constraint-Based Trajectory Planning
Max Grobbel, Tristan Schneider, Daniel Flögel et al. (+1 more)
In teleoperation, the human operator typically controls only the end-effector pose, which often leads to self-collisions of the manipulator and collisions with environmental obstacles, since joints and links are not controlled individually. A common strategy to mitigate this issue is to enhance the operator's input using optimal-control-based trajectory planning. As derivative-based solvers require differentiable constraints, existing approaches either approximate
robots and obstacles with spheres, reducing
geometric accuracy, or approximate derivatives, degrading convergence and increasing computation times. We address these limitations by adapting a recent formulation of differentiable collision-avoidance constraints, based on duality in convex optimization, to the teleoperation setting. The
robot is approximated with capsules and the environment with polytopes. We compare the resulting trajectory planning method against state-of-the-art techniques in simulation with varying numbers of obstacles and evaluate it on a UR5e manipulator in a real-world teleoperation test. Results show that our approach achieves lower computation times while enabling more accurate obstacle modeling, leading to smoother and collision-free end-effector teleoperation.
Read the full abstract ▾
All authors Max Grobbel, Tristan Schneider, Daniel Flögel, Sören Hohmann
Robotics · Cross-listed from eess.SY
Hybrid Neural Network and Conventional Controller Approach for Robust Control of Highly Unstable Systems: Application to Tilt-Rotor Control
Ali Kafili Gavgani, Amin Talaeizadeh, Aria Alasty et al. (+1 more)
Multirotors are widely used in applications ranging from surveillance to precision agriculture, yet conventional designs remain limited by their under-actuation. Tilt-rotor configurations overcome this limitation by enabling full actuation. This paper investigates neural-network-based control strategies for a fully actuated tilt-rotor system with four thrust-vectoring inputs. Our work is structured in two parts. First, we deliberately present a negative result by evaluating a direct input-output control approach. In this method, multilayer perceptrons (MLPs), long short-term
memory (LSTM) networks, and
transformer models are trained to map system states and their desired values directly to control signals. We show that this strategy fails to stabilize the system, highlighting the inherent difficulty of applying direct input-output learning to highly unstable plants. Second, as the main contribution, we propose a neural-network-enhanced sliding mode controller (SMC). The method decomposes the system dynamics into input-independent and input-dependent components, with the former learned from a small
dataset using lightweight networks, thereby reducing real-time computational demands. Moreover, the proposed method can be trained using flight logs collected from low-performance controllers, and the resulting dynamic model learned from real-world data can be used in simulation. We further compare MLP- and LSTM-based implementations under model uncertainties and external disturbances, demonstrating the robustness and effectiveness of the proposed approach; in particular, the controller with the LSTM plant dynamics predictor achieves superior performance to its MLP-based counterpart while also exhibiting lower runtime.
Read the full abstract ▾
All authors Ali Kafili Gavgani, Amin Talaeizadeh, Aria Alasty, Hossein Nejat Pishkenari
Robotics
PhysAgent: Automating Physics-Based 4D Synthesis via Trajectory-Grounded Multi-Agent Feedback
Chunji Lv, Jiaxi Ye, Yuchen Jiang et al. (+2 more)
Achieving fully automated, physically plausible 3D motion synthesis is a core objective in graphics and generative AI. However, configuring complex environmental force fields still relies entirely on manual expert intervention, creating a severe bottleneck for large-scale simulation data generation. Existing automated methods primarily focus on material optimization and exhibit severe modality gaps and technical flaws when applied to the vastly more complex force field optimization space: naive Large
Language Models (LLMs) lack underlying simulation feedback, causing severe physical inaccuracies, while traditional Score
Distillation Sampling (SDS) suffers from sluggish gradients, local optima entrapment, and a mathematical inability to dynamically switch discrete force fields. To address this, we propose PhysAgent, the first simulator-in-the-loop multi-
agent framework that leverages
multimodal inputs for automated, physically grounded 4D synthesis. By decoupling intrinsic materials from extrinsic dynamics, PhysAgent utilizes a Semantic
Agent equipped with an externalized Force Field Skill module to master simulation rules and generate valid initializations. Subsequently, the Refine
Agents, driven by Trajectory-Grounded Multi-
Agent Feedback, leverage
vision foundation models to extract dense point trajectories from rendered frames. By converting these explicit motion trajectories into structured textual descriptors, the
agent harnesses LLM commonsense
reasoning to execute zero-shot macroscopic leaps, effectively escaping local optima and dynamically switching discrete force fields. Extensive experiments demonstrate that PhysAgent rapidly generates stable, diverse physical scenes from arbitrary
multimodal prompts, significantly outperforming existing baselines in both generation diversity and physical accuracy.
Read the full abstract ▾
All authors Chunji Lv, Jiaxi Ye, Yuchen Jiang, Rexar Lin, Changsheng Li
Robotics · Cross-listed from cs.CV
Distortion-Aware PETR for BEV Object Detection with Mixed Pinhole-Fisheye Cameras
Xiangzhong Liu
Fisheye cameras are widely deployed in autonomous driving perception suites for their low cost and full-coverage field of view (FOV), yet their potential remains underleveraged in 3D object detection. Severe radial distortion challenges most BEV detectors by violating the fundamental assumption of uniform sampling. To bridge this gap, we propose Distortion-Aware PETR (DAPETR), a projection-free detector tailored for mixed pinhole-fisheye camera setups. DAPETR incorporates two key learned-adaptive modules: a unified distortion-aware positional
embedding that harmonizes positional encodings for image representations with fisheye geometry, and a bidirectional feature-geometry co-modulation module that mutually adapts image features and 3D positional
embeddings. In our experiments on a converted KITTI-360
benchmark, we systematically compare our learned adaptive approach against PETR in polar coordinates (PolarPETR). We find that while both methods improve over the baseline, our learned modules achieve superior performance. Crucially, we uncover a negative interaction when combining both strategies, revealing that learned adaptation and explicit
geometric reparameterization can conflict. Our final DAPETR model significantly advances the research and
benchmark for fisheye BEV detection, providing critical insights into effective distortion-aware 3D perception design other than image rectification.
Read the full abstract ▾
Robotics
Language as a Sensor: Calibrated Spatial Belief Estimation in 3D Scenes from Natural Language
Aryan Naveen, Jason Xinyu Liu, Luca Carlone et al. (+1 more)
Robots deployed in human-centric environments routinely receive natural-
language descriptions of spatial information ("I left my backpack on the table") that reference parts of the world beyond their perceptual field of view. Traditional metric-semantic mapping ignores this signal, while off-the-shelf
multimodal models remain limited in 3D spatial
reasoning and are not directly amenable to fusion with other sensor modalities. To convert
language observations into a calibrated spatial distribution, we train a
Language Sensor Model (LSM) that maps each utterance and its scene-graph context to a
multimodal distribution, with mixture weights encoding referential ambiguity (e.g., "which table") and component covariances encoding spatial uncertainty (e.g., where "on the table" the target lies). We then introduce VL-Map (
Vision-
Language Metric-Semantic Mapping), a probabilistic framework that treats these
language predictions as
stochastic observations and fuses them with onboard perception within a unified belief map. On the VLA-3D
benchmark as well as on a real-world mobile
robot, LSM is the only
language predictor whose covariance estimates remain within the calibrated regime; fused into VL-Map, it leads to more accurate predictions of the target object location (~70% more probability mass on the true target compared to the strongest foundation-model baseline).
Read the full abstract ▾
All authors Aryan Naveen, Jason Xinyu Liu, Luca Carlone, Andreea Bobu
Robotics
Latent Diffusion Policy: Shaping Latent Spaces for Diffusion-Based Robotic Manipulation
Zhexuan Zhou, Yichen Lai, Jinhao Zhang et al. (+3 more)
Diffusion-based visuomotor policies operating directly in raw action spaces conflate scene comprehension with trajectory generation within a single denoising process. The resulting velocity field must simultaneously encode scene information and generate precise trajectories, increasing learning complexity and limiting performance on tasks demanding precise temporal coordination across multiple arms. To simplify this joint learning problem, we introduce Latent
Diffusion Policy (LDP), a two-stage framework performing flow matching in a deliberately shaped latent space. By absorbing scene understanding into an observation-conditioned CVAE encoder, LDP concentrates the conditional distribution of each observation. Consequently, the flow model avoids implicitly resolving scene-dependent structures; instead, it generates within a pre-concentrated distribution featuring a smoother velocity field, simplifying learning from limited demonstrations. Furthermore, to capture temporal dependencies among latent
tokens, LDP trains with per-
token diffusion forcing and employs staircase inference sampling to resolve the resulting distributional mismatch. We also propose reconstruction FID (rFID) as a lightweight proxy predicting downstream task success solely from latent space statistics. On coordination-intensive tasks from RoboTwin 2.0, LDP outperforms DP3 by a substantial margin and transfers effectively to real-world bimanual deployments.
Read the full abstract ▾
All authors Zhexuan Zhou, Yichen Lai, Jinhao Zhang, Huizhe Li, Youmin Gong, Jie Mei
Robotics
PhysGraph: A Physics-aware 3D Scene Graph for Perception and Reasoning
Haoyu Li, Aaron Thomas, Shuyan Zhou et al. (+1 more)
To perform a wide range of daily tasks,
robots need to construct a 3D representation that is semantically rich, physically grounded, and structured enough to support task planning and affordance prediction. However, existing approaches primarily focus on semantic
retrieval, often overlooking physical and kinematic factors. Methods that attempt to model physical properties typically rely on narrow training sets or single-object modeling, limiting scalability and generalization across diverse object types. To address these challenges, we present PhysGraph, a framework that unifies symbolic
reasoning with structured 3D geometry to model kinematic and physical properties in cluttered scenes. Given RGB-D observations, PhysGraph reconstructs object-centric 3D geometry and associates object instances across views. It then decomposes objects into functional parts and infers materials and articulations through visual
reasoning. Evaluated on both synthetic and real-world
datasets, PhysGraph achieves state-of-the-art results in semantic
segmentation, multi-object mass estimation, and articulation prediction. With its simple yet effective design, PhysGraph produces physically consistent and semantically structured scene graphs, serving as a structured 3D representation for downstream tasks such as constraint-aware 3D affordance prediction and real-to-sim transfer, both of which are demonstrated in our experiments.
Read the full abstract ▾
All authors Haoyu Li, Aaron Thomas, Shuyan Zhou, Xianyi Cheng
Robotics · Cross-listed from cs.CV
FiberTune: Preserving Action-Fiber Visual Residuals in Vision-Language-Action Fine-Tuning
Haihao Lin, Xiangsheng Huang, Xiao Yang et al. (+7 more)
Action-supervised fine-tuning of
vision-
language-action (VLA) policies can compress visual structure that is weakly constrained by the action objective. We formalize this as residual visual collapse along local action fibers and propose FiberTune, a training-time objective that preserves teacher-structured visual residuals without adding inference-time overhead. FiberTune uses an online action probe to filter action-predictive directions from intermediate visual-
token representations. It preserves the residuals through centered directional
alignment with a frozen visual teacher and effective-rank regularization. Under identical training conditions, FiberTune improves over task-loss-only fine-tuning in every one of six simulation settings spanning two
benchmarks and two architectures ($\pi_{0.5}$ and OpenVLA-OFT), as well as on physical SO-101 pick-and-place; representative gains include +10.7 pp SR(5) on long-horizon CALVIN ABC$\rightarrow$D and physical SO-101 task success rising from 72.7% to 78.1%. Representation analyses support the preservation of visual information in the residuals.
Read the full abstract ▾
All authors Haihao Lin, Xiangsheng Huang, Xiao Yang, Weibang Zhou, Yiqi Zhang, Bo Yang, Simin Zeng, Jiawei Yang, Zhengyang Wang, Jiahui Du
Robotics
HARBOR: A Harness Framework for Agentic Robot Reinforcement Learning
Zechu Li, Yufeng Jin, Xiaoyang Liu et al. (+4 more)
Reinforcement learning (RL) has become a powerful paradigm for
robot learning, particularly in sim-to-real settings, but its broader adoption remains limited by the engineering pipeline surrounding the
algorithms. Building tasks, shaping rewards, and tuning hyperparameters require substantial expert effort, making RL workflows costly and difficult to scale. We introduce HARBOR, an agentic framework that frames
robot RL automation as a harness-engineering problem: given a simulator codebase and a task specification, it automates the workflow from environment setup to policy training in simulation. HARBOR decomposes such high-level objectives into bounded stages executed by specialized
agents through standardized commands, persistent artifacts, executable gates, and reusable knowledge, and scales iteration via decentralized parallel trials and experience learning across runs. We evaluate HARBOR across 6
benchmarks and 16 tasks in total, spanning manipulation, locomotion, and bimanual dexterous control. We demonstrate that HARBOR automates the simulation RL workflow end-to-end, designs rewards, tunes
algorithms to match or improve over default configurations, and reduces engineering effort at practical
token and wall-clock cost; the resulting policies can also be transferred to real
robots.
Read the full abstract ▾
All authors Zechu Li, Yufeng Jin, Xiaoyang Liu, Puze Liu, Vignesh Prasad, Carlo D'Eramo, Georgia Chalvatzaki
Robotics
Real-IKEA: Physical Fidelity is the Prerequisite for Robust Manipulation
Kunqi Xu, Zhenhao Huang, Siyuan Luo et al. (+2 more)
Robotic manipulation robustness often founders on the physics gap between simplified simulations and the resistance-laden real world. In this work, we emphasize that physical realism in articulated interaction is an important ingredient for robust policy learning. We present Real-IKEA, a
dataset and simulation framework designed with physical accuracy as a first-class goal. Real-IKEA provides 1,079 articulated asset configurations, derived from 83 authentic IKEA handles and knobs processed through a meticulous six-step physical workflow. For contact-geometry accuracy, we introduce a bidirectional surface-deviation metric to quantify collision meshes. For dynamics realism, we establish resistance-calibrated configurations that vary damping and friction. Crucially, we demonstrate through a Reinforcement Learning (RL) policy that high-fidelity assets enable the discovery of robust "hooking" and "levering" strategies that prioritize mechanical advantage over fragile friction-pulling. Together, these results position Real-IKEA as a critical
benchmark for developing manipulation policies capable of human-level robustness in articulated object tasks.
Read the full abstract ▾
All authors Kunqi Xu, Zhenhao Huang, Siyuan Luo, Ziqiu Zeng, Fan Shi
Robotics
FAWAM: Force-Aware World Action Models for Closed-Loop Contact-Rich Manipulation
Haotian He, Zeyu Yan, Qipeng Liu et al. (+2 more)
Force signals provide critical interaction cues for contact-rich robotic manipulation. However, existing methods mostly use force as an additional observation modality, without fully exploiting its role in modeling future interaction dynamics or guiding execution-time feedback correction. In this paper, we propose FAWAM, a force-aware world action model that incorporates force information at three levels: perception, prediction, and closed-loop execution. FAWAM first encodes historical 6-axis force/torque signals to modulate action generation, then jointly predicts future actions and end-effector wrenches to explicitly model contact
evolution. It further introduces a residual correction module that uses the predicted wrench trajectory as an execution-time reference to refine actions online based on real-time force feedback. Real-world experiments across multiple contact-rich tasks show that FAWAM improves the average success rate by 36.25% over
vision-only baselines and 21.25% over existing force-aware baselines, demonstrating the effectiveness of our force-aware framework for robust contact-rich manipulation.
Read the full abstract ▾
All authors Haotian He, Zeyu Yan, Qipeng Liu, Ning Guo, Wenzhao Lian
Robotics
OASIS: From Simulation Data Collection to Real-World Humanoid Loco-Manipulation
Zehao Yu, Jiakun Zheng, Weiji Xie et al. (+4 more)
Recent progress in
robot manipulation has been largely driven by learning from large-scale demonstrations. For
humanoid robot loco-manipulation tasks, however, existing data sources force an unsatisfying tradeoff between trajectory quality and scalability. Real-world teleoperation provides the highest-quality trajectories but requires dedicated physical space and time-consuming scene resets. Simulation offers an alternative way out of this dilemma: it can produce clean, embodiment-aligned data at scale without any physical hardware. In this paper, we propose OASIS, a simulation-data-driven framework for
humanoid loco-manipulation. OASIS automatically reconstructs realistic object assets from real-world images using a 3D generative model. Based on these assets, trajectories are first collected through teleoperation in simulation, and then augmented under diverse domain randomizations in a post-processing stage. With the resulting simulation data, we further design a hierarchical visuomotor policy for
humanoid loco-manipulation. Extensive experiments on the real
humanoid robot show that, under zero-shot deployment, the policy trained on our simulation data achieves higher success rates on most tasks than that trained on real-
robot teleoperation data, owing largely to the broad lighting and environmental variations covered by our simulation rendering, which real-
robot data fails to capture. The project page is available at https://oasis-
humanoid.github.io/.
Read the full abstract ▾
All authors Zehao Yu, Jiakun Zheng, Weiji Xie, Jiyuan Shi, Chenyun Zhang, Chenjia Bai, Xuelong Li
Robotics
When Video Misreads: Closed-Loop Distillation of Reading Heuristics for Exploratory Manipulation Trace QA
Haizhou Ge, Yufei Jia, Yue Li et al. (+5 more)
Exploratory manipulation often turns an apparent failed attempt into the key evidence for what to do next. For example, a
robot pulls a locked cabinet drawer, fails, and only succeeds after opening the lock. The failed pull reveals a latent precondition (the drawer is locked) that determines the minimal-success action chain (the fewest actions that complete the task), here [lock-open, drawer-pull]. Correctly reading this trace is therefore the prerequisite for recovering that chain. We formalize this setting as Exploratory Manipulation Trace QA (EMT-QA): given synchronized
video and proprioception from an exploratory trace, predict the minimal-success action chain under the latent precondition revealed by the probe. However, even state-of-the-art VLMs and
embodied multimodal LLMs misread this evidence: they do not reliably recover the chain from raw
video, raw proprioception, or their combination. We introduce Closed-Loop Trace
Distillation, a pipeline that uses a per-task coding
agent to inspect labeled training traces and distill a one-line natural-
language prompt over the trace, which we call the Distilled Reading Heuristic (DRH). At inference, no
agent is invoked and no model weights are updated; a frozen VLM receives the raw trace plus the DRH as a
prompt entry. Across three simulator and two real-
robot tasks, the DRH improves chain accuracy by +0.38 to +0.47 over the best raw-modality baseline. The same DRH also serves as the sole specification for one-shot programmatic classifiers that match the prompted VLM.
Read the full abstract ▾
All authors Haizhou Ge, Yufei Jia, Yue Li, Zhixing Chen, Lu Shi, Lei Han, Guyue Zhou, Ruqi Huang
Robotics · Cross-listed from cs.LG
Autonomous Aerial Manipulation via Contextual Contrastive Meta Reinforcement Learning
Lixuan Jin, Bingxuan Lan, Xinyi Bao et al. (+9 more)
Unmanned aerial vehicles (
UAVs) are increasingly being deployed in logistics, service robotics, and other real-world applications, creating a growing demand for autonomous payload acquisition and delivery. Existing approaches typically assume pre-attached payloads or rely on specialized grippers, leaving versatile end-to-end aerial delivery largely unresolved, where different payloads induce highly variable flight dynamics, requiring a single policy to adapt online without manual calibration or explicit system identification. To this end, we study Autonomous Aerial Manipulation via Contextual Contrastive Meta Reinforcement Learning (\textbf{Aco2}), a fully autonomous aerial delivery setting in which a quadrotor equipped with a lightweight hook continuously picks up, transports, and delivers diverse handle-equipped objects between randomized locations, all without human intervention. First, we design a contextual observation encoder that infers a compact latent context from recent interaction history, enabling the policy to adapt online to payload-dependent dynamics. To further improve the quality of this context, we introduce a contrastive objective that structures the context
embedding around task-relevant variations, improving generalization across diverse payloads without requiring explicit system identification. Trained entirely in simulation with extensive domain randomization, Aco2 can be directly deployed on a physical quadrotor without real-world fine-tuning.
Read the full abstract ▾
All authors Lixuan Jin, Bingxuan Lan, Xinyi Bao, Xiangyuan Xie, Chunjie Zhang, Zheng Chen, Tianshuo Liu, Ruijie Tian, Jinyu Ru, Gang Wang, Lei Yuan, Yang Yu
Robotics
GEAR-VLA: Learning Geometry-Aware Action Representations for Generalizable Robotic Manipulation
Yuan Zhang, Shiqi Zhang, Yedong Shen et al. (+11 more)
Vision-
Language-Action (VLA) models achieve strong
benchmark performance but still struggle in real-world deployment with unseen objects, background shifts, and different
robot embodiments. We argue that this stems from the lack of a unified geometry-aware manipulation representation, leaving existing VLAs vulnerable to low-level trajectory supervision, misaligned 3D features, and embodiment differences. To address this, we propose GEAR-VLA, a VLA framework for learning unified geometry-aware action representations for generalizable robotic manipulation. GEAR-VLA adopts coarse-to-fine action learning, where multi-source
embodied pretraining equips the VLM with
embodied reasoning and discrete action understanding before latent action
tokens connect action semantics to a gradient-decoupled DiT continuous action expert. It further performs semantic-aligned 3D integration by aligning a trainable 3D spatial backbone with the VLA representation while freezing the original VLM-aligned visual pathway. To share this representation across
robots, GEAR-VLA uses embodiment canonicalization, where embodiment-aware states and embodiment-invariant actions confine
robot differences to the low-level interface. Extensive simulation and real-world experiments demonstrate strong generalization: GEAR-VLA achieves state-of-the-art performance on LIBERO, zero-shot LIBERO-Plus, and RoboTwin 2.0, reaches 85.9% success on AgileX and 81.0% on the pretraining-unseen LDT-01 embodiment, and obtains 90.1% success on a 6,360-trial universal grasping
benchmark with 212 unseen objects. Code and models will be released at https://github.com/babynabeauty/GEAR-VLA.
Read the full abstract ▾
All authors Yuan Zhang, Shiqi Zhang, Yedong Shen, Shuai Dong, Jiajun Deng, Xin Zhang, Yuxuan Gao, Jiajia Wu, Xin Nie, Zhiyuan Cheng, Jianmin Ji, Yanyong Zhang, Xingyi Zhang, Jia Pan
Robotics
Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data
Linqi Yin, Shiduo Zhang, Shenling Qiu et al. (+11 more)
Vision-
language models (VLMs) are powerful general-purpose reasoners, yet converting them into
robot control policies (VLAs) is surprisingly difficult. The root cause is a two-fold gap: VLMs are trained on internet-scale images with
language-understanding objectives, while VLAs must perceive
robot scenes and predict motor actions. Fine-tuning a VLM directly on
robot action data forces the model to cross both gaps at once – the learning curve is steep and the rich generalizations learned during pretraining tend to degrade rather than transfer. We argue that this gap can be bridged gradually with the right intermediate data. We introduce
embodied trajectory-coupled (ETC) data –
vision-
language supervision derived from the same
robot scenes and trajectories used for action learning. Because ETC data shares the visual context of
robot operation while retaining familiar
language-understanding objectives, it provides a natural stepping stone between VLM pretraining and VLA fine-tuning. Building on this, we design a three-stage training recipe. Distribution Bridging first adapts the VLM to
embodied visual-
language semantics. Objective Bridging then gradually shifts the model toward action prediction while preserving the acquired representations. Retentive Adaptation finally specializes the policy to the target deployment domain. We further show that mixing task-relevant out-of-distribution ETC data with a small amount of action data enables the model to generalize to novel visual-
language conditions without requiring additional
robot demonstrations. Simulation and real-
robot experiments confirm that this gradual bridging strategy is the key to transferring VLM generalization into robust, deployable
robot policies.
Read the full abstract ▾
All authors Linqi Yin, Shiduo Zhang, Shenling Qiu, Chenxin Li, Zhaoyang Fu, Lei Xiao, Xiang Wang, Chenchen Yang, Zhe Xu, Pengfang Qian, Jingjing Gong, Xipeng Qiu, Xuanjing Huang, Yu-Gang Jiang
Robotics
Towards End to End Motion Planning and Execution for Autonomous Underwater Vehicles Using Reinforcement Learning
Elisei Shafer and Oren Gal
Autonomous
Underwater Vehicles (AUVs) traditionally rely on complex, heavily engineered pipelines for perception, path planning, and motion control. This paper explores the feasibility of an end-to-end Deep Reinforcement Learning (DRL) approach that maps raw sensor data directly to thruster commands, reducing manual engineering. We propose a hierarchical reinforcement learning (HRL) architecture splitting the problem into two Markov Decision Processes. A High-Level (HL) policy operating at 2Hz processes raw $84 \times 84$ pixel
monocular camera frames, stacked $100 \times 100$ pixel forward-looking imaging sonar, and proprioceptive data to generate spatial subgoals. Simultaneously, a Low-Level (LL) policy operating at 10Hz converts these subgoals into thruster commands. The HL policy is trained using Reinforcement Learning from Prior Demonstrations (RLPD) within a modified Sample-Efficient Robotic Reinforcement Learning (SERL) framework, while the LL policy utilizes Soft Actor-Critic (SAC) combined with Hindsight Experience Replay (HER). Evaluated in the high-fidelity HoloOcean simulator, our method demonstrates successful obstacle avoidance, achieving trajectory lengths closely approximating (within 4% to 6% of) an $\text{RRT}^*$ planning baseline. Furthermore, the learned policy exhibits strong robustness to simulated sensor noise and decreased visibility. While the system navigates familiar geometries effectively, experiments reveal generalization limitations when encountering unvisited areas with novel obstacle shapes. Ultimately, this work demonstrates the promise of sample-efficient, end-to-end DRL for
underwater navigation using minimal computational hardware.
Read the full abstract ▾
Robotics
ActProbe: Action-Space Probe for Early Failure Detection of Generative Robot Policies
Bingjia Huang, Xiangyu Li, Xiang Wang et al. (+7 more)
Generative
robot policies fail unpredictably at deployment: they hesitate at critical moments, drift off-task, or commit to unrecoverable actions. Existing online failure detectors either require white-box access to policy internals or add runtime overhead through resampling and observation-side signals. Our empirical analysis shows that emitted action chunks themselves already carry strong predictive signal for impending failures in generative
robot policies. Motivated by this observation, we introduce ActProbe, a lightweight, pure action-space detector that uses two compact signals available from a single forward pass: Temporal Consistency Error (TCE) between consecutive action chunks and Action Chunk Magnitude (ACM) of the current chunk. ActProbe maps these signals to per-step failure probabilities with a task-conditioned LSTM-MLP architecture. Across a diverse suite of generative
robot policies and
benchmarks, ActProbe raises alerts before failures become visually recognizable, improving the accuracy (F1)-timeliness Pareto frontier of failure detection by an average hypervolume gain of +12.7% over both internal- and external-feature baselines, with a +9.0% early-detection ROC-AUC lead on unseen tasks. ActProbe further transfers to deployment, predicting failures on unseen real-
robot pick tasks and accelerating RL fine-tuning (PPO) with 2.9x fewer environment interactions.
Read the full abstract ▾
All authors Bingjia Huang, Xiangyu Li, Xiang Wang, Liang Mi, Zixu Hao, Weijun Wang, Hao Wu, Kun Li, Yunxin Liu, Ting Cao
Robotics
EgoPriMo: Egocentric Motion Generation for Interactive Humanoid Control
Haoyang Ge, Peng Ren, Yukun Shi et al. (+3 more)
Humanoid robots require whole-body motions that adapt to scene context, task requirements, and user intent. Motion tracking reproduces specified trajectories, and
humanoid vision-
language-action systems provide semantic interfaces, but neither offers a scalable and interactive prior for broad full-body behavior. We introduce EgoPriMo (
Egocentric Motion Prior for
Humanoid Robots), a unified framework that learns such priors from
egocentric human demonstrations. Given
egocentric observations and a text
prompt, EgoPriMo reconstructs, generates, and forecasts SMPL-based full-body motion.
Language is used as a high-level control signal rather than a complete motion specification. At the core of EgoPriMo is a Triple-stream DiT that jointly models body dynamics,
egocentric visual context, and text; task-conditioning masks route different tasks and missing-modality data through the same checkpoint. Experiments on Nymeria and EgoExo4D show that one checkpoint improves
egocentric motion generation over UniEgoMotion while supporting reconstruction and
forecasting; the generated SMPL motions can also be executed by a Unitree
humanoid controller. These results indicate a practical path from scalable
egocentric observations to generalizable and interactive
humanoid motion priors.
Read the full abstract ▾
All authors Haoyang Ge, Peng Ren, Yukun Shi, Cong Huang, Kun Li, Kai Chen
Robotics
LUNA-AD: Lightweight Uncertainty-Aware Language Model with Lifelong Learning for Autonomous Driving
Ruoyu Yao, Pei Liu, Ruiguo Zhong et al. (+3 more)
While large
language models (LLMs) offer promising
reasoning capabilities, their integration into safety-critical driving systems is hindered by limited
reasoning diversity, high computational overhead, and static learning paradigms. To address these challenges, we propose LUNA-AD, a lightweight uncertainty-aware
language model with lifelong learning for autonomous driving (AD). LUNA-AD features a tri-system architecture that reconciles complex
multimodal behavioral
reasoning, efficient deployment, and continual refinement. We design a multi-
agent analytical system to generate uncertainty-aware decision-making demonstrations through diverse hypothesis exploration. A dual-head lightweight heuristic model is distilled to unify the inference of decision distributions and textual explanations while enabling efficient deployment. Furthermore, a reflection-driven lifelong learning mechanism operates on
multimodal decision outputs and preserves strategic diversity, allowing for the refinement of candidate decisions and rationales via closed-loop feedback to enhance driving robustness. Extensive experiments on nuPlan
benchmarks demonstrate that LUNA-AD achieves state-of-the-art success rates under both non-reactive and reactive modes, with drastically reduced inference latency compared to existing knowledge-driven AD frameworks.
Read the full abstract ▾
All authors Ruoyu Yao, Pei Liu, Ruiguo Zhong, Mingxing Peng, Rui Yang, Jun Ma
Robotics
Personalized and Robust Proactive Robot Assistance with Uncertainty-Guided LLM Reasoning
Alvaro Gonzalez, M.H. Hasan Shovo and Ali Ayub
Proactive
robot assistance in household environments requires accurate prediction of human activities and object usage under dynamic and noisy conditions. Existing approaches often rely on complex spatio-temporal models, which can be computationally expensive and sensitive to environmental variability. In this paper, we propose GLOBE, a lightweight framework that combines n-gram Markov models for capturing temporal behavioral patterns with uncertainty-guided large
language model (LLM)
reasoning. The framework performs sequential prediction efficiently while selectively invoking LLM
reasoning only when the model confidence is low. To evaluate performance under realistic conditions, we introduce HOMER-Noise, a noisy extension of the HOMER+
dataset that simulates structured disturbances such as object movements caused by humans, pets, and toddlers. Experimental results show that GLOBE achieves competitive performance with state-of-the-art methods while improving robustness and computational efficiency across both clean and noisy settings. The framework is further validated through a proof-of-concept integration with a Stretch 3 mobile manipulator, demonstrating its potential application in real-world human-
robot interaction scenarios.
Read the full abstract ▾
Robotics
GraspFoM: Towards Reconstruction-Driven Robotic Grasping with 3D Foundation Priors
Dongli Wu, Xiaobao Wei, Hao Wang et al. (+5 more)
Robotic grasping is a fundamental capability in robotic manipulation. Yet grasping remains challenging under partial observations. Reliable grasping depends on both local contact cues and object-level 3D structure. Existing geometry-aware grasping methods recognize the value of reconstruction, but they typically treat geometry as an intermediate prediction rather than a reusable object prior for grasping. In this paper, we present GraspFoM, a unified framework that leverages 3D foundation priors (SAM3D) to build a shared 3D object latent for both reconstruction and grasp pose prediction. Built on this shared object latent, we introduce an anchor-initialized truncated pose-
reasoning diffuser that predicts continuous and
multimodal grasp poses without directly relying on discrete grasp candidates. We further investigate the interaction between reconstruction and grasping through a reconstruction-aware scorer and a residual latent updater. Reconstruction provides grounded
geometric cues, while grasp supervision refines the shared object latent toward grasp-relevant affordances. GraspFoM jointly predicts grasp poses and reconstructs high-fidelity 3D assets in mesh and 3DGS forms. Comprehensive experiments demonstrate that GraspFoM achieves state-of-the-art results on both reconstruction and grasping. Notably, these improvements require only a small number of additional trainable parameters. Component-wise ablation studies also demonstrate the contribution of each component.
Read the full abstract ▾
All authors Dongli Wu, Xiaobao Wei, Hao Wang, Qiaochu Dong, Ying Li, Qingpo Wuwu, Ming Lu, Wufan Zhao
Robotics
PACT: Self-Evolving Physical Safety Alignment for Diffusion Policies in Embodied Manipulation
Lingxuan Wu, Zijian Zhu, Lizhong Wang et al. (+5 more)
Diffusion policies have achieved remarkable success in robotic manipulation, yet they often fail to satisfy strict physical constraints required for safe deployment. Existing approaches impose safety either prematurely during training or reactively via external guardrails at test time, limiting policy expressivity and overall scalability. We propose Physical safety
Alignment for Constrained Trajectories (PACT), a self-evolving post-training framework that projects pretrained
diffusion policies onto constraint-feasible regions without accessing demonstration data or task rewards. PACT distills constraint gradients into the
diffusion model through a reverse-KL objective with dense supervision across timesteps. It incorporates a curriculum that progressively tightens constraints while maintaining theoretically bounded policy shift and monotone improvement, mitigating the safety-performance trade-off from catastrophic forgetting. On simulated and real-world
embodied manipulation
benchmarks, PACT significantly reduces safety violations by 31.0% on average while improving task success by 30.7%.
Read the full abstract ▾
All authors Lingxuan Wu, Zijian Zhu, Lizhong Wang, Chengyang Ying, Huayu Chen, Xiao Yang, Fangming Liu, Jun Zhu