Robotics
SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving
Shaoyuan Xie, Zihan Zhang, Jingxuan Wang et al. (+6 more)
Vision-
Language-Action (VLA) models are gaining
attention in autonomous driving, inheriting commonsense priors from large
language models (LLMs) and widening the navigation interface from a fixed command set to free-form natural-
language instructions. However, this open-ended interface exposes a new vulnerability surface: an inattentive driver may issue instructions that are unsafe given the current scene, and a malicious user may issue dangerous ones on purpose. We present the first
benchmark suite that evaluates navigation following and conflict awareness together, along with
adversarial robustness. Existing VLAs fall short on both: SimLingo completes no
language-instructed lane change in closed loop, and all models degrade sharply under conflicting signals. We propose SafeDriveVLA, the first driving VLA that (1) explicitly predicts a three-level navigation-compliance mode to decouple conflict
reasoning from action generation, and (2) uses navigation-conditioned world model dreaming to inform this prediction. Unlike prior world models that roll out expert or planned actions, ours dreams of strictly following the navigation signal, showing when following is safe. SafeDriveVLA reaches 88.16 Driving Score (DS) and 68.64% Success Rate (SR) on Bench2Drive, lifts closed-loop
language following by 49.3% and reduces collisions under unsafe instructions from 60 to 38 compared to the best SimLingo variant.
Video demonstrations are available on our project page.
Read the full abstract ▾
All authors Shaoyuan Xie, Zihan Zhang, Jingxuan Wang, Jiashu Qu, Xiaoqing Liang, Lingdong Kong, Junchi Lu, Henrik I. Christensen, Qi Alfred Chen
Robotics
Hierarchical Residual Policy Learning for Real-World Mobile Manipulation with Sparse Human Guidance
Zhefei Gong, Hefeifei Jiang, Zehao Wang et al. (+1 more)
Mobile Manipulation (MoMa) is a cornerstone of autonomous robotic systems, yet real-world fine-tuning remains a formidable challenge because of high-dimensional action spaces and intricate coupling between heterogeneous modules. In this paper, we propose HiRe-MoMa (Hierarchical Residual Mobile Manipulation), a systematic framework for stable and sample-efficient real-world fine-tuning of complex MoMa systems. At its core, HiRe-MoMa adopts a Hierarchical Residual Architecture that fine-tunes a frozen base policy. A Policy-Agnostic Residual learns additive corrections atop the base policy, while Hierarchical Decoupling splits the residual into mobile-base and manipulator-arm heads with unidirectional conditioning, mitigating gradient interference and aligning corrections with the platform's physical hierarchy. To improve sample efficiency in data-expensive real-world settings, we propose Expert-Guided Value
Alignment, which leverages sparse human interventions with under 20% training coverage to reshape the critic's value landscape, effectively guiding autonomous exploration through expert priors. Extensive real-world experiments on the TIAGo++ platform demonstrate that HiRe-MoMa consistently enhances task performance with only around 30 minutes of online interaction, regardless of the base policy architecture. To our knowledge, this is among the first frameworks to enable efficient on-
robot fine-tuning for mobile manipulation in the real world, providing a practical paradigm for deploying complex robotic systems in dynamic, human-centric environments. Further details can be found on our project website: https://hire-moma.github.io/
Read the full abstract ▾
All authors Zhefei Gong, Hefeifei Jiang, Zehao Wang, Jiachen Li
Robotics
Modeling and Understanding Neural Weights of Humanoid Policies with Diffusion
Huixuan Cheng, Bike Zhang, Shankar Sastry et al. (+1 more)
Recent advances in
robot learning have driven rapid progress in humanoid control, enabling increasingly agile behaviors in the real world. However, humanoid control networks are typically obtained through conventional training procedures, while comparatively little
attention has been given to directly modeling the distribution of their learned weights and generating functional networks in weight space. In this work, we investigate whether high-performing humanoid control networks can be generatively modeled directly through their weights. We adopt a latent
diffusion framework that couples a linear autoencoder with a
Diffusion Transformer. The autoencoder compresses actor network parameters into a compact latent space, and the
Diffusion Transformer models the distribution of these latents. Trained on networks collected from reinforcement learning for the Unitree G1
robot, our approach generates deployable networks that perform various motions in the real world, including walking, dancing, and
spin kick. Beyond generation, we further study the learned weight space to better understand the structure captured by these humanoid control networks. We identify a structural principle in the first-layer weights that consistently reflects the
robot's kinematic prior. Notably, this kinematic prior emerges under both Reinforcement Learning (RL) and Imitation Learning (IL) training, suggesting that it is not specific to a particular optimization process but may instead reflect common knowledge associated with humanoid embodiment. Together, these results demonstrate the feasibility of generative modeling in humanoid network weight space, while providing initial insights into the information encoded by their learned weights. Additional results are available on our project website: https://humanoid-weight-
diffusion.github.io/
Read the full abstract ▾
All authors Huixuan Cheng, Bike Zhang, Shankar Sastry, Koushil Sreenath
Robotics
CoWear: Towards Collaborative Wearable Inertial Localization with Multiple Smart Devices
Zhuoyuan She, Zhanpeng Zhang, Yapeng Chen et al. (+3 more)
Consumer smart devices observe the same human motion from different body locations, yet existing inertial-localization
benchmarks rarely provide synchronized heterogeneous wearables with device-specific motion ground truth. We present CoWear, a
dataset and reference framework for collaborative inertial localization using a smartphone, smartwatch, and smart glasses. The
dataset contains over 30 participants, over 300 walking sessions, and more than 30 hours and 100 km of aggregate device trajectories, with each device individually tracked by motion capture. To enable collaboration across distinct body motions, we train role-specific
estimators to predict the displacement of a common reference device over synchronized temporal windows. Their predictions are transformed to the world frame using gyroscope integration and fused according to learned uncertainty. Experiments using phone, glasses, and watch as reference targets show that multi-device fusion consistently improves both displacement and trajectory rollout accuracy over individual-device estimation. CoWear provides a
dataset and reproducible baseline for studying collaborative inertial localization across heterogeneous consumer wearables.
Read the full abstract ▾
All authors Zhuoyuan She, Zhanpeng Zhang, Yapeng Chen, Jiale Wang, Xin Xie, Ming Xia
Robotics
PingpongDuo: Agile Humanoid Table Tennis Doubles with Safe Team Coordination
Haoran Jiang, Yichao Zhong, Kai Zhang et al. (+6 more)
Unlike singles, humanoid table tennis doubles requires teammates to strike in strict alternation while sharing a confined workspace. Rapid repositioning can compromise striking accuracy, while uncoordinated teammate movements introduce collision risks. We present PingpongDuo, a hierarchical framework for agile humanoid doubles with safe team coordination. At the team level, a planner jointly selects positioning targets and uses learned risk estimates to filter risky proposals, preferentially preserving the current striker's target. At the individual level, separate footwork and striking skills combine agile movement with precise racket control. The skills are guided by human motion references, including footwork collected from human doubles play, and trained from states produced by one another to support rapid transitions. A racket-arm residual further refines strikes after movement using return feedback from a simulator calibrated with real ball-flight and contact measurements. In simulation, PingpongDuo improves the ball-return success rate by 13.2% over the state-of-the-art baseline, while achieving approximately 2 cm positioning error during clearance. In real-world experiments, a
robot completes a 2.15 m clearance-and-recovery cycle in 2.4 s with a peak speed of 2.23 m/s, while two
robots sustain more than 10 consecutive strikes in strict alternation.
Read the full abstract ▾
All authors Haoran Jiang, Yichao Zhong, Kai Zhang, Cheng Liu, Yixuan Pan, Zhenjie Yang, Jingbo Wang, Tianyu Li, Li Chen
Robotics
KinoGen: Customizable Humanoid Loco-Manipulation References via Kinematically Grounded Video Generation
Qi Wu, Lulu Zhao, Sanghyun Hahn et al. (+6 more)
We present KinoGen, a framework that synthesizes customizable humanoid loco-manipulation references through kinematically grounded
video generation. KinoGen conditions a pretrained
video generator on calibrated renders of the target
robot and object in a simulated task instance and recovers the reference by fitting the same models to the generated two-view
video, so that the reference lies in the
robot configuration space by construction without a human body proxy or retargeting. Body, hand, and object priors initialize the reconstruction, and direct optimization against cross-view evidence refines it. Since an edited keyframe is simply another rendered state, a user can prescribe a goal object pose, regenerate the
video around the edited keyframe, and obtain a reference that satisfies the pose exactly. Across eight loco-manipulation tasks, KinoGen yields lower penetration and foot skating than
video-based baselines and preserves over 90% of annotated contacts, and its customized references satisfy every prescribed target with substantially less penetration than direct interpolation. Tracking policies trained on the synthesized references are deployed on a Unitree G1 humanoid with dexterous hands. Project webpage: https://kinogen-humanoid.github.io/.
Read the full abstract ▾
All authors Qi Wu, Lulu Zhao, Sanghyun Hahn, Rishi Yennu, Emmy Yang, Dingqi Zhang, Tao Pang, Jiuguang Wang, Kuan Fang
Robotics
EarlyCommit: Calibrated Word-by-Word Commitment for Robots Acting Before You Finish Speaking
Muhammad Haris Khan
Human listeners can often identify the intended object several words before an instruction ends.
Robots, however, are typically designed to wait until the speaker has finished before deciding how to act. We present EarlyCommit, an open-source tool that lets a
robot commit to the intended object while the instruction is being spoken. After every recognized word, a fine-tuned text encoder scores each object in the scene. The
robot acts when the calibrated probability of the best object reaches a threshold tau set by the user, so 1 - tau is the accepted risk of choosing the wrong object. Otherwise, it waits or asks. The tool includes a streaming
speech front end, a scene and instruction generator with exact word-by-word ground truth, and scripts to train, calibrate, and evaluate. On instructions containing words never seen in training, EarlyCommit commits correctly 1.6 to 2 times as often as keyword matching, is right at least 92 percent of the time when it commits at tau = 0.9, and needs 31 to 35 ms per word on consumer GPUs.
Read the full abstract ▾
Robotics
Physical Time as a Missing Dimension of Robot Policy Memory Dynamics
Yushen Liang, Yue Peng, Baosheng Jin et al. (+6 more)
Existing
robot policy
memories typically evolve with observation sequences, implicitly treating each update as the same amount of physical time. Yet identical observation histories can span different durations and require different actions, exposing physical time as a missing dimension of policy
memory. We introduce Temporal Flow Policy (TFP), which learns
memory dynamics to retain past events and evolve the
memory state over measured time intervals. Observations and prior
memory determine the content and time scales of each transition; elapsed time determines how far the state advances. TFP conditions
diffusion and flow-matching policies through this
memory state and is trained with their native imitation objectives. We evaluate TFP with
Diffusion Policy and $\pi_{0.5}$ through controlled comparisons of
memory designs. These experiments show that successful timing depends on both measured intervals and retained history. TFP achieves higher average success than the recurrent and state-space
memories we equip with the same intervals and than a variant that feeds the interval to a learned gate.
Memory readouts recover the time since an event and motion information after occlusion. Changing the supplied intervals systematically delays or advances actions, linking time inputs to behavior. Real-
robot tasks requiring timing or
memory, together with LIBERO evaluations, demonstrate benefits in physical and general manipulation. These results suggest viewing
robot memory not as stored observation history, but as a state that retains past events and evolves with physical time.
Read the full abstract ▾
All authors Yushen Liang, Yue Peng, Baosheng Jin, Tianluo Zhang, Xinyu Zhang, Shuyi Zhou, Zhuoran Chen, Xinqi Liu, Shenji Wan
Robotics
Graph-Reweighted Behavior Cloning with Human Interventions for Precise Robotic Manipulation
Xiaoman Li, Zhiqiang Yang, Jiayu Hu et al. (+6 more)
Long-horizon precise manipulation produces mixed-quality data for post-training
vision-
language-action (VLA) models: even successful demonstrations and intervention-assisted rollouts may contain detours, hesitations, failed attempts, and imperfect recoveries. The key challenge is to identify which action chunks actually advance the task, rather than uniformly cloning or discarding imperfect trajectories. We propose Graph-Reweighted Behavior Cloning (GR-BC), an offline iterative post-training framework that builds a task-conditioned goal-reachability graph over demonstrations and iteratively collected rollouts, including both successful and failed episodes with human interventions. Reverse graph search computes minimal remaining steps to task success within the
dataset support, and each action chunk is weighted by how much its execution reduces this potential. This consequence-based weighting reuses recoverable segments from failed trajectories, leverages interventions as recovery paths, and down-weights locally regressive chunks even inside successful trajectories. New rollouts are incorporated by appending them to the graph and rerunning search, without training a reward model, action-conditioned critic, or progress predictor. Across four precise manipulation tasks and three VLA backbones, GR-BC improves single-round supervised fine-tuning and reaches 91% success after a few offline iterations, without trajectory comparisons or progress labels. More details and
videos are available at: https://gr-bc.github.io
Read the full abstract ▾
All authors Xiaoman Li, Zhiqiang Yang, Jiayu Hu, Jipeng Kong, Yang Zhang, Jingwei Zhang, Fangzheng Yan, Chunhua Shen, Chenjia Bai
Robotics
Towards Certifiable State Estimation under Observability Degeneracy
Baoshan Song
Certifiable optimization provides global optimality guarantees for non-convex state estimation in robotics. However, under severe physical sensing degeneracy, convex Semidefinite Programming relaxations solved via Interior-Point Methods suffer from rank expansion, rendering direct semidefinite solving and spectral rounding algebraically invalid. In this paper, we establish a universal candidate-driven dual certification framework that resolves this challenge without incurring rank expansion. First, from an epigraph perspective, we prove that while sensing degeneracy expands the primal optimal set into a flat equi-cost face, cost tightness and multiplier recovery remain strictly intact, enabling global verification without direct semidefinite solving. Second, we propose an observability-aware local search using solution remapping, coupled with a closed-form dual multiplier recovery scheme that executes in sub-microsecond latency. Third, we derive a model-specific adaptive dual safety threshold parameterized by the minimum non-zero
eigenvalue of the observable Hessian and uneradicated residual leakage, preventing false rejections of optimal candidates under weak observability. We validate our universal framework with robust registration under degenerate measurements.
Read the full abstract ▾
Robotics
Tiered-Resolution Volumetric Mapping for Real-Time Simultaneous Planning and Mapping on Natural Outdoor Environments
Afonso E. Carvalho, David Portugal, Paulo Peixoto et al. (+1 more)
Real-time volumetric mapping for a planning
robot is dominated by the cost of integrating and storing information at the finest resolution, yet only a small fraction of any map ever lies on or near a path the planner actually considers. We exploit this with a tiered-resolution mapping mechanism that integrates
LiDAR into a single map structure at two resolutions simultaneously: fine inside a corridor around the
robot's current plan, and coarse everywhere else. As opposed to traditional navigation pipelines where path planning uses the mapping result and provides no feedback, we propose a feedback loop where planning also informs mapping to dynamically integrate information at different leaf resolutions. Across two real-world teleoperated
datasets, replayed and scored offline against an independent full-resolution map, the tiered maps are 13-15x smaller and require 6-8x less integration CPU time than a uniform fine map, while producing paths at a similar cost per meter as the ones generated from fine resolution maps on both scenarios. A uniform coarse map reaches the same size savings but its path quality collapses (+210% cost per meter) on the harder scenario, because it over-discretizes terrain the
robot must traverse. Tiering approximates the coarse map's cheapness without its failure mode.
Read the full abstract ▾
All authors Afonso E. Carvalho, David Portugal, Paulo Peixoto, George Kantor
Robotics · Cross-listed from cs.CL
AdaRSI: On-Demand Recursive Self-Improvement for Embodied Agents
Peidong Wang, Ying Chang, Zhiming Ma et al. (+8 more)
Pretrained
vision-
language-action models (VLAs) often fail midway through unseen long-horizon tasks. Recursive self-improvement (RSI) offers a remedy, but embodied RSI methods either refine the harness around a frozen executor and so cannot add a missing motion, or train the executor on persistent failures without asking whether a failure lies in how the executor is called or in what it can do. We ask when a self-improving embodied system should change the weights of its executor, and propose AdaRSI, an on-demand RSI framework for embodied
agents. A coding
agent first improves its skills and harness, then attributes persistent failures by retrying failing sub-steps from saved states, and fine-tunes only the sub-steps that remain unreliable, with demonstrations it collects itself and experience carried over from earlier tasks. On all 50 target tasks of RoboCasa365, with GR00T N1.5 as the executor and no human demonstrations, AdaRSI improves the VLA on all three suites and raises its success on unseen composite tasks from 4.6% to 13.8%. Compared with fine-tuning every sub-step, it reaches slightly higher success while using only 28% of the GPU time and 22% of the self-collected demonstrations, and fine-tuning without a harness trains more yet gains far less.
Read the full abstract ▾
All authors Peidong Wang, Ying Chang, Zhiming Ma, Xiaocui Yang, Shi Feng, Daqi Liu, Yuzhe Yang, Xu Cao, Chuang Wang, Jian Su, Daling Wang
Robotics
DD-GBP: Anytime Multi-Arm Planning and Continuous Execution via Distributed Dynamic Gaussian Belief Propagation
Zhao Jin, Fengyi Dai, Yuetong Yin et al. (+3 more)
Safely executing long-horizon multi-arm tasks online in complex dynamic environments remains challenging. We propose a new distributed dynamic Gaussian belief propagation (DD-GBP) method that directly plans joint-space trajectories through parallel local inference. A unified Gaussian-process (GP) parameterization connects adaptive planning horizons, interpolation-time factors, and continuous execution. Each
robot maintains a local tree-structured factor graph and exchanges messages asynchronously over active inter-arm links. Dynamic subnet reconstruction and event-triggered updates reduce communication and inference costs while responding to moving obstacles. We also provide theoretical analyses of local inference accuracy, anytime early stopping, triggered-replanning consistency, and GP-consistent execution. In 50 long-horizon, multi-target four-arm scenes with moving obstacles, DD-GBP achieves $92.5\%$ collision-free all-target completion and the highest success rate among six representative baselines. Scenario ablations demonstrate long-horizon safety and robustness to communication impairment and extreme conditions; success remains $86.0\%$ under $90\%$ packet loss. Execution results further show substantially suppressed extreme velocity and acceleration peaks relative to Syn-GBP. On one workstation, obstacle-free tests with 30 coexisting arms retain $99.87\%$ success and a $15.80$ ms mean warm solve, exceeding 60 Hz and outperforming the compared methods in scalability. Dual-arm transport experiments under dynamic interference demonstrate online response and practical deployability. To the best of our knowledge, these results establish a new state of the art for online long-horizon multi-arm planning in dynamic scenes.
Read the full abstract ▾
All authors Zhao Jin, Fengyi Dai, Yuetong Yin, Yunan Wang, Wenjie Chen, Chuxiong Hu
Robotics
Towards Agentic Robots: Model, Data, Environment, and Harness
Chonghan Qin, Dongming Wu, Kaituo Feng et al. (+13 more)
Foundation models are driving a paradigm shift in robotic intelligence, moving beyond low-level action prediction to goal-directed orchestration of data collection, model training, task planning, and real-world execution. We characterize this emerging paradigm as an agentic
robots, in which
agents reason over human instructions, interact with robotic environments, rectify execution errors autonomously, and dynamically adjust operational strategies. Although this field develops fast, current studies are not well connected. Researchers have no consistent view of how to combine different agentic functions into a complete
robot system. This paper provides a unified overview of existing research on agentic
robots. We first trace the development from
robot policies to agentic systems, covering
vision-
language-action models, world models, and world-action models. We then clarify the definition and scope of agentic
robots and organize existing research into seventeen categories across four functional domains: Model, Data, Environment, and Harness. Across these domains, we examine how
agents shape model learning and
robot decisions, collect and curate
robot data, construct worlds, tasks, and tests for learning and evaluation, and organize capabilities for
robot use. Finally, we discuss open challenges and outline directions for future research. Looking forward, we hope this work contributes to a clearer understanding of agentic
robots and supports new approaches to making them more capable, reliable, and adaptive.
Read the full abstract ▾
All authors Chonghan Qin, Dongming Wu, Kaituo Feng, Jiaxu Wang, Xinlei Yu, Peiwen Sun, Yijia Fan, Xilin He, Bingao Chen, Weitong Lian, Ryan Ji, Wenqi Ren, Xiaobin Hu, Dahua Lin, Yu-Gang Jiang, Xiangyu Yue
Robotics
When Do Discrete Actions Make Good Continuous Controllers?
Zeji Yi, Soham Dasgupta, Xinyang Zhang et al. (+5 more)
Though modern AI models like LLMs predict discrete
tokens, today's robotic foundation models represent actions as continuous vectors. In this paper, we challenge this common consensus by asking if continuous action parametrizations are superior to their discrete counterparts. We conduct an extensive comparison of single-step regression and multi-step flow variants of both continuous and discrete policies in both from-scratch single-task and multi-task VLA training, in both simulation and on real hardware. In all regimes, we find that discrete action prediction, even the most naive, uniform-bin quantization, can match or even outperform continuous-action policies in task success rate. Moreover, we find that the challenge to discrete action policies is not action precision, but rather factorization error introduced by product-distribution predictions across time and action dimensions. We show that, when paired with the correct interventions—iterative computation through discrete flows, or distribution sharpening through classifier-free guidance—discrete flow policies can turn this weakness into a strength, enabling more responsive action changes during a predicted action chunk. Lastly, we show that attempts to explicitly inject
geometric information into discrete policies, e.g. through corruption processes that emulate continuous flow matching, do not yield consistent improvements. Taken together, our findings question existing assumptions that robotic action prediction requires distinct design decisions compared to other AI domains, and motivates the possibility of discrete action prediction as a compelling alternative to today's continuous action parametrizations. Link for the webpage \href{https://iscoyizj.github.io/discrete-action-website/}{https://iscoyizj.github.io/discrete-action-website/}
Read the full abstract ▾
All authors Zeji Yi, Soham Dasgupta, Xinyang Zhang, Koushil Sreenath, Guannan Qu, Xi Chen, Max Simchowitz, Guanya Shi
Robotics
Human Demonstrations as Post-Training Data for Any Robot Body: Retargeting the Object, Not the Hand
Nagasrikanth Kallakuri
Human demonstrations are today used largely for pretraining
vision-
language-action models, with little ability to use the same data for target-embodiment or task-oriented training, which scalable deployment of
robots in industrial and general environments depends on. A human demonstration recorded on a simple and scalable device such as a mobile phone can serve as post-training data for a
robot whose body differs in structure and capability from the human who recorded it. This is made possible not by retargeting the hand's motion but the object's. A capture yields a body-free record: the object's pose over time, its contact intervals, and a reconstructed 3D object with provenance. Any
robot, given as a URDF and a short descriptor, is solved against that record and scored in simulation against the demonstrated object motion as ground truth: the human-to-simulation leg of human-to-simulation-to-
robot transfer, measured per
robot. Each (
robot, episode) pair gets one outcome: pass, failure with a named mode, or refusal, with its reason, before any motion. The method runs as a working system over a growing collection; the set analysed here is a sample of it: 42 captures solved on 8
robots, 157 pairs, of which 31.2% pass, 52.2% fail, 14.0% are refused and 2.5% are unauditable (kinematic feasibility, +/-50 mm / +/-15 deg, in simulation). Refusals held up: every pair the checks refuse on stored evidence yet ran anyway failed (13 of 13; 5 on a body whose public model welds its hand). A description file could not be shown to screen a
robot: its outcomes were barely more predictable than guessing. No policy has yet been trained on the record. The record decides what a
robot may post-train on; whether the policy then works on the real
robot is the simulation-to-
robot question, answered downstream.
Read the full abstract ▾
Robotics
AutoRSR: Automated Real-to-Sim-to-Real Data Generation for Scalable Robot Learning
Jiawei Zhang, Chang Liu, Litao Deng et al. (+2 more)
Scaling embodied
robot learning requires large and diverse manipulation
datasets, but collecting real-world demonstrations remains expensive in human effort and
robot time. Simulation provides a scalable alternative, yet converting a real workspace into a
robot-learning-ready environment still requires substantial manual effort for object extraction, asset processing, coordinate calibration, and trajectory specification. We present AutoRSR, a highly automated real-to-sim-to real framework that converts scanned real-world workspaces into reusable embodied simulation assets and scalable manipulation data. Given multi-view observations,
robot states, and open-vocabulary semantic queries, AutoRSR reconstructs the workspace, extracts individual object assets, and automatically registers the reconstructed scene to the
robot base frame at metric scale. Starting from a small number of reference motions, AutoRSR then generates executable manipulation trajectories and scales them through trajectory interpolation and domain randomization. After workspace scanning and semantic queries, the remaining pipeline requires no manual object annotation, scene-to-
robot calibration, or
robot teleoperation. Across three real-world manipulation tasks, policies trained on 1K automatically generated episodes per task achieve 73.3–86.7% real-world success rates, comparable to pi 0.5 trained from 150 human teleoperated demonstrations, while reducing the human time required for data generation by an order of magnitude. These results demonstrate that real-world observations can be converted into metrically aligned, executable, and scalable simulation data with substantially reduced human effort.
Read the full abstract ▾
All authors Jiawei Zhang, Chang Liu, Litao Deng, Tao Zhang, Qin Jin
Robotics
VIGOR-MPC: Vision-Guided Objective Regulation MPC
Jelena Trisovic, Rahel Rickenbach, Marek Landert et al. (+2 more)
Controlling an
agent in an environment solely perceived through onboard
vision is challenging: perception yields noisy, variable-length, and partially occluded context, while onboard compute is shared between perception and control, limiting the planning horizon executable in real time. We propose VIGOR-MPC, a modular student-teacher framework, agnostic to the perception pipeline, which addresses reference tracking across navigation tasks by distilling a privileged, field-of-view-aware long-horizon MPC into a computationally lightweight short-horizon MPC whose cost parameters are determined online from
vision. The teacher operates in simulation with full state knowledge and a multi-trajectory formulation that guarantees the
agent can stay within its field of view. The student observes only the noisy, occluded context produced by a perception module and learns, via a multi-level recurrent mapping, to infer cost parameters that aim to recover the teacher's behavior despite missing information. We evaluate VIGOR-MPC on
vision-based autonomous racing in simulation and on hardware. Using only onboard perception of the track, VIGOR-MPC generalizes to tracks unseen during training and approaches the lap times of the privileged long-horizon teacher while reducing per-step computation by over 90%.
Read the full abstract ▾
All authors Jelena Trisovic, Rahel Rickenbach, Marek Landert, Panagiotis Minos, Melanie N. Zeilinger
Robotics
TacClaw-Infinity: A Large-Scale Vision-Tactile-Language-Action Dataset for Dexterous Manipulation
Fucai Zhu, Xidong Zhang, Yichi Zhang et al. (+23 more)
Large-scale
robot learning increasingly relies on diverse human demonstrations, yet tactile manipulation data remain difficult to scale, and existing
multimodal datasets often lack consistent quality control and structured supervision. We present TacClaw-Infinity, a large-scale structured
Vision-Tactile-
Language-Action (VTLA)
dataset for contact-rich manipulation, collected with TacClaw, a tactile-enabled UMI-style handheld interface that records synchronized visual observations, fingertip tactile signals, and end-effector trajectories without requiring a target
robot during data collection. TacClaw-Infinity is built through a distributed worldwide collection across homes, laboratories, industrial workplaces, retail stores, and outdoor environments including tasks contributed by academic and industrial collaborators. Together, these efforts yield over 10,000 hours, 1 million episodes, and 1 billion frames spanning more than 1,000 task types. We develop a scalable pipeline for trajectory reconstruction,
multimodal quality control, embodiment-aware filtering, and structured annotation, covering contact-centric segmentation, subtask description, and spatial grounding. We validate TacClaw-Infinity at both the data and model levels: At the data level, we assess the collection interface, the action-space compatibility of the curated demonstrations, and the structured annotations, while for the model level, we demonstrate cross-embodiment transfer across three heterogeneous
robots without
robot-specific demonstrations.
Read the full abstract ▾
All authors Fucai Zhu, Xidong Zhang, Yichi Zhang, Yipan Wang, Fubo Wang, Zibin Wu, Kaiqi Li, Jianhao Jiao, Shihan Lu, Di Hu, Mengdi Xu, Siyu Zhu, Xiaoxiao Long, Yao Yao, Xun Cao, Zhuo Xu, Koichi Hashimoto, Qifeng Chen, Zhe Li, Jianfei Yang, Haofei Lu, Danica Kragic, Shuicheng Yan, Jianghua Duan, Michael Yu Wang, Weihao Yuan
Robotics
Efficient Test-Time Adaptation For Robot Policies
Motasem Alfarra, Pietro Mazzaglia, Markus Peschl et al. (+2 more)
Reinforcement-learning policies for
robot locomotion can fail at deployment when operating conditions shift (e.g., rough or slippery terrain, payload changes, sensing degradation), and retraining or manual recalibration is often infeasible under on-device latency and
memory constraints. We study test-time adaptation (TTA) for robotic control: adapting a pretrained policy online and in a self-supervised manner during execution, without access to rewards. Our starting point is a lightweight baseline that updates only the actor using a frozen pretrained critic as a deployment-time performance surrogate. Building on this, we introduce TEMPO, a test-time
entropy-regularized,
memory-guided policy optimization method that stabilizes critic-driven updates under severe shift. TEMPO combines (i) an
entropy-promoting regularizer on the critic's predictive distribution to mitigate overconfident extrapolation, and (ii) a score-based
memory, inspired by prioritized replay, that prioritizes low-value ("hard") observations and applies importance sampling to focus limited computation on the most informative deployment conditions. Across Go1, Go2, and G1 in MuJoCo Playground under four representative shifts and multiple architectures, TEMPO yields consistent gains, reaching up to 80% improvement in a single-device setting. We further validate on a physical Unitree Go2 with an added calf payload, where fully on-device adaptation improves performance over the non-adapted baseline across real-world episodes.
Read the full abstract ▾
All authors Motasem Alfarra, Pietro Mazzaglia, Markus Peschl, Daniel Dijkman, Christos Louizos
Robotics
Learning-Based Adaptive MPC for Magnetorquer-Only Satellite Attitude Control
Patrick Schwartz and Abhishek Cauligi
Nanosatellites (e.g., CubeSats) have the potential to carry out useful science and exploration activities for outer solar system exploration, but their small size leads to tradeoffs between attitude control and science capabilities. Magnetorquers offer a unique compromise as compact actuators that enable three-axis attitude control, but suffer from a loss of control authority and require full knowledge of the planetary
magnetic field. In this work, we present a learning-based adaptive model predictive control (MPC)
algorithm that plans rotational trajectories for the satellite while accounting for this loss of control authority. Key to our approach is the inclusion of a learned
magnetic field model that is trained in simulation using low-fidelity
magnetic field data. Using measurements collected online, our proposed
algorithm adapts this learned
magnetic field model using a simple last-layer update scheme, allowing the MPC controller to quickly adapt to the true planetary
magnetic field model and more effectively plan attitude trajectories. We demonstrate the results of our proposed approach for two use cases, (1) Earth-orbit and (2) Uranus-orbit, and show that our learning-based adaptive MPC
algorithm improves controller success for these system simulations by 5% and 61%, respectively.
Read the full abstract ▾
Robotics
Safe Reinforcement Learning with Receding-Horizon Safety Critic and Differentiable Action Shields
Siwei Ju, Jan Peters and Oleg Arenz
Despite the Markov assumption, safety failures in
robot reinforcement learning rarely occur instantaneously; they typically emerge over several steps as the policy enters unfamiliar states under disturbances and domain shift. We address this limitation by learning a receding-horizon safety critic that predicts whether a state will remain non-terminating over the next receding horizon, enabling anticipatory detection of impending locomotion failures. The critic is trained from rollout-derived horizon labels with a composite objective that combines horizon classification, control-barrier-function-inspired temporal consistency, and smoothness regularization, and we aggregate an ensemble of critics with a conservative Conditional Value at Risk estimate to account for uncertainty. Building on the learned critic, we derive a closed-form, differentiable CBF-style safety filter that minimally modifies the policy action to improve predicted safety, and integrate it into PPO with an anticipation loss to reduce policy-filter mismatch during training. In simulated quadruped locomotion, our method improves robustness under action noise, external perturbations, and terrain shifts, reducing catastrophic failures while maintaining competitive returns.
Read the full abstract ▾
Robotics · Cross-listed from cs.CG
FPTAS for Polygonal Multi-Robot Motion Planning
Sayan Bandyapadhyay, William Lochet, Saket Saurabh et al. (+1 more)
We study multi-
robot motion planning in a polygonal environment with polygonal
robots. The setting of the problem consists of a polygonal domain $\mathcal{W}$ with $n$ vertices and $k$
robots $R_1,\dots,R_k$ translating in $\mathcal{W}$ each of which is represented by a polygon of constant complexity. Given a source configuration $\mathbf{s}$ and a target configuration $\mathbf{t}$ of the
robots in $\mathcal{W}$, the goal of the problem is to find a collision-free motion plan from $\mathbf{s}$ to $\mathbf{t}$ that is shortest with respect to some cost function. We present the first fully
polynomial-time approximation schemes (FPTAS) for motion planning with any fixed number $k$ of
robots, under the min-sum objective, the min-max objective, and the min-makespan objective. Here, the min-sum variant aims to minimize the total traveling distance of all
robots, the min-max variant aims to minimize the maximum traveling distance of a single
robot, and the min-makespan variant aims to minimize the total time cost under the assumption that all
robots move with a maximum speed $1$. The
algorithm directly generalizes to the problem in any fixed dimension $d$, with a polytope environment and polytope
robots. Our result settles an open question of Agarwal, Halperin, Sharir, Steiger [SODA'24] and Agarwal, Holmgren, Steiger [SODA'26], who gave an FPTAS for $k=2$ unit-square
robots and a bicriteria-approximation
algorithm for a fixed number $k$ of unit-square or unit-disk
robots, respectively. Compared to the previous results, our
algorithm is substantially simpler.
Read the full abstract ▾
All authors Sayan Bandyapadhyay, William Lochet, Saket Saurabh, Jie Xue
Robotics
Learning Navigation from Multimodal Demonstrations with Geometric Candidate Modes
Chengyang Peng, Zhihao Zhang, Upama Pant et al. (+3 more)
Offline navigation demonstrations are
multimodal, because one observation often allows several valid actions, such as passing an obstacle on either side. Behavior cloning averages these actions, potentially leading to unsafe routes.
Multimodal policies must discover the alternatives from the data, but similar states can lead to different routes whose current actions overlap. In navigation, the alternatives have a
geometric cause and can be computed from one observation. We propose a hierarchical neural-network policy that computes candidate modes from local geometry and learns from demonstrations which mode to follow and how. A high-level policy selects a mode using the goal and its previous choice among a set of candidate modes, and a low-level policy turns it into an action. In simulation, the policy succeeds in over 95% of seen and unseen test cases, compared with at most 43% and 20% for four offline learning baselines and 79% and 67% for a heuristic that uses the same modes without learning. The simulation-trained policy also transfers to a LimX Tron 1 bipedal
robot.
Read the full abstract ▾
All authors Chengyang Peng, Zhihao Zhang, Upama Pant, Yiming Wang, Keith A. Redmill, Ayonga Hereid
Robotics
Smart Fusion for Adaptive Visual-IMU Weighting in cuVSLAM Sparse Bundle Adjustment
Wonju Lee, Dara Molloy, Abhilash Nand Kumar et al. (+5 more)
In cuVSLAM, visual and inertial measurement unit (IMU) residuals are fused in sparse bundle adjustment (SBA) with a weighting fixed by pre-calibrated noise covariances, leaving no mechanism to adapt the visual-inertial balance at runtime. We introduce a scalar $\alpha\geq1$ that upscales the visual information
matrix relative to the IMU preintegration constraint. Across the candidate set $\{1.0,2.0,5.0,10.0,20.0\}$, the resulting trajectories form two empirical performance regimes. The IMU-TRUST regime contains $\alpha\in\{1,2\}$ near the calibrated baseline, while the
VISION-TRUST regime contains $\alpha\in\{10,20\}$. The intermediate value $\alpha=5$ is treated as a transition case. Differences within each regime are small relative to those observed between the regimes. This structure motivates reducing $\alpha$-selection to a binary decision. We train a lightweight Visual Quality
Estimator (VQE), implemented as a multilayer perceptron with 914 trainable parameters. It maps a 10-dimensional visual quality feature vector assembled from statistics already maintained by the tracker to a per-SBA regime prediction. The proposed method reduces mean RMSE APE by $27.7\%$ and by $14.9\%$ under leave-one-out cross-validation (LOOCV) on EuRoC MAV and TUM-VI Room, respectively.
Read the full abstract ▾
All authors Wonju Lee, Dara Molloy, Abhilash Nand Kumar, Onias Castelo, Petar Trslic, David Adley, Conor Ferguson, Hesam Rabeti
Robotics
ObjDream: Guiding Vision-Language-Action Models with Object-Level Understanding and Future State Prediction
Huailiang Ma, Yuxuan Li, Ziyi Zhao et al. (+7 more)
Vision-
language-action (VLA) models have emerged as a powerful paradigm for robotic manipulation, enabling
robots to generate actions from visual observations and
language instructions. However, most existing VLA models still operate on image-level visual representations inherited from pretrained
vision-
language backbones, while lacking fine-grained object understanding for manipulation. As a result, their robustness can degrade under out-of-distribution object layouts, clutter, and background variations. To address these limitations, we introduce ObjDream, an object-centric VLA framework centered on object-level understanding and future object state prediction. ObjDream uses a spatio-temporal object encoder to organize visual observations into temporal object slots, where persistent object identity and dynamic object state are represented through separate channels. These structured object representations are inserted into the
token sequence of the
vision-
language model (VLM), where identity-guided routing modulates VLM
attention to the corresponding object
tokens. In addition, an action-conditioned object dynamics module predicts compact future object states relevant to manipulation during training, explicitly supervising object state
evolution under
robot actions without requiring future prediction at inference. Experiments on LIBERO, LIBERO-Plus, and RoboTwin 2.0 show that ObjDream improves generalization under diverse distribution shifts and scene perturbations. Real-world
robot experiments further demonstrate that ObjDream transfers to physical manipulation settings with novel object arrangements and background changes.
Read the full abstract ▾
All authors Huailiang Ma, Yuxuan Li, Ziyi Zhao, Xinkai Wang, Mingyu Li, Xinyu Guo, Lifeng Zhu, Aiguo Song, Cewu Lu, Jianhua Sun
Robotics
Investigating Behavior Cloning from Few Demonstrations for Autonomous Driving based on Bird's-Eye View in Simulated Cities
Eric Aislan Antonelo, Gustavo Claudio Karl Couto, Christian Möller et al. (+1 more)
This paper investigates the use of Behavior Cloning (BC) for autonomous driving from a bird's-eye view (BEV) perspective in simulated urban environments. BC uses supervised learning to mimic expert driving behaviors. Previous works have applied BC in the CARLA simulator but did not fully address the challenges of traffic light compliance. Our approach enhances BC by integrating a kernel density
estimator to adjust training sample weights based on action density, thereby improving the learning of rare but critical actions such as stopping at red lights and accelerating at green lights, specially in scenarios of scarce number of expert demonstrations. Using BEV inputs, which provide an abstract top-down view of the driving environment, our method simplifies the policy learning process. The trained convolutional neural network (CNN) outputs steering and acceleration actions based on these BEV inputs and additional state variables. Experimental results in the CARLA simulator demonstrate that our weighted BC method significantly improves driving performance, achieving higher route completion compared to standard BC. This weighted approach proved to be crucial in learning correct driving behaviors, particularly in test environments not encountered during training, highlighting its potential for enhancing autonomous vehicle navigation.
Read the full abstract ▾
All authors Eric Aislan Antonelo, Gustavo Claudio Karl Couto, Christian Möller, Pedro Henrique Fernandes
Robotics
Atomic-WAM: Execution-Grounded Atomic-Motion Reasoning for World Action Models
Shijun Shi, Zhe Liu, Zhi Hou et al. (+5 more)
World Action Models (WAMs) typically adapt
video foundation models to robotic control through joint
video and action prediction. However, task-level instructions may leave the next physical interaction underspecified, requiring semantic grounding and state-dependent motion selection. To this end, we introduce Atomic-WAM, an understand-then-execute framework that connects task
reasoning to action generation through atomic-motion descriptions. Specifically, a
vision-
language reasoner describes the next local motion from the task instruction and visual context, guiding a WAM to predict future
video and an action chunk. Furthermore, an automatic annotation pipeline curates temporally aligned, contact-enriched supervision from multi-view DROID demonstrations. To make atomic-motion predictions executable by the WAM, we post-train the reasoner with Execution-Grounded GRPO (EG-GRPO). With the WAM fixed, EG-GRPO adapts the reasoner through rewards for motion appropriateness and execution success. Finally, Atomic-WAM achieves 39.6% task success under default instructions and 31.6% under vague instructions in a real-to-sim evaluation on RoboLab. Project page: https://ssj9596.github.io/Atomic-WAM-project
Read the full abstract ▾
All authors Shijun Shi, Zhe Liu, Zhi Hou, Yinan Chen, Tao Zhang, Kai Hu, Yong Liu, Jiangning Zhang
Robotics
Enabling Tidying Tasks in Robotic Vacuum Cleaners via a Geometry-Induced Self-Locking Deployable Linkage
Eunseok Song, inwoong Kim, Sohee Lim et al. (+5 more)
Robotic
vacuum cleaners must extend their functionality within tight limits on size, cost, and durability. Deployable linkages with few actuators meet these limits, but their backdrivability makes it difficult to hold a load-bearing posture without continuous motor torque. We introduce a two-stage deployable linkage that is self-locking, in the sense used for toggle and singularity-based linkages, through its own geometry: at full deployment an internal three-link subchain approaches a collinear, near-singular configuration where the input–output transmission ratio approaches zero, so that plate loads produce a small residual back-driving torque, absorbed by drive-train friction and a lightweight
magnetic detent. A single actuator performs deployment and locking, a passive chassis-mounted trigger releases the lock for retraction, and a second actuator adjusts only the deployment height. We integrate the mechanism into a commercial robotic
vacuum cleaner and validate it through a 100-cycle repeatability test, a quasi-static load test in which the locked plate held 50\,N with the motor de-energized, and 140 pushing-type tidying trials on seven household objects against a commercial arm-equipped
vacuum robot under a documented protocol. We derive the analytical holding condition and discuss its limits as an intermediate design point between passive compliance and articulated manipulation.
Read the full abstract ▾
All authors Eunseok Song, inwoong Kim, Sohee Lim, Taehyeong Kim, Yongkook Kim, Hyeontaek Lim, Hyungmin Son, Seungyong Hyung
Robotics
Tactile-Guided Robot Learning: From Representation Learning to Foundation Models
Kiyanoush Nazari, Luchen Li, Haoshu Fang et al. (+4 more)
Tactile sensing is becoming central to robust, adaptive, and dexterous robotic manipulation. Beyond its traditional role in contact detection and reactive control, touch is increasingly used to learn transferable representations, condition manipulation policies, and support embodied physical
reasoning. Yet existing surveys largely emphasize tactile hardware and perception, leaving the rapid shift toward learning-based tactile manipulation insufficiently synthesized. This paper reviews tactile-guided skill learning from a learning-centric perspective, organized by the functional role of touch across the learning and control pipeline. We examine tactile representation and policy learning, tactile foundation models, predictive and adaptive control, and generalization across objects, tasks, sensors, embodiments, and simulation-to-real transfer. Across these directions, we identify common design principles and remaining challenges in learning transferable physical representations, integrating semantic
reasoning with contact-level control, and scaling tactile learning across heterogeneous data and hardware. Finally, we outline emerging directions toward tactile foundation and world models that can predict, reason about, and adapt physical interactions. Together, this review characterizes the transition of touch from a task-specific feedback signal toward a core modality for general-purpose embodied intelligence. An accompanying database of the reviewed literature and associated resources is available at https://sites.google.com/view/touch-in-
robot-learning-review.
Read the full abstract ▾
All authors Kiyanoush Nazari, Luchen Li, Haoshu Fang, Nima Fazeli, Zhibin Li, Fumiya Iida, Thomas George Thuruthel