Robotics
Fiber-Normalized Manipulability and Determinant Proxies: Intrinsic Redundancy Optimization Across and Within Task Fibers
Antonio Franchi and Mirko Mizzoni
This work establishes that determinant-based manipulability is an exact objective for fixed-task redundancy optimization, despite its dependence on task coordinates and the choice of task-space metric used for volume measurement. On every regular task fiber, the determinant proxy, its representation in any task chart, and every metric-completed manipulability differ only by positive constants. They consequently induce the same complete ordering, constrained extrema, gradient directions, critical points, and local optimality classifications. For comparisons and trajectory optimization across task fibers, this work introduces fiber-normalized manipulability: the capability attained at an internal state divided by the best capability available on the same fiber. The resulting dimensionless scalar is invariant under coordinate changes on the internal-state and task
manifolds and independent of the task-space metric. Its associated loss provides an intrinsic objective for physically admissible cross-fiber trajectories, including problems with prescribed or free task
evolution and temporal coupling. Planar-manipulator and redundant aerodynamic-allocation examples demonstrate fixed-fiber equivalence, task-dependent cross-fiber differences, and invariant fiber-normalized trajectory optimization.
Read the full abstract ▾
Robotics
Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models
Yuan Fang, Zechu Li, Haolei Tong et al. (+2 more)
Vision–
language–action (VLA) models provide strong priors for robotic manipulation but are typically deployed as frozen policies, unable to improve from their own failures. Real-world reinforcement learning (RL) offers a path to continued improvement, yet manual environment resets and task-success supervision hinder autonomous learning. We introduce FIND, an agentic real-world RL framework that closes the loop between scene understanding, weakness-aware practice, self-evaluation, and policy improvement in a persistent workspace. FIND reframes autonomous practice as a scene-conditioned, performance-aware task-selection problem: instead of restoring a predefined scene after each rollout, it uses the resulting scene to determine what to practice next. A
vision–
language agent identifies feasible tasks from a predefined library, prioritizes those with lower recent success rates, and evaluates outcomes using paired pre- and post-execution observations. We instantiate FIND with a frozen $\pi_{0.5}$ VLA and residual off-policy RL. Across eight real-world manipulation tasks, the independent human-assessed success rate improves from $55\%$ to $71.9\%$. A representative run completes 456 autonomous episodes within 6 hours of interaction, requiring 30 scene-recovery interventions and no human-provided reward labels during online learning. Ablations and systematic evaluations further examine key design choices,
agent evaluation accuracy, and human intervention requirements. Our website is made publicly available at: FIND.github.io.
Read the full abstract ▾
All authors Yuan Fang, Zechu Li, Haolei Tong, Puze Liu, Georgia Chalvatzaki
Robotics
Grasp2Twist: Learning Bimanual Dexterous Jar Opening by Reinforcement Learning
Mo Xu, Yunfu Deng, Jianuo Wang et al. (+2 more)
This paper presents Grasp2Twist, a bimanual dexterous manipulation system that learns to grasp and twist open jar lids using reinforcement learning. Learning this task raises three challenges: learning a unified policy for a multi-stage task, sustaining lid twisting, and sim-to-real transfer. To address the first challenge, we introduce a continuous enclosure measure to guide grasp formation and a binary enclosure indicator to guide the grasp-to-twist transition for unified policy learning. We derive both from the
geometric relationship between the object center and the convex hull formed by the hand's palm and fingertips. Kinematic constraints limit how far the hand can rotate the lid with fixed contacts, so sustained twisting requires finger contact reconfiguration. We use a three-stage curriculum to facilitate exploration of these contact changes and also improve robustness for sim-to-real transfer. With our approach, the learned policy demonstrates finger gaiting, reconfiguring hand-object contacts to sustain lid rotation. It transfers zero-shot to the physical system and achieves an 88% task success rate across six household containers, including peanut-butter, vitamin, and instant-coffee jars. Ablations further validate the roles of the
geometric enclosure in grasp formation and the curriculum in contact-reconfiguration exploration.
Read the full abstract ▾
All authors Mo Xu, Yunfu Deng, Jianuo Wang, Josiah Hanna, Bilge Mutlu
Robotics · Cross-listed from cs.CV
Depth Any Seen: Which Surfaces and How Far?
Xiaohao Xu and Xiaonan Huang
When several surfaces are visible along a ray, recovering visible 3D structure from one image requires jointly estimating their presence and metric depth. Depth Any Seen represents these surfaces as image-conditioned multi-Bernoulli depth sets, whose components each contribute one depth or remain absent. Its auxiliary-free Exact Multi-Bernoulli objective (ExactMB) learns depth and presence by marginalizing one-to-one assignments to complete, distinct targets. Our analysis shows that matching expected count can leave component-surface assignment unresolved. We extend real and synthetic layered-depth
benchmarks to evaluate depth accuracy, recovered support, and overprediction. Compared to depth stacking, ExactMB reduces overprediction by a relative 88.2% on LD-Real and 80.5% on MD-3K while retaining most ordinal accuracy, with comparable conditional metric-depth error on LD-Syn. Further ablation studies show that ordered assignment improves depth-accurate recall and precision over marginalization, whereas the count-regularized configuration achieves higher deeper-rank precision than ordered assignment at lower recall. Our code will be publicly released.
Read the full abstract ▾
Robotics
Low Cost Eye Tracking for Vision Screening
Maissa Abir Smaili, Yaman Ismaiel and Zafer Iscan
This engineering study assesses gaze-assisted data acquisition during visual screening with a Gaze Quest implementation via webcam and GC308 near infrared camera with an Orlosky eye tracking pipeline. Ten subjects performed under both acquisition conditions. Accuracy of Tumbling E orientation, reported application logMAR value,and response latency were determined from keyboard responses and hence are behavioral outputs as opposed toeye tracker screening results. Frame-to-frame change in theangle between successive normalized gaze vectors (28.2degrees) served as the GC308 eye tracker derived diagnostic with an effective sampling rate of roughly 8 FPS. No screen target calibration or gaze to screen translation was done on the GC308 eye tracker output data for the current analysis and hence, these data represent acquisition stability and not gaze accuracy in relation to the target. The webcam pipeline relied on 25-point screen calibration. Because the display resolution and pixel pitch could not be retained in the records of the study, the reported application logMAR values cannot be viewed as measures of visual acuity in any way.
Read the full abstract ▾
Robotics
Inertia-Corrected Newton Method For Generalized Nash Equilibria in Dynamic Games with Optimality Verification
Zhiyuan Zhang and Panagiotis Tsiotras
Newton methods efficiently find Generalized Nash
Equilibria (GNE) in dynamic games by solving for the KKT necessary conditions. These methods are fast and can support multi-
agent Model Predictive Control (MPC) for highly dynamic
robots. However, a small KKT residual alone does not certify that the returned solution satisfies the second-order sufficient conditions for a local GNE. In this paper, we propose an efficient numerical method to verify the second-order sufficient conditions (SOSC) for a local GNE. We connect the inertia of the
agent KKT
matrix with the positive definiteness of the reduced Hessian of the cost function, projected onto the null space of the constraints. Furthermore, we introduce an inertia-corrected update step that improves convergence to local GNEs by destabilizing strict saddle points with weak cross-
agent coupling. Our main contribution is a fast Newton solver for Constrained Dynamic Games that provides efficient optimality checking. Through numerical
benchmarks, we demonstrate the proposed solver's runtime and convergence performance in practical multi-
agent planning problems. We also validate the solver's real-time capabilities in physical experiments using a platform of miniature autonomous race cars.
Read the full abstract ▾
Robotics
HapticWorld: an Interactive World Simulator with Real-time Torque Feedback
Shaoting Peng, Litian Liang, Yixuan Wang et al. (+4 more)
Contact-rich manipulation depends on force sensing that is hard to infer from visual signals alone, both for collecting demonstrations and for training policies. Force-annotated data, however, remains hard to obtain at scale: real-
robot collection ties every demonstration to physical hardware, physics simulators report contact forces that deviate systematically from real measurements, and learned world simulators, though scalable and realistic, are
vision-only, so operators feel nothing during data collection and the data carries no force/torque (F/T) labels. We present HapticWorld, an interactive world simulator that predicts joint torque together with observations and renders it back to the operator in real time, closing the haptic loop between a human and a learned world model. Across three contact-rich tasks, torque feedback raises data collection throughput by 1.6 times on average. Policies trained on HapticWorld-generated demonstrations succeed in 54/60 real-world trials, approaching the 56/60 upper bound of real-world data, and far exceeding the 19/60 success rate of the
vision-only baseline. Moreover, the success rates measured inside HapticWorld closely match real-world evaluation, demonstrating that HapticWorld can serve as a stand-alone F/T-conditioned policy evaluation platform.
Read the full abstract ▾
All authors Shaoting Peng, Litian Liang, Yixuan Wang, Ming Yang, Katherine Driggs-Campbell, Mark Cutkosky, James Jingxi Xu
Robotics
End-to-end QP-based policies: A unified perspective on robust control and robot learning
Fausto Vega, Priyanka Supraja Balaji, Chase Dunaway et al. (+3 more)
We present an end-to-end QP-based policy framework that enables systematic policy construction with minimal domain-specific design, while preserving the transparency and interpretability of model-based control. The proposed policy representation supports both domain-randomized model-based auto-tuning, where policy parameters are optimized over distributions of disturbances and model variations, and black-box policy construction, where the problem is formulated in terms of a (possibly) unknown model without requiring explicit notions of states, inputs, or the underlying system dynamics. We establish connections to existing policy representations and control paradigms, including robust control, multilayer perceptrons, and
robot learning, and interpret our end-to-end QP policies as a common abstraction of these approaches. We validate the resulting framework in both simulation and hardware, demonstrating a broad range of applications spanning robustness, automatic policy tuning, and control under unknown system dynamics.
Read the full abstract ▾
All authors Fausto Vega, Priyanka Supraja Balaji, Chase Dunaway, Joe Koszut, Jon Arrizabalaga, Zachary Manchester
Robotics
GT-VLA: Target-Conditioned Trace Guidance for Generalizable Robotic Manipulation
Ninghan Zhong, Jing-Chen Peng and Sriram Vishwanath
Vision-
Language-Action (VLA) models have shown strong performance on robotic manipulation, but they often struggle to generalize to unseen tasks, configurations, and long-horizon settings. A key challenge is that VLAs overfit to training scenes and fail to follow novel
language instructions. Off-the-shelf
vision-
language models (VLMs) often provide stronger generalization, but cannot directly control
robot actions. To combine the common sense of VLMs with VLA control, we propose Guided Trace VLA (GT-VLA), a steerable framework that accepts guidance from an external generalist VLM through trace-conditioned action generation. GT-VLA uses a generalist model to identify semantic guidance for the current skill, converts this guidance into a 2D visual trace, and conditions its action policy on the resulting trace-rendered observation. This design separates semantic target acquisition, trace generation, and low-level action execution, allowing high-level guidance to propagate to
robot actions. GT-VLA uses a Mixture-of-Experts architecture with skill-specific trace and action modules for robust execution. We evaluate GT-VLA on LIBERO and a physical
robot platform, showing improved generalization over recent VLA baselines in both settings. The code and additional supplemental materials are available on our project website at https://ivaniz.github.io/gt-vla/.
Read the full abstract ▾
Robotics
Game-Theoretic Control with Constrained Potential Surgery
Zhiyuan Zhang and Panagiotis Tsiotras
Constrained general-sum dynamic games are a popular formulation for highly interactive multi-
agent planning problems. In recent years, Generalized Nash
Equilibrium (GNE) solvers have achieved real-time performance for small dynamic games. However, solution speed still remains a bottleneck, and controlling even a small number of
agents (e.g., more than four) in a dynamic task remains elusive. In addition, Newton solvers that focus on the first-order conditions are vulnerable to non-Nash saddle points, limiting the usefulness of the provided solution. In this work, we propose a fast and versatile interior point solver for constrained dynamic games, along with a computationally efficient second-order correction that increases the probability of converging to a local GNE solution in constrained dynamic games. The performance of the proposed method is evaluated on numerical
benchmarks and a physical experiment involving scaled race cars.
Read the full abstract ▾
Robotics
Differentiable Dynamics for Autonomous Micro-Mobility Navigation
Grace Cai, Joey Lee, Nithin Parepally et al. (+2 more)
Autonomous micro-mobility vehicles (MMVs) such as wheelchairs, scooters, and bicycles have the potential to improve mobility access and support safe low-speed transportation in pedestrian-shared spaces. Achieving MMV autonomy will require realistic, predictable MMV motion. However, many existing autonomous vehicle stacks rely on simplified kinematic models that fail to capture key MMV characteristics such as tire slip, friction, and wheel layouts, limiting realism and gradient-based optimization. In this paper, we explore differentiable formulations of dynamics models for autonomous micro-mobility systems. We first construct DiffKBM, a differentiable version of the kinematic bicycle model (KBM). Then, we introduce DiffGM3, a differentiable formulation of the General Micro-Mobility Model (GM3), a unified tire-based dynamics formulation for micro-mobility vehicles that supports a wide range of MMV configurations. DiffKBM and DiffGM3 enable end-to-end differentiable optimization through MMV dynamics, making them suitable for integration into differentiable autonomy stacks. We evaluate these dynamics models in both open-loop and closed-loop settings: (1) open-loop trajectory matching, where DiffKBM and DiffGM3 are integrated as a dynamics layer within DiffStack and optimized to reproduce real-world MMV trajectories, and (2) closed-loop autonomous navigation, where DiffKBM and DiffGM3 are paired with a differentiable MPC controller in CrowdNav pedestrian scenarios. In the open-loop setting, DiffGM3 outperforms DiffKBM in reproducing trajectories with improvements in ADE and NLL across bicycle, scooter, and motorcycle modes, and reductions in planning loss for bicycle and motorcycle trajectories. We also find that, in closed-loop settings, DiffGM3 improves on DiffKBM's CrowdNav performance by producing 55% fewer collisions and a 75% lower discomfort frequency for the bicycle mode.
Read the full abstract ▾
All authors Grace Cai, Joey Lee, Nithin Parepally, Laura Zheng, Ming C. Lin
Robotics
Efficient Bezier Velocity Optimization for Free-Floating Space Manipulators
Duo Zhang, Zhizhuo Zhang, Xiaoli Bai et al. (+1 more)
We present FAVOR (Free-floating Arm Velocity Optimization with Recursive Sensitivities), a planner for collision-free reaching, tracking, and prescribed-time pre-grasp interception on an unactuated spacecraft. It optimizes Bezier joint-velocity curves with linear velocity, acceleration, and continuity constraints. Decision dimension is independent of rollout resolution. Analytical recursive sensitivities provide task and clearance gradients through the coupled base-arm motion. Parallel evaluation, caching, and incremental collision discovery reduce computation. With a seven-DoF arm and five simulated spacecraft models, FAVOR achieves 99.8% point-to-point success with 1.779 s mean computation, versus 62.9% and 50.270 s for an IK-initialized position-spline baseline. Means include failures and timeouts. FAVOR completes 30 of 36 tracking cases, versus 15 for single-step QP, and all 36 interception instances, versus 22 for the spline baseline. A controlled ablation shows that finite differences increase mean planning time 4.7-fold.
Read the full abstract ▾
All authors Duo Zhang, Zhizhuo Zhang, Xiaoli Bai, Jingjin Yu
Robotics · Cross-listed from cs.AI
Metro-WM: Long-Horizon Latent Planning with Realisable Sub-Goals
Royson Lee, Fady Rezk, Titouan Parcollet et al. (+2 more)
Model-predictive control with Joint-
Embedding Predictive Architectures (JEPAs) provides a strong zero-shot goal-reaching planner, but it is only effective over short planning horizons. Hierarchical extensions attempt to bridge this gap by learning a macro planner to predict intermediate latent sub-goals to guide the micro planner. In this work, we demonstrate that unconstrained latent sub-goal prediction is fundamentally flawed. A rigorous evaluation reveals that a leading state-of-the-art macro planner routinely emits physically unrealisable sub-goals. To resolve this, we introduce Metro-WM, a hierarchical framework that issues sub-goals by retrieving genuine states from prior experience rather than generating ungrounded latent vectors. Specifically, Metro-WM constructs a graph whose vertices are observed frames from offline expert demonstrations or random-action trajectories, allowing frames from different episodes to be connected and stitched into routes to the goal. Planning over the full graph also makes the system highly robust to execution errors: if the micro planner drifts off course, Metro-WM instantly finds a new optimal path from the current state. Our experiments show that Metro-WM achieves superior long-horizon success rates of up to 37.33 percentage points over the next best hierarchical approach while being up to 10.9 times faster, requiring both 13-56 times less offline compute and fewer tuned hyperparameters. Additional analysis reveals that Metro-WM finds shorter paths than the offline demonstrations, outperforms an oracle relying on the query's own demonstration, and maintains robust performance under extremely sparse
dataset conditions.
Read the full abstract ▾
All authors Royson Lee, Fady Rezk, Titouan Parcollet, Timothy Hospedales, Cristina Cornelio
Robotics
PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning
Hang Yu, James Staley, Cheng Xi Tsou et al. (+8 more)
Human demonstrations provide dense policy-level information but sometimes lack local precision. Human feedback presents accurate local critiques, but offers sparse evaluations rather than direct policy guidance. We propose Progress-Heuristicized Inverse Reinforcement Learning (PHIRL), a data-efficient framework that learns robust reward functions by jointly leveraging demonstrations and feedback. Specifically, we use progress, a feedback modality that describes cumulative task completion. PHIRL iteratively infers a reward function from demonstrations via inverse reinforcement learning, calculates the learned rewards over the progress-annotated demonstrations, and aligns the rewards with progress annotations over four dimensions. We evaluate PHIRL on real and simulated
robot tasks, with additional exploration using a fine-tuned
vision-
language model to provide progress feedback. Results demonstrate that PHIRL significantly outperforms the baselines, achieving substantially higher environmental return rewards and task success with only twenty percent of demonstrations annotated. Analysis of reward-hacking scenarios demonstrates that PHIRL learned reward functions are reliable against exploitation.
Read the full abstract ▾
All authors Hang Yu, James Staley, Cheng Xi Tsou, Xiujin Liu, Wenchang Gao, Jindan Huang, Shijie Fang, Zhegong Shangguan, Angelo Cangelosi, Reuben Aronson, Elaine Short
Robotics
Humanoid Badminton: Learning Dynamic Racket Skills from Limited Human Motion Data
Jingzhi Cui, Zhexiong Wang, Bangjie Xu et al. (+9 more)
High-speed racket sports provide a demanding testbed for humanoid
robots, requiring time-critical decisions, precise striking, and dynamic whole-body coordination. In badminton, fast-changing shuttle trajectories require timely contact decisions, while successful returns demand precise racket pose and velocity within a brief contact window and across a broad three-dimensional striking workspace. Human motion data provide valuable priors for such athletic skills, but usable badminton references are limited and imperfect. Direct tracking provides insufficient executable variation for diverse shuttle conditions, while purely task-driven optimization may produce unnatural motion. To address these challenges, we present a three-stage hierarchical reinforcement learning framework for dynamic humanoid badminton. First, task-randomized motion augmentation expands sparse annotated hitting events into executable target-conditioned stroke variations, forming a continuous latent skill space. Second, a high-level planner outputs continuous latent skill codes to compose these skills online according to the observed shuttle state. Third, a context-conditioned
adversarial regularizer encourages more natural planner-level skill usage while preserving return performance. When deployed on a real humanoid
robot, our system achieves sustained multi-skill rallies with human players, including forehand, backhand, and highly dynamic jump returns. This is the first real-world humanoid racket-sport system to demonstrate multi-skill human–
robot rallies including highly dynamic jump returns.
Read the full abstract ▾
All authors Jingzhi Cui, Zhexiong Wang, Bangjie Xu, Pengyu Zhao, Youyuan Li, Zhi Su, Peng Ren, Mengdi Xu, Chao Yu, Yi Wu, Luyang Wang, Zhongyu Li
Robotics
Path Planning with Motion Primitives in Dynamic Environments: SIPP on Lattices
Marat Agranovskiy
Autonomous navigation in dynamic environments is a critical challenge, particularly when spaces are shared with other mobile
agents whose future trajectories are known. While traditional grid-based planners efficiently find collision-free paths, their reliance on stop-and-turn mechanics over $2^k$-connected grids produces piecewise-linear trajectories that are kinodynamically highly sub-optimal for differentially constrained
robots. In this paper, we present an adaptation of Safe Interval Path Planning (SIPP) that operates on state
lattices, utilizing precomputed, kinodynamically smooth motion primitives. To efficiently handle dynamic environments, we rasterize the spatiotemporal swept volumes of moving obstacles directly onto the grid, treating grid
cells as atomic units of space, whose resolution is typically dictated by inherent localization noise. We perform a comprehensive comparative analysis between our
lattice-based approach and $2^k$-connected grid planners across diverse topological environments. Our evaluation considers a broad spectrum of performance metrics, including planning time, path angularity, cumulative heading change (angle-over-length), and bending energy. The results demonstrate that while the expanded state space of
lattice-based search increases computational overhead, it yields trajectories with significantly superior kinodynamic properties. Specifically, our method achieves a reachability comparable to highly connected grids while ensuring smooth, continuous, and physically executable paths ready for real-world deployment.
Read the full abstract ▾
Robotics · Cross-listed from cs.AI
ConflictVLA-Bench: Benchmarking Behavioral Responses of Vision-Language-Action Models to Premise Conflicts
Liyu Hou, Yuan Wu and Yi Chang
While
Vision-
Language-Action (VLA) models perform strongly on manipulation tasks, their responses to invalid task premises remain underexplored. Existing evaluations of premise conflicts often focus on terminal task outcomes, yet task failure alone cannot distinguish behavioral disengagement from continued pursuit followed by an execution error. We call the latter pattern Failed Persistence. To study this phenomenon, we introduce ConflictVLA-Bench, which pairs conflict rollouts with premise-consistent reference rollouts and evaluates both outcomes and execution processes. Built on LIBERO, the
benchmark contains 2,826
prompt-conditioned conflict tasks spanning four conflict families, four structural configurations, and two
prompt conditions. Across all eight VLAs, invalid premises reduce original goal completion by at least 17.3 percentage points, with the reduction reaching 56.2 percentage points for OpenVLA. Crucially, even when models succeed on premise-consistent tasks and fail on their matched conflict tasks, they often continue to approach the original targets, retain early trajectory structure, and show limited action magnitude suppression. Failed Persistence therefore recurs across the evaluated models. Explicit premise checking does not consistently produce selective and coordinated behavioral changes. These findings show that terminal failure alone establishes neither behavioral disengagement nor refusal and that outcomes alone are insufficient for VLA evaluation. Experimental data and additional details are available on the project page: https://github.com/EmbodiedAISurvey/ConflictVLA-Bench
Read the full abstract ▾
Robotics
Learning Robot Policies from Sparse Success Signals via STL-Guided Stein Variational Policy Gradient
Hongrui Zheng, Cristian Ioan Vasile, Antonio Loquercio et al. (+1 more)
Learning
robot policies for tasks with sparse success signals is challenging when completion depends on coordinated actions, precise contact outcomes, or satisfying several conditions together. Intricate physical interactions with the world further complicate these requirements. Prior work using conventional reward shaping mechanisms provides dense feedback but local progress might not translate into eventual task completion. We present Signal Temporal Logic-guided Stein Variational Policy Gradient (STL-SVPG), a population-based method that uses smooth STL robustness as a trajectory-level training objective. Differentiating this objective through the dynamics assigns credit to policy actions according to their effect on the complete task specification, rather than local progress alone. We evaluate the approach on six quadcopter and manipulator tasks that involves event-triggered responses, strictly ordered behavior, responses within specified deadlines, and physical interaction with the world. STL-SVPG achieves the highest mean success rate among the compared methods on five of six
benchmarks. Simulation-trained policies trained in simulation transfer temporal and contact task behavior to the real world.
Read the full abstract ▾
All authors Hongrui Zheng, Cristian Ioan Vasile, Antonio Loquercio, Rahul Mangharam
Robotics
Generate, Track, Improve: Perceptive Multi-Skill Humanoid Locomotion with RL-Fine-Tuned Motion Generators
Zachary Olkin, William D. Compton and Aaron D. Ames
General purpose humanoids require locomotion controllers that are multi-skill, perceptive, dynamic, and robust enough to go anywhere humans can. In this work, we present a two layer locomotion architecture: (1) a perceptive flow matching motion generator plans whole body trajectories from raw depth images while a (2) perceptive tracking policy trained with control-guided RL follows these motions. Both policies are trained on a library of terrain consistent motion clips created with dynamically optimized human data which yields both accurate velocity tracking and terrain consistent references. Our central contribution is a simple yet effective off-policy RL fine tuning loop that improves the motion generator. A structured search method is used with the generator to gather data for advantage weighted regression. This off-policy loop is much more sample efficient than on-policy residual fine tuning and improves terrain consistency on unseen geometries and skill compositions. We find that successful terrain traversals increased by up to 25 percentage points and skill selection improved by up to 80 percentage points. By using raw depth images to perceive the environment no odometry or height maps are needed, and outdoor deployment is easy. With two cameras, the policy can see terrain coming from further away and adjust its velocity regardless of the commanded speed so it can traverse the terrain. A single policy pair enables a Unitree G1 humanoid to walk, run, stand, jump on and off of boxes, and traverse stairs in outdoor environments. Project page: https://zolkin1.github.io/generate-track-improve/
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
SatNav: A Scalable Benchmark for Long-Horizon UAV Vision-Language Navigation from Satellite Imagery
Jiajun Jiang, Chunliang Hua, Zichun Chen et al. (+4 more)
Urban uncrewed aerial vehicle (
UAV)
vision-
language navigation (VLN) requires
agents to follow instructions across extended urban spaces, inherently demanding long-term
memory and geospatial grounding. However, scaling existing
benchmarks remains difficult because of their reliance on costly reconstructed 3D assets, limiting geographic diversity and episode scale. To address this, we introduce SatNav, a scalable, long-horizon
UAV VLN
benchmark built from high-resolution satellite imagery. SatNav targets city-level navigation missions and uses satellite crops as approximations of
UAV nadir views for visual observations. Through an automated cue-to-episode pipeline, SatNav constructs 118K episodes from 59 scenes across 18 cities, with an average trajectory length of 379 m. To stress-test long-horizon
memory and geospatial
reasoning, SatNav defines three task families: Boundary, Landmark, and Route, targeting loop progress tracking, landmark-based spatial grounding, and route following with counting cues. Benchmarking classical VLN
agents and recent
agents based on large
vision-
language models (LVLMs) on SatNav shows that city-scale navigation remains challenging. We further introduce SwiftVLN, a modular framework with switchable
memory components, and conduct systematic
memory-design ablations. Finally, satellite-to-
UAV transfer experiments show that satellite-trained navigation models can operate on real-flight
UAV observations, showing the practical relevance of SatNav. Our project page: https://eku127.github.io/SatNav/
Read the full abstract ▾
All authors Jiajun Jiang, Chunliang Hua, Zichun Chen, Yanxing Wu, Zeyuan Yang, Jie Song, Xiao Hu
Robotics
Vision-Based 6-DoF Grasp Pose Estimation for Robot Cloth Unfolding
Domen Tabernik, Peter Nimac, Jan Jerićević et al. (+2 more)
Cloth manipulation is a challenging task due to the deformable and high-dimensional nature of cloth, which leads to complex interaction dynamics and perceptual ambiguity arising from frequent occlusions of critical visual cues such as folds, edges, and grasp points. In this work, we tackle cloth unfolding using a regrasping-in-the-air strategy, where one manipulator holds the cloth while the other grasps it at an optimally selected point to unfold it. To this end, we propose CeDiRNet-6DoF, a deep learning framework that jointly predicts effective grasp points and the complete 6-DoF grasp pose from the observed cloth configuration. By integrating dense 3D grasp regression with segmentation and sine-cosine-encoded Euler angles, the proposed method reliably estimates the grasp configuration that maximizes the unfolded cloth area. We extensively evaluated CeDiRNet-6DoF on a bimanual robotic setup within the ICRA 2024 Cloth Competition framework, achieving state-of-the-art performance. An ablation study further validates the benefits of key design components, including joint segmentation, background randomization, and image cropping. These results establish CeDiRNet-6DoF as a robust and versatile foundation for reliable robotic cloth manipulation in unstructured environments.
Read the full abstract ▾
All authors Domen Tabernik, Peter Nimac, Jan Jerićević, Danijel Skočaj, Andrej Gams
Robotics
Learning to Leverage Compliance: A Policy-Admittance Learning Framework for Robotic Insertion
Chongren Wang, Minghe Li, Honghua Dai et al. (+3 more)
Policy learning and compliant control offer a promising route to reliable autonomous assembly under pose errors and contact uncertainty. However, combining them does not ensure coordination: the policy may continue pushing against contact while the controller yields, producing sustained loading with limited progress. To address this problem, we propose LeCo (Leverage Compliance), a policy-admittance learning framework that guides a visual policy through execution-time interaction under fixed admittance. A multirate feedback mechanism aggregates high-rate contact-interaction records into policy-transition rewards. An integrated conflict cost then characterizes sustained policy-loading/controller-unloading opposition, while a directional high-force tail cost captures continued-loading events within a transition. Together with task completion, these costs encourage the policy to leverage compliance with less unproductive loading. We evaluate LeCo on four real connector-assembly tasks, obtaining an aggregate success rate of 94%. Across tasks, mean successful-trial resultant-force and torque peaks decrease by approximately 30% and 64% relative to the comparison baseline. Reward ablation further shows that adding conflict shaping reduces median successful-trial contact-conditioned conflict density by approximately 53%. These results support learning to leverage fixed compliance by turning multirate policy-admittance interaction into complementary reward signals for effective, lower-load insertion.
Read the full abstract ▾
All authors Chongren Wang, Minghe Li, Honghua Dai, Zhicheng Lin, Shiyang Wei, Xiaokui Yue