Robotics
RSC: Decentralized Rigid Formation Flocking for Large-Scale Swarms via Hybrid Predictive Control and Online Reconfiguration
Ganyu Zou, Linhan Wang, Chen Dai et al. (+2 more)
Decentralized rigid formation flocking requires a swarm of autonomous
agents to maintain a predetermined
geometric configuration while moving, relying solely on local sensing and communication. However, existing decentralized control methods struggle to maintain strict inter-
agent distance constraints in cluttered environments, often suffering from local minima deadlocks, high frequency control oscillations, or limited flexibility during obstacle
navigation, resulting in low success rate. To address these limitations, we propose Rigid Swarm Control (RSC), a decentralized control framework for large-scale rigid formation flocking. To escape local minima via robust long-term planning while ensuring short-term safety, RSC integrates finite-horizon trajectory predictions with a reactive artificial potential field (APF) safety controller within a hybrid architecture. Furthermore, to accelerate formation reassembly after obstacle traversal without interrupting task execution, RSC introduces an online leader-follower reconfiguration mechanism based on stable role exchange. Extensive evaluations in challenging cluttered environments with 25
UAVs demonstrate that RSC reliably unifies rigid formation maintenance, obstacle avoidance, and target tracking. Under strict success criteria - collision-free operation with a maximum relative edge-length error below 10%, RSC achieves an 83% success rate, significantly outperforming existing heuristic and learning-based baselines that fall below 5%.
Read the full abstract ▾
All authors Ganyu Zou, Linhan Wang, Chen Dai, Siji Chen, Chang-Tien Lu
Robotics
What Are We Actually Benchmarking in Robot Manipulation?
Tianchong Jiang, Xiangshan Tan, Samuel Wheeler et al. (+3 more)
A robotics
benchmark score measures success under one fixed evaluation setup, yet is routinely treated as evidence of general manipulation capability. We identify four failure modes, each of which weakens or invalidates a
benchmark's role as a valid proxy for that capability: shortcut solvability, lack of statistical significance, creeping overfitting, and data-source dependence. We propose one diagnostic per failure mode. We audit LIBERO, CALVIN, SimplerEnv, RoboCasa, and RoboTwin 2.0 under these diagnostics. LIBERO and CALVIN fail multiple diagnostics. RoboCasa and RoboTwin 2.0 fail fewer, despite appearing far less often in recent progress claims. On LIBERO, a 0.09B probe with no
language encoder scores at or near reported SOTA, and most reported gains are not provably statistically significant. On CALVIN, randomizing block poses within the training range drops performance for every tested policy. We release the four diagnostics with reference implementations for authors and reviewers to apply before treating a
benchmark score as evidence of progress. Code and artifacts are available at https://ripl.github.io/manipulation_
benchmark_audit/.
Read the full abstract ▾
All authors Tianchong Jiang, Xiangshan Tan, Samuel Wheeler, Luzhe Sun, Tewodros W. Ayalew, Matthew Walter
Robotics
PerceptTwin: Semantic Scene Reconstruction for Iterative LLM Planning and Verification
Charlie Gauthier, Sacha Morin and Liam Paull
Simulation environments are useful for both
robot policy learning and planning verification and validation. Traditionally, the process of creating a simulation was onerous. Creating a bespoke simulation environment for each individual environment that a
robot would operate in was simply infeasible. In this work, we introduce PerceptTwin, a fully automatic pipeline that constructs interactive simulations directly from semantic scene representations produced by a
robot's perception stack. PerceptTwin combines open-vocabulary object maps with 3D asset generation, affordance prediction, and commonsense condition checking. These interactive simulations can be used to validate and refine plans before they are executed on the
robot hardware. Borrowing from the AI
alignment literature, we also introduce an LLM judge that verifies plan correctness and
alignment with human preferences. Experiments show that PerceptTwin feedback allows LLM planners to refine plans, enhance safety, and resist harmful black-box prompting attacks. In our suite of tasks, PerceptTwin improves plan success by an average of approximately 39% for GPT5, GPT5Mini, and GPT5Nano planners. Additionally, PerceptTwin also improves human plan verification by up to 18% on average for plans that fail due to unfilled skill preconditions. Our results demonstrate the potential of open-vocabulary scene simulation from
robot perception as a foundation for safer, more reliable
robot planning.
Read the full abstract ▾
Robotics
Towards Estimating Normal and Shear Interface Pressures in Prosthetic Sockets via Least Squares and Mechanics Modeling
Axel González Cornejo, Tianhao Yu, Chi Hwan Lee et al. (+1 more)
Prosthetic socket fitting remains largely manual and iterative, and objective fit metrics are still limited. Part of the challenge is the lack of long-term real-life pressure data at the residual limb–socket interface. Traditional pressure sensors are prone to drift over time, and capture only normal pressures at sparse locations within the socket, missing a critical component for biomechanical analysis: shear. Although some sensors can report both normal and shear interface stresses, these components are often difficult to decouple because of measurement crosstalk. One potential path forward is to develop models that can augment available measurements. This work introduces a testbed to evaluate model performance under sparse pressure sensing using two complementary validation signals: (i) the global wrench (\ie, total forces and moments expressed in an orthonormal frame) transmitted through the socket, by an artificial residual-limb, and (ii) local interface loads (\ie, decoupled normal and shear pressure components in a right-hand-rule orthogonal frame that lives in each instrumented location) measured by sparse sensing clusters, each composed of four capacitance-sensing channels. Rather than presenting full-field pressure estimates, the focus is on an analysis sequence that quantifies how well candidate mechanical models explain both global and local measurements under controlled conditions. A quasi-static spring–mass contact model is evaluated, and its parameters are identified via a two-stage convex least-squares problem. Validation under static loading shows that estimating constant bias terms reduces steady offsets in the wrench channels and improves agreement with local measurements. A Pareto-front sensitivity analysis further illustrates how the trade-off between global and local objectives changes when bias terms are included.
Read the full abstract ▾
All authors Axel González Cornejo, Tianhao Yu, Chi Hwan Lee, Edgar Bolívar-Nieto
Robotics
DLO-Lab: Benchmarking Deformable Linear Object Manipulations with Differentiable Physics
Junyi Cao, Yian Wang, Ziyan Xiong et al. (+3 more)
We address the challenge of enabling
robots to manipulate deformable linear objects (DLOs), such as ropes, cables, and rubber bands. Prior work has primarily focused on narrow, task-specific problems, often relying on real-world demonstrations or handcrafted heuristics. Such approaches, however, struggle to scale to the wide variety of materials and tasks encountered in practice, and collecting sufficiently diverse real-world data is often impractical. Additionally, existing simulation environments offer limited support for the broad spectrum of material behaviors necessary for generalizable DLO manipulation. To overcome these limitations, we introduce a differentiable simulator explicitly designed for versatile DLO manipulation. Our simulator models a wide range of material properties-including (in)extensibility, elasticity, bending plasticity, and complex interactions with other objects-providing a robust foundation for learning and evaluating manipulation skills. Building on this simulator, we propose a
benchmark suite of representative tasks that highlight the unique challenges of DLO manipulation. The successful execution of these tasks is often hindered by the topological complexity and grasp sensitivity inherent to DLOs. Therefore, we introduce a specialized DLO
agent that explicitly manages these challenges by proposing strategic grasping points and decomposing long-horizon tasks to maximize control authority. Finally, we evaluate various policy-learning
algorithms using our framework, alongside sim-to-real transfer experiments, demonstrating our platform's potential to advance DLO manipulation.
Read the full abstract ▾
All authors Junyi Cao, Yian Wang, Ziyan Xiong, Chunru Lin, Zhehuan Chen, Chuang Gan
Robotics · Cross-listed from cs.LG
Dual Advantage Fields
Alexey Zemtsov, Maxim Bobrin, Alexander Nikulin et al. (+5 more)
Offline goal-conditioned reinforcement learning requires both long-horizon reachability estimates and local action comparisons. Dual goal representations provide value fields that capture global goal reachability, but they do not directly specify which action should be preferred at a given state. We propose Dual Advantage Fields (DAF), a policy-extraction method that turns a bilinear dual value model into a local advantage signal. Under bilinear dual parameterization, the goal
embedding is the gradient of the value field with respect to the state representation, which directly specifies which action should be preferred. DAF introduces an action-effect model that predicts the discounted direction of change in state representation space induced by actions and scores them by the similarity between this change and the goal direction. In the realizable case, this score equals the goal-conditioned Bellman advantage, yielding a standard local policy-improvement guarantee. On OGBench locomotion, manipulation, and puzzle tasks, DAF improves aggregate RLiable metrics and performs strongly in settings where locally correct actions differ from direct movement toward the final goal.
Read the full abstract ▾
All authors Alexey Zemtsov, Maxim Bobrin, Alexander Nikulin, Dmitry V. Dylov, Fakhri Karray, Vladislav Kurenkov, Martin Takáč, Arip Asadulaev
Robotics
Distribution-Free Risk-Aware Planning and Control Under Uncertainty Using Conformal Spectral Risk Control
Junsik Eom and Tulga Ersal
Safe
navigation in dynamic and uncertain environments often relies on accurate estimation of, or assumptions about, the true underlying uncertainty. However, accurately characterizing the true uncertainty distribution is often difficult due to limited data or imperfect information. An incorrect understanding of the uncertainty and its associated risk may lead to dangerous decisions even under high levels of risk aversion. To address this issue, we propose a risk-aware model predictive control (RA-MPC) framework that incorporates prediction sets to guarantee risk control below a user-specified threshold without requiring assumptions about the underlying uncertainty distribution. To generate the prediction sets, we develop a distribution-free risk quantification framework that extends conformal risk control (CRC) to general spectral risk measures. We then show that incorporating the prediction sets into the MPC framework provides statistical safety guarantees in terms of spectral risk constraint satisfaction even under uncertainty misspecification. We validate the proposed framework in simulated vehicle obstacle avoidance scenarios, demonstrating improved safety and reduced solve time compared to a baseline RA-MPC framework.
Read the full abstract ▾
Robotics
Affordance2Action: Task-Conditioned Scene-level Affordance Grounding for Real-Time Manipulation
Litao Liu, Yifan Han, Pengfei Yi et al. (+9 more)
Task-conditioned manipulation requires grounding instructions to task-relevant functional parts rather than object categories. This setting is scene-dependent and often one-to-many in cluttered scenes: the same object may afford different interactions across tasks, while a single task may correspond to either one functional region or multiple valid functional regions, depending on the scene layout. Existing affordance
datasets and
benchmarks remain misaligned with this setting, as they typically focus on grasping or object-level affordances, rely on synthetic scenes, or assume a single instruction-region correspondence. We present Affordance2Action (A2A), a
benchmark-centered learning framework for scene-level, task-conditioned part affordance grounding. At its core is A2A-Bench, a manipulation-oriented
benchmark that covers both single-region and multi-region instruction correspondences in everyday scenes, with the latter highlighting the ambiguity and diversity of affordance grounding in realistic multi-object environments. To construct it at scale, we build A2A-AffordGen, an
agent-assisted annotation pipeline that combines
language-model filtering, interactive part
segmentation, instance-level mask-out refinement, task-
reasoning instruction generation, and human verification. A2A-Bench's supervision further supports diverse downstream applications, with real-time affordance grounding and affordance-conditioned manipulation policies as two representative examples. Experiments show that A2A exposes substantial gaps in generic
segmentation, VLM-based grounding, and affordance
distillation baselines, while improving task-level localization and providing useful spatial priors for downstream manipulation.
Read the full abstract ▾
All authors Litao Liu, Yifan Han, Pengfei Yi, Wenbo Yu, Hanqing Wang, Haoran Du, Enze Yuan, Zilin Yuan, Ruiding Feng, Michael Liu, Qi Zhang, Jingjin Yu
Robotics
Multi-Agent Next-Best-View Optimization for Risk-Averse Planning
Amirhossein Mollaei Khass, Vivek Pandey, Guangyi Liu et al. (+3 more)
Multi-
agent Next-Best-View (NBV) selection for safe path planning in uncertain and unknown environments requires informative, safety-aware, and efficient coordination. Centralized approaches rely on sharing raw sensor data or significant communication overhead, resulting in limited scalability. We propose a distributed, risk-aware multi-
agent NBV framework in which each
robot maintains a private local 3D Gaussian Splatting map and the team jointly maximizes expected information gain (EIG) restricted to masked zones along planned trajectories. The resulting distributed objective is solved by Consensus ADMM (C-ADMM) over a communication graph, with each
robot exchanging only candidate viewpoints, planned trajectory descriptors, and scalar EIG contributions. Collision risk along each trajectory is modeled via Average Value-at-Risk (AV@R) over the local 3DGS map and used both to shape the masking radius and to score planned paths. Experiments in Gibson environments at multiple team sizes show that the distributed formulation approaches the centralized baseline in mapping quality and trajectory safety while reducing communication by orders of magnitude.
Read the full abstract ▾
All authors Amirhossein Mollaei Khass, Vivek Pandey, Guangyi Liu, Athanasios Cosse, Emrah Bayrak, Nader Motee
Robotics
Selecting haptic guidance models in teleoperation: guidelines from a comparative user study
Alexis Boulay, Margot Vulliez and David Daney
Haptic guidance in teleoperation enhances operator performance through force feedback. This paper presents guidelines to select the most appropriate model considering the task, the environment and the operator. We define a unified formulation expressing most common models (spring-damper, potential field, and guiding tube) as variations of a stiffness-damping system with model-specific guiding functions. We conducted a user study comparing the three classical models across six scenarios with varying environmental conditions in a vertical farming task. Results show no universally superior model: spring-damper excels in cluttered environments, potential field in free spaces (but it shows risks near obstacles), and guiding tube offers a balanced compromise. We propose novel objective metrics to evaluate the interaction, and show that guiding force magnitude correlates with comfort and trust scores. These findings provide practical model selection guidelines through environmental characteristics and real-time evaluation metrics.
Read the full abstract ▾
Robotics
CoPark: Learning Reactive Parking via Self-Play
Jiarong Wei, Yanxing Chen, Sinuo Song et al. (+3 more)
Learning a single policy that reaches a goal with high
geometric precision while interacting safely with nearby
agents poses conflicting objectives. Precision favors commitment to a fixed
geometric plan, whereas interaction requires immediate deviation when another
agent intrudes, causing policies optimized for one objective to often fail at the other. We study this problem in the context of reactive autonomous parking, where multiple vehicles must reach assigned slots with sub-meter terminal accuracy while remaining responsive to neighboring vehicles throughout the maneuver. We propose CoPark, a multi-
agent self-play RL approach built on a residual-policy architecture. A precomputed offline plan provides a fixed action prior, while a residual head learns the reactive corrections. The residual policy learns behaviors under self-play, where data and scripting fall short, while the fixed prior holds the slot-frame geometry that pure policies struggle to reach reliably. The key design is a partner-threat-modulated, channel-asymmetric release of the prior. A continuous threat signal shifts authority of the longitudinal channel to the residual head to enable yielding, while the lateral channel remains anchored to the precomputed reference to preserve sub-meter slot
alignment. A closed-loop refinement layer corrects residual terminal error from action-grid discretization. We train our policy on six parking lots and evaluate zero-shot on our new reactive-parking
benchmark spanning Dragon Lake Parking (DLP) and DeepScenario Open 3D (DSC3D). CoPark achieves ~70-85% success with only 3-6% collision rate, substantially outperforming classical, imitation-learning, and large-scale RL baselines. Importantly, the results demonstrate emergent interaction behaviors such as reverse-yielding, mid-maneuver yielding, tight-corridor passing, and queuing.
Read the full abstract ▾
All authors Jiarong Wei, Yanxing Chen, Sinuo Song, Yin Wu, Anna Rehr, Abhinav Valada
Robotics
CLAW: Learning Continuous Latent Action World Models via Adversarial Latent Regularization
Tewodros Ayalew, Matthew Jeung, Samuel Wheeler et al. (+5 more)
We introduce CLAW, a fully end-to-end self-supervised framework for learning a world model jointly with continuous latent action representations directly from action-free
videos. Our approach leverages
adversarial latent regularization and
diffusion-based
video generation to capture structured and semantically meaningful action representations while modeling rich, predictive environment dynamics, without relying on any action labels or annotations. By simultaneously training the Latent Action Model and world model, CLAW learns to reason about how inferred actions induce environment transitions from visual observations alone. We show that the resulting latent action world model supports both imitation learning from observation and goal-directed planning. In imitation learning, latent actions extracted from raw
videos enable behavior cloning. For planning, CLAW generates sequences of latent actions and maps them to executable actions to reach desired goals. Extensive experiments across diverse tasks and embodiments demonstrate that CLAW produces semantically meaningful latent action representations, supports effective action transfer, and enables planning and imitation from observation, outperforming existing methods.
Read the full abstract ▾
All authors Tewodros Ayalew, Matthew Jeung, Samuel Wheeler, Xiao Zhang, Andre de la Cruz Arce, Kaylene Stocking, Michael Maire, Matthew R. Walter
Robotics · Cross-listed from math.OC
Semantic Constraint Synthesis for Adaptive Trajectory Optimization via Large Language Models
Eleanor Brosius, Yuji Takubo, Daniele Gammelli et al. (+2 more)
Trajectory optimization is a critical component for enabling safe and reliable autonomous operations in space exploration. As space missions increase in frequency, complexity, and scope, there is a growing need to rapidly formulate mathematically sound trajectory optimization problems that accurately reflect mission objectives and operational constraints. However, translating mission intent into tractable analytical formulations for trajectory optimization requires substantial domain expertise. This paper presents a framework that leverages large
language models (LLMs) to translate natural
language descriptions of mission requirements and constraints into executable trajectory optimization code and corresponding mathematical formulations. Experiments in spacecraft rendezvous scenarios demonstrate a high success rate in reconditioning a convex trajectory optimization problem from semantic mission requirements. Ultimately, this work highlights the potential of LLMs to bridge high-level intent and formal optimization models, enabling more flexible and efficient trajectory design of spacecraft.
Read the full abstract ▾
All authors Eleanor Brosius, Yuji Takubo, Daniele Gammelli, Simone D'Amico, Marco Pavone
Robotics
Faryal Batool, Muhammad Ahsan Mustafa, Fawad Mehboob et al. (+2 more)
Vision-based
UAV navigation becomes challenging when
navigation targets are distributed across complementary camera views and cannot be reliably observed from a single viewpoint. We propose AgenticDiffusion, an agentic multi-view
UAV navigation framework that semantically coordinates first-person-view (FPV) and top-view observations for mission-level
navigation. Given a natural-
language instruction, AgenticDiffusion identifies the requested targets, selects the most appropriate camera view for each
navigation task, determines the corresponding
navigation goal, and invokes the appropriate view-conditioned
diffusion planner for trajectory generation. The resulting trajectories are executed using Nonlinear Model Predictive Control (NMPC). AgenticDiffusion was evaluated in four real-world indoor scenarios, achieving an overall mission success rate of 80% across 40 physical-flight trials. In mixed-visibility scenarios, where the requested targets were distributed across FPV and top-view observations, coordinated multi-view
navigation reduced average mission time by 50.8% relative to FPV-only
navigation and by 26.8% relative to Top-only
navigation. The semantic view-selection mechanism was also robust to lexical variation in target descriptions, achieving 100% accuracy across 66 test cases, compared with 63.64% for a confidence-based view-selection baseline. In a substantially larger Gazebo environment, AgenticDiffusion achieved a 90% mission success rate and completed the multi-stage mission, whereas the FPV-only and Top-only variants were unable to complete all requested
navigation tasks.
Read the full abstract ▾
All authors Faryal Batool, Muhammad Ahsan Mustafa, Fawad Mehboob, Valerii Serpiva, Dzmitry Tsetserukou
Robotics
CADET: A Modular Platform for Evaluating Distributed Cooperative Autonomy in Connected Autonomous Vehicles
Pragya Sharma, Brian Wang and Mani Srivastava
Deep learning models are increasingly central to autonomous vehicle (AV) pipelines, yet their integration has traditionally followed a monolithic design where perception, planning, and control execute on a single onboard computer. This design overlooks the emerging paradigm of cooperative autonomy, where vehicles interact with roadside units (RSUs), edge servers, and cloud-hosted intelligence through vehicle-to-everything (V2X) connectivity. Cooperative perception and control improve safety and efficiency, but also introduce systems-level challenges: network latency, compute heterogeneity, and multi-tenant contention, all critically affect real-time decision-making. These challenges are further amplified by the increasing reliance on large foundation models, whose scale necessitates cloud deployment. We present CADET (Cooperative Autonomy through Distributed Experimentation Toolkit), a modular platform for systematic and reproducible evaluation of distributed cooperative autonomy systems under realistic deployment conditions. CADET decouples the AV stack into composable modules that can be flexibly deployed across vehicles, infrastructure, and edge/cloud tiers. The framework integrates state-of-the-art models, incorporates trace-driven network and workload emulation, and provides synchronized model-, system-, and task-level instrumentation. Through V2V and V2I experiments, we show that distributed deployment choices fundamentally shape safety, with V2V intent packets outperforming cloud-based perception and RSU-assisted perception sustaining safety until overloaded by concurrent requests. Although designed for AV pipelines, CADET also supports
dataset-driven experimentation, enabling systems and ML researchers to
benchmark distributed inference workloads independently of full vehicle simulation. CADET is open source, with code and demo available at https://nesl.github.io/cadet-web.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generation
Boyuan Xiao, Bohong Chen, Yumeng Li et al. (+3 more)
In
embodied vision-
language decision making tasks such as robotic manipulation and
navigation,
Vision-
Language and
Vision-
Language-Action Models (VLMs & VLAs) are powerful tools with different benefits: VLMs are better at long-term planning, while VLAs are better at reactive control. However, their performance is limited by the same perceptual bottleneck: visual
hallucinations arise due to the models' inability to distinguish task-relevant objects from distractors. In principle, accurate identification and focus on critical objects while filtering out irrelevant ones is the key to break this limitation. A straightforward solution is one-step focus: directly attending to essential objects. However, this approach proves ineffective because effective focus inherently requires deep scene understanding. To this end, we propose SceneDiver, a coarse-to-fine focus plan generation method for VLMs leveraging their long-term planning abilities, that first constructs a holistic scene graph to establish initial comprehension, then progressively decomposes the task into simpler sub-problems through an iterative cycle of recognition, understanding, and analysis. To enable reactive control, we also design a lightweight adapter for distilling the deliberate focus ability into VLAs. Evaluations on standard
embodied AI
benchmarks confirm that our method substantially reduces visual
hallucinations for both VLMs and VLAs, while preserving computational efficiency in tasks requiring fast execution. Our code and data are released at: https://future-item.github.io/SceneDiver.
Read the full abstract ▾
All authors Boyuan Xiao, Bohong Chen, Yumeng Li, Ji Feng, Yao-Xiang Ding, Kun Zhou
Robotics · Cross-listed from cs.CV
SimuScene: Simulation-Ready Compositional 3D Scene Reconstruction from a Single Image
Inhee Lee, Sangwon Baik, Sungjoo Kim et al. (+3 more)
Reconstructing interactive, simulation-ready 3D scenes from a single image is a critical bottleneck for robotic manipulation. While recent single-image lifters recover plausible per-object shapes, composing them yields scenes that collapse under physical simulation due to interpenetrating, hovering, or sinking objects. Existing physics-aware methods address this strictly as a post-hoc layout correction, leaving the underlying
geometric errors unresolved. To address this, we introduce SimuScene, a compositional 3D reconstruction pipeline that puts physics in the loop of shape and layout estimation. Rather than using physics merely for layout cleanup, we utilize the physics engine as a diagnostic measurement tool during the generative process itself. By diagnostically simulating reconstructed objects under
gravity, we convert penetration and support failures into quantitative correction signals that drive
gravity-axis stretching and amodal shape resampling. This physics-informed feedback loop mitigates accumulated reconstruction errors and produces a stable, simulation-ready compositional 3D scene. Extensive experiments demonstrate state-of-the-art performance on physical stability and
geometric alignment benchmarks. We further highlight SimuScene's utility by deploying reconstructed environments in
humanoid control and
robot-arm manipulation tasks.
Read the full abstract ▾
All authors Inhee Lee, Sangwon Baik, Sungjoo Kim, Hyeonwoo Kim, Hyunsoo Cha, Hanbyul Joo
Robotics · Cross-listed from cs.CV
Exploring Easy Boosts for Lidar Semantic Scene Completion
Tetiana Martyniuk, Jonathan Seele, Alexandre Boulch et al. (+3 more)
This paper investigates "free lunch" strategies to boost the performance of
lidar semantic scene completion (SSC) without requiring complex architectural redesigns. We first demonstrate that endowing input point clouds with semantic pseudo-labels from off-the-shelf segmentors significantly improves the performance of existing architectures. By evaluating these models against an oracle, we establish that high-quality semantic priors are a primary driver of mIoU gains. Furthermore, we equip the input
lidar scan with visibility information that distinguishes between empty and unknown spaces, which provides a secondary performance boost across the tested architectures. Using these simple enhancements, we observe that older models remain competitive with state-of-the-art systems, and can even outperform them. Our code is available at https://github.com/astra-
vision/SSC-Priors.
Read the full abstract ▾
All authors Tetiana Martyniuk, Jonathan Seele, Alexandre Boulch, Gilles Puy, Renaud Marlet, Raoul de Charette
Robotics
Humanoid-GPT: Scaling Data and Structure for Zero-Shot Motion Tracking
Zekun Qi, Xuchuan Chen, Dairu Liu et al. (+10 more)
We introduce
Humanoid-GPT, a GPT-style
Transformer with causal
attention trained on a billion-scale motion corpus for whole-body control. Unlike prior shallow MLP trackers constrained by scarce data and an agility-generalization trade-off,
Humanoid-GPT is pre-trained on a 2B-frame retargeted corpus that unifies all major mocap
datasets with large-scale in-house recordings. Scaling both data and model capacity yields a single generative
Transformer that tracks highly dynamic behaviors while achieving unprecedented zero-shot generalization to unseen motions and control tasks. Extensive experiments and scaling analyses show that our model establishes a new performance frontier, demonstrating robust zero-shot generalization to unseen tasks while simultaneously tracking highly dynamic and complex motions.
Read the full abstract ▾
All authors Zekun Qi, Xuchuan Chen, Dairu Liu, Chenghuai Lin, Yunrui Lian, Sikai Liang, Zhikai Zhang, Yu Guan, Jilong Wang, Wenyao Zhang, Xinqiang Yu, He Wang, Li Yi
Robotics
AgenticRL: Agentic Reinforcement Learning with Self-Refinement for Complex UAV Navigation
Roohan Ahmed Khan, Yasheerah Yaqoot, Amir Atef Habel et al. (+2 more)
Deep reinforcement learning enables autonomous
robots to learn complex
navigation tasks, but still relies heavily on time consuming manual reward design and fine tuning. Existing automated reward generation and refinement methods reduce this effort, yet often lack task-level behavioral diagnosis for directing subsequent reward revisions. We introduce AgenticRL, a
multimodal closed loop framework in which role-specialized
agents generate executable rewards, diagnose failures of the resulting policies, formulate targeted refinement instructions, and regenerate improved rewards. Before training, a task grounding stage automatically selects a compatible action profile, together with its observation and reward interfaces. Each generated reward is used to train a policy using Proximal Policy Optimization (PPO), which is subsequently evaluated under randomized conditions. Task-level behavioral,
geometric, and safety measurements are organized into a structured diagnosis packet and jointly analyzed with the current reward code, task specification, behavioral summary, and visual scene context. Unlike one-shot reward generation, human-guided refinement, or broad candidate search, AgenticRL uses automated diagnosis of the behavior induced by a reward to direct its next revision. We evaluate the framework across eight
UAV tasks covering
navigation, obstacle interaction, trajectory tracking, agile manoeuvres, and cluttered flight. Under the reported comparative evaluation, AgenticRL achieves success rates of 100% in racing and 88% in cluttered
navigation, exceeding the strongest Eureka and Text2Reward baselines, respectively. Reward refinement increases mean simulation success from 37.2% to 96.4%, while the resulting policies achieve a collective real-world success rate of 90.0% and a sim-to-real accuracy of 93.4%.
Read the full abstract ▾
All authors Roohan Ahmed Khan, Yasheerah Yaqoot, Amir Atef Habel, Muhammad Ahsan Mustafa, Dzmitry Tsetserukou
Robotics · Cross-listed from cs.CV
Hanjiang Hu, Yiyuan Pan, Jiaxing Li et al. (+5 more)
As AI systems increasingly assist humans in physical tasks, ensuring safety becomes paramount – physical actions carry immediate and irreversible consequences that digital errors do not. We introduce the
Vision-
Language Embodied Safety
Agent (VLESA), a framework that monitors human activities from
egocentric video and triggers real-time safety interventions when dangerous actions are predicted. VLESA addresses intent-dependent safety where identical actions can be safe or dangerous depending on context. A
dataset pairing
egocentric frames with goal-conditioned safety annotations is introduced, enabling a goal-conditioned safety Q-filter trained via GRPO that evaluates actions with respect to inferred intent without retraining. On top of that, an intent-action prediction
agent is proposed to jointly infer goals and predict future actions from
video. On the ASIMOV-2.0
benchmark, VLESA achieves higher intervention accuracy at the exact ground-truth frame compared to baselines, while the GRPO-trained Q-filter improves action safety by over 41 percentage points through goal-conditioned constrained decoding. Code is available at https://github.com/HanjiangHu/VLESA.
Read the full abstract ▾
All authors Hanjiang Hu, Yiyuan Pan, Jiaxing Li, Xusheng Luo, Alexander Robey, Na Li, Yebin Wang, Changliu Liu
Robotics
Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation
Zeyi Liu, Guangyao Liu, Yinuo Qu et al. (+6 more)
Human-in-the-loop reinforcement learning (HIL-RL) improves sample efficiency in real-
robot manipulation through online human intervention. However, successful trajectories may include suboptimal actions that deviate from the desired task-execution path and force human intervention. Existing HIL-RL methods typically apply the consistent credit assignment principle to all transitions, uniformly propagating discounted terminal rewards through suboptimal segments, ignoring the actual contribution of each transition to task success. This overestimates Q-values for critic learning and indirectly misguides actor updates toward suboptimal behavior patterns. To this end, we propose PACT, a Preference-calibrated Actor-Critic Training framework that leverages the implicit preference signals induced by intervention to perform credit reassignment on identified suboptimal segments while directly guiding policy training for unbiased critic-actor learning. Specifically, we first design a progress model that learns from human demonstration and identifies suboptimal segments for credit correction. Then, from the human action and resampled policy action at the intervention state, we build preference pairs to define a counterfactual advantage that penalizes Bellman targets of the identified suboptimal segment, enabling directional credit calibration. Moreover, we directly align the policy with human corrective actions in the bounded mean space, providing an additional signal beyond critic-guided updates. Across five real-
robot manipulation tasks, PACT improves the average success rate by 24.5% and achieves 1.3 times faster convergence, thereby improving both RL sample efficiency and performance. Code is available at https://anonymous.4open.science/r/HILRL-A1X-BC05.
Read the full abstract ▾
All authors Zeyi Liu, Guangyao Liu, Yinuo Qu, Yuquan Xue, Bofang Jia, Chunhua Yang, Weihua Gui, Keke Huang, Ziwei Wang
Robotics
PointAction: 3D Points as Universal Action Representations for Robot Control
Mutian Tong, Han Jiang, Qiao Feng et al. (+2 more)
Video-Action Models (VAMs) leverage the broad visual dynamics captured by pre-trained
video diffusion models, offering a promising path toward generalizable
robot manipulation. However, RGB-only
video rollouts are not directly actionable: they leave metric 3D motion, contact geometry, and fine-grained spatial constraints under-specified, making action grounding ambiguous. Meanwhile, scaling action supervision across diverse tasks and embodiments remains costly. We present PointAction, a framework that bridges
video predictions to
robot actions through explicit point-based 4D modeling. PointAction fine-tunes a foundation
video generation model to jointly predict future RGB frames and dynamic 3D pointmaps, producing temporally consistent 3D motion of task-relevant scene geometry. These point dynamics serve as a structured, embodiment-agnostic action interface, which a
diffusion-based action decoder maps to executable
robot actions. By using metric 3D point dynamics as the interface between
video prediction and control, PointAction reduces the ambiguity of RGB-only action grounding and supports transfer across tasks and embodiments with limited action supervision. Experiments show that PointAction achieves state-of-the-art 4D generation quality on
robot scenes, outperforms existing baselines in simulation, and generalizes to two real
robot arms unseen during pretraining.
Read the full abstract ▾
All authors Mutian Tong, Han Jiang, Qiao Feng, Lingjie Liu, Jiatao Gu
Robotics · Cross-listed from eess.IV
SEAOTTER: Sensor Embedded Autoencoding with One-Time Transcode for Efficient Reconstruction
Dan Jacobellis and Neeraja J. Yadwadkar
In robotics systems, vast amounts of visual data are easily captured at high resolution using low-cost, low-power hardware. Yet, limited bandwidth and on-device compute resources prevent full utilization when transmitted via conventional codecs like JPEG/MPEG. Newer codecs, like AV1/AVIF, improve the rate-distortion trade-off, but demand far more resources for encoding, impractical without custom ASICs. Recent asymmetric autoencoders deliver high quality under extreme power and bandwidth constraints, but add prohibitive decoding cost and use bespoke formats that ignore decades of infrastructure built around standards like JPEG. To address these limitations, we introduce a compression framework for cloud robotics based on a Sensor Embedded Autoencoder paired with a One-Time Transcode for Efficient Reconstruction (SEAOTTER). Because the sensor, cloud, and consumer stages face very different power and bandwidth budgets, SEAOTTER combines the compactness of a learned latent with the broad usability of a standard JPEG file. Since naive transcoding degrades performance, we propose a learnable JPEG color and quantization transform that enables increased accuracy for global, dense, and
vision-
language-based perception. Using SEAOTTER, we train both general-purpose and task-aware transcoding pipelines for a pre-trained, frozen encoder. At a compression ratio of 200:1 and compared to AVIF, we observe 7 times faster encoding, 3.5 times faster decoding, and +8% ImageNet top-1 accuracy, while retaining compatibility with JPEG infrastructure. Our code is available at https://github.com/UT-SysML/seaotter .
Read the full abstract ▾
Robotics
Multi-Robot Bearing-based Pose Estimation via Angle Rigidity
J. Francisco Presenza, Leonardo J. Colombo, Ignacio Mas et al. (+1 more)
This letter proposes a novel distributed pose
estimator for multi-
robot systems evolving on $\mathrm{SE}(3)$. The
robots' positions are estimated in $\mathbb{R}^3$ using angles computed from body-frame bearings, without requiring orientation knowledge. The
robots' orientations are then recovered in $\mathrm{SO}(3)$ from the estimated positions, together with bearing and bearing-rate measurements. The
estimator accommodates directed sensing
topologies and requires only infinitesimal angle
rigidity (IAR), thereby relaxing the requirement, common in bearing-based approaches, that every
robot acquire at least two bearings. Unlike existing angle-based schemes, the proposed method also estimates the
robots' orientations. We prove local uniform exponential stability of the observer, assuming that a subset of
robots executes persistently exciting motions. These theoretical results are corroborated through numerical simulations.
Read the full abstract ▾
All authors J. Francisco Presenza, Leonardo J. Colombo, Ignacio Mas, Juan I. Giribet
Robotics
Semantic-weighted ICP for LiDAR Odometry: Class-Aware Residual Reweighting for Robust Scan Registration
Vasco Carvalho, Tiago Barros and Urbano J. Nunes
LiDAR odometry is a fundamental component of autonomous robotic systems, relying on
geometric registration between consecutive point clouds to estimate ego-motion. However, traditional
geometric approaches often degrade in dynamic or unstructured environments due to unreliable correspondences caused by moving objects, sparse
geometric features, vegetation, and semantically ambiguous structures. Existing works have shown that, some of these limitations can be addressed by introducing semantic information from the environment in the registration process. In this work, we build on this, and show that not all elements in the environment are equally relevant for registration. Hence, we propose a semantic class-weighted ICP for
LiDAR odometry. Instead of strictly filtering out points belonging to specific semantic classes, the proposed approach weights the residuals of points belonging to semantic categories based on their expected
geometric stability. This strategy enables informative but potentially unstable structures, to contribute to the registration process while mitigating the influence of dynamic objects. The experimental evaluation was conducted on the SemanticKITTI and RELLIS-3D
datasets, which include urban, highway, rural, and off-road environments. The empirical results show that the proposed Semantic-weighted ICP improves pose estimation, especially in challenging off-road scenarios where conventional rigid features are scarce. Furthermore, the analysis reveals that the effectiveness of this weighting strategy is highly environment-dependent, influenced by the structural and semantic composition of the scene.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
DyaPlex: Full-Duplex Speech-Motion Model for Dyadic Interaction
Koki Nagano, Hongyu Liu, Seonwook Park et al. (+9 more)
We present DyaPlex, a streaming, full-duplex
speech-and-motion model designed for dyadic interaction. To capture the continuous and reciprocal nature of human communication, this full-duplex capability empowers the
agent to simultaneously perceive and generate both
speech and physical motion in a streaming fashion. At its core, our method leverages the strong priors of a foundational full-duplex
speech model and integrates a novel motion pathway, thereby achieving fully synchronized multi-modal interaction. Specifically, we design a dual-tower
Transformer architecture that preserves the zero-shot conversational
reasoning of a frozen base
speech model while constructing a deeply coupled, streaming motion pathway. By introducing a unified dyadic
token interleaving mechanism and guiding cross-
attention via a time-aligned
speech-motion RoPE, our model effectively aligns autoregressive motions with rich latent
speech features. Trained on the 4,000-hour Seamless Interaction
dataset, our model effectively captures cross-speaker dependencies and establishes new state-of-the-art performance across both monadic and dyadic human interaction
benchmarks.
Read the full abstract ▾
All authors Koki Nagano, Hongyu Liu, Seonwook Park, Tianye Li, Amrita Mazumdar, Christian Jacobsen, Shengze Wang, Michael Stengel, Rajarshi Roy, Ka Chun Cheung, Simon See, Shalini De Mello
Robotics
Denoising Tells When to Replan: Denoising-Variance Adaptive Chunking for Flow-Based Robot Policies
Xiangdong Feng, Yuxuan Cheng, Chen Shi et al. (+5 more)
Action chunking has become a common inference strategy for flow-based
robot policies, improving action coherence by modeling multi-step temporal dependencies in demonstrations. However, the execution horizon is still typically set as an empirical fixed value, overlooking that predictable free-space motions and precision-critical interaction phases often require different replanning frequencies. In this work, we first show that the denoising process of flow-based policies contains an intrinsic signal of task phases: clean-action estimates remain stable during predictable motion phases, but fluctuate more strongly around contact-rich or precision-sensitive operations. Motivated by this observation, we propose DVAC (Denoising-Variance Adaptive Chunking), a test-time method that adaptively determines how many actions to execute from each predicted chunk. DVAC measures the variance of clean-action estimates over the final denoising steps, executes the stable low-variance prefix, and replans before high-variance future actions are committed. To transfer across tasks and rollouts, DVAC further calibrates the threshold with a rolling estimate of the local variance scale. Experiments on LIBERO, RoboTwin, CALVIN, and real-world manipulation show that DVAC improves task success while reducing replanning frequency. With a $\pi_{0.5}$-based policy, DVAC improves LIBERO success from 94.75% to 98.00% and reduces replanning by 43.0%, while also yielding aggregate gains on RoboTwin and CALVIN and improving real-world execution efficiency.
Read the full abstract ▾
All authors Xiangdong Feng, Yuxuan Cheng, Chen Shi, Boyao Han, Yuxuan Yan, Yitong Hong, Zhuotao Tian, Li Jiang
Robotics
Let the Dynamics Flow: Stable Flow Matching Dynamical Systems
Rodrigo Pérez-Dattari, Francisco Leiva, Andrea Testa et al. (+3 more)
Flow matching has recently emerged as a powerful approach for imitation learning, enabling scalable, expressive, and
multimodal motion policies. However, when modeling these policies as dynamical systems, incorporating formal stability guarantees into these generative models is a prerequisite to ensure safe and generalizable
robot behaviors, which remains a significant challenge. This paper introduces Stable Flow Matching Dynamical Systems (SFMDS), a novel framework that bridges the gap between highly expressive generative modeling and formal stability guarantees. SFMDS parametrizes dynamical systems via flow matching while constraining the model to satisfy positive invariance and/or Lyapunov stability conditions. We propose two variants: a soft constraint based on a penalty term, and a hard structural constraint embedded directly into the model architecture. We further extend both formulations to Lie groups to robustly handle orientation trajectories. Experiments on
benchmark datasets, in simulation, and on a
humanoid robot show that SFMDS learns stable, scalable, and
multimodal dynamical systems in low- and high-dimensional state spaces, enabling safe and expressive
robot motion generation. SFMDS matches or outperforms state-of-the-art methods on unimodal
datasets, while substantially improving performance on
multimodal datasets, where competing approaches fail to capture multi-modal behaviors. Accompanying source code and
video are available at: https://let-the-dynamics-flow.github.io/SFMDS/.
Read the full abstract ▾
All authors Rodrigo Pérez-Dattari, Francisco Leiva, Andrea Testa, Leonel Rozo, Javier Ruiz del Solar, Noémie Jaquier
Robotics
Optimal Design and Analytical Modeling of a Soft Fin-Ray Effect Gripper Finger Using the Finite Rigid Elements Method
Sara Adeli and Hassan Sayyaadi
Fin Ray-inspired soft grippers offer a promising solution for gently handling delicate, irregular objects, especially in agriculture. The objective of this research is to design, fabricate, and model a Fin Ray Effect (FRE) soft gripper finger to enable precise force control in future applications. This design aims to gently grasp delicate agricultural products, such as tomatoes, that require both adaptability and accurate force application. To address the inherent challenges of soft robotics, including nonlinear behavior, infinite degrees of freedom, and variable material properties, the Finite Rigid Elements Method (FREM) was employed for modeling. This method preserves analytical accuracy while providing a reliable foundation for the development of a force controller in later stages. A detailed Finite Element Model (FEM) was created using ANSYS, and the analytical results were validated through simulation and experimental testing. The gripper's fingers were optimized based on four key criteria: tip displacement, total deflection, stress distribution, and contact force. The optimal finger configuration includes a length of 30 mm, rib spacing of 10 mm, seven ribs angled at -15 deg, and a rib thickness of 1 mm. Theoretical modeling using the FREM predicted finger deformation with a 3% error, while the ANSYS numerical model achieved 2% error.
Read the full abstract ▾