Robotics
Contact as a Decision Variable: Capability-Tradeoff Contact Selection for Legged Loco-Manipulation
Al Jaber Mahmud, Shuai Li and Xuan Wang
In this paper, we study the joint selection of an environmental support contact and a whole-body configuration for a prescribed loco-manipulation task. A contact may provide greater physical support while restricting the motion required for the task. We formulate this problem through three capability measures: residual wrench, end-effector reach, and base mobility available after satisfying the task requirements, and we balance them against contact acquisition cost. Evaluating these capabilities for every candidate requires repeated whole-body optimizations. To reduce this computational cost, we propose Capability-Tradeoff Contact Selection (CTCS). CTCS screens candidates for contact and task feasibility, groups similar candidates within each surface, and predicts their capabilities from exact anchor evaluations using local sensitivity analysis. It checks these predictions through selective exact evaluations, ranks candidates by capability, and evaluates a shortlist exactly for final selection. We evaluate CTCS in simulations and hardware experiments using a Unitree Go2 quadruped with an AgileX NERO arm across $392$ task conditions with nine available support surfaces. Results show that CTCS outperforms ground-only and fixed-contact support, as it can select support surfaces that provide favorable capability trade-offs for the task. Compared with evaluating every candidate exactly, CTCS achieves approximately $3\times$ speedup while closely matching the resulting mean objective value.
Read the full abstract ▾
Robotics
Training-free Behavior Cloning
Maximilian Adang, Timothy Chen, Lars Osterberg et al. (+2 more)
Neural behavior cloning compresses demonstrations into large models, making individual actions difficult to trace and policy updates costly.
Retrieval policies retain access to demonstrations but struggle with mismatch between recorded and live behavior. We introduce Behavior Predictive Control (BPC), which synthesizes policies without end-to-end policy training by combining an action-aware
retrieval metric, a Hankel-based action-continuation prior, and a closed-form one-step residual correction. Inspired by behavioral systems theory, BPC predicts future actions by blending stored observation-action data that best reconstructs the recent runtime observation–action history. Across simulated
benchmarks and real-
robot deployments, BPC is competitive with learned policies such as $\pi_{0.5}$ (surpassing it in some cases), while reducing policy fitting from hours to seconds on consumer GPUs and supporting closed-loop control upwards of 75 Hz on a Jetson Orin Nano. The retrieved demonstration windows and their coefficients also provide an intrinsic estimate of task progress. Retaining demonstrations within the deployed policy makes its predictions traceable to supporting trajectories and enables behavior revision through the demonstration bank.
Read the full abstract ▾
All authors Maximilian Adang, Timothy Chen, Lars Osterberg, Aiden Swann, Mac Schwager
Robotics
Faster Visuomotor Policy Learning on Action Manifolds via Riemannian MeanFlow
S. Talha Bukhari, Austin Garrett, Yi Wei et al. (+3 more)
Visuomotor policies learn a direct map from raw sensory observations to
robot action sequences. Policies based on
Diffusion and Flow Matching capture the
multimodal distribution over action sequences in an end-to-end manner. This expressivity comes at the cost of multi-step numerical integration of the learned vector field for action generation, which can be expensive and time-consuming, impeding fast control rates required in robotics applications. Furthermore,
robot action sequences are usually defined on a smooth, differentiable
manifold, requiring that the learned policy respects the intrinsic geometry of the
robot's action space. Here, we present Riemannian MeanFlow Policy (RMFP), which learns the conditioned flow map of the probability path on the
robot action
manifold. Our formulation employs a flow map consistency objective grounded in the data by a Riemannian Conditional Flow Matching anchor. The flow map consistency condition is stable to train and constrains the learned model to finite-time transport, which yields on-
manifold action sequence generation with as few as one network function evaluation. We present results on the spherical LASA and Push-T
benchmarks, on the Tool Hang and Transport tasks of the Robomimic suite, and on the Franka Kitchen task with
manifold-constrained action generation, and demonstrate that RMFP attains performance competitive with prior work at a lower sampling cost. We also employ RMFP on a real-world robotic manipulation task to demonstrate fast action generation under imperfect sensor measurements in the physical world.
Read the full abstract ▾
All authors S. Talha Bukhari, Austin Garrett, Yi Wei, Ruiqi Ni, Zachary Kingston, Aniket Bera
Robotics
Self-Adaptive VLA for Robust Robot Deployment
Hongxin Zhang, Chunru Lin, Tsun-Hsuan Wang et al. (+2 more)
While
Vision-
Language-Action (VLA) models demonstrate impressive capabilities in robotic manipulation, their memoryless nature renders them brittle to test-time environment shifts, particularly hardware shifts caused by wear or imperfect calibration. Enabling these models to self-adapt during deployment without requiring continuous on-site recalibration remains a critical bottleneck for real-world scalability. In this work, we introduce Self-Adaptive VLA, a novel post-training recipe that enables the policy to iteratively adapt to deployment-time hardware shifts leveraging its own rollouts as context. To do so, we first collect policy rollouts under deliberately injected hardware shifts. We then transform the base policy's training data into shift-conditioned expert demonstrations by pre-compensating the expert actions for these known shifts. Next, we introduce a lightweight, plug-in context encoder that compresses the context, including visual observation, proprioception, and actions in the shifted environment, into a latent context
token. This
token modulates the policy through adaptive layer normalization (AdaLN). Furthermore, we find that context
tokens can be ensembled, allowing the policy to iteratively self-correct and mitigate failures step by step. Extensive experiments across four precision-critical bi-manual and dexterous manipulation tasks show that Self-Adaptive VLA recovers over 80% of the base policy's performance under hardware shifts, such as actuation bias and joint encoder offsets. Moreover, Self-Adaptive VLA enables more robust deployment to new workstations compared to the base policy. Our approach provides a pathway for robust large-scale real-world
robot deployments and easier maintenance. See
videos at https://icefoxzhx.github.io/self-adaptive-vla.
Read the full abstract ▾
All authors Hongxin Zhang, Chunru Lin, Tsun-Hsuan Wang, Zhenjia Xu, Chuang Gan
Robotics
Real-Time Force Regulation for Whole-Hand Dexterous Grasping
Sang Min Kim, Alexander Alexiev, Tzu-Yuan Lin et al. (+3 more)
Robust dexterous grasping requires maintaining physical stability despite contacts interactively evolving across the entire hand. A precomputed force distribution can easily fail under object motion, modeling errors, or external disturbances. In this paper, we present a framework for real-time force regulation over dynamically changing whole-hand contacts. Our method geometrically estimates contacts across all hand links using a tracked object model and proprioception, without requiring tactile sensing at those contacts. It repeatedly recomputes the desired contact-force distribution subject to friction constraints, actuator limits, and an actuation-consistency constraint motivated by classical whole-limb force analysis. We integrate this force-regulation controller with reactive reaching, enabling the hand to acquire a grasp, maintain it under disturbances, and regrasp after losing the object. Simulation experiments without
gravity demonstrate improved grasp retention over fixed-allocation and fingertip-only execution under controlled perturbations, while real-world experiments on a 27-DoF arm-hand system demonstrate grasp maintenance and recovery under human-applied disturbances as contacts evolve across the whole hand. Project page: https://sangminkim-99.github.io/reactive-grasp-whole-hand/
Read the full abstract ▾
All authors Sang Min Kim, Alexander Alexiev, Tzu-Yuan Lin, Sangbae Kim, Young Min Kim, Yonghyeon Lee
Robotics
M3GD: Multi-Modal Multi-View Geometric Diffusion for Camera–LiDAR Novel View Synthesis
Yang Zhou, Jiuhong Xiao, Shizhao Ye et al. (+3 more)
Robotic novel view synthesis (NVS) must recover both visual appearance and metric 3D structure, yet most generative NVS methods rely only on images, overlooking
LiDAR, a complementary sensor common on robotic platforms. We present M3GD, a Camera–
LiDAR multimodal representation for generative NVS that composes independently pretrained 2D image and 3D point-cloud foundation models without separately pretraining a cross-modal translator. We show that, after camera projection, frozen
LiDAR and image features exhibit substantial shared spatial structure, providing a natural cross-modal representation. M3GD conditions generation on
LiDAR through this structure: it combines explicit geometry statistics with learned point-cloud descriptors into view-aligned packets on the image-latent grid, injected through a lightweight residual adapter into a multi-view flow-matching generator whose latent space, decoders, and training objective remain intact. On the GrandTour
dataset, M3GD improves target-view RGB and depth synthesis over an image-only version of the same backbone. Ablations show that the gains come from pixel-aligned
LiDAR content and that target-view
LiDAR acts as a
geometric query linking the requested view to source observations. Deployment on a ground
robot demonstrates practical real-world operation, with a configurable quality–cost trade-off controlled by the number of Euler integration steps.
Read the full abstract ▾
All authors Yang Zhou, Jiuhong Xiao, Shizhao Ye, Long Quang, Carlos Nieto-Granda, Giuseppe Loianno
Robotics · Cross-listed from cs.LG
Aim Short to Reach Far: Your Frozen World Model Can Plan Better Than You Think
Xvyuan Liu, Jianjie Fang, Wei Wu et al. (+2 more)
Latent world models plan toward goal images with a frozen pretrained predictor, without task rewards or extra trained heads. However, their planners struggle with long-range goals, and prior work addresses this by training extra components such as value functions or subgoal models. We show that the planning target itself can cause this failure: even with exact dynamics and globally optimal short-horizon search, scoring predictions by their distance to the final goal rejects the first steps of a route that initially moves away from the goal. Building on this insight, we propose Anchored Planning (AP), a training-free method that reuses the world model's own offline trajectories. AP retrieves a segment that leads from the current observation toward the goal and aims the frozen planner at an observation shortly after the segment's start. Across four diverse tasks, AP substantially improves frozen LeWM planners for both action synthesis and action ranking, and it outperforms both additional final-goal search and the LeWM planner on long-range goals.
Read the full abstract ▾
All authors Xvyuan Liu, Jianjie Fang, Wei Wu, Chen Gao, Yong Li
Robotics
Body-Grounded Replanning for Physically Adaptive Manipulation
Namiko Saito and Hiroshi Kera
Manipulation requires not only
reasoning about the external environment, but also about the
robot's physical condition. A strategy may remain geometrically feasible while becoming physically unsuitable due to increased joint load or limited mobility, yet internal physical state is typically used only for low-level control. We propose body-grounded high-level replanning, which uses internal physical state to adapt manipulation strategies during execution. Body-state events trigger strategy replanning, and an LLM interprets the underlying joint-level state, recent execution statistics, and execution history to select a context-dependent alternative, while leaving the task objective and low-level controller unchanged. We evaluate the framework on a reaching task under controlled load and asymmetric mobility constraints in simulation and on a real
robot. Our experiments show that body-grounded replanning maintains high task success while reducing physical effort and enabling more efficient strategy adaptation. Additional contact-rich manipulation experiments demonstrate the applicability of the same replanning interface beyond reaching. These results show that internal physical state can inform not only low-level control, but also high-level decisions about how a manipulation task should be performed.
Read the full abstract ▾
Robotics
Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation
Mariia Iavorskaia, Christian Dietz, Sebastian Albrecht et al. (+1 more)
Imitation learning enables
robots to acquire manipulation skills from demonstrations, but the resulting policies can fail outside the training data, while collecting more demonstrations requires substantial human effort. Human-in-the-loop reinforcement learning uses corrective feedback during online training, but typically learns the complete task policy rather than refining a pretrained imitation policy. We introduce Res-HIL, a human-in-the-loop residual reinforcement learning framework that learns corrective actions on top of a frozen imitation policy. Each human intervention provides two complementary learning signals: direct supervision of the residual policy and reward shaping of preceding autonomous behavior. Res-HIL combines these signals with zero initialization of the residual policy to stabilize and accelerate online learning. We evaluate Res-HIL on five contact-rich manipulation tasks spanning high-precision and long-horizon behaviors. With only 20 initial demonstrations, Res-HIL outperforms state-of-the-art full-policy human-in-the-loop reinforcement learning and residual fine-tuning without human guidance on every task after ten minutes of online training. Res-HIL improves its pretrained base policies and outperforms imitation policies trained with five times more demonstrations. An ablation study shows that direct residual supervision is critical to performance, while intervention-aware reward shaping substantially improves training efficiency.
Read the full abstract ▾
All authors Mariia Iavorskaia, Christian Dietz, Sebastian Albrecht, Majid Khadiv
Robotics
World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal
Yehang Zhang, Haojian Huang, Yifan Chang et al. (+13 more)
General-purpose
vision-
language models (VLMs) bring broad knowledge and spatial
reasoning to
robot manipulation, yet existing systems either use them indirectly, to predict constraints or write programs, or give them a view of the scene rather than a world in which to act. We present World Action
Agent (WAA), a multi-
agent harness through which VLMs pilot
robots with basic tools, making every decision within a visual action workspace. The workspace has three properties. Contact views, selected automatically from the scene geometry, present the scene around the current interaction. Action rehearsal turns each action into an editable proposal that the
agent, alone or through an Imagination
Agent, previews and revises against planning feedback before execution. In-view correction closes the loop between observation, rehearsal, and low-level execution, letting the
agent remove residual offsets in the view where it observes them. Through the same workspace, WAA acquires embodied procedural knowledge in two ways: it evolves
multimodal skills from expert
videos and human teaching under evidence-based review and consults them through a Skill
Agent, and its interaction traces train smaller VLMs to pilot the same harness. On LIBERO-Pro, WAA with skills evolved only from LIBERO-90 reaches a state-of-the-art 75.6% average success, outperforming end-to-end VLAs, code-as-policy
agents, and a visual-harness baseline with the same backbone; the same skills remain effective on robosuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises its out-of-domain success from 1.7% to 43.3%.
Read the full abstract ▾
All authors Yehang Zhang, Haojian Huang, Yifan Chang, Jianchong Su, Bohan Zhou, Yingjie Xu, Wosong Chen, Tianhao Zhou, Chenxu Wang, Tianyi Zhang, Yangkai Wei, Wenqian Li, Shiyuan Deng, Yinchuan Li, Ying-Cong Chen, Zexi Li
Robotics · Cross-listed from cs.CV
Beyond Spatial Benchmarks: From Spatial Reasoning to Navigation
Xun Huang, Shijia Zhao, Rongsheng Qu et al. (+5 more)
Does progress on spatial
reasoning benchmarks translate into better navigation? Existing
benchmarks test isolated inferences from images or
videos, with little connection to downstream navigation. Our analysis reveals a gap between
benchmark-oriented spatial specialization and navigation performance, and shows how aligning spatial supervision with navigation goals, phases, and decision learning improves navigation. Guided by these findings, we build Spatial-Nav-100K and fine-tune in two stages, i.e. first learning a shared spatial-navigation foundation, and then specializing each phase with the abilities it relies on. We further introduce Spatial-NPD, where a teacher conditioned on spatial priors produces grounded action preferences and distills them into a student policy, so no explicit spatial
reasoning is needed at inference. With 45 A100 GPU-hours of policy training, our 8B model reaches SR/SPL of 77.4/35.4 on HM3D-v0.2, 60.2/30.5 on HM3D-v0.1, and 47.9/20.6 on train-unseen MP3D. It outperforms several systems that rely on closed-source models or thousands of GPU-hours of training, at 148 ms per action step. All code and
datasets will be publicly available at https://github.com/ylwhxht/Spatial-Nav.
Read the full abstract ▾
All authors Xun Huang, Shijia Zhao, Rongsheng Qu, Jiayuan Li, Xin Lu, Weixin Li, Chenglu Wen, Cheng Wang
Robotics
Pairwise Approximation Can Select the Wrong Multi-Robot Plan
William Teo
Multi-
robot coordination methods often score a joint plan from singleton and pairwise terms, leaving out the terms that involve three or more
robots. We measure the plan-selection regret of two pairwise approximations to delivered coverage using frozen multi-
robot trajectories. For each four-
robot plan on an indoor exploration
benchmark, replaying all 16
robot subsets gives the exact delivered-coverage set function $F$. From the same subset values we compute two pairwise scores: the exact order-2 Möbius truncation $F_2$, which depends only on the singleton and pair values, and an equal-weight least-squares two-additive fit $G$. Ranking by $F_2$ instead of $F$ changes the selected plan on six of seven maps at the 15 m candidate-generation range in each of two candidate families, with regret up to 0.337 of map coverage. Switching to $G$ reduces the regret but still changes the selection on three of seven maps in each family. The additive score $F_1$, which keeps only the singleton terms, selects the exact winner on six of seven maps in one family and four of seven in the other, against one of seven for $F_2$. We also find that lower average reconstruction error does not guarantee lower selection regret.
Read the full abstract ▾
Robotics
High-Voltage Optocoupler Amplifier for Electrostatic Actuators
George C. Jurgiel, Alex S. Miller and Jeffrey H. Lang
Many electrostatic actuators require multi-kilovolt drive voltages at sub-milliamp currents, a task poorly suited for conventional switching devices. As an alternative, we demonstrate a high-voltage amplifier using optocouplers as active elements. The amplifier produces a 20-kV peak-to-peak output with up to 500 Hz bandwidth while maintaining a minimal component count. By using optocouplers as linear devices in feedback, lower harmonic distortion and higher bandwidth are achieved than offered by equivalent PWM amplifiers. This design improves the viability of electrostatic actuators by providing a simpler method to achieve useful drive waveforms.
Read the full abstract ▾
Robotics
MorphIK: Morphology-Conditioned Neural Inverse Kinematics for Unknown Robots
Lennart Clasmeier, Jan Gerrit Habekost, Cornelius Weber et al. (+1 more)
Neural models can learn to generate various solutions to the inverse kinematics problem from data, but are usually limited to a single
robot. We present MorphIK, a flow-matching model that solves inverse kinematics for revolute-joint-based kinematic chains it has never seen during training. The model uses a
transformer architecture to encode the
robot's morphology along with the target pose. This encoding then conditions a flow-matching head that generates poses from noise. Trained on purely synthetic data from procedurally generated
robots, the model reaches a precision of about 5 cm on unseen real-world
robots with 6 to 9 Degrees of Freedom. For higher precision, the model serves as an excellent Prior for further optimization
algorithms, reducing error to less than 1 cm after a single step of Damped Least Squares optimization and to sub-1 mm error after 3 steps in most cases. Building on flow matching's generative capabilities to produce highly diverse outputs, our model can efficiently sample the
robot's null space, providing a wide variety of configurations for the same pose. Thus, overall, MorphIK allows learning and generalizing neural inverse kinematics for a multitude of known and unknown
robots.
Read the full abstract ▾
All authors Lennart Clasmeier, Jan Gerrit Habekost, Cornelius Weber, Stefan Wermter
Robotics
GPT-6-Astra Lights Up Embodied Navigation: Evaluation in Zero-Shot Vision-and-Language Navigation in Continuous Environments
Guangzhao Dai, Qianru Sun, Qi Wu et al. (+1 more)
We investigate whether GPT-6-Astra, a general-purpose foundation model, can navigate unfamiliar environments using its own perception,
reasoning, and decision-making capabilities. Our evaluation focuses on zero-shot
vision-and-
language navigation in continuous environments (VLN-CE) through a minimal interface in the Codex harness, aiming to unleash GPT-6-Astra's full potential for navigation. Using monocular RGB, GPT-6-Astra decides when to observe, how to move, and when to stop, without navigation-specific fine-tuning, a trained waypoint predictor, or a pre-built scene map. Our evaluation yields four key findings and implications. First, GPT-6-Astra achieves strong zero-shot navigation performance using only monocular RGB observations. On R2R-CE-100, GPT-6-Astra (ultra
reasoning) achieves a success rate of 81.3%, exceeding the strongest reported zero-shot and even train-based success rates by 15.3 and 9.2 percentage points, respectively. Second, GPT-6-Astra exhibits promising capabilities in interpreting multi-stage instructions, understanding the environment, and adjusting routes. Third, execution and goal-verification failures persist even with ultra
reasoning. Fourth, these results motivate combining general-purpose model capabilities with navigation-specific expertise. Based on these findings, future VLN research should investigate which aspects of instruction interpretation, spatial understanding, and navigation decision-making general-purpose models can handle directly, and where navigation-specific learning can extend their capabilities. This includes exploring how spatial representations, navigation experience, and learned control skills can improve progress tracking, error recovery, and goal verification while preserving the flexibility to adjust routes.
Read the full abstract ▾
All authors Guangzhao Dai, Qianru Sun, Qi Wu, Bin Zhu
Robotics
BeyondRetarget: Learning Executable Humanoid Motions Directly from Monocular Video
Tianyu Xiong, Yi Lu, Jinrui Wang et al. (+6 more)
Learning executable motions from human
videos offers a scalable solution for humanoid
robots to acquire demonstration motions. However, existing pipelines typically first construct an explicit human motion representation and then convert it into
robot motions via motion retargeting. Although such methods can effectively leverage large volumes of existing human data for training, the substantial differences between humans and humanoid
robots in locomotion mechanisms and joint degree-of-freedom configurations make motions generated by this human-representation-centric approach difficult to execute on
robots. Furthermore, errors introduced during human motion estimation inevitably propagate to the retargeting stage and cannot be eliminated via joint optimization. We propose BeyondRetarget, an end-to-end framework that directly maps monocular RGB
videos to
robot motions. Discarding the explicit human representation, this framework learns
robot-oriented implicit representations directly from visual observations, enabling the model to capture cross-morphology motion structures. To generate motions more suitable for
robot execution, we further design a contact-aware motion optimization mechanism to improve temporal consistency and physical plausibility. Experiments show that BeyondRetarget significantly improves the accuracy and robustness of generated
robot motions, while achieving higher execution success rates and lower latency in both simulation environments and real humanoid
robots.
Read the full abstract ▾
All authors Tianyu Xiong, Yi Lu, Jinrui Wang, Ziqi Liang, Dandan Lei, Xiaoyang Zhou, Xiao-xiao Long, Qiu Shen, Xun Cao
Robotics · Cross-listed from cs.CV
SplatLabel: Pseudo-Labelling through 4D Gaussian Splatting
Nitya Nanvani, Andras Palffy and Holger Caesar
While 2D
Vision Foundation Models offer a pathway to automate 3D semantic pseudo-labelling, translating these priors into robust 3D representations typically requires complex heuristics or multi-model ensembles. We introduce SplatLabel, an automated pipeline that leverages a 4D Gaussian representation to extract
LiDAR segmentation with predictive confidence, as well as semantic occupancy grids at arbitrary voxel resolutions. At its core, SplatLabel handles dynamic environments through an explicit temporal
manifold that models the trajectories and lifespans of individual 3D primitives. This allows the system to accurately track moving actors and strictly define when objects appear and disappear, completely eliminating the need for pre-annotated 3D bounding boxes. To robustly support this dynamic tracking, the representation is grounded by structural and semantic priors: we guide scene geometry in unobserved regions by integrating 360-degree
LiDAR via virtual depth maps, and rather than relying on domain-specific
prompt engineering, we directly distill continuous soft probabilities from 2D models to inherently resolve semantic ambiguities over time and space. Finally, to accurately reflect the real-world trade-off between precision and recall, we reframe pseudo-label evaluation as a selective classification task using a generalized risk-recall metric. Experiments on SemanticKITTI demonstrate that SplatLabel consistently outperforms state-of-the-art baselines across multiple recall levels, establishing a highly robust framework for both 3D
LiDAR segmentation and occupancy prediction.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Retrieve-to-Localize: Bridging Large Language Models and LiDAR Geometry for Spatial Grounding
Byounggun Park, Giyong Moon, Jusung Kim et al. (+1 more)
LiDAR provides precise
geometric information for spatial perception tasks such as object detection in autonomous driving and outdoor robotics. However, recognizing and localizing individual objects is not sufficient to answer questions that require composing spatial relations and grounding the intended target. Motivated by recent advances in large
language models (LLMs) for autonomous driving, we leverage their
language priors to interpret complex spatial questions and ground the referred target in
LiDAR geometry. To support this spatial grounding capability, we introduce SpatialLiDAR-QA, which combines single- and multi-step relational grounding with complementary spatial understanding tasks. We further propose SpatialLiDAR-LM, which aligns
LiDAR point features with an LLM and grounds target coordinates through
language-conditioned, position-aware proposal
retrieval and local point refinement. This design derives target coordinates directly from local
LiDAR geometry rather than through textual
language decoding. Experiments demonstrate substantial improvements over representative
LiDAR–
language models and multi-camera VLMs on precise coordinate prediction tasks. Our
dataset and model training code will be publicly released.
Read the full abstract ▾
All authors Byounggun Park, Giyong Moon, Jusung Kim, Soonmin Hwang
Robotics · Cross-listed from eess.SY
System Identification of an Octocopter in Hover using Full-Harmonic Orthogonal Multisine Inputs
Justin J. Matt and George V. Altamirano
A new method for multi-input flight maneuver design for system identification is presented. The method consists of injecting "full-harmonic" orthogonal multisine signals into the flight control system. Orthogonality is achieved by repeating maneuvers with changing multisine polarities. The multisines can contain the same frequency content, which can simplify frequency response estimation and allow for long flight maneuvers to be split into several shorter maneuvers while maintaining the same frequency resolution and minimum frequency. An input allocation scheme is presented that augments the multisines to size the vehicle response amplitude about a specific degree of freedom. The developed approach was demonstrated through flight testing of a small octocopter in near-hover conditions. The input allocation scheme was utilized successfully to increase excitation about the yaw axis. Electrical power, motor speed, and rigid-body dynamic models were identified and are shown to predict the vehicle and motor responses accurately. The models are parameterized primarily by rotor thrust and torque coefficients, making them suitable for analysis of aircraft flight dynamics and individual rotor aerodynamics. The results demonstrate that the near-hover flight dynamics can be modeled accurately by neglecting rotor hub moments, variations in rotor coefficients, gyroscopic moments in roll and pitch, and aerodynamic interaction effects.
Read the full abstract ▾
Robotics
Self-Supervised Anchoring of Fingertip Sensing to Proprioception and Proactive Actions for Robot Imitation Learning
Tomohiro Motoda, Masaki Murooka, Keisuke Shirai et al. (+6 more)
Robotic imitation learning often relies on external cameras, yet local interaction cues such as object proximity, contact onset, and grasp state are difficult to observe near the fingertips because of occlusion and limited temporal resolution. We study how to effectively incorporate complementary fingertip sensing into imitation learning using pressure-sensitive tactile and reflective proximity sensors, along with pretrained sensor encoders. The two modalities provide information at different manipulation phases: proximity sensing is informative before contact, whereas tactile sensing becomes informative after contact. However, naively adding these signals to a policy does not consistently improve performance and can even underperform
vision-only policies, suggesting that sparse, phase-dependent sensor signals are difficult to exploit from limited demonstrations. We therefore propose a proprioception-anchored pretraining method, PROprioceptive-and-PRoactive Anchoring (PROPRA), which independently aligns each fingertip sensor history with proprioceptive and action segments. This provides a continuously available sensorimotor reference, allowing each sensor to be aligned independently during its informative phases. Experiments on real-world manipulation tasks show that our pretraining method improves average success rates over
vision-only policies and image-anchored pretraining baselines. Representation analysis further shows that it preserves richer information about pre-contact states, enabling more effective use of complementary fingertip sensing. Please refer to our project page: https://tomohiromotoda.github.io/nia.propra/
Read the full abstract ▾
All authors Tomohiro Motoda, Masaki Murooka, Keisuke Shirai, Hanbit Oh, Ryoichi Nakajo, Shotaro Miwa, Roman Mykhailyshyn, Hugo Duarte, Yukiyasu Domae
Robotics
PolyUMI: Accessible Visual-Tactile-Audio Data Collection for Object Inference and Manipulation
Conor W. Hayes, Rickmer Krohn, Aravind Ramaswami et al. (+6 more)
Humans typically rely on
vision, touch, hearing, and proprioception to perceive contact and adapt their actions during manipulation. Providing
robots with comparable responsiveness therefore requires hardware that can retain and use these complementary sensory signals. Most imitation-learning systems, however, observe demonstrations primarily through
vision and proprioception, limiting access to contact information that is difficult to infer visually. We present PolyUMI, an open-source platform for scalable visual–tactile–audio demonstration collection and
robot deployment. Its lightweight, wireless handheld gripper records synchronized wrist-camera, optical tactile, contact-audio, and proprioceptive observations without requiring a tethered workstation. The same sensing finger can be transferred to the
robot end effector, preserving the sensing geometry between demonstration collection and policy execution. To effectively use these heterogeneous observations, we further introduce VisTA, a
token-level
multimodal policy that integrates information across sensors and time to predict contact-aware
robot actions. Experiments spanning object inference, slip control, and contact-rich manipulation show that touch and audio reveal task-relevant information beyond
vision and that VisTA is competitive with or outperforms existing
multimodal policies. Together, PolyUMI and VisTA provide an accessible pipeline for collecting
multimodal demonstrations and learning policies that perceive physical interaction beyond
vision. Project Page: https://polyumi-vista.github.io
Read the full abstract ▾
All authors Conor W. Hayes, Rickmer Krohn, Aravind Ramaswami, Anunth Ramaswami, Nils Dengler, Kevin M. Lynch, J. Edward Colgate, Georgia Chalvatzaki, Matthew L. Elwin
Robotics
From Target Selection to Digging: A Learning-Based Framework for Continuous Autonomous Excavation
Shuai Zhao, Ji-an Pan, Quantao Yang et al. (+4 more)
Repeated excavation continuously reshapes pile geometry, requiring an autonomous excavator to adapt its digging targets and coordinate motion across successive excavation cycles. We present a learning-based framework for continuous autonomous excavation that integrates terrain-aware target selection with reinforcement- and imitation-learning controllers. The framework separates target-conditioned motion from local digging: a shared task-conditioned RL policy controls waypoint-guided approach and loaded transport, while an IL policy learns
vision-based digging and lifting from expert demonstrations. Digging targets are selected from
LiDAR elevation maps and converted into bucket-tip waypoints for motion control. The control architecture coordinates the learned policies and deterministic unloading through a shared motion interface. The complete system is deployed on a scaled hydraulic excavator with
multimodal sensing and closed-loop actuator control. Offline replay and physical experiments demonstrate more consistent target selection, shorter local motion time, and increased payload compared with the respective baselines. The learned digging policy achieves a mean payload of 6.52 kg per completed cycle, compared with 2.68 kg for Fixed Dig. Three five-scoop runs further demonstrate consecutive autonomous excavation under continuously changing pile geometry.
Read the full abstract ▾
All authors Shuai Zhao, Ji-an Pan, Quantao Yang, Zheng Wang, Chaoyi Chen, Qing Xu, Keqiang Li
Robotics
Combining Evasive and Braking Reactions for Safety Reference Models in Automated Vehicles
Riccardo Donà, Konstantinos Mattas and Biagio Ciuffo
Computational models of careful and competent human drivers are essential for scenario-based evaluation of automated driving systems (ADS). However, most existing safety reference models primarily focus on longitudinal braking, neglecting the role of evasive steering in human collision avoidance. This paper proposes a hybrid Fuzzy-Safety Model (FSM-H) that integrates longitudinal mitigation and lateral avoidance within a unified behavioral framework. The braking component is governed by Proactive Fuzzy Safety (PFS) metrics, representing the erosion of longitudinal safety margins, while the steering component is driven by Criticality Fuzzy Safety for lane-change (CFS-LC), capturing lateral conflict severity and maneuver feasibility. A finite-state architecture models the sequential escalation from nominal driving to braking and, when necessary, to evasive steering, incorporating perception-reaction time and lane-check delays to reflect human decision processes. The model is evaluated in reconstructed high-criticality cut-in scenarios and compared with braking-only and steering-only reference strategies. Results show that the hybrid approach expands the preventability envelope while maintaining behavioral plausibility and computational tractability. The proposed framework provides a transparent and explainable human reference model suitable for simulation-based ADS safety benchmarking and regulatory assessment.
Read the full abstract ▾
Robotics
Markerless Multi-Modal Autonomous Robotic Inspection of Large Space Structures
Juan De Dios Alfaro, Arturo Ríos, David Rodríguez-Martínez et al. (+1 more)
Future orbital infrastructures, such as deployable
antennas, solar farms, and large orbital platforms will require autonomous inspection systems able to operate with limited prior knowledge and without cooperative markers. Current on-orbit servicing approaches often rely on predefined trajectories, standard interfaces, fiducial markers or accurate target models, which limits scalability for large, heterogeneous or partially unknown structures. This paper presents a markerless autonomous robotic inspection pipeline in which 3D reconstruction is used as an inspection-support representation. The system integrates a Kinova Gen2 manipulator with an end-effector-mounted
multimodal sensor head composed of an RGB-D camera, a thermal camera and a 2D
LiDAR. The pipeline estimates an approximate inspection volume, generates viewpoints, plans collision-free motions with MoveIt, and synchronously records RGB-D images, thermal data, and
robot poses in ROS2. Candidate reconstruction methods were evaluated to select a practical method for this pipeline, with Nerfacto used for
geometric reconstruction and Thermal-Nerfacto used to demonstrate thermal-aware rendering for inspection. Validation in a Gazebo-based simulator and preliminary laboratory tests reveal that the proposed system can autonomously acquire spatially coherent inspection data and produce reconstructions suitable for visual and
geometric assessment, representing a step towards inspection of large non-cooperative space structures.
Read the full abstract ▾
All authors Juan De Dios Alfaro, Arturo Ríos, David Rodríguez-Martínez, Carlos Pérez-del-Pulgar
Robotics
Coupled State-Space Modelling, Control, and Policy Distillation for Hybrid Rigid-Pneumatic Manipulators
Alan Royce Gabriel Samuel and Pulkit Verma
Hybrid manipulators combine motorized rigid joints with pressure-actuated origami segments. Published arms of this kind are controlled with decoupled per-DOF loops, and the cost of this approximation has not been quantified, because the coupled model needed to measure it has not been built. This paper derives such a model for a chain of $N$ alternating revolute joints and Kresling origami segments, including pneumatic chamber dynamics and crease hysteresis. Using the model, we measure the coupling directly and show that its strength varies joint by joint, and that decoupled control loses precisely on the strongly coupled joints while remaining competitive on the one nearly decoupled joint. Coupled model-based controllers track $2.5\times$ tighter than a decoupled PID baseline at lower torque. However, the model predictive controller (MPC) is too slow for real time, and model-free reinforcement learning stalls far below acceptable success rates on a strict settling metric. We therefore distill the MPC into a small neural policy with behavior cloning and DAgger. The distilled policy settles 93-94$\%$ of goals with zero collisions, within a few points of its teacher, and runs inside the 5 ms control step where the MPC does not. Where the teacher itself fails, we trace the failure to a limit cycle with the bellows' lightly damped mode, and we remove it by selecting goal postures holdable at low pressure.
Read the full abstract ▾
Robotics
Temperament Engineering: Designing Strategic Behavioural Diversity in Robot Swarms
Edmund R. Hunt
No two
robots are truly identical: calibration, battery state, sensor drift and wear give every swarm a distribution of behaviour rather than a single point, usually treated as an imperfection to be minimised. In animal collectives the reverse holds: consistent individual differences in behaviour ('temperament') are shaped by natural selection and often decisive for group performance. This perspective proposes 'temperament engineering', a bio-inspired framework that treats the swarm's distribution of temperaments, rather than the individual controller, as the design object. It borrows five evolutionarily validated axes of animal temperament (shyness-boldness, exploration-avoidance, activity, aggressiveness and sociability) as a design vocabulary, rendering each as a continuous control parameter $\tau \in [0,1]$ above the controller, realisable as a module threshold, a policy-conditioning vector in multi-
agent reinforcement learning, or a constraint on a foundation-model planner. A three-phase workflow maps mission success criteria onto relevant axes, plans the shape of the $\tau$ distribution, and tunes reaction norms governing how temperament responds to environmental cues. The payoff is greatest under decentralisation: where a central planner can reassign behaviour online, a temperament distribution is a planner output, but in a swarm without global knowledge it must be an offline, anticipatory design input. Behavioural and platform heterogeneity are thereby co-design variables, and I sketch tentative
robot-native axes (self-model plasticity, forcefulness, initiative and expressiveness) arising from features
robots have and animals do not. Engineered heterogeneity has been shown to outperform homogeneous swarms in tasks such as aggregation and exploration; establishing when, and how much, heterogeneity repays its cost is the work the field can now take forward.
Read the full abstract ▾
Robotics
UCON: Uncertainty-aware Navigation with Historical Re-association in Dynamic Environments
Bing Sun, Yue Lin, Yongsheng Yuan et al. (+3 more)
Autonomous navigation in dynamic environments is hindered by two fundamental challenges: perception instability and uncertainty-optimization mismatch. The former leads to identity switches and unreliable motion estimation, while the latter prevents principled incorporation of motion uncertainty into trajectory optimization. To address these challenges, we propose UCON, an uncertainty-aware navigation
algorithm in dynamic environments. For perception instability, we present a point-level historical re-association mechanism that leverages historical point cloud fragments to recover lost targets while maintaining identity continuity. Subsequently, a Kalman filter is employed to provide anisotropic motion state estimation and covariance propagation. To resolve the uncertainty-optimization mismatch, we transform predicted states and their covariances into uncertainty sectors, which are embedded as differentiable cost terms within a trajectory optimization framework. This achieves consistent uncertainty-aware dynamic obstacle avoidance while maintaining smoothness and feasibility. Extensive simulations and real-world experiments demonstrate that, while maintaining high computational efficiency, UCON achieves superior perception stability and robust navigation performance in dynamic environments compared to state-of-the-art methods. The code will be open-sourced to facilitate further research.
Read the full abstract ▾
All authors Bing Sun, Yue Lin, Yongsheng Yuan, Yang Liu, Dong Wang, Huchuan Lu
Robotics
Singularity Analysis for the Perspective-Four and Five-Line Problems
Jorge García Fontán, Abhilash Nayak, Sébastien Briot et al. (+1 more)
This paper deals with image-based visual servoing and pose estimation by observing four and five lines. Our main interest is to determine the relative configurations of the camera and the observed lines that lead to problems in control and stability. Since it is equivalent to finding the singularities of the corresponding Jacobian
matrix, we use tools from computational algebraic geometry to seek configurations such that all of its minors vanish simultaneously. By choosing a suitable basis for this
matrix, we revisit the problem in the case of three lines to show that one type of the singularities is when the camera lies on the hyperboloid of one sheet uniquely defined by the lines. This result is further exploited to prove that the one-dimensional singularities, if any, in the case of $n$ lines appear when the camera lies on the transversals to the observed lines. Thus, by forcing the transversals to be complex, we can avoid the aforementioned type of singularities in the case of four lines although the
algebra shows that there can always be up to 10 inevitable singular locations of the camera for the other type of singularity. For five lines, we find out that there are no singularities in the generic case. The singularities are also characterized for four and five lines with orthogonality and parallelism constraints. Furthermore, a visual servoing library is used to conduct some simulated experiments to substantiate the theoretical results. As expected, we observe problems in control in the vicinity of a singularity as well as increased errors in pose estimation.
Read the full abstract ▾
All authors Jorge García Fontán, Abhilash Nayak, Sébastien Briot, Mohab Safey El Din
Robotics
WRAP: Fixtureless Wrench-aware Multi-Robot Assembly Planning
Valentin N. Hartmann, Huang Su, Yijiang Huang et al. (+1 more)
Assembly using
robots often requires specially designed fixtures, or relies on top-down only assembly strategies. Using multiple
robots, we can avoid using fixtures and make robotic assembly more flexible. Planning assembly sequences for multiple
robots is challenging due to the high number of possible task assignments and orders. In addition, we need to reason over forces that occur during the assembly process, e.g., to decide if multiple
robots are required for support, or if external support such as a table should be used. We present Wrap, a multi-
robot assembly planner for multi-part assemblies, given the inter-part ordering-dependencies, the part meshes, and their initial state. We formulate a linear program to reason about valid grasps for supporting the forces that occur during assembly. The search leverages the assembly sequence, and greedily finds a feasible solution per assembly step by computing a heuristic via a cheap backwards search, and using the heuristic in the more expensive forward search. We then solve the multi-
robot, multi-goal motion planning problem, and for execution, we split the plan into contact-rich assembly skills, and free space motion. We
benchmark the planner on a variety of multi-part assemblies, and apply the planner to groups of
robots differing in size and kinematics. We validate the work both in a physics simulation, and in real.
Videos and code are available at https://www.vhartmann.com/wrap.
Read the full abstract ▾
All authors Valentin N. Hartmann, Huang Su, Yijiang Huang, Stelian Coros
Robotics
RACaP: Agentic Reasoning, Acting, and Coding as Policies for Evolvable Robot Learning
Zexi Li, Yehang Zhang, Haojian Huang et al. (+10 more)
General-purpose
robot agents must learn from experience, transfer to new tasks, and act efficiently. Code as Policies (CaP) methods generate and repair programs at runtime, incurring latency and entangling reusable mechanisms with task-specific decisions. We introduce RACaP, an agentic framework that moves coding to
evolution and uses a
Reasoning-and-Acting (ReAct) loop to call frozen, typed Policy APIs at deployment. A two-phase strategy combines capability curriculum learning with autonomous self-
evolution to improve the APIs, the ReAct harness, and experience
memory. The APIs encode reusable physical mechanisms while exposing arguments for runtime adaptation. ReAct combines task-specific working
memory, long-term experience
memory, and visual feedback to select actions, verify outcomes, and recover from failures without modifying source code. RACaP achieves 54.4% success on LIBERO-90, 45.0% on zero-shot LIBERO-PRO, and 46.0% on LIBERO-Long, compared with at most 4.0% for CaP baselines on long-horizon tasks. On LIBERO-PRO, it achieves 2.5 times the success rate of CaP baselines and a 1.9-fold speedup in median policy time. For efficient on-
robot deployment, rejection-sampled fine-tuning distills GPT-5.6 ReAct decisions into Qwen3-VL-8B-Instruct, yielding a 13.2-fold per-decision inference speedup and reducing repeated physical calls from 16 to 4. These results show that separating reusable code from runtime decisions supports continued
evolution, effective transfer, and efficient long-horizon control.
Read the full abstract ▾
All authors Zexi Li, Yehang Zhang, Haojian Huang, Bohan Zhou, Wenqian Li, Chenxu Wang, Yifan Chang, Yangkai Wei, Tianyi Zhang, Ying-Cong Chen, Kaiwen Zhou, Yinchuan Li, James Cheng