Robotics
Reasoning Knowledge-Gap in Drone Planning via LLM-based Active Elicitation
Zeyu Fang, Beomyeol Yu, Cheng Liu et al. (+5 more)
Human-AI joint planning in Unmanned Aerial Vehicles (
UAVs) typically relies on control handover when facing environmental uncertainties, which is often inefficient and cognitively demanding for non-expert operators. To address this, we propose a novel framework that shifts the collaboration paradigm from control takeover to active information elicitation. We introduce the Minimal Information Neuro-Symbolic Tree (MINT), a
reasoning mechanism that explicitly structures knowledge gaps regarding obstacles and goals into a queryable format. By leveraging large
language models, our system formulates optimal binary queries to resolve specific ambiguities with minimal human interaction. We demonstrate the efficacy of this approach through a comprehensive workflow integrating a
vision-
language model for perception, voice interfaces, and a low-level
UAV control module in both high-fidelity NVIDIA Isaac simulations and real-world deployments. Experimental results show that our method achieves a significant improvement in the success rate for complex search-and-rescue tasks while significantly reducing the frequency of human interaction compared to exhaustive querying baselines.
Read the full abstract ▾
All authors Zeyu Fang, Beomyeol Yu, Cheng Liu, Zeyuan Yang, Rongqian Chen, Yuxin Lin, Mahdi Imani, Tian Lan
Robotics
Knowing When to Ask: Resolving Uncertainty in Human-Robot Joint Planning via Explicit Dialogue and Implicit Intent Cues
Zeyu Fang, Yuxin Lin, Cheng Liu et al. (+6 more)
Effective human-
robot collaboration in open-world environments requires joint planning under uncertainty about the task, the environment, and the human teammate. Communication is the most direct means of resolving such uncertainty, yet most existing systems support only one-way communication:
robots listen and act, treating humans as passive supervisors rather than conversational teammates capable of two-way dialogue. We propose a unified human-
robot joint planning system in which the
robot actively resolves uncertainty through two complementary communication channels. When uncertainty is decision-critical, an uncertainty-mitigation joint planning module engages the human in clarification dialogue: it grounds ambiguous instructions via an LLM-assisted active elicitation mechanism, enumerates traversability hypotheses through a hypothesis-augmented A* search, and computes a cost-optimal querying policy via dynamic programming, so that the
robot asks only the questions whose answers actually matter for the plan. When explicit dialogue is unnecessary or impractical, a real-time intent-aware collaboration module instead reads implicit, nonverbal cues, maintaining a probabilistic belief over the human's latent task intent from spatial and directional signals to enable coordination-aware task selection without any communication overhead. We validate the proposed system in both Gazebo simulations and real-world
UAV deployments, integrated with a voice dialogue interface and a
Vision-
Language Model (VLM)-based 3D semantic perception pipeline. Experimental results show that cost-optimal clarification dialogue cuts the interaction cost by 51.9% while maintaining a 100% task success rate, and implicit intent reading reduces the cooperative task execution time by 25.4% compared to the baselines.
Read the full abstract ▾
All authors Zeyu Fang, Yuxin Lin, Cheng Liu, Beomyeol Yu, Zeyuan Yang, Rongqian Chen, Taeyoung Lee, Mahdi Imani, Tian Lan
Robotics
Preference-Conditioned Reinforcement Learning for Space-Time Efficient Online 3D Bin Packing
Nikita Sarawgi, Omey M. Manyar, Fan Wang et al. (+3 more)
Robotic bin packing is widely deployed in warehouse automation, with current systems achieving robust performance through heuristic and learning-based strategies. These systems must balance compact placement with rapid execution, where selecting alternative items or reorienting them can improve space utilization but introduce additional time. We propose a selection-based formulation that explicitly reasons over this trade-off: at each step, the
robot evaluates multiple candidate actions, weighing expected packing benefit against estimated operational time. This enables time-aware strategies that selectively accept increased operational time when it yields meaningful spatial improvements. Our method, STEP (Space-Time Efficient Packing), uses a preference-conditioned,
Transformer-based reinforcement learning policy, and allows generalization across candidate set sizes and integration with standard placement modules. It achieves a 44% reduction in operational time without compromising packing density. Additional material is available at https://step-packing.github.io.
Read the full abstract ▾
All authors Nikita Sarawgi, Omey M. Manyar, Fan Wang, Thinh H. Nguyen, Daniel Seita, Satyandra K. Gupta
Robotics · Cross-listed from cs.CV
MWM: Mobile World Models for Action-Conditioned Consistent Prediction
Han Yan, Zishang Xiang, Zeyu Zhang et al. (+1 more)
World models enable planning in imagined future predicted space, offering a promising framework for
embodied navigation. However, existing
navigation world models often lack action-conditioned consistency, so visually plausible predictions can still drift under multi-step rollout and degrade planning. Moreover, efficient deployment requires few-step
diffusion inference, but existing
distillation methods do not explicitly preserve rollout consistency, creating a training-inference mismatch. To address these challenges, we propose MWM, a mobile world model for planning-based image-goal
navigation. Specifically, we introduce a two-stage training framework that combines structure pretraining with Action-Conditioned Consistency (ACC) post-training to improve action-conditioned rollout consistency. We further introduce Inference-Consistent State
Distillation (ICSD) for few-step
diffusion distillation with improved rollout consistency. Our experiments on
benchmark and real-world tasks demonstrate consistent gains in visual fidelity, trajectory accuracy, planning success, and inference efficiency. Code: https://github.com/AIGeeksGroup/MWM. Website: https://aigeeksgroup.github.io/MWM.
Read the full abstract ▾
All authors Han Yan, Zishang Xiang, Zeyu Zhang, Hao Tang
Robotics
Toward Global Intent Inference for Human Motion by Inverse Reinforcement Learning
Sarmad Mehrdad, Maxime Sabbah, Vincent Bonnet et al. (+1 more)
This paper investigates whether a single, unified cost function can explain and predict human reaching movements, in contrast with existing approaches that rely on subject- or posture-specific optimization criteria. Using the Minimal Observation Inverse Reinforcement Learning (MO-IRL)
algorithm, together with a seven-dimensional set of candidate cost terms, we efficiently estimate time-varying cost weights for a standard planar reaching task. MO-IRL provides orders-of-magnitude faster convergence than bilevel formulations, while using only a fraction of the available data, enabling the practical exploration of time-varying cost structures. Three levels of generality are evaluated: Subject-Dependent Posture-Dependent, Subject-Dependent Posture-Independent, and Subject-Independent Posture-Independent. Across all cases, time-varying weights substantially improve trajectory reconstruction, yielding an average 27% reduction in RMSE compared to the baseline. The inferred costs consistently highlight a dominant role for joint-acceleration regulation, complemented by smaller contributions from torque-change smoothness. Overall, a single subject- and posture-agnostic time-varying cost function is shown to predict human reaching trajectories with high accuracy, supporting the existence of a unified optimality principle governing this class of movements.
Read the full abstract ▾
All authors Sarmad Mehrdad, Maxime Sabbah, Vincent Bonnet, Ludovic Righetti
Robotics
Inverse Resistive Force Theory (I-RFT): Learning granular properties through robot-terrain physical interactions
Shipeng Liu, Feng Xue, Yifeng Zhang et al. (+2 more)
For
robots to navigate safely and efficiently on soft, granular terrains, it is crucial to gather information about the terrain's mechanical properties, which directly affect locomotion performance. Recent research has developed robotic legs that can accurately sense ground reaction forces during locomotion. However, existing tests of granular property estimation often rely on specific foot trajectories, such as vertical penetration or horizontal shear, limiting their applicability during natural locomotion. To address this limitation, we introduce a physics-informed machine learning framework, Inverse Resistive Force Theory (I-RFT), which integrates the Granular Resistive Force Theory model with Gaussian Processes to infer terrain properties from proprioceptively measured contact forces under arbitrary gait trajectories. By
embedding the granular force model within the learning process, I-RFT preserves physical consistency while enabling generalization across diverse motion primitives. Experimental results demonstrate that I-RFT accurately estimates terrain properties across multiple gait trajectories and toe shapes. Moreover, we show that the quantified uncertainty over the terrain resistance stress map could enable
robots to optimize foot design and gait trajectories for efficient information gathering. This approach establishes a new foundation for data-efficient characterization of complex granular environments and opens new avenues for locomotion strategies that actively adapt gait for autonomous terrain exploration.
Read the full abstract ▾
All authors Shipeng Liu, Feng Xue, Yifeng Zhang, Tarunika Ponnusamy, Feifei Qian
Robotics
A Robust Antenna Provides Tactile Feedback in a Multi-legged Robot
Zhaochen J. Xu, Juntao He, Delfin Aydan et al. (+5 more)
Multi-legged elongate
robots hold promise for maneuvering through complex environments. Prior work has demonstrated that reliable locomotion can be achieved using open-loop body undulation and foot placement on rugose terrain. However, robust
navigation through confined spaces remains challenging when body-environment contact is extensive and terrain rheology varies rapidly. To address this challenge, we develop a pair of tactile
antennae for multi-legged
robots that enable real-time sensing of surrounding geometry, modeling the morphology and function of biological centipede
antennae. Each
antenna features gradient compliance, with a stiff base and soft tip, allowing repeated deformation and elastic recovery. Robophysical experiments reveal a relationship between continuous
antenna curvature and contact force, leading to a simplified mapping from
antenna deformation to inferred discrete collision states. We incorporate this mapping into a controller that selects among a set of locomotor maneuvers based on the inferred collision state. Experiments in obstacle-rich and confined environments demonstrate that tactile feedback enables reliable steering and allows the
robot to recover from near-stuck conditions without requiring global environmental information or real-time
vision. These results highlight how mechanically tuned tactile appendages can simplify sensing and enhance autonomy in elongate multi-legged
robots operating in constrained spaces.
Read the full abstract ▾
All authors Zhaochen J. Xu, Juntao He, Delfin Aydan, Malaika Taylor, Tianyu Wang, Jianfeng Lin, Wesley Dyer, Daniel I. Goldman
Robotics
Stability-Aligned Residual Adaptation for Rapid Recovery from Dynamics Shifts
Nethmi Jayasinghe, Diana Gontero, Francesco Migliarba et al. (+4 more)
Robotic systems deployed beyond controlled environments encounter abrupt and unobserved dynamics shifts that induce substantial transient performance loss, even when the nominal closed loop remains locally stabilizing. We formulate rapid inference-time recovery for physical AI as a constrained residual learning problem, in which an online correction added to a frozen reinforcement learning policy is regulated to minimize recovery time while remaining inside the robustness margin of the nominal controller. We specifically focus upon the regulation of residual's corrective authority relative to an already competent frozen policy. The learned residual produces bounded action-space corrections online, and a Stability
Alignment Gate regulates this corrective authority through magnitude constraints, directional coherence, performance-conditioned regulation, and adaptive gain modulation. The method requires no environment resets, policy retraining, explicit system identification, or privileged shift information. Across mid-episode changes in actuation, mass, and contact conditions, it reduces recovery time relative to frozen SAC by up to 87% on Go1, 48% on Cassie, 30% on H1, and 20% on Scout in simulation, while preserving near-nominal steady-state performance. The framework is also characterized on two physical
robots, a Trossen ViperX manipulator and an AgileX Scout Mini, cutting recovery time by 19.5% on hardware without modifying onboard controller or accessing shift information.
Read the full abstract ▾
All authors Nethmi Jayasinghe, Diana Gontero, Francesco Migliarba, Spencer T. Brown, Vinod K. Sangwan, Mark C. Hersam, Amit Ranjan Trivedi
Robotics · Cross-listed from cs.HC
Directing the Robot: Scaffolding Creative Human-AI-Robot Interaction
Jordan Aiko Deja, Isidro Butaslac, Nicko Reginio Caluya et al. (+1 more)
Robots are moving beyond industrial settings into creative, educational, and public environments where interaction is open-ended and improvisational. Yet much of human-AI-
robot interaction remains framed around performance and efficiency, positioning humans as supervisors rather than collaborators. We propose a re-framing of AI interaction with
robots as scaffolding: infrastructure that enables humans to shape robotic behaviour over time while remaining meaningfully in control. Through scenarios from creative practice, learning-by-teaching, and
embodied interaction, we illustrate how humans can act as executive directors, defining intent and steering revisions, while AI mediates between human expression and robotic execution. We outline design and evaluation implications that foreground creativity, agency, and flow. Finally, we discuss open challenges in social, scalable, and mission-critical contexts. We invite the community to rethink interacting with
Robots and AI not as autonomy, but as sustained support for human creativity.
Read the full abstract ▾
All authors Jordan Aiko Deja, Isidro Butaslac, Nicko Reginio Caluya, Maheshya Weerasinghe
Robotics
AeroPlace-Flow: Language-Grounded Object Placement for Aerial Manipulators via Visual Foresight and Object Flow
Sarthak Mishra, Rishabh Dev Yadav, Naveen Nair et al. (+2 more)
Precise object placement remains underexplored in aerial manipulation, where most systems rely on predefined target coordinates and focus primarily on grasping and control. Specifying exact placement poses, however, is cumbersome in real-world settings, where users naturally communicate goals through
language. In this work, we present AeroPlace-Flow, a training-free framework for
language-grounded aerial object placement that unifies visual foresight with explicit 3D
geometric reasoning and object flow. Given RGB-D observations of the object and the placement scene, along with a natural
language instruction, AeroPlace-Flow first synthesizes a task-complete goal image using image editing models. The imagined configuration is then grounded into metric 3D space through depth
alignment and object-centric
reasoning, enabling the inference of a collision-aware object flow that transports the grasped object to a
language and contact-consistent placement configuration. The resulting motion is executed via standard trajectory tracking for an aerial manipulator. AeroPlace-Flow produces executable placement targets without requiring predefined poses or task-specific training. We validate our approach through extensive simulation and real-world experiments, demonstrating reliable
language-conditioned placement across diverse aerial scenarios with an average success rate of 75% on hardware.
Read the full abstract ▾
All authors Sarthak Mishra, Rishabh Dev Yadav, Naveen Nair, Wei Pan, Spandan Roy
Robotics
C$^2$-Explorer: Contiguity-Driven Task Allocation with Connectivity-Aware Task Representation for Decentralized Multi-UAV Exploration
Xinlu Yan, Mingjie Zhang, Yuhao Fang et al. (+5 more)
Efficient multi-
UAV exploration under limited communication is severely bottlenecked by inadequate task representation and allocation. Previous task representations either impose heavy communication requirements for coordination or lack the flexibility to handle complex environments, often leading to inefficient traversal. Furthermore, short-horizon allocation strategies neglect spatiotemporal contiguity, causing non-contiguous assignments and frequent cross-region detours. To address this, we propose C$^2$-Explorer, a decentralized framework that constructs a connectivity graph to decompose disconnected unknown components into independent task units. We then introduce a contiguity-driven allocation formulation with a graph-based neighborhood penalty to discourage non-adjacent assignments, promoting more contiguous task sequences over time. Extensive simulation experiments show that C$^2$-Explorer consistently outperforms state-of-the-art (SOTA) baselines, reducing average exploration time by 43.1% and path length by 33.3%. Real-world flights further demonstrate the system's feasibility. The code will be released at https://github.com/Robotics-
STAR-Lab/C2-Explorer
Read the full abstract ▾
All authors Xinlu Yan, Mingjie Zhang, Yuhao Fang, Yanke Sun, Jun Ma, Youmin Gong, Boyu Zhou, Jie Mei
Robotics
RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation
Zhanqi Xiao, Ruiping Wang and Xilin Chen
Understanding spatial affordances – comprising the contact regions of object interaction and the corresponding contact poses – is essential for
robots to effectively manipulate objects and accomplish diverse tasks. However, existing spatial affordance prediction methods mainly focus on locating the contact regions while delegating the pose to independent pose estimation approaches, which can lead to task failures due to inconsistencies between predicted contact regions and candidate poses. In this work, we propose RoboPCA, a pose-centered affordance prediction framework that jointly predicts task-appropriate contact regions and poses conditioned on instructions. To enable scalable data collection for pose-centered affordance learning, we devise Human2Afford, a data curation pipeline that automatically recovers scene-level 3D information and infers pose-centered affordance annotations from human demonstrations. With Human2Afford, scene depth and the interaction object's mask are extracted to provide 3D context and object localization, while pose-centered affordance annotations are obtained by tracking object points within the contact region and analyzing hand-object interaction patterns to establish a mapping from the 3D hand mesh to the
robot end-effector orientation. By integrating geometry-appearance cues through an RGB-D encoder and incorporating mask-enhanced features to emphasize task-relevant object regions into the
diffusion-based framework, RoboPCA outperforms baseline methods on image
datasets, simulation, and real
robots, and exhibits strong generalization across tasks and categories.
Read the full abstract ▾
Robotics
UniUncer: Unified Dynamic Static Uncertainty for End to End Driving
Yu Gao, Jijun Wang, Zongzheng Zhang et al. (+7 more)
End-to-end (E2E) driving has become a cornerstone of both industry deployment and academic research, offering a single learnable pipeline that maps multi-sensor inputs to actions while avoiding hand-engineered modules. However, the reliability of such pipelines strongly depends on how well they handle uncertainty: sensors are noisy, semantics can be ambiguous, and interaction with other road users is inherently
stochastic. Uncertainty also appears in multiple forms: classification vs. localization, and, crucially, in both static map elements and dynamic
agents. Existing E2E approaches model only static-map uncertainty, leaving planning vulnerable to overconfident and unreliable inputs. We present UniUncer, the first lightweight, unified uncertainty framework that jointly estimates and uses uncertainty for both static and dynamic scene elements inside an E2E planner. Concretely: (1) we convert deterministic heads to probabilistic Laplace regressors that output per-vertex location and scale for vectorized static and dynamic entities; (2) we introduce an uncertainty-fusion module that encodes these parameters and injects them into object/map queries to form uncertainty-aware queries; and (3) we design an uncertainty-aware gate that adaptively modulates reliance on historical inputs (ego status or temporal perception queries) based on current uncertainty levels. The design adds minimal overhead and drops throughput by only $\sim$0.5 FPS while remaining plug-and-play for common E2E backbones. On nuScenes (open-loop), UniUncer reduces average L2 trajectory error by 7%. On NavsimV2 (pseudo closed-loop), it improves overall EPDMS by 10.8% and notable stage two gains in challenging, interaction-heavy scenes. Ablations confirm that dynamic-
agent uncertainty and the uncertainty-aware gate are both necessary.
Read the full abstract ▾
All authors Yu Gao, Jijun Wang, Zongzheng Zhang, Anqing Jiang, Yiru Wang, Yuwen Heng, Shuo Wang, Hao Sun, Zhangfeng Hu, Hao Zhao
Robotics
Low-Cost Teleoperation Extension for Mobile Manipulators
Danil Belov, Artem Erkhov, Yaroslav Savotin et al. (+3 more)
Teleoperation of mobile bimanual manipulators requires simultaneous control of high-dimensional systems, often necessitating expensive specialized equipment. We present an open-source teleoperation framework that enables intuitive whole body control using readily available commodity hardware. Our system combines smartphone-based head tracking for camera control, leader arms for bilateral manipulation, and foot pedals for hands-free base
navigation. Using a standard smartphone with IMU and display, we eliminate the need for costly VR helmets while maintaining immersive visual feedback. The modular architecture integrates seamlessly with the XLeRobot framework, but can be easily adapted to other types of mobile manipulators. We validate our approach through user studies that demonstrate improved task performance and reduced cognitive load compared to keyboard-based control.
Read the full abstract ▾
All authors Danil Belov, Artem Erkhov, Yaroslav Savotin, Tatiana Podladchikova, Pavel Osinenko, Dzmitry Tsetserukou
Robotics
DAISS: Phase-Aware Imitation Learning for Dual-Arm Robotic Ultrasound-Guided Interventions
Feng Li, Pei Liu, Shiting Wang et al. (+4 more)
Imitation learning has shown strong potential for automating complex robotic manipulation. In medical robotics, ultrasound-guided needle insertion demands precise bimanual coordination, as clinicians must simultaneously manipulate an ultrasound probe to maintain an optimal acoustic view while steering an interventional needle. Automating this asymmetric workflow – and reliably transferring expert strategies to
robots – remains highly challenging. In this paper, we present the Dual-Arm Interventional
Surgical System (DAISS), a teleoperated platform that collects high-fidelity dual-arm demonstrations and learns a phase-aware imitation policy for ultrasound-guided interventions. To avoid constraining the operator's natural behavior, DAISS uses a flexible NDI-based leader interface for teleoperating two coordinated follower arms. To support robust execution under real-time ultrasound feedback, we develop a lightweight, data-efficient imitation policy. Specifically, the policy incorporates a phase-aware architecture and a dynamic mask loss tailored to asymmetric bimanual control. Conditioned on a planned trajectory, the network fuses real-time ultrasound with external visual observations to generate smooth, coordinated dual-arm motions. Experimental results show that DAISS can learn personalized expert strategies from limited demonstrations. Overall, these findings highlight the promise of phase-aware imitation-learning-driven dual-arm
robots for improving precision and reducing cognitive workload in image-guided interventions.
Read the full abstract ▾
All authors Feng Li, Pei Liu, Shiting Wang, Ning Wang, Zhongliang Jiang, Nassir Navab, Yuan Bi
Robotics
Multi-Agent Off-World Exploration for Sparse Evidence Discovery via Gaussian Belief Mapping and Dual-Domain Coverage
Zhuoran Qiao, Tianxin Hu, Thien-Minh Nguyen et al. (+1 more)
Off-world multi-
robot exploration is challenged by sparse targets, limited sensing, hazardous terrain, and restricted communication. Many scientifically valuable clues are visually ambiguous and often require close-range observations, making efficient and safe informative path planning essential. Existing methods often rely on predefined areas of interest (AOIs), which may be incomplete or biased, and typically handle terrain risk only through soft penalties, which are insufficient for avoiding non-recoverable regions. To address these issues, we propose a multi-
agent informative path planning framework for sparse evidence discovery based on Gaussian belief mapping and dual-domain coverage. The method maintains Gaussian-process-based interest and risk beliefs and combines them with trajectory-intent representations to support coordinated sequential decision-making among multiple
agents. It further prioritizes search inside the AOI while preserving limited exploration outside it, thereby improving robustness to AOI bias. In addition, the risk-aware design helps
agents balance information gain and operational safety in hazardous environments. Experimental results in simulated lunar environments show that the proposed method consistently outperforms sampling-based and greedy baselines under different budgets and communication ranges. In particular, it achieves lower final uncertainty in risk-aware settings and remains robust under limited communication, demonstrating its effectiveness for cooperative off-world robotic exploration.
Read the full abstract ▾
All authors Zhuoran Qiao, Tianxin Hu, Thien-Minh Nguyen, Shenghai Yuan
Robotics
AtomicVLA: Unlocking the Potential of Atomic Skill Learning in Robots
Likui Zhang, Tao Tang, Zhihao Zhan et al. (+9 more)
Recent advances in Visual-
Language-Action (VLA) models have shown promising potential for robotic manipulation tasks. However, real-world robotic tasks often involve long-horizon, multi-step problem-solving and require generalization for continual skill acquisition, extending beyond single actions or skills. These challenges present significant barriers for existing VLA models, which use monolithic action decoders trained on aggregated data, resulting in poor scalability. To address these challenges, we propose AtomicVLA, a unified planning-and-execution framework that jointly generates task-level plans, atomic skill abstractions, and fine-grained actions. AtomicVLA constructs a scalable atomic skill library through a Skill-Guided Mixture-of-Experts (SG-MoE), where each expert specializes in mastering generic yet precise atomic skills. Furthermore, we introduce a flexible routing encoder that automatically assigns dedicated atomic experts to new skills, enabling continual learning. We validate our approach through extensive experiments. In simulation, AtomicVLA outperforms $\pi_{0}$ by 2.4% on LIBERO, 10% on LIBERO-LONG, and outperforms $\pi_{0}$ and $\pi_{0.5}$ by 0.22 and 0.25 in average task length on CALVIN. Additionally, our AtomicVLA consistently surpasses baselines by 18.3% and 21% in real-world long-horizon tasks and continual learning. These results highlight the effectiveness of atomic skill abstraction and dynamic expert composition for long-horizon and lifelong robotic tasks. The project page is \href{https://zhanglk9.github.io/atomicvla-web/}{here}.
Read the full abstract ▾
All authors Likui Zhang, Tao Tang, Zhihao Zhan, Xiuwei Chen, Zisheng Chen, Jianhua Han, Jiangtong Zhu, Pei Xu, Hang Xu, Hefeng Wu, Liang Lin, Xiaodan Liang
Robotics
TempoFit: Plug-and-Play Layer-Wise Temporal KV Memory for Long-Horizon Vision-Language-Action Manipulation
Jun Sun, Boyu Yang, Jiahao Zhang et al. (+7 more)
Pretrained
Vision-
Language-Action (VLA) policies have achieved strong single-step manipulation, but their inference remains largely memoryless, which is brittle in non-Markovian long-horizon settings with occlusion, state aliasing, and subtle post-action changes. Prior approaches inject history either by stacking frames, which scales visual
tokens and latency while adding near-duplicate pixels, or by learning additional temporal interfaces that require (re-)training and may break the original single-frame inference graph. We present TempoFit, a training-free temporal retrofit that upgrades frozen VLAs through state-level
memory. Our key insight is that prefix
attention K/V already form a model-native, content-addressable runtime state; reusing them across timesteps introduces history without new
tokens or trainable modules. TempoFit stores layer-wise FIFO prefix K/V at selected intermediate layers, performs parameter-free K-to-K
retrieval with Frame-Gap Temporal Bias (FGTB), a fixed recency bias inspired by positional biases in NLP, to keep decisions present-dominant, and injects the retrieved context via pre-
attention residual loading with norm-preserving rescaling to avoid distribution shift under frozen weights. On LIBERO-LONG, TempoFit improves strong pretrained backbones by up to +4.0% average success rate while maintaining near-real-time latency, and it transfers consistently to CALVIN and real-
robot long-horizon tasks.
Read the full abstract ▾
All authors Jun Sun, Boyu Yang, Jiahao Zhang, Ning Ma, Chencheng Wu, Siqing Zhang, Yiou Huang, Qiufeng Wang, Shan Liang, Yaran Chen
Robotics
PanoDP: Learning Collision-Free Navigation with Panoramic Depth and Differentiable Physics
Hao Zhong, Pei Chi, Jiang Zhao et al. (+4 more)
Autonomous collision-free
navigation in cluttered environments requires safe decision-making under partial observability with both static structure and dynamic obstacles. We present PanoDP, a communication-free learning framework that combines four-view panoramic depth perception with differentiable-physics-based training signals. PanoDP encodes panoramic depth using a lightweight CNN and optimizes policies with dense differentiable collision and motion-feasibility terms, improving training stability beyond sparse terminal collisions. We evaluate PanoDP on a controlled ring-to-center
benchmark with systematic sweeps over
agent count, obstacle density/layout, and dynamic behaviors, and further test out-of-distribution generalization in an external simulator (e.g., AirSim). Across settings, PanoDP increases collision-free and completion rates over single-view and non-physics-guided baselines under matched training budgets, and ablations (view masking, rotation augmentation) confirm the policy leverages 360-degree information. Code will be open source upon acceptance.
Read the full abstract ▾
All authors Hao Zhong, Pei Chi, Jiang Zhao, Shenghai Yuan, Xuyang Gao, Thien-Minh Nguyen, Lihua Xie
Robotics
Exoskeleton Control through Learning to Reduce Biological Joint Moments in Simulations
Zihang You and Xianlian Zhou
Data-driven joint-moment predictors offer a scalable alternative to laboratory-based inverse-dynamics pipelines for biomechanics estimation and exoskeleton control. Meanwhile, physics-based reinforcement learning (RL) enables simulation-trained controllers to learn dynamics-aware assistance strategies without extensive human experimentation. However, quantitative verification of simulation-trained exoskeleton torque predictors, and their impact on human joint power injection, remains limited. This paper presents (1) an RL framework to learn exoskeleton assistance policies that reduce biological joint moments, and (2) a validation pipeline that verifies the trained control networks using an open-source gait
dataset through inference and comparison with biological joint moments. Simulation-trained multilayer perceptron (MLP) controllers are developed for level-ground and ramp walking, mapping short-horizon histories of bilateral hip and knee kinematics to normalized assistance torques. Results show that predicted assistance preserves task-intensity trends across speeds and inclines. Agreement is particularly strong at the hip, with cross-correlation coefficients reaching 0.94 at 1.8 m/s and 0.98 during 5{\deg} decline walking, demonstrating near-matched temporal structure. Discrepancies increase at higher speeds and steeper inclines, especially at the knee, and are more pronounced in joint power comparisons. Delay tuning biases assistance toward greater positive power injection; modest timing shifts increase positive power and improve agreement in specific gait intervals. Together, these results establish a quantitative validation framework for simulation-trained exoskeleton controllers, demonstrate strong sim-to-data consistency at the torque level, and highlight both the promise and the remaining challenges for sim-to-real transfer.
Read the full abstract ▾
Robotics
GeoLoco: Leveraging 3D Geometric Priors from Visual Foundation Model for Robust RGB-Only Humanoid Locomotion
Yufei Liu, Xieyuanli Chen, Hainan Pan et al. (+5 more)
The prevailing paradigm of perceptive
humanoid locomotion relies heavily on active depth sensors. However, this depth-centric approach fundamentally discards the rich semantic and dense appearance cues of the visual world, severing low-level control from the high-level
reasoning essential for general
embodied intelligence. While
monocular RGB offers a ubiquitous, information-dense alternative, end-to-end reinforcement learning from raw 2D pixels suffers from extreme sample inefficiency and catastrophic sim-to-real collapse due to the inherent loss of
geometric scale. To break this deadlock, we propose GeoLoco, a purely RGB-driven locomotion framework that conceptualizes
monocular images as high-dimensional 3D latent representations by harnessing the powerful
geometric priors of a frozen, scale-aware Visual Foundation Model (VFM). Rather than naive feature concatenation, we design a proprioceptive-query multi-head cross-
attention mechanism that dynamically attends to task-critical topological features conditioned on the
robot's real-time gait phase. Crucially, to prevent the policy from overfitting to superficial textures, we introduce a dual-head auxiliary learning scheme. This explicit regularization forces the high-dimensional latent space to strictly align with the physical terrain geometry, ensuring robust zero-shot sim-to-real transfer. Trained exclusively in simulation, GeoLoco achieves robust zero-shot transfer to the Unitree G1
humanoid and successfully negotiates challenging terrains.
Read the full abstract ▾
All authors Yufei Liu, Xieyuanli Chen, Hainan Pan, Chenghao Shi, Yanjie Chen, Kaihong Huang, Zhiwen Zeng, Huimin Lu
Robotics
SMAT: Staged Multi-Agent Training for Co-Adaptive Exoskeleton Control
Yifei Yuan, Ghaith Androwis and Xianlian Zhou
Effective exoskeleton assistance requires co-adaptation: as the device alters joint dynamics, the user reorganizes neuromuscular coordination, creating a non-stationary learning problem. Most learning-based approaches do not explicitly account for the sequential nature of human motor adaptation, leading to training instability and poorly timed assistance. We propose Staged Multi-
Agent Training (SMAT), a four-stage curriculum designed to mirror how users naturally acclimate to a wearable device. In SMAT, a musculoskeletal human actor and a bilateral hip exoskeleton actor are trained progressively: the human first learns unassisted gait, then adapts to the added device mass; the exoskeleton subsequently learns a positive assistance pattern against a stabilized human policy, and finally both
agents co-adapt with full torque capacity and bidirectional feedback. We implement SMAT in the MyoAssist simulation environment using a 26-muscle lower-limb model and an attached hip exoskeleton. Our musculoskeletal simulations demonstrate that the learned exoskeleton control policy produces an average 10.1% reduction in hip muscle activation relative to the no-assist condition. We validated the learned controller in an offline setting using open-source gait data, then deployed it to a physical hip exoskeleton for treadmill experiments with five subjects. The resulting policy delivers consistent assistance and predominantly positive mechanical power without the need for any explicitly imposed timing shift (mean positive power: 13.6 W at 6 Nm RMS torque to 23.8 W at 9.3 Nm RMS torque, with minimal negative power) consistently across all subjects without subject-specific retraining.
Read the full abstract ▾
Robotics
Model-Based and Neural-Aided Approaches for Dog Dead Reckoning
Gal Versano, Itai Savin and Itzik Klein
Modern canine applications span medical and service roles, while robotic legged dogs serve as autonomous platforms for high-risk industrial inspection, disaster response, and search and rescue operations. For both, accurate positioning remains a significant challenge due to the cumulative drift inherent in inertial sensing. To bridge this gap, we propose three
algorithms for accurate positioning using only inertial sensors, collectively referred to as dog dead reckoning (DDR). To evaluate our approaches, we designed DogMotion, a wearable unit for canine data recording. Using DogMotion, we recorded a
dataset of 13 minutes. Additionally, we utilized a robotic legged dog
dataset with a duration of 116 minutes. Across the two distinct
datasets we demonstrate that our neural-aided methods consistently outperform model-based approaches, achieving an absolute distance error of less than 10%. Consequently, we provide a lightweight and low-cost positioning solution for both biological and legged robotic dogs. To support reproducibility, our codebase and associated
datasets have been made publicly available.
Read the full abstract ▾
Robotics
FeasibleCap: Real-Time Embodiment Constraint Guidance for In-the-Wild Robot Demonstration Collection
Zi Yin, Fanhong Li, Yun Gui et al. (+1 more)
Gripper-in-hand data collection decouples demonstration acquisition from
robot hardware, but whether a trajectory is executable on the target
robot remains unknown until a separate replay-and-validate stage. Failed demonstrations therefore inflate the effective cost per usable trajectory through repeated collection, diagnosis, and validation. Existing collection-time feedback systems mitigate this issue but rely on head-worn AR/VR displays,
robot-in-the-loop hardware, or learned dynamics models; real-time executability feedback has not yet been integrated into the gripper-in-hand data collection paradigm. We present FeasibleCap, a gripper-in-hand data collection system that brings real-time executability guidance into
robot-free capture. At each frame, FeasibleCap checks reachability, joint-rate limits, and collisions against a target
robot model and closes the loop through on-device visual overlays and haptic cues, allowing demonstrators to correct motions during collection without learned models, headsets, or
robot hardware. On pick-and-place and tossing tasks, FeasibleCap improves replay success and reduces the fraction of infeasible frames, with the largest gains on tossing. Simulation experiments further indicate that enforcing executability constraints during collection does not sacrifice cross-embodiment transfer across
robot platforms. Hardware designs and software are available at https://github.com/aod321/FeasibleCap.
Read the full abstract ▾
All authors Zi Yin, Fanhong Li, Yun Gui, Jia Liu
Robotics
Approximate Imitation Learning for Event-based Quadrotor Flight in Cluttered Environments
Nico Messikommer, Jiaxu Xing, Leonard Bauersfeld et al. (+3 more)
Event cameras offer high temporal resolution and low latency, making them ideal sensors for high-speed robotic applications where conventional cameras suffer from motion blur. However, their widespread adoption in
robot learning is severely bottlenecked by the computational cost of simulating high-frequency event data during online training. In this work, we present Approximate Imitation Learning, a novel framework that fundamentally resolves this bottleneck, reducing policy training time for complex, agile drone flight from 52.44 hours to just 1.86 hours - a 28x computational speedup. Our key insight is to separate representation learning from policy search. We first leverage a large-scale offline
dataset to learn a task-specific representation space. Subsequently, the policy is fine-tuned through online interactions that rely solely on lightweight state information, completely eliminating the need to render events during the active policy search phase. This training paradigm drastically reduces development overhead and enables event-based control policies to scale to complex environments. Furthermore, our approach eliminates the reliance on standard cameras or intermediate representations during deployment, mapping events directly to control commands. In simulation, our method matches or exceeds the performance of standard imitation learning baselines that require full online event rendering. Finally, we successfully validate the framework in the real world, demonstrating that a policy trained via this ultra-efficient paradigm enables a quadrotor to fly through highly cluttered environments at remarkable speeds of up to 9.8 m/s.
Read the full abstract ▾
All authors Nico Messikommer, Jiaxu Xing, Leonard Bauersfeld, Marco Cannici, Elie Aljalbout, Davide Scaramuzza
Robotics · Cross-listed from cs.CV
ReconDrive: Fast Feed-Forward 4D Gaussian Splatting for Autonomous Driving Scene Reconstruction
Haibao Yu, Kuntao Xiao, Jiahang Wang et al. (+7 more)
High-fidelity visual reconstruction and novel-view synthesis are essential for realistic closed-loop evaluation in autonomous driving. While 4D Gaussian Splatting (4DGS) offers a promising balance of accuracy and efficiency, existing per-scene optimization methods require costly iterative refinement, rendering them unscalable for extensive urban environments. Conversely, current feed-forward approaches often suffer from degraded photometric quality. To address these limitations, we propose ReconDrive, a feed-forward framework that leverages and extends the 3D foundation model VGGT for rapid, high-fidelity 4DGS generation. Our architecture introduces two core adaptations to tailor the foundation model to dynamic driving scenes: (1) Hybrid Gaussian Prediction Heads, which decouple the regression of spatial coordinates and appearance attributes to overcome the photometric deficiencies inherent in generalized foundation features; and (2) a Static-Dynamic 4D Composition strategy that explicitly captures temporal motion via velocity modeling to represent complex dynamic environments. Benchmarked on nuScenes, ReconDrive significantly outperforms existing feed-forward baselines in reconstruction, novel-view synthesis, and 3D perception. It achieves performance competitive with per-scene optimization while being orders of magnitude faster, providing a scalable and practical solution for realistic driving simulation.
Read the full abstract ▾
All authors Haibao Yu, Kuntao Xiao, Jiahang Wang, Ruiyang Hao, Yuxin Huang, Guoran Hu, Haifang Qin, Bowen Jing, Yuntian Bo, Ping Luo
Robotics
ACCURATE: Arbitrary-shaped Continuum Reconstruction Under Robust Adaptive Two-view Estimation
Yaozhi Zhang, Shun Yu, Yugang Zhang et al. (+1 more)
Accurate reconstruction of arbitrary-shaped long slender continuum bodies, such as guidewires, catheters and other soft continuum manipulators, is essential for accurate mechanical simulation. However, existing image-based reconstruction approaches often suffer from limited accuracy because they often underutilize camera geometry, or lack generality as they rely on rigid
geometric assumptions that may fail for continuum
robots with complex and highly deformable shapes. To address these limitations, we propose ACCURATE, a 3D reconstruction framework integrating an image
segmentation neural network with a geometry-constrained
topology traversal and dynamic programming
algorithm that enforces global biplanar
geometric consistency, minimizes the cumulative point-to-epipolar-line distance, and remains robust to occlusions and epipolar ambiguities cases caused by noise and discretization. Our method achieves high reconstruction accuracy on both simulated and real phantom
datasets acquired using a clinical X-ray C-arm system, with mean absolute errors below 1.0 mm.
Read the full abstract ▾
All authors Yaozhi Zhang, Shun Yu, Yugang Zhang, Yang Liu
Robotics
ICLR: In-Context Imitation Learning with Visual Reasoning
Toan Nguyen, Weiduo Yuan, Songlin Wei et al. (+3 more)
In-context imitation learning enables
robots to adapt to new tasks from a small number of demonstrations without additional training. However, existing approaches typically condition only on state-action trajectories and lack explicit representations of task intent. This limitation hinders performance in complex and ambiguous task settings where the same actions may be consistent with different objectives. To address this, we present In-Context Imitation Learning with Visual
Reasoning (ICLR), a novel framework that augments demonstration
prompts with structured visual
reasoning traces representing anticipated future
robot trajectories in image space. ICLR also jointly learns to generate
reasoning traces and low-level actions within a unified autoregressive
transformer, enabling the model to mimic not only action prediction but also the
reasoning process that leads to those actions. We extensively evaluate ICLR in both simulation and real-world manipulation tasks and demonstrate consistent improvements in success rates and generalization to unseen tasks and novel object configurations compared to other in-context imitation learning methods. These results suggest that incorporating
embodied visual
reasoning represents a promising direction for enhancing the robustness and generalization of robotic in-context learning systems.
Read the full abstract ▾
All authors Toan Nguyen, Weiduo Yuan, Songlin Wei, Hui Li, Daniel Seita, Yue Wang
Robotics
InterReal: A Unified Physics-Based Imitation Framework for Learning Human-Object Interaction Skills
Dayang Liang, Yuhang Lin, Xinzhe Liu et al. (+3 more)
Interaction is one of the core abilities of
humanoid robots. However, most existing frameworks focus on non-interactive whole-body control, which limits their practical applicability. In this work, we develop InterReal, a unified physics-based imitation learning framework for Real-world human-object Interaction (HOI) control. InterReal enables
humanoid robots to track HOI reference motions, facilitating the learning of fine-grained interactive skills and their deployment in real-world settings. Within this framework, we first introduce a HOI motion data augmentation scheme with hand-object contact constraints, and utilize the augmented motions to improve policy stability under object perturbations. Second, we propose an automatic reward learner to address the challenge of large-scale reward shaping. A meta-policy guided by critical tracking error metrics explores and allocates reward signals to the low-level reinforcement learning objective, which enables more effective learning of interactive policies. Experiments on HOI tasks of box-picking and box-pushing demonstrate that InterReal achieves the best tracking accuracy and the highest task success rate compared to recent baselines. Furthermore, we validate the framework on the real-world
robot Unitree G1, which demonstrates its practical effectiveness and robustness beyond simulation.
Read the full abstract ▾
All authors Dayang Liang, Yuhang Lin, Xinzhe Liu, Jiyuan Shi, Yunlong Liu, Chenjia Bai
Robotics · Cross-listed from eess.SY
Inverse-dynamics observer design for a linear single-track vehicle model with distributed tire dynamics
Luigi Romano, Ole Morten Aamo, Jan Åslund et al. (+1 more)
Accurate estimation of the vehicle's sideslip angle and tire forces is essential for enhancing safety and handling performances in unknown driving scenarios. To this end, the present paper proposes an innovative observer that combines a linear single-track model with a distributed representation of the tires and information collected from standard sensors. In particular, by adopting a comprehensive representation of the tires in terms of
hyperbolic partial differential equations (PDEs), the proposed estimation strategy exploits dynamical inversion to reconstruct the lumped and distributed vehicle states solely from yaw rate and lateral acceleration measurements. Simulation results demonstrate the effectiveness of the observer in estimating the sideslip angle and tire forces even in the presence of noise and model uncertainties.
Read the full abstract ▾
All authors Luigi Romano, Ole Morten Aamo, Jan Åslund, Erik Frisk