Robotics
One Interface, Many Robots: Unified Real-Time Low-Level Motion Planning for Collaborative Arms
Yue Feng, Weicheng Huang and I-Ming Chen
This paper proposes a common interface for real-time low-level motion planning of collaborative robotic arms, aimed at enabling broader applicability and improved portability across heterogeneous hardware platforms. In previous work, we introduced WinGs Operating Studio (WOS), a middleware solution that abstracts diverse robotic components into uniform software resources and provides a broad suite of
language-agnostic APIs. This paper specifically focuses on its minimal yet flexible interface for real-time end-effector trajectory control. By employing an n-degree
polynomial interpolator in conjunction with a quadratic programming solver, the proposed method generates smooth, continuously differentiable trajectories with precise position, velocity, and acceleration profiles. We validate our approach in three distinct scenarios. First, in an offline demonstration, a collaborative arm accurately draws various
geometric shapes on paper. Second, in an interruptible, low-frequency re-planning setting, a robotic manipulator grasps a dynamic object placed on a moving mobile
robot. Finally, we conducted a teleoperation experiment in which one robotic arm controlled another to perform a series of dexterous manipulations, confirming the proposed method's reliability, versatility, and ease of use.
Read the full abstract ▾
Robotics
Morphology-Conditioned World Model for Cross-Embodiment Quadrupedal Locomotion
Mohamad H. Danesh, Chenhao Li, Amin Abyaneh et al. (+5 more)
World models promise a paradigm shift in robotics, where an
agent learns the physics of its environment once and then acquires behaviors efficiently. Yet the learned dynamics models at their core are typically morphology locked. In legged locomotion, a dynamics model trained on an ANYmal-D quadruped fails on a Unitree Go1 because it overfits to one
robot's embodiment rather than capturing the locomotion dynamics shared across
robots, so even a small change in actuator dynamics or limb length forces retraining from scratch. However, if we formalize a
robot's unique physical traits into a morphology specification, a controller for a family of
robots can utilize this blueprint in two ways. It can feed the specification to a model-free policy, or it can feed the specification to a learned dynamics model and extract the policy in imagination. We argue for the second route and introduce the Quadrupedal World Model (QWM), which conditions a single generative dynamics model on scale-invariant physical features and trains policies entirely inside it, through a physical morphology encoder, an adaptive reward normalizer, and morphology conditioning in the latent dynamics. Holding the morphology information identical, a model-free policy matches QWM on the training cohort but degrades on unseen morphologies, while QWM transfers zero-shot with no fine-tuning, adaptation, or warm-up in such cases. To our knowledge, this is the first world model to demonstrate zero-shot cross-embodiment transfer within the quadrupedal family.
Read the full abstract ▾
All authors Mohamad H. Danesh, Chenhao Li, Amin Abyaneh, Anas Houssaini, Kirsty Ellis, Glen Berseth, Marco Hutter, Hsiu-Chin Lin
Robotics
Task-Aware Bimanual Affordance Prediction via VLM-Guided Semantic-Geometric Reasoning
Fabian Hahne, Vignesh Prasad, Georgia Chalvatzaki et al. (+2 more)
Bimanual manipulation requires
reasoning about where to interact with an object and which arm should perform each action, a joint affordance localization and arm allocation problem that geometry-only planners cannot resolve without semantic understanding of task intent. Existing approaches either treat affordance prediction as coarse part
segmentation or rely on
geometric heuristics for arm assignment, failing to jointly reason about task-relevant contact regions and arm allocation. We reframe bimanual manipulation as a joint affordance localization and arm allocation problem and propose a hierarchical framework for task-aware bimanual affordance prediction that leverages a
Vision-
Language Model (VLM) to generalize across object categories and task descriptions without requiring category-specific training. Our approach fuses multi-view RGB-D observations into a consistent 3D scene representation and generates global 6-DoF grasp candidates, which are then spatially and semantically filtered by querying the VLM for task-relevant affordance regions on each object, as well as for arm allocation to the individual objects, thereby ensuring
geometric validity while respecting task semantics. We evaluate our method on a dual-arm platform across nine real-world manipulation tasks spanning four categories: parallel manipulation, coordinated stabilization, tool use, and human handover. Our approach achieves consistently higher task success rates than
geometric and semantic baselines for task-oriented grasping, demonstrating that explicit semantic
reasoning over affordances and arm allocation helps enable reliable bimanual manipulation in unstructured environments.
Read the full abstract ▾
All authors Fabian Hahne, Vignesh Prasad, Georgia Chalvatzaki, Jan Peters, Alap Kshirsagar
Robotics · Cross-listed from cs.CV
LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving
Hao Shao, Letian Wang, Yang Zhou et al. (+5 more)
Recent years have seen remarkable progress in autonomous driving, yet generalization to long-tail and open-world scenarios remains a major bottleneck for large-scale deployment. To address this challenge, some works use LLMs and VLMs for
vision-
language understanding and
reasoning, enabling vehicles to interpret rare and safety-critical situations when generating actions. Others study generative world models to capture the spatio-temporal
evolution of driving scenes, allowing
agents to imagine possible futures before acting. Inspired by human intelligence, which unifies understanding and imagination, we explore a unified model for autonomous driving. We present LMGenDrive, the first framework that combines LLM-based
multimodal understanding with generative world models for end-to-end closed-loop driving. Given multi-view camera inputs and natural-
language instructions, LMGenDrive generates both future driving
videos and control signals. This design provides complementary benefits:
video prediction improves spatio-temporal scene modeling, while the LLM contributes strong semantic priors and instruction grounding from large-scale pretraining. We further propose a progressive three-stage training strategy, from
vision pretraining to multi-step long-horizon driving, to improve stability and performance. LMGenDrive supports both low-latency online planning and autoregressive offline
video generation. Experiments show that it significantly outperforms prior methods on challenging closed-loop
benchmarks, with clear gains in instruction following, spatio-temporal understanding, and robustness to rare scenarios. These results suggest that unifying
multimodal understanding and generation is a promising direction for more generalizable and robust
embodied decision-making systems.
Read the full abstract ▾
All authors Hao Shao, Letian Wang, Yang Zhou, Yuxuan Hu, Zhuofan Zong, Steven L. Waslander, Wei Zhan, Hongsheng Li
Robotics · Cross-listed from cs.CV
Xinmiao Xiong, Bangya Liu, Hao Wang et al. (+7 more)
Geometric Foundation Models (GFMs) have recently advanced
monocular SLAM by providing robust, calibration-free 3D priors. However, deploying these models on dense
video streams introduces significant computational redundancy. Current GFM-based SLAM systems typically rely on post hoc keyframe selection. Because of this, they must perform expensive dense
geometric decoding simply to determine whether a frame contains novel geometry, resulting in late rejection and wasted computation. To mitigate this inefficiency, we propose LeanGate, a lightweight feed-forward frame-gating network. LeanGate predicts a
geometric utility score to assess a frame's mapping value prior to the heavy GFM feature extraction and matching stages. As a predictive plug-and-play module, our approach bypasses over 90% of redundant frames. Evaluations on standard SLAM
benchmarks demonstrate that LeanGate reduces tracking FLOPs by more than 85% and achieves a 5x end-to-end throughput speedup. Furthermore, it maintains the tracking and mapping accuracy of dense baselines. Project page: https://lean-gate.github.io/
Read the full abstract ▾
All authors Xinmiao Xiong, Bangya Liu, Hao Wang, Dayou Li, Nuo Chen, Andrew Feng, Mingyu Ding, Suman Banerjee, Yang Zhou, Zhiwen Fan
Robotics
GenPHRI: Agentic Generative Simulation for Physical Human-Robot Interaction
Junxiang Wang, Nir Pechuk, Xinwen Xu et al. (+4 more)
Real-world physical human-
robot interaction (pHRI) development is costly and slow, while simulation requires substantial task-specific scene and motion authoring. We present GenPHRI, a generative simulation framework for producing deployment-ready pHRI scenarios entirely from natural-
language task descriptions using
vision-
language model (VLM)
agents. Generated scenarios contain a room with furniture, a task-appropriate human pose and placement, and a
robot executing the requested interaction. Generator and critic
agents iterate on each component, and an orchestrator
agent controls cross-stage decisions. With GenPHRI, we generate 50 different assistive scenarios in simulation with no manual intervention and deploy two tasks in a 12-participant user study.
Vision policies trained on these scenarios achieve zero-shot task completion rates of ~80% in real-world trials against ~90% in simulation, while producing behaviors that participants rate as consistent with textual task
prompts. Further, we demonstrate direct execution of generated motion as a complementary, training-free deployment pathway that enables rapid real-world inspection of generated trajectories at the cost of contact coverage. We release GenPHRI as a platform for generating, training, and deploying pHRI tasks from textual
prompts. Additional information may be found on our project website: https://rchi-lab.github.io/gen_phri/
Read the full abstract ▾
All authors Junxiang Wang, Nir Pechuk, Xinwen Xu, Tiancheng Wu, Nayon Kim, Julian Millan, Zackory Erickson
Robotics · Cross-listed from cs.CV
3D-VCD: Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decoding
Makanjuola Ogunleye, Eman Abdelrahman and Ismini Lourentzou
Large
multimodal models are increasingly used as the
reasoning core of
embodied agents operating in 3D environments, yet they remain prone to
hallucinations that can produce unsafe and ungrounded decisions. Existing inference-time
hallucination mitigation methods largely target 2D
vision-
language settings and do not transfer to
embodied 3D
reasoning, where failures arise from object presence, spatial layout, and
geometric grounding rather than pixel-level inconsistencies. We introduce 3D-VCD, the first inference-time visual contrastive decoding framework for
hallucination mitigation in 3D
embodied agents. 3D-VCD constructs a distorted 3D scene graph by applying semantic and
geometric perturbations to object-centric representations, such as category substitutions and coordinate or extent corruption. By contrasting predictions under the original and distorted 3D contexts, our method suppresses
tokens that are insensitive to grounded scene evidence and are therefore likely driven by
language priors. We evaluate 3D-VCD on the 3D-POPE and HEAL
benchmarks and show that it consistently improves grounded
reasoning without any retraining, establishing inference-time contrastive decoding over structured 3D representations as an effective and practical route to more reliable
embodied intelligence.
Read the full abstract ▾
Robotics
LEGO: Latent-space Exploration for Geometry-aware Optimization of Humanoid Kinematic Design
Jihwan Yoon, Taemoon Jeong, Jeongeun Park et al. (+5 more)
Designing
robot morphologies and kinematics has traditionally relied on human intuition, with little systematic foundation. Motion-design co-optimization offers a promising path toward automation, but two major challenges remain: (i) the vast, unstructured design space and (ii) the difficulty of constructing task-specific loss functions. We propose a new paradigm that minimizes human involvement by (i) learning the design search space from existing mechanical designs, rather than hand-crafting it, and (ii) defining the loss directly from human motion data via motion retargeting and Procrustes analysis. Using screw-theory-based joint axis representation and isometric
manifold learning, we construct a compact, geometry-preserving latent space of
humanoid upper body designs in which optimization is tractable. We then solve design optimization in this latent space using gradient-free optimization. Our approach establishes a principled framework for data-driven
robot design and demonstrates that leveraging existing designs and human motion can effectively guide the automated discovery of novel
robot design.
Read the full abstract ▾
All authors Jihwan Yoon, Taemoon Jeong, Jeongeun Park, Chanwoo Kim, Jaewoon Kwon, Yonghyeon Lee, Kyungjae Lee, Sungjoon Choi
Robotics
SIM1: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worlds
Yunsong Zhou, Hangxu Liu, Xuekun Jiang et al. (+12 more)
Robotic manipulation with deformable objects represents a data-intensive regime in
embodied learning, where shape, contact, and
topology co-evolve in ways that far exceed the variability of rigids. Although simulation promises relief from the cost of real-world data acquisition, prevailing sim-to-real pipelines remain rooted in rigid-body abstractions, producing mismatched geometry, fragile soft dynamics, and motion primitives poorly suited for cloth interaction. We posit that simulation fails not for being synthetic, but for being ungrounded. To address this, we introduce SIM1, a physics-aligned real-to-sim-to-real data engine that grounds simulation in the physical world. Given limited demonstrations, the system digitizes scenes into metric-consistent twins, calibrates deformable dynamics through elastic modeling, and expands behaviors via
diffusion-based trajectory generation with quality filtering. This pipeline transforms sparse observations into scaled synthetic supervision with near-demonstration fidelity. Experiments show that policies trained on purely synthetic data achieve parity with real-data baselines at a 1:15 equivalence ratio, while delivering 90% zero-shot success and 50% generalization gains in real-world deployment. These results validate physics-aligned simulation as scalable supervision for deformable manipulation and a practical pathway for data-efficient policy learning.
Read the full abstract ▾
All authors Yunsong Zhou, Hangxu Liu, Xuekun Jiang, Xing Shen, Yuanzhen Zhou, Hui Wang, Baole Fang, Yang Tian, Mulin Yu, Qiaojun Yu, Li Ma, Hengjie Li, Hanqing Wang, Jia Zeng, Jiangmiao Pang
Robotics
Fail2Drive: Benchmarking Closed-Loop Driving Generalization
Simon Gerstenecker, Andreas Geiger and Katrin Renz
Generalization under distribution shift remains a central bottleneck for closed-loop autonomous driving. Although simulators like CARLA enable safe and scalable testing, existing
benchmarks rarely measure true generalization: they typically reuse training scenarios at test time. Success can therefore reflect memorization rather than robust driving behavior. We introduce Fail2Drive, the first paired-route
benchmark for closed-loop generalization in CARLA, with 200 routes and 17 new scenario classes spanning appearance, layout, behavioral, and robustness shifts. Each shifted route is matched with an in-distribution counterpart, isolating the effect of the shift and turning qualitative failures into quantitative diagnostics. Evaluating multiple state-of-the-art models reveals consistent degradation, with an average success-rate drop of 22.8%. Our analysis uncovers unexpected failure modes, such as ignoring objects clearly visible in the
LiDAR and failing to learn the fundamental concepts of free and occupied space. To accelerate follow-up work, Fail2Drive includes an open-source toolbox for creating new scenarios and validating solvability via a privileged expert policy. Together, these components establish a reproducible foundation for benchmarking and improving closed-loop driving generalization. We open-source all code, data, and tools at https://github.com/autonomousvision/fail2drive .
Read the full abstract ▾
Robotics
ActiveGlasses: Learning Manipulation with Active Vision from Ego-centric Human Demonstration
Yanwen Zou, Chenyang Shi, Wenye Yu et al. (+5 more)
Large-scale real-world
robot data collection is a prerequisite for bringing
robots into everyday deployment. However, existing pipelines often rely on specialized handheld devices to bridge the embodiment gap, which not only increases operator burden and limits scalability, but also makes it difficult to capture the naturally coordinated perception-manipulation behaviors of human daily interaction. This challenge calls for a more natural system that can faithfully capture human manipulation and perception behaviors while enabling zero-shot transfer to robotic platforms. We introduce ActiveGlasses, a system for learning
robot manipulation from ego-centric human demonstrations with active
vision. A stereo camera mounted on smart glasses serves as the sole perception device for both data collection and policy inference: the operator wears it during bare-hand demonstrations, and the same camera is mounted on a 6-DoF perception arm during deployment to reproduce human active
vision. To enable zero-transfer, we extract object trajectories from demonstrations and use an object-centric point-cloud policy to jointly predict manipulation and head movement. Across several challenging tasks involving occlusion and precise interaction, ActiveGlasses achieves zero-shot transfer with active
vision, consistently outperforms strong baselines under the same hardware setup, and generalizes across two
robot platforms.
Read the full abstract ▾
All authors Yanwen Zou, Chenyang Shi, Wenye Yu, Han Xue, Jun Lv, Ye Pan, Chuan Wen, Cewu Lu
Robotics
A-SLIP: Acoustic Sensing for Continuous In-hand Slip Estimation
Uksang Yoo, Yuemin Mao, Jean Oh et al. (+1 more)
Reliable in-hand manipulation requires accurate real-time estimation of slip between a gripper and a grasped object. Existing tactile sensing approaches based on
vision, capacitance, or force-torque measurements face fundamental trade-offs in form factor, durability, and their ability to jointly estimate slip direction and magnitude. We present A-SLIP, a multi-channel acoustic sensing system integrated into a parallel-jaw gripper for estimating continuous slip in the grasp plane. The A-SLIP sensor consists of piezoelectric microphones positioned behind a textured silicone contact pad to capture structured contact-induced vibrations. The A-SLIP model processes synchronized multi-channel audio as log-mel spectrograms using a lightweight convolutional network, jointly predicting the presence, direction, and magnitude of slip. Across experiments with
robot- and externally induced slip conditions, the fine-tuned four-microphone configuration achieves a mean absolute directional error of 14.1 degrees, outperforms baselines by up to 12 percent in detection accuracy, and reduces directional error by 32 percent. Compared with single-microphone configurations, the multi-channel design reduces directional error by 64 percent and magnitude error by 68 percent, underscoring the importance of spatial acoustic sensing in resolving slip direction ambiguity. We further evaluate A-SLIP in closed-loop reactive control and find that it enables reliable, low-cost, real-time estimation of in-hand slip. Project
videos and additional details are available at https://a-slip.github.io.
Read the full abstract ▾
All authors Uksang Yoo, Yuemin Mao, Jean Oh, Jeffrey Ichnowski
Robotics · Cross-listed from cs.CV
Hang Ye, Xiaoxuan Ma, Fan Lu et al. (+3 more)
Digital human generation has been studied for decades and supports a wide range of real-world applications. However, most existing systems are passively animated, relying on privileged state or scripted control, which limits scalability to novel environments. We instead ask: how can digital humans actively behave using only visual observations and specified goals in novel scenes? Achieving this would enable populating any 3D environments with digital humans at scale that exhibit spontaneous, natural, goal-directed behaviors. To this end, we introduce Visually-grounded
Humanoid Agents, a coupled two-layer (world-
agent) paradigm that replicates humans at multiple levels: they look, perceive, reason, and behave like real people in real-world 3D scenes. The World Layer reconstructs semantically rich 3D Gaussian scenes from real-world
videos via an occlusion-aware pipeline and accommodates animatable Gaussian-based human avatars. The
Agent Layer transforms these avatars into autonomous
humanoid agents, equipping them with first-person RGB-D perception and enabling them to perform accurate,
embodied planning with spatial awareness and iterative
reasoning, which is then executed at the low level as full-body actions to drive their behaviors in the scene. We further introduce a
benchmark to evaluate
humanoid-scene interaction in diverse reconstructed environments. Experiments show our
agents achieve robust autonomous behavior, yielding higher task success rates and fewer collisions than ablations and state-of-the-art planning methods. This work enables active digital human population and advances human-centric
embodied AI. Data, code, and models will be open-sourced.
Read the full abstract ▾
All authors Hang Ye, Xiaoxuan Ma, Fan Lu, Wayne Wu, Kwan-Yee Lin, Yizhou Wang
Robotics
Sumo: Dynamic and Generalizable Whole-Body Loco-Manipulation
John Z. Zhang, Maks Sorokin, Jan Brüdigam et al. (+14 more)
This paper presents a sim-to-real approach that enables legged
robots to dynamically manipulate large and heavy objects with whole-body dexterity. Our key insight is that by performing test-time steering of a pre-trained whole-body control policy with a sample-based planner, we can enable these
robots to solve a variety of dynamic loco-manipulation tasks. Interestingly, we find our method generalizes to a diverse set of objects and tasks with no additional tuning or training, and can be further enhanced by flexibly adjusting the cost function at test time. We demonstrate the capabilities of our approach through a variety of challenging loco-manipulation tasks on a Spot quadruped
robot in the real world, including uprighting a tire heavier than the
robot's nominal lifting capacity and dragging a crowd-control barrier larger and taller than the
robot itself. Additionally, we show that the same approach can be generalized to
humanoid loco-manipulation tasks, such as opening a door and pushing a table, in simulation. Project code and
videos are available at https://sumo.rai-inst.com/.
Read the full abstract ▾
All authors John Z. Zhang, Maks Sorokin, Jan Brüdigam, Brandon Hung, Stephen Phillips, Dmitry Yershov, Farzad Niroui, Tong Zhao, Leonor Fermoselle, Xinghao Zhu, Chao Cao, Duy Ta, Tao Pang, Jiuguang Wang, Preston Culbertson, Zachary Manchester, Simon Le Cléac'h
Robotics · Cross-listed from math.OC
Density-Driven Optimal Control: Convergence Guarantees for Stochastic LTI Multi-Agent Systems
Kooktae Lee
This paper addresses the decentralized non-uniform area coverage problem for multi-
agent systems, a critical task in missions with high spatial priority and resource constraints. While existing density-based methods often rely on computationally heavy Eulerian PDE solvers or heuristic planning, we propose
Stochastic Density-Driven Optimal Control (D$^2$OC). This is a rigorous Lagrangian framework that bridges the gap between individual
agent dynamics and collective distribution matching. By formulating a
stochastic MPC-like problem that minimizes the Wasserstein distance as a running cost, our approach ensures that the time-averaged empirical distribution converges to a non-parametric target density under
stochastic LTI dynamics. A key contribution is the formal convergence guarantee established via reachability analysis, providing a bounded tracking error even in the presence of process and measurement noise. Numerical results verify that
Stochastic D$^2$OC achieves robust, decentralized coverage while outperforming previous heuristic methods in optimality and consistency.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
CrashSight: A Phase-Aware, Infrastructure-Centric Video Benchmark for Traffic Crash Scene Understanding and Reasoning
Rui Gan, Junyi Ma, Pei Li et al. (+4 more)
Cooperative autonomous driving requires traffic scene understanding from both vehicle and infrastructure perspectives. While
vision-
language models (VLMs) show strong general
reasoning capabilities, their performance in safety-critical traffic scenarios remains insufficiently evaluated due to the ego-vehicle focus of existing
benchmarks. To bridge this gap, we present CrashSight, a large-scale
vision-
language benchmark for roadway crash understanding using real-world roadside camera data. The
dataset comprises 250 crash
videos, annotated with 13K multiple-choice question-answer pairs organized under a two-tier taxonomy. Tier 1 evaluates the visual grounding of scene context and involved parties, while Tier 2 probes higher-level
reasoning, including crash mechanics, causal attribution, temporal progression, and post-crash outcomes. We
benchmark 8 state-of-the-art VLMs and show that, despite strong scene description capabilities, current models struggle with temporal and causal
reasoning in safety-critical scenarios. We provide a detailed analysis of failure scenarios and discuss directions for improving VLM crash understanding. The
benchmark provides a standardized evaluation framework for infrastructure-assisted perception in cooperative autonomous driving. The CrashSight
benchmark, including the full
dataset and code, is accessible at https://mcgrche.github.io/crashsight.
Read the full abstract ▾
All authors Rui Gan, Junyi Ma, Pei Li, Xingyou Yang, Kai Chen, Sikai Chen, Bin Ran
Robotics
A Soft Robotic Interface for Chick-Robot Affective Interactions
Jue Chen, Alexander Mielke, Kaspar Althoefer et al. (+1 more)
The potential of Animal-
Robot Interaction (ARI) in welfare applications depends on how much an animal perceives a robotic
agent as socially relevant, non-threatening and potentially attractive (acceptance). Here, we present an animal-centered soft robotic affective interface for newly hatched chicks (Gallus gallus). The soft interface provides safe and controllable cues, including warmth, breathing-like rhythmic deformation, and face-like visual stimuli. We evaluated chick acceptance of the interface and chick-
robot interactions by measuring spontaneous approach and touch responses during
video tracking. Overall, chicks approached and spent increasing time on or near the interface, demonstrating acceptance of the device. Across different layouts, chicks showed strong preference for warm thermal stimulation, which increased over time. Face-like visual cues elicited a swift and stable preference, speeding up the initial approach to the tactile interface. Although the breathing cue did not elicit any preference, neither did it trigger avoidance, paving the way for further exploration. These findings translate affective interface concepts to ARI, demonstrating that appropriate soft, thermal and visual stimuli can sustain early chick-
robot interactions. This work establishes a reliable evaluation protocol and a safe baseline for designing
multimodal robotic devices for animal welfare and neuroscientific research.
Read the full abstract ▾
All authors Jue Chen, Alexander Mielke, Kaspar Althoefer, Elisabetta Versace
Robotics
Exploring Temporal Representation in Neural Processes for Multimodal Action Prediction
Marco Gabriele Fedozzi, Yukie Nagai, Francesco Rea et al. (+1 more)
Inspired by the human ability to understand and predict others, we study the applicability of Conditional Neural Processes (CNP) to the task of self-supervised
multimodal action prediction in robotics. Following recent results regarding the ontogeny of the Mirror
Neuron System (MNS), we focus on the preliminary objective of self-actions prediction. We find a good MNS-inspired model in the existing Deep Modality Blending Network (DMBN), able to reconstruct the visuo-motor sensory signal during a partially observed action sequence by leveraging the probabilistic generation of CNP. After a qualitative and quantitative evaluation, we highlight its difficulties in generalizing to unseen action sequences, and identify the cause in its inner representation of time. Therefore, we propose a revised version, termed DMBN-Positional Time Encoding (DMBN-PTE), that facilitates learning a more robust representation of temporal information, and provide preliminary results of its effectiveness in expanding the applicability of the architecture. DMBN-PTE figures as a first step in the development of robotic systems that autonomously learn to forecast actions on longer time scales refining their predictions with incoming observations.
Read the full abstract ▾
All authors Marco Gabriele Fedozzi, Yukie Nagai, Francesco Rea, Alessandra Sciutti
Robotics · Cross-listed from cs.CV
BLaDA: Bridging Language to Functional Dexterous Actions within 3DGS Fields
Fan Yang, Wenrui Chen, Guorun Yan et al. (+7 more)
In unstructured environments, functional dexterous grasping calls for the tight integration of semantic understanding, precise 3D functional localization, and physically interpretable execution. Modular hierarchical methods are more controllable and interpretable than end-to-end VLA approaches, but existing ones still rely on predefined affordance labels and lack the tight semantic–pose coupling needed for functional dexterous manipulation. To address this, we propose BLaDA (Bridging
Language to Dexterous Actions in 3DGS fields), an interpretable zero-shot framework that grounds open-vocabulary instructions as perceptual and control constraints for functional dexterous manipulation. BLaDA establishes an interpretable
reasoning chain by first parsing natural
language into a structured sextuple of manipulation constraints via a Knowledge-guided
Language Parsing (KLP) module. To achieve pose-consistent spatial
reasoning, we introduce the Triangular Functional Point Localization (TriLocation) module, which utilizes 3D Gaussian Splatting as a continuous scene representation and identifies functional regions under triangular
geometric constraints. Finally, the 3D Keypoint Grasp
Matrix Transformation Execution (KGT3D+) module decodes these semantic-
geometric constraints into physically plausible wrist poses and finger-level commands. Extensive experiments on complex
benchmarks demonstrate that BLaDA significantly outperforms existing methods in both affordance grounding precision and the success rate of functional manipulation across diverse categories and tasks. Code will be publicly available at https://github.com/PopeyePxx/BLaDA.
Read the full abstract ▾
All authors Fan Yang, Wenrui Chen, Guorun Yan, Ruize Liao, Wanjun Jia, Dongsheng Luo, Jiacheng Lin, Kailun Yang, Zhiyong Li, Yaonan Wang
Robotics
A Unified Multi-Layer Framework for Skill Acquisition from Imperfect Human Demonstrations
Zi-Qi Yang and Mehrdad R. Kermani
Current Human-
Robot Interaction (HRI) systems for skill teaching are fragmented, and existing approaches in the literature do not offer a cohesive framework that is simultaneously efficient, intuitive, and universally safe. This paper presents a novel, layered control framework that addresses this fundamental gap by enabling robust, compliant Learning from Demonstration (LfD) built upon a foundation of universal
robot compliance. The proposed approach is structured in three progressive and interconnected stages. First, we introduce a real-time LfD method that learns both the trajectory and variable impedance from a single demonstration, significantly improving efficiency and reproduction fidelity. To ensure high-quality and intuitive {kinesthetic teaching}, we then present a null-space optimization strategy that proactively manages singularities and provides a consistent interaction feel during human demonstration. Finally, to ensure generalized safety, we introduce a foundational null-space compliance method that enables the entire
robot body to compliantly adapt to post-learning external interactions without compromising main task performance. This final contribution transforms the system into a versatile HRI platform, moving beyond end-effector (EE)-specific applications. We validate the complete framework through comprehensive comparative experiments on a 7-DOF KUKA LWR
robot. The results demonstrate a safer, more intuitive, and more efficient unified system for a wide range of human-
robot collaborative tasks.
Read the full abstract ▾
Robotics
EMMa: End-Effector Stability-Oriented Mobile Manipulation for Tracked Rescue Robots
Yifei Wang, Hao Zhang, Jidong Huang et al. (+2 more)
The autonomous operation of tracked mobile manipulators in rescue missions requires not only ensuring the reachability and safety of
robot motion but also maintaining stable end-effector manipulation under diverse task demands. However, existing studies have overlooked many end-effector motion properties at both the planning and control levels. This paper presents a motion generation framework for tracked mobile manipulators to achieve stable end-effector operation in complex rescue scenarios. The framework formulates a coordinated path optimization model that couples end-effector and mobile base states and designs compact cost/constraint representations to mitigate nonlinearities and reduce computational complexity. Furthermore, an isolated control scheme with feedforward compensation and feedback regulation is developed to enable coordinated path tracking for the
robot. Extensive simulated and real-world experiments on rescue scenarios demonstrate that the proposed framework consistently outperforms SOTA methods across key metrics, including task success rate and end-effector motion stability, validating its effectiveness and robustness in complex mobile manipulation tasks.
Read the full abstract ▾
All authors Yifei Wang, Hao Zhang, Jidong Huang, Shuohang Fang, Haoyao Chen
Robotics
EvoGymCM: Harnessing Continuous Material Stiffness for Soft Robot Co-Design
Le Shen, Kangyao Huang, Wentao Zhao et al. (+1 more)
In the automated co-design of soft
robots, precisely adapting the material stiffness field to task environments is crucial for unlocking their full physical potential. However, mainstream platforms (e.g., EvoGym) strictly discretize the material dimension, artificially restricting the design space and performance of soft
robots. To address this, we propose EvoGymCM (EvoGym with Continuous Materials), a
benchmark suite formally establishing continuous material stiffness as a first-class design variable alongside morphology and control. Aligning with real-world material mechanisms, EvoGymCM introduces two settings: (i) EvoGymCM-R (Reactive), motivated by programmable materials with dynamically tunable stiffness; and (ii) EvoGymCM-I (Invariant), motivated by traditional materials with invariant stiffness fields. To tackle the resulting high-dimensional coupling, we formulate two Morphology-Material-Control co-design paradigms: (i) Reactive-Material Co-Design, which learns real-time stiffness tuning policies to guide programmable materials; and (ii) Invariant-Material Co-Design, which jointly optimizes morphology and fixed material fields to guide traditional material fabrication. Systematic experiments across diverse tasks demonstrate that continuous material optimization boosts performance and unlocks synergy across morphology, material, and control.
Read the full abstract ▾
All authors Le Shen, Kangyao Huang, Wentao Zhao, Huaping Liu
Robotics
State and Trajectory Estimation of Tensegrity Robots via Factor Graphs and Chebyshev Polynomials
Edgar Granados, Patrick Meng, Charles Tang et al. (+4 more)
Tensegrity
robots offer compliance and adaptability, but their nonlinear, and underconstrained dynamics make state estimation challenging. Reliable continuous-time estimation of all rigid links is crucial for closed-loop control, system identification, and machine learning; however, conventional methods often fall short. This paper proposes a two-stage approach for robust state or trajectory estimation (i.e., filtering or smoothing) of a cable-driven tensegrity
robot. For online state estimation, this work introduces a factor-graph-based method, which fuses measurements from an RGB-D camera with on-board cable length sensors. To the best of the authors' knowledge, this is the first application of factor graphs in this domain. Factor graphs are a natural choice, as they exploit the
robot's structural properties and provide effective sensor fusion solutions capable of handling nonlinearities in practice. Both the Mahalanobis distance-based clustering
algorithm, used to handle noise, and the Chebyshev
polynomial method, used to estimate the most probable velocities and intermediate states, are shown to perform well on simulated and real-world data, compared to an ICP-based
algorithm. Results show that the approach provides high fidelity, continuous-time state and trajectory estimates for complex tensegrity
robot motions.
Read the full abstract ▾
All authors Edgar Granados, Patrick Meng, Charles Tang, Shrimed Sangani, William R. Johnson III, Rebecca Kramer-Bottiglio, Kostas Bekris
Robotics
ViVa: A Video-Generative Value Model for Robot Reinforcement Learning
Jindi Lv, Hao Li, Jie Li et al. (+11 more)
Vision-
language-action (VLA) models have advanced
robot manipulation through large-scale pretraining, but real-world deployment remains challenging due to partial observability and delayed feedback. Reinforcement learning addresses this via value functions, which assess task progress and guide policy improvement. However, existing value models built on
vision-
language models (VLMs) struggle to capture temporal dynamics and physical interactions, undermining reliable value estimation in long-horizon tasks. In this paper, we propose ViVa, a
video-generative value model that repurposes a pretrained
video generator to jointly predict future proprioception and a scalar value. By grounding value estimation in anticipated embodiment dynamics, ViVa leverages spatiotemporal priors to intrinsically couple value with foresight beyond static snapshots. ViVa achieves state-of-the-art results in metric-based evaluation across three tasks, producing reliable value signals that accurately track task progress and detect execution errors. Integrated into RECAP, it achieves an average success rate of 80%, highlighting the promise of
video-generative models for value estimation.
Read the full abstract ▾
All authors Jindi Lv, Hao Li, Jie Li, Fankun Kong, Yang Wang, Pengfei Yi, Yifei Nie, Xiaofeng Wang, Zheng Zhu, Chaojun Ni, Qiuping Deng, Hengtao Li, Jiancheng Lv, Guan Huang
Robotics
Semantic-Aware UAV Command and Control for Efficient IoT Data Collection
Assane Sankara, Daniel Bonilla Licea and Hajar El Hammouti
Unmanned Aerial Vehicles (
UAVs) have emerged as a key enabler technology for data collection from Internet of Things (IoT) devices. However, effective data collection is challenged by resource constraints and the need for real-time decision-making. In this work, we propose a novel framework that integrates semantic communication with
UAV command-and-control (C&C) to enable efficient image data collection from IoT devices. Each device uses Deep Joint Source-Channel Coding (DeepJSCC) to generate a compact semantic latent representation of its image to enable image reconstruction even under partial transmission. A base station (BS) controls the
UAV's trajectory by transmitting acceleration commands. The objective is to maximize the average quality of reconstructed images by maintaining proximity to each device for a sufficient duration within a fixed time horizon. To address the challenging trade-off and account for delayed C&C signals, we model the problem as a Markov Decision Process and propose a Double Deep Q-Learning (DDQN)-based adaptive flight policy. Simulation results show that our approach outperforms baseline methods such as greedy and traveling salesman
algorithms, in both device coverage and semantic reconstruction quality.
Read the full abstract ▾
Robotics · Cross-listed from eess.SY
Complementary Filtering on SO(3) for Attitude Estimation with Scalar Measurements
Alessandro Melis, Soulaimane Berkane, Robert Mahony et al. (+1 more)
This paper proposes a complementary filter on SO(3) for attitude estimation from scalar measurements corresponding to projections of known inertial vectors onto body-frame sensing directions. The observer evolves directly on SO(3) and employs a constant-gain innovation tailored to the scalar-output structure. Under suitable persistence-of-excitation conditions, almost-global asymptotic stability is established when at least three inertial vectors are measured along a common body-frame direction. For configurations involving only two scalar measurements, sufficient conditions for asymptotic convergence are derived together with an explicit characterization of the region of attraction. Numerical simulations illustrate the proposed results.
Read the full abstract ▾
All authors Alessandro Melis, Soulaimane Berkane, Robert Mahony, Tarek Hamel
Robotics
Governed Capability Evolution: Lifecycle-Time Compatibility Checking and Rollback for AI-Component-Based Systems, with a Proof-of-Concept Evaluation on Embodied Agents
Xue Qin, Simin Luan, John See et al. (+3 more)
Systems built from versioned AI components need lifecycle-time governance over how a new module version is admitted, monitored, and withdrawn. Established deployment patterns (canary, blue-green, feature flags, MLOps) monitor aggregate service-level signals of stateless services, not the stateful, policy-constrained runtimes that drive AI components. We formulate governed capability
evolution as a software-lifecycle problem and ask whether staged upgrade governance prevents unsafe activations where practice-adapted canary and blue-green deployment do not, and at what cost. We propose four compatibility checks (interface, policy, behavioral, recovery) and a seven-stage pipeline (validation, sandbox, shadow, gated activation, monitoring, rollback, audit). A reference prototype is evaluated in a behavioral simulation of an
embodied manipulation stack; a PyBullet backend replicates the strategy ordering. Upgrade candidates come from a seeded fault-operator generator, including out-of-taxonomy probes, and thresholds are fixed on a disjoint development set. Across five strategies, five upgrade rounds and 15 seeds, naive upgrade reaches 73.3% final-round success but activates unsafe versions in 74.7% of rounds. Canary and blue-green adaptations keep comparable success yet still admit faulty versions in 61.3% and 64.0% of rounds, because aggregate success hides policy and recovery regressions. Governed upgrade admits no in-taxonomy fault and one out-of-taxonomy probe in 75 rounds, at a 4.6-point success cost (Holm-adjusted p<=0.024 against all three baselines); the full pre-activation stack detects 97.3-100% of in-taxonomy faults across deployment profiles, and rollback restores verified operation in 72.4% of drift scenarios. Bounded-window aggregate-success gating proved insufficient against these fault operators; staged governance closes the gap at a modest, tunable cost.
Read the full abstract ▾
All authors Xue Qin, Simin Luan, John See, Zeyd Boukhers, Cong Yang, Zhijun Li