Robotics
Learning from Watching: Scalable Extraction of Manipulation Trajectories from Human Videos
X. Hu and G. Ye
Collecting high-quality data for training large-scale robotic models typically relies on real
robot platforms, which is labor-intensive and costly, whether via teleoperation or scripted demonstrations. To scale data collection, many researchers have turned to leveraging human manipulation
videos available online. However, current methods predominantly focus on hand detection or object pose estimation, failing to fully exploit the rich interaction cues embedded in these
videos. In this work, we propose a novel approach that combines large foundation models for
video understanding with point tracking techniques to extract dense trajectories of all task-relevant keypoints during manipulation. This enables more comprehensive utilization of Internet-scale human demonstration
videos. Experimental results demonstrate that our method can accurately track keypoints throughout the entire manipulation process, paving the way for more scalable and data-efficient
robot learning.
Read the full abstract ▾
Robotics
TACO: Trajectory-Aware Controller Optimization for Quadrotors
Hersh Sanghvi, Spencer Folk, Vijay Kumar et al. (+1 more)
Controller performance in quadrotor trajectory tracking depends heavily on parameter tuning, yet standard approaches often rely on fixed, manually tuned parameters that sacrifice task-specific performance. We present Trajectory-Aware Controller Optimization (TACO), a framework that adapts controller parameters online based on the upcoming reference trajectory and current quadrotor state. TACO employs a learned predictive model and a lightweight optimization scheme to optimize controller gains in real time with respect to a broad class of trajectories, and can also be used to adapt trajectories to improve dynamic feasibility while respecting smoothness constraints. To enable large-scale training, we also introduce a parallelized quadrotor simulator supporting fast data collection on diverse trajectories. Experiments on a variety of trajectory types show that TACO outperforms conventional, static parameter tuning while operating orders of magnitude faster than black-box optimization baselines, enabling practical real-time deployment on a physical quadrotor. Furthermore, we show that adapting trajectories using TACO significantly reduces the tracking error obtained by the quadrotor.
Read the full abstract ▾
All authors Hersh Sanghvi, Spencer Folk, Vijay Kumar, Camillo Jose Taylor
Robotics
TurboMap: GPU-Accelerated Local Mapping for Visual SLAM
Parsa Hosseininejad, Kimia Khabiri, Shishir Gopinath et al. (+3 more)
In real-time Visual SLAM systems, local mapping must operate under strict latency constraints, as delays degrade map quality and increase the risk of tracking failure. GPU parallelization offers a promising way to reduce latency. However, parallelizing local mapping is challenging due to synchronized shared-state updates and the overhead of transferring large map data structures to the GPU. This paper presents TurboMap, a GPU-parallelized and CPU-optimized local mapping backend that holistically addresses these challenges. We restructure Map Point Creation to enable parallel Keypoint Correspondence Search on the GPU, redesign and parallelize Map Point Fusion, optimize Redundant Keyframe Culling on the CPU, and integrate a fast GPU-based Local Bundle Adjustment solver. To minimize data transfer and synchronization costs, we introduce persistent GPU-resident keyframe storage. Experiments on the EuRoC and TUM-VI
datasets show average local mapping speedups of 1.3x and 1.6x, respectively, while preserving accuracy.
Read the full abstract ▾
All authors Parsa Hosseininejad, Kimia Khabiri, Shishir Gopinath, Soudabeh Mohammadhashemi, Karthik Dantu, Steven Y. Ko
Robotics · Cross-listed from cs.LG
Path-Coordinated Continual Learning with Neural Tangent Kernel-Justified Plasticity: A Theoretical Framework with Near State-of-the-Art Performance
Rathin Chandra Shit
Catastrophic forgetting is one of the fundamental issues of continual learning because neural networks forget the tasks learned previously when trained on new tasks. The proposed framework is a new path-coordinated framework of continual learning that unites the Neural Tangent Kernel (NTK) theory of principled plasticity bounds, statistical validation by Wilson confidence intervals, and evaluation of path quality by the use of multiple metrics. Experimental evaluation shows an average accuracy of 66.7% at the cost of 23.4% catastrophic forgetting on Split-CIFAR10, a huge improvement over the baseline and competitive performance achieved, which is very close to state-of-the-art results. Further, it is found out that NTK condition numbers are predictive indicators of learning capacity limits, showing the existence of a critical threshold at condition number $>10^{11}$. It is interesting to note that the proposed strategy shows a tendency of lowering forgetting as the sequence of tasks progresses (27% to 18%), which is a system stabilization. The framework validates 80% of discovered paths with a rigorous statistical guarantee and maintains 90-97% retention on intermediate tasks. The core capacity limits of the continual learning environment are determined in the analysis, and actionable insights to enhance the adaptive regularization are offered.
Read the full abstract ▾
Robotics
Stein-based Optimization of Sampling Distributions in Model Predictive Path Integral Control
Jace Aldrich and Odest Chadwicke Jenkins
This paper introduces a method for Model Predictive Path Integral (MPPI) control that optimizes sample generation towards an optimal trajectory through Stein Variational Gradient Descent (SVGD). MPPI relies upon predictive rollout of trajectories sampled from a distribution of possible actions. Traditionally, these action distributions are assumed to be unimodal and represented as Gaussian. This assumption can lead to suboptimal rollout predictions due to sample deprivation and, in the case of differentiable simulation, sensitivity to noise in the cost gradients. By introducing SVGD updates in between MPPI environment steps, we present Stein-Optimized Predictive Path-Integral Inference (SOPPI), an MPPI/SVGD
algorithm that can dynamically update noise distributions during rollouts to better capture action sampling distributions without an excessive increase in computational requirements. We demonstrate the efficacy of SOPPI through experiments on a planar cart-pole, 7-DOF
robot arm, a planar bipedal walker, and a Unitree G1 29-DOF
Robot. Results indicate improved inference of an underlying multi-modal action distribution compared to other Stein-based MPPI
algorithms, with improved system performance and similar or improved efficiency across a range of hyper-parameters.
Read the full abstract ▾
Robotics
TRACE: Textual Reasoning for Affordance Coordinate Extraction
Sangyun Park, Jin Kim, Yuchen Cui et al. (+1 more)
Vision-
Language Models (VLMs) struggle to translate high-level instructions into the precise spatial affordances required for robotic manipulation. While visual Chain-of-Thought (CoT) methods exist, they are often computationally intensive. In this work, we introduce TRACE (Textual
Reasoning for Affordance Coordinate Extraction), a novel methodology that integrates a textual Chain of
Reasoning (CoR) into the affordance prediction process. We use this methodology to create the TRACE
dataset, a large-scale collection created via an autonomous pipeline that pairs instructions with explicit textual rationales. By fine-tuning a VLM on this data, our model learns to externalize its spatial
reasoning before acting. Our experiments show that our TRACE-tuned model achieves state-of-the-art performance, reaching 48.1% accuracy on the primary Where2Place (W2P)
benchmark (a 9.6% relative improvement) and 55.0% on the more challenging W2P(h) subset. Crucially, an ablation study demonstrates that performance scales directly with the amount of
reasoning data used, confirming the CoR's effectiveness. Furthermore, analysis of the model's
attention maps reveals an interpretable
reasoning process where focus shifts dynamically across
reasoning steps. This work shows that training VLMs to generate a textual CoR is an effective and robust strategy for enhancing the precision, reliability, and interpretability of VLM-based
robot control. Our
dataset and code are available at https://github.com/jink-ucla/TRACE
Read the full abstract ▾
All authors Sangyun Park, Jin Kim, Yuchen Cui, Matthew S. Brown
Robotics
Hybrid Neural Network-Based Indoor Localisation System for Mobile Robots Using CSI Data in a Robotics Simulator
Javier Ballesteros-Jerez, Jesus Martínez-Gómez, Ismael García-Varea et al. (+2 more)
We present a hybrid neural network model for inferring the position of mobile
robots using Channel State Information (CSI) data from a Massive
MIMO system. By leveraging an existing CSI
dataset, our approach integrates a Convolutional Neural Network (CNN) with a Multilayer Perceptron (MLP) to form a Hybrid Neural Network (HyNN) that estimates 2D
robot positions. CSI readings are converted into synthetic images using the TINTO tool. The localisation solution is integrated with a robotics simulator, and the
Robot Operating System (ROS), which facilitates its evaluation through heterogeneous test cases, and the adoption of state
estimators like Kalman filters. Our contributions illustrate the potential of our HyNN model in achieving precise indoor localisation and
navigation for mobile
robots in complex environments. The study follows, and proposes, a generalisable procedure applicable beyond the specific use case studied, making it adaptable to different scenarios and
datasets.
Read the full abstract ▾
All authors Javier Ballesteros-Jerez, Jesus Martínez-Gómez, Ismael García-Varea, Luis Orozco-Barbosa, Manuel Castillo-Cara
Robotics · Cross-listed from cs.LG
Fractional Diffusion Bridge Models
Gabriel Nobis, Maximilian Springenberg, Arina Belova et al. (+5 more)
We present Fractional
Diffusion Bridge Models (FDBM), a novel generative
diffusion bridge framework driven by an approximation of the rich and non-Markovian fractional Brownian motion (fBM). Real
stochastic processes exhibit a degree of
memory effects (correlations in time), long-range dependencies, roughness and anomalous
diffusion phenomena that are not captured in standard
diffusion or bridge modeling due to the use of Brownian motion (BM). As a remedy, leveraging a recent Markovian approximation of fBM (MA-fBM), we construct FDBM that enable tractable inference while preserving the non-Markovian nature of fBM. We prove the existence of a coupling-preserving generative
diffusion bridge and leverage it for future state prediction from paired training data. We then extend our formulation to the Schrödinger bridge problem and derive a principled loss function to learn the unpaired data translation. We evaluate FDBM on both tasks: predicting future
protein conformations from aligned data, and unpaired image translation. In both settings, FDBM achieves superior performance compared to the Brownian baselines, yielding lower root mean squared deviation (RMSD) of C$_\alpha$ atomic positions in
protein structure prediction and lower Fréchet Inception Distance (FID) in unpaired image translation.
Read the full abstract ▾
All authors Gabriel Nobis, Maximilian Springenberg, Arina Belova, Rembert Daems, Christoph Knochenhauer, Manfred Opper, Tolga Birdal, Wojciech Samek
Robotics
GenDexHand: Generative Simulation for Dexterous Hands
Feng Chen, Zhuxiu Xu, Tianzhe Chu et al. (+7 more)
Data scarcity remains a fundamental bottleneck for
embodied intelligence. Existing approaches use large
language models (LLMs) to automate gripper-based simulation generation, but they transfer poorly to dexterous manipulation, which demands more specialized environment design. Meanwhile, dexterous manipulation tasks are inherently more difficult due to their higher degrees of freedom. Massively generating feasible and trainable dexterous hand tasks remains an open challenge. To this end, we present GenDexHand, a generative simulation pipeline that autonomously produces diverse robotic tasks and environments for dexterous manipulation. GenDexHand introduces a closed-loop refinement process that adjusts object placements and scales based on
vision-
language model (VLM) feedback, substantially improving the average quality of generated environments. Each task is further decomposed into sub-tasks to enable sequential reinforcement learning, reducing training time and increasing success rates. Our work provides a viable path toward scalable training of diverse dexterous hand behaviors in
embodied intelligence by offering a simulation-based solution to synthetic data generation. Our website: https://winniechen2002.github.io/GenDexHand/.
Read the full abstract ▾
All authors Feng Chen, Zhuxiu Xu, Tianzhe Chu, Xunzhe Zhou, Li Sun, Zewen Wu, Shenghua Gao, Zhongyu Li, Yanchao Yang, Yi Ma
Robotics
MOBIUS: A Multi-Modal Bipedal Robot that can Walk, Crawl, Climb, and Roll
Alexander Schperberg, Yusuke Tanaka, Stefano Di Cairano et al. (+1 more)
This paper presents the MOBIUS platform, a bipedal
robot capable of walking, crawling, climbing, and rolling. MOBIUS features four limbs, two 6-DoF arms with two-finger grippers for manipulation and climbing, and two 4-DoF legs for locomotion–enabling smooth transitions across diverse terrains without reconfiguration. A hybrid control architecture combines reinforcement learning for locomotion and force control for compliant contact interactions during manipulation. A high-level MIQCP planner autonomously selects locomotion modes to balance stability and energy efficiency. Hardware experiments demonstrate robust gait transitions, dynamic climbing, and full-body load support via pinch grasp. Overall, MOBIUS demonstrates the importance of tight integration between morphology, high-level planning, and control to enable mobile loco-manipulation and grasping, substantially expanding its interaction capabilities, workspace, and traversability.
Read the full abstract ▾
All authors Alexander Schperberg, Yusuke Tanaka, Stefano Di Cairano, Dennis Hong
Robotics
Lightweight Learning from Actuation-Space Demonstrations via Flow Matching for Whole-Body Soft Robotic Grasping
Liudi Yang, Yang Bai, Yuhao Wang et al. (+4 more)
Robotic grasping under uncertainty remains a fundamental challenge due to its uncertain and contact-rich nature. Traditional rigid robotic hands, with limited degrees of freedom and compliance, rely on complex model-based and heavy feedback controllers to manage such interactions. Soft
robots, by contrast, exhibit
embodied mechanical intelligence: their underactuated structures and passive flexibility of their whole body, naturally accommodate uncertain contacts and enable adaptive behaviors. To harness this capability, we propose a lightweight actuation-space learning framework that infers distributional control representations for whole-body soft robotic grasping, directly from deterministic demonstrations using a flow matching model (Rectified Flow),without requiring dense sensing or heavy control loops. Using only 30 demonstrations (less than 8% of the reachable workspace), the learned policy achieves a 97.5% grasp success rate across the whole workspace, generalizes to grasped-object size variations of +-33%, and maintains stable performance when the
robot's dynamic response is directly adjusted by scaling the execution time from 20% to 200%. These results demonstrate that actuation-space learning, by leveraging its passive redundant DOFs and flexibility, converts the body's mechanics into functional control intelligence and substantially reduces the burden on central controllers for this uncertain-rich task.
Read the full abstract ▾
All authors Liudi Yang, Yang Bai, Yuhao Wang, Ibrahim Alsarraj, Gitta Kutyniok, Zhanchi Wang, Ke Wu
Robotics · Cross-listed from cs.CV
3EED: Ground Everything Everywhere in 3D
Rong Li, Yuhao Dong, Tianshuai Hu et al. (+7 more)
Visual grounding in 3D is the key for
embodied agents to localize
language-referred objects in open-world environments. However, existing
benchmarks are limited to indoor focus, single-platform constraints, and small scale. We introduce 3EED, a multi-platform, multi-modal 3D grounding
benchmark featuring RGB and
LiDAR data from vehicle, drone, and quadruped platforms. We provide over 128,000 objects and 22,000 validated referring expressions across diverse outdoor scenes – 10x larger than existing
datasets. We develop a scalable annotation pipeline combining
vision-
language model prompting with human verification to ensure high-quality spatial grounding. To support cross-platform learning, we propose platform-aware normalization and cross-modal
alignment techniques, and establish
benchmark protocols for in-domain and cross-platform evaluations. Our findings reveal significant performance gaps, highlighting the challenges and opportunities of generalizable 3D grounding. The 3EED
dataset and
benchmark toolkit are released to advance future research in
language-driven 3D
embodied perception.
Read the full abstract ▾
All authors Rong Li, Yuhao Dong, Tianshuai Hu, Ao Liang, Youquan Liu, Dongyue Lu, Liang Pan, Lingdong Kong, Junwei Liang, Ziwei Liu
Robotics
Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Denoising Diffusion Process
Jiayi Chen, Wenxuan Song, Pengxiang Ding et al. (+5 more)
Vision-
language-action (VLA) models aim to understand natural
language instructions and visual observations and to execute corresponding actions as an
embodied agent. Recent work integrates future images into the understanding-acting loop, yielding unified VLAs that jointly understand, generate, and act – reading text and images and producing future images and actions. However, these models either rely on external experts for modality unification or treat image generation and action prediction as separate processes, limiting the benefits of direct synergy between these tasks. Our core philosophy is to optimize generation and action jointly through a synchronous denoising process, where the iterative refinement enables actions to evolve from initialization, under constant and sufficient visual guidance. We ground this philosophy in our proposed Unified
Diffusion VLA and Joint Discrete Denoising
Diffusion Process (JD3P), which is a joint
diffusion process that integrates multiple modalities into a single denoising trajectory to serve as the key mechanism enabling understanding, generation, and acting to be intrinsically synergistic. Our model and theory are built on a unified tokenized space of all modalities and a hybrid
attention mechanism. We further propose a two-stage training pipeline and several inference-time techniques that optimize performance and efficiency. Our approach achieves state-of-the-art performance on
benchmarks such as CALVIN, LIBERO, and SimplerEnv with 4$\times$ faster inference than autoregressive methods, and we demonstrate its effectiveness through in-depth analysis and real-world evaluations. Our project page is available at https://irpn-eai.github.io/UD-VLA.github.io/.
Read the full abstract ▾
All authors Jiayi Chen, Wenxuan Song, Pengxiang Ding, Ziyang Zhou, Han Zhao, Feilong Tang, Donglin Wang, Haoang Li
Robotics
MARS: Multi-Agent Robotic System with Multimodal Large Language Models for Assistive Intelligence
Renjun Gao
Multimodal large
language models (MLLMs) have shown remarkable capabilities in cross-modal understanding and
reasoning, offering new opportunities for intelligent assistive systems, yet existing systems still struggle with risk-aware planning, user personalization, and grounding
language plans into executable skills in cluttered homes. We introduce MARS - a Multi-
Agent Robotic System powered by MLLMs for assistive intelligence and designed for smart home
robots supporting people with disabilities. The system integrates four
agents: a visual perception
agent for extracting semantic and spatial features from environment images, a risk assessment
agent for identifying and prioritizing hazards, a planning
agent for generating executable action sequences, and an evaluation
agent for iterative optimization. By combining
multimodal perception with hierarchical multi-
agent decision-making, the framework enables adaptive, risk-aware, and personalized assistance in dynamic indoor environments. Experiments on multiple
datasets demonstrate the superior overall performance of the proposed system in risk-aware planning and coordinated multi-
agent execution compared with state-of-the-art
multimodal models. The proposed approach also highlights the potential of collaborative AI for practical assistive scenarios and provides a generalizable methodology for deploying MLLM-enabled multi-
agent systems in real-world environments.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Model
Wenqi Liang, Gan Sun, Yao He et al. (+5 more)
Vision-
Language-Action models (VLAs) are emerging as powerful tools for learning generalizable visuomotor control policies. However, current VLAs are mostly trained on large-scale image-text-action data and remain limited in two key ways: (i) they struggle with pixel-level scene understanding, and (ii) they rely heavily on textual
prompts, which reduces their flexibility in real-world settings. To address these challenges, we introduce PixelVLA, the first VLA model designed to support both pixel-level
reasoning and
multimodal prompting with text and visual inputs. Our approach is built on a new visuomotor instruction tuning framework that integrates a multiscale pixel-aware encoder with a visual promptaware encoder. To train PixelVLA effectively, we further propose a two-stage automated annotation pipeline that generates Pixel-160K, a large-scale
dataset with pixel-level annotations derived from existing
robot data. Experiments on three standard VLA
benchmarks and two VLA model variants show that PixelVLA improves manipulation success rates by 10.1%-28.7% over OpenVLA, while requiring only 1.5% of its pretraining cost. These results demonstrate that PixelVLA can be integrated into existing VLAs to enable more accurate, efficient, and versatile
robot control in complex environments.
Read the full abstract ▾
All authors Wenqi Liang, Gan Sun, Yao He, Jiahua Dong, Suyan Dai, Ivan Laptev, Salman Khan, Yang Cong
Robotics
Phy-Tac: Toward Human-Like Grasping via Physics-Conditioned Tactile Goals
Shipeng Lyu, Lijie Sheng, Fangyuan Wang et al. (+5 more)
Humans naturally grasp objects with minimal level required force for stability, whereas
robots often rely on rigid, over-squeezing control. To narrow this gap, we propose a human-inspired physics-conditioned tactile method (Phy-Tac) for force-optimal stable grasping (FOSG) that unifies pose selection, tactile prediction, and force regulation. A physics-based pose selector first identifies feasible contact regions with optimal force distribution based on surface geometry. Then, a physics-conditioned latent
diffusion model (Phy-LDM) predicts the tactile imprint under FOSG target. Last, a latent-space LQR controller drives the gripper toward this tactile imprint with minimal actuation, preventing unnecessary compression. Trained on a physics-conditioned tactile
dataset covering diverse objects and contact conditions, the proposed Phy-LDM achieves superior tactile prediction accuracy, while the Phy-Tac outperforms fixed-force and GraspNet-based baselines in grasp stability and force efficiency. Experiments on classical robotic platforms demonstrate force-efficient and adaptive manipulation that bridges the gap between robotic and human grasping.
Read the full abstract ▾
All authors Shipeng Lyu, Lijie Sheng, Fangyuan Wang, Wenyao Zhang, Weiwei Lin, Zhenzhong Jia, David Navarro-Alarcon, Guodong Guo
Robotics · Cross-listed from cs.CV
Discriminately Treating Motion Components Evolves Joint Depth and Ego-Motion Learning
Mengtan Zhang, Zizhan Guo, Hongbo Zhao et al. (+6 more)
Unsupervised learning of depth and ego-motion, two fundamental 3D perception tasks, has made significant strides in recent years. However, most methods treat ego-motion as an auxiliary task, either mixing all motion types or excluding depth-independent rotational motions in supervision. Such designs limit the incorporation of strong
geometric constraints, reducing reliability and robustness under diverse conditions. This study introduces a discriminative treatment of motion components, leveraging the
geometric regularities of their respective rigid flows to benefit both depth and ego-motion estimation. Given consecutive
video frames, network outputs first align the optical axes and imaging planes of the source and target cameras. Optical flows between frames are transformed through these
alignments, and deviations are quantified to impose
geometric constraints individually on each ego-motion component, enabling more targeted refinement. These
alignments further reformulate the joint learning process into coaxial and coplanar forms, where depth and each translation component can be mutually derived through closed-form
geometric relationships, introducing complementary constraints that improve depth robustness. DiMoDE, a general depth and ego-motion joint learning framework incorporating these designs, achieves state-of-the-art performance on multiple public
datasets and a newly collected diverse real-world
dataset, particularly under challenging conditions. Our source code will be publicly available at mias.group/DiMoDE upon publication.
Read the full abstract ▾
All authors Mengtan Zhang, Zizhan Guo, Hongbo Zhao, Yi Feng, Zuyi Xiong, Yue Wang, Shaoyi Du, Hanli Wang, Rui Fan
Robotics · Cross-listed from cs.CV
SE(3)-PoseFlow: Estimating 6D Pose Distributions for Uncertainty-Aware Robotic Manipulation
Yufeng Jin, Niklas Funk, Vignesh Prasad et al. (+4 more)
Object pose estimation is a fundamental problem in robotics and computer
vision, yet it remains challenging due to partial observability, occlusions, and object
symmetries, which inevitably lead to pose ambiguity and multiple hypotheses consistent with the same observation. While deterministic deep networks achieve impressive performance under well-constrained conditions, they are often overconfident and fail to capture the multi-modality of the underlying pose distribution. To address these challenges, we propose a novel probabilistic framework that leverages flow matching on the SE(3)
manifold for estimating 6D object pose distributions. Unlike existing methods that regress a single deterministic output, our approach models the full pose distribution with a sample-based estimate and enables
reasoning about uncertainty in ambiguous cases such as symmetric objects or severe occlusions. We achieve state-of-the-art results on Real275, YCB-V, and LM-O, and demonstrate how our sample-based pose estimates can be leveraged in downstream robotic manipulation tasks such as active perception for disambiguating uncertain viewpoints or guiding grasp synthesis in an uncertainty-aware manner.
Read the full abstract ▾
All authors Yufeng Jin, Niklas Funk, Vignesh Prasad, Zechu Li, Mathias Franzius, Jan Peters, Georgia Chalvatzaki
Robotics
Floor Plan-Guided Visual Navigation Incorporating Depth and Directional Cues
Weiqi Huang, Jiaxin Li, Zan Wang et al. (+3 more)
Current visual
navigation strategies mainly follow an exploration-first and then goal-directed
navigation paradigm. This exploratory phase inevitably compromises the overall efficiency of
navigation. Recent studies propose leveraging floor plans alongside RGB inputs to guide
agents, aiming for rapid
navigation without prior exploration or mapping. Key issues persist despite early successes. The modal gap and content misalignment between floor plans and RGB images necessitate an efficient approach to extract the most salient and complementary features from both for reliable
navigation. Here, we propose GlocDiff, a novel framework that employs a
diffusion-based policy to continuously predict future waypoints. This policy is conditioned on two complementary information streams: (1) local depth cues derived from the current RGB observation, and (2) global directional guidance extracted from the floor plan. The former handles immediate
navigation safety by capturing surrounding geometry, while the latter ensures goal-directed efficiency by offering definitive directional cues. Extensive evaluations on the FloNa
benchmark demonstrate that GlocDiff achieves superior efficiency and effectiveness. Furthermore, its successful deployment in real-world scenarios underscores its strong potential for broad practical application.
Read the full abstract ▾
All authors Weiqi Huang, Jiaxin Li, Zan Wang, Huijun Di, Wei Liang, Zhu Yang
Robotics
MO-SeGMan: Rearrangement Planning Framework for Multi Objective Sequential and Guided Manipulation in Constrained Environments
Cankut Bora Tuncer, Marc Toussaint and Ozgur S. Oguz
In this work, we introduce MO-SeGMan, a Multi-Objective Sequential and Guided Manipulation planner for highly constrained rearrangement problems. MO-SeGMan generates object placement sequences that minimize both replanning per object and
robot travel distance while preserving critical dependency structures with a lazy evaluation method. To address highly cluttered, non-monotone scenarios, we propose a Selective Guided Forward Search (SGFS) that efficiently relocates only critical obstacles and to feasible relocation points. Furthermore, we adopt a refinement method for adaptive subgoal selection to eliminate unnecessary pick-and-place actions, thereby improving overall solution quality. Extensive evaluations on nine
benchmark rearrangement tasks demonstrate that MO-SeGMan generates feasible motion plans in all cases, consistently achieving faster solution times and superior solution quality compared to the baselines. These results highlight the robustness and scalability of the proposed framework for complex rearrangement planning problems.
Read the full abstract ▾
Robotics
AERMANI-VLM: Structured Prompting and Reasoning for Aerial Manipulation with Vision Language Models
Sarthak Mishra, Rishabh Dev Yadav, Avirup Das et al. (+3 more)
The rapid progress of
vision–
language models (VLMs) has sparked growing interest in robotic control, where natural
language can express the operation goals while visual feedback links perception to action. However, directly deploying VLM-driven policies on aerial manipulators remains unsafe and unreliable since the generated actions are often inconsistent,
hallucination-prone, and dynamically infeasible for flight. In this work, we present AERMANI-VLM, the first framework to adapt pretrained VLMs for aerial manipulation by separating high-level
reasoning from low-level control, without any task-specific fine-tuning. Our framework encodes natural
language instructions, task context, and safety constraints into a structured
prompt that guides the model to generate a step-by-step
reasoning trace in natural
language. This
reasoning output is used to select from a predefined library of discrete, flight-safe skills, ensuring interpretable and temporally consistent execution. By decoupling symbolic
reasoning from physical action, AERMANI-VLM mitigates hallucinated commands and prevents unsafe behavior, enabling robust task completion. We validate the framework in both simulation and hardware on diverse multi-step pick-and-place tasks, demonstrating strong generalization to previously unseen commands, objects, and environments.
Read the full abstract ▾
All authors Sarthak Mishra, Rishabh Dev Yadav, Avirup Das, Saksham Gupta, Wei Pan, Spandan Roy
Robotics
Designing for Distributed Heterogeneous Modularity: On Software Architecture and Deployment of MoonBots
Elian Neppel, Shamistan Karimov, Ashutosh Mishra et al. (+5 more)
This paper presents the software architecture and deployment strategy behind the MoonBot platform: a modular space robotic system composed of heterogeneous components distributed across multiple computers, networks and ultimately celestial bodies. We introduce a principled approach to distributed, heterogeneous modularity, extending modular robotics beyond physical reconfiguration to software, communication and orchestration. We detail the architecture of our system that integrates component-based design, a data-oriented communication model using ROS2 and Zenoh, and a deployment orchestrator capable of managing complex multi-module assemblies. These abstractions enable dynamic reconfiguration, decentralized control, and seamless collaboration between numerous operators and modules. At the heart of this system lies our open-source Motion Stack software, validated by months of field deployment with self-assembling
robots, inter-
robot cooperation, and remote operation. Our architecture tackles the significant hurdles of modular robotics by significantly reducing integration and maintenance overhead, while remaining scalable and robust. Although tested with space in mind, we propose generalizable patterns for designing robotic systems that must scale across time, hardware, teams and operational environments.
Read the full abstract ▾
All authors Elian Neppel, Shamistan Karimov, Ashutosh Mishra, Gustavo Hernan Diaz Huenupan, Hazal Gozbasi, Kentaro Uno, Shreya Santra, Kazuya Yoshida