Robotics
LabInstruct: Benchmarking Situated Instructional Video Generation for Lab Procedures
Yuming Fu, Weijia Wu, Jing Chen et al. (+5 more)
Self-driving laboratories (SDLs) aim to automate the full experimental loop, from scientific decision-making to physical execution. Ideally, AI-generated plans could be carried out directly by robotic systems, but reliable automation remains difficult in complex, open-world laboratory environments, where experiments often involve fine-grained manipulation, long-horizon procedures, and substantial variation across tasks and setups. Humans therefore remain an important execution interface between AI-generated plans and physical experiments, creating a need for clear and effective human-facing experimental guidance. Because laboratory procedures are inherently visual, spatial, and dynamic,
video is particularly well suited to communicating apparatus configurations, manipulation actions, temporal dependencies, and state changes. Recent advances in
video generation now make it possible to synthesize experimental demonstrations directly from an initial workspace image and a natural-
language instruction. However, whether such models can reliably communicate real laboratory procedures has not been systematically studied. We introduce LabInstruct, a
benchmark for situated instructional
video generation in real laboratories. LabInstruct contains 204 tasks across 5 scientific disciplines, with real reference executions and structured annotations of objects, actions, contacts, and state transitions. Evaluating 8 frontier image-to-
video models, we find that visually plausible generations frequently remain procedurally incorrect, revealing a substantial gap between visual realism and the reliability required for experimental instruction.
Read the full abstract ▾
All authors Yuming Fu, Weijia Wu, Jing Chen, Jiahao Tang, Feifei Chen, Hongyu Zhu, Xin Jin, Alex Jinpeng Wang
Robotics
RoboHarness: A Simple Harness Outperforms VLA and World Actions Models
Bince Qu, Wei Chen and Bo Zhang
The rise of large
language models (LLMs) has opened a path toward generalist embodied AI, raising the question of how LLMs can act in the physical world. One straightforward approach is training the model to generate actions: collect
robot-action data and
video demonstration, refine LLM architecture with an action head, and train a
vision-
language-action (VLA) model at scale. However, we argue that coding
agents have the potential to realize generalist embodied manipulation. All they need is a simple yet effective
robot harness. We present RoboHarness: a robotic harness that gives LLM
agents a visual-
geometric control panel so that they can directly understand and invoke embodied tasks. With RoboHarness, Qwen3.8-Flash-Next outperforms state-of-the-art VLA models on BEHAVIOR Challenge 2025 tasks. The implementation will be released as open source.
Read the full abstract ▾
Robotics · Cross-listed from cond-mat.soft
Tactile Perception through Fluid-Solid Interaction
Arman Goshtasbi, Minke Berghuis, Aida Parvaresh et al. (+3 more)
Soft tactile sensors elevate robotic touch through enhanced flexibility and adaptability, yet most existing designs depend on embedded electronics that are susceptible to interference and environmental limitations. In this work, we leverage fluid-solid interactions to develop a class of soft tactile sensors that operate entirely without electronics at the sensing site. The sensor comprises a fluid-filled elastomeric channel connected to only two external pressure sensors. Touching different regions of the elastomeric surface displaces the viscous fluid, producing distinct pressure patterns that encode both touch position and force. These signals are decoded through a machine learning framework that integrates feature extraction, soft clustering, and adaptive neuro-fuzzy inference to achieve accurate localization and force estimation. We validate this concept through single-point touch localization and force estimation in a linear (1D) sensor and extend the same sensing principle to 2D tactile mapping by routing the channel across the surface using space-filling curves, while maintaining the same minimal hardware setup. This simple approach remains effective in environments where conventional electronic sensors often fail, such as underwater or in the presence of
magnetic interference.
Read the full abstract ▾
All authors Arman Goshtasbi, Minke Berghuis, Aida Parvaresh, Saravana Prashanth Murali Babu, Robert W. Style, Ahmad Rafsanjani
Robotics
Retrospective Open-Vocabulary Memory for Long-Term Object Search
Jiaming Wang, Zhiwei Xue, Chen Jizhuo et al. (+2 more)
Long-term object search requires learning where objects usually appear from repeated but uneven observations of a changing environment. We formulate retrospective open-vocabulary
memory as probabilistic inference from censored observations, where the key idea is to reason with evidence per opportunity: a detection or non-detection should influence belief only in proportion to the
robot's opportunity to observe the corresponding location. We introduce ECROM, which uses this principle to estimate long-term prevalence for concepts specified only at query time and converts the resulting belief directly into an active-search prior. To evaluate this problem, we introduce a controlled long-term
benchmark in ten HM3D homes that independently varies object placement and observation opportunity across repeated traversals. ECROM improves support-level AP on held-out queries by 4.5 points and search SPL by 4.2 points over the strongest competing
memory in each metric. The
benchmark,
dataset, and code will be open-sourced. Project page: https://jiaming.im/ecrom/
Read the full abstract ▾
All authors Jiaming Wang, Zhiwei Xue, Chen Jizhuo, Peng Shiqi, Harold Soh
Robotics · Cross-listed from cs.CV
EgoRefine: Ego-Referenced Predictive Alignment and Trajectory-Conditioned Reliability-Aware Fusion for Asynchronous Collaborative Perception
Lingzhao Kong, Yongsheng Zang, Yu Kang et al. (+4 more)
Collaborative perception enables connected
agents to share complementary observations for 3D object detection, extending sensing range and mitigating occlusion. Under asynchronous communication, however, cooperative features arrive with temporal delay. Existing prediction-based methods compensate for these features mainly from the transmitting
agent's own history, leaving residual misalignment with the ego
agent's current observation; subsequent fusion also often overlooks spatial variations in
alignment quality. We propose EgoRefine, an ego-referenced predictive
alignment and reliability-aware fusion framework for asynchronous collaborative perception. Its Ego-referenced Predictive
Alignment module uses the current ego feature to guide cooperative trajectory-field prediction and refines the sampling offsets along an ego-referenced trajectory direction. Its Trajectory-conditioned Reliability-aware Fusion module treats the trajectory discrepancy between the ego and cooperative streams and the directional refinement magnitude as
alignment cues, using them to condition the relation between aligned features and adaptively reweight the two streams before convolutional fusion. Experiments on V2V4Real and DAIR-V2X-Seq show that EgoRefine outperforms TraF-Align by 1.6 and 2.9 points on average in AP@0.5 and AP@0.7, respectively. The source code will be made publicly available at https://github.com/godk0509/EgoRefine.
Read the full abstract ▾
All authors Lingzhao Kong, Yongsheng Zang, Yu Kang, Kailun Yang, Jie Fu, Yukun Zuo, Zhiyong Li
Robotics
DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies
Youngjun Jun, Kyumin Choi, Youngmin Kim et al. (+4 more)
Vision-
Language-Action (VLA) models increasingly rely on action experts that generate short action chunks under receding-horizon control. While chunk-level training is convenient across
robot embodiments, it optimizes local action likelihood without explicitly accounting for long-horizon task success. Sequence-level reinforcement learning can address this limitation, but typically requires policy rollouts and closed-loop interaction, which are costly for real-
robot manipulation. We introduce DriftOPD, a teacher-free, rollout-free framework for sequence-level on-policy
distillation of continuous VLA action experts. We show that the sequence-level reverse Kullback-Leibler (KL) divergence decomposes into a chunk-level reverse-KL term and a future-potential term that captures the long-horizon effect of the current action. DriftOPD optimizes these two terms using a one-step drifting objective and a Q-function critic learned from offline demonstrations, respectively, enabling sequence-level optimization with only offline data and one-step action generation. Across multiple VLA architectures in simulation and real-world manipulation, DriftOPD generally outperforms existing one-step
distillation baselines while achieving task success performance comparable to multi-step teacher policies. These results demonstrate that long-horizon behavior can be effectively distilled into one-step VLA action experts without online interaction or a separate teacher.
Read the full abstract ▾
All authors Youngjun Jun, Kyumin Choi, Youngmin Kim, Seonghyun Jin, Sunwoo Park, Jangho Park, Jong Chul Ye
Robotics
TERRA: Terrain-Aware Reconstruction, Retargeting and Control for Musculoskeletal Locomotion
Merkourios Simos, Chengkun Li, Bianca Ziliotto et al. (+1 more)
Recent advances in musculoskeletal modeling and reinforcement learning have enabled muscle-actuated
agents to reproduce increasingly complex human motions. Yet these capabilities remain largely confined to flat ground, in part because motion
datasets rarely include aligned terrain geometry and because retargeting terrain interactions to complex musculoskeletal bodies is challenging. We present TERRA, an end-to-end pipeline for terrain-aware retargeting and control of musculoskeletal locomotion. From kinematic trajectories alone, TERRA combines terrain priors, estimated contacts, and negative free-space evidence to recover task-relevant support geometry. TERRA further considers anatomical, tendon-continuity, and contact constraints during retargeting. Using the resulting motion-terrain pairs from five
datasets, we successfully train a single muscle-actuated control policy on 9.4 hours of diverse locomotion. Across reconstruction, retargeting, and held-out tracking
benchmarks, TERRA improves terrain accuracy, sharply reduces anatomical and interaction violations, and achieves the highest observed completion rate over supported terrain families. Overall, TERRA provides a practical route from scene-less motion data to muscle-actuated locomotion over diverse non-flat terrain. Project website: https://cnai.epfl.ch/terra/
Read the full abstract ▾
All authors Merkourios Simos, Chengkun Li, Bianca Ziliotto, Alexander Mathis
Robotics · Cross-listed from cs.CV
Kai Yan, Xiangyu Chen, Yulong Cao et al. (+9 more)
Vision-
Language-Action (VLA) foundation models have recently emerged as one of the prevailing solutions for autonomous driving, as they can utilize knowledge acquired during
vision-
language pretraining for accurate and interpretable driving. However, VLAs can take only a limited number of frames as visual input due to the high
token cost of an image, which is problematic for
memory-dependent tasks such as determining the arrival order at all-way stops and long-horizon driving scene understanding. Existing solutions use latent vector
memories accessed through cross-
attention, which are neither interpretable nor portable. In this paper, we propose AD-Memo, a general-purpose VLA driving
agent with
language-based
memory. The
agent outputs
memory as an extension of its Chain-of-Thought (CoT) to record surrounding objects critical to driving; this
memory becomes part of the
agent's future input. We curate
memory-based
datasets and train VLAs with a two-stage recipe: Supervised Fine-Tuning (SFT) and Da Capo, a novel semi-closed-loop Reinforcement Learning (RL)
algorithm which uses trajectory-level advantage for
memory and step-level advantage for driving, leading to better credit assignment. Across scenarios such as all-way stops and general driving, AD-Memo improves driving quality, enables better question answering on driving scenes, and produces plug-and-play
memory for other models.
Read the full abstract ▾
All authors Kai Yan, Xiangyu Chen, Yulong Cao, Alex Naumann, Peter Karkus, Yan Wang, Jef Packer, Alex Schwing, Yuxiong Wang, Boris Ivanovic, Wenjie Luo, Marco Pavone
Robotics
Yggdrasil: a Layer-First 3D Scene Graph for Real-Time Querying
Arshia Akhavan, Ermanno Bartoli, Afnan Algharbi et al. (+3 more)
Robotic
agents use 3D scene graphs (3DSG) to perform tasks ranging from scene understanding to scene interaction. Although an extensive body of work addresses scene graph generation, little
attention has been paid to optimizing the graph for consumption, which leaves state-of-the-art perception pipelines to work around their own scene graph and to pay a latency cost that does not fit the real-time budget a perception loop runs on. We present Yggdrasil, the first 3D scene graph designed to be efficient for both generation and consumption: a layer-first hierarchical graph built from generic nodes, edges, and layers, which expresses the representations existing pipelines already produce, indoor or outdoor, flat or hierarchical, while natively answering the positional and semantic queries downstream tasks issue. Against a published DSG baseline, Yggdrasil answers queries up to $121\times$ faster, and every query we measure falls between 2 and 127 microseconds, three to five orders of magnitude inside the 200 microsecond keyframe budget a 3DSG consumer lives in, on both a workstation and embedded class device. We integrate Yggdrasil into three published pipelines spanning human trajectory prediction, object-goal navigation, and human-aware motion planning, where it removes up to 99% of the time each spends on its scene graph. The implementation,
benchmark harness, and all three integrations are available online.
Read the full abstract ▾
All authors Arshia Akhavan, Ermanno Bartoli, Afnan Algharbi, Alireza Hoseinpur, Iolanda Leite, Bryan Donyanavard
Robotics · Cross-listed from cs.CV
HIGS: Hierarchical Implicit Grids for Joint Geometric and Semantic Scene Understanding
Hanwen Cao, Wenqiang Wu, Kuang-Ting Tu et al. (+5 more)
Neural implicit representations have had a significant impact on scene reconstruction by enabling
robots to build continuous, differentiable, and high-fidelity 3D maps. Most existing works focus on
geometric reconstruction and lack semantic information for high-level spatial understanding and task planning. Also, as the scale and complexity of the environment increase, neural representations face the challenge of maintaining computational efficiency in back-end optimization. To resolve these two challenges, we introduce a hierarchical neural field that leverages multiresolution submaps to achieve an efficient and scalable implicit representation, and a unified query and decoding mechanism to support both
geometric and semantic features. More specifically, the learnable map features can be converted to the output with the query and decoding process for both training and inference. For large-scale representation, we decompose a scene into overlapping submaps and do hierarchical optimization within each local submap, thus enabling scalable computation. To further improve efficiency, we design feature encoders that predict initial hierarchical grid features to substantially reduce the time needed to optimize the submap features from scratch. To correct estimation drift among submaps, we align and fuse them entirely within the implicit feature space, leading to substantial acceleration by avoiding the need to decode the final output. Building upon this efficient hierarchical representation, we embed both
geometric features and
vision-
language latent features into the map, and demonstrate it on both Signed Distance Field (SDF) construction and open-vocabulary object grounding. Our approach significantly improves computation and
memory efficiency, maintains high estimation accuracy, and endows the
robot with spatial awareness on large-scale real-world
benchmarks.
Read the full abstract ▾
All authors Hanwen Cao, Wenqiang Wu, Kuang-Ting Tu, Mathias Otnes, Jeffrey Delmerico, Rui Wang, Yulun Tian, Nikolay Atanasov
Robotics
Dense Temporal Motion Retargeting for Legged Robots
Jaeryeong Kim, Taerim Yoon, Jin Cheng et al. (+2 more)
Legged
robots can learn expressive whole-body skills from the motions of humans and animals. Due to the morphology gap between the source and the
robot, however, the motion must be tailored to the dynamic properties of the
robot. In particular, dynamic motions such as a jump require careful adjustment, since their timing and control are interdependent. We propose dense temporal motion retargeting (DTMR), which jointly optimizes timing and control within a single optimization, where dense means that the timing is adjusted for every control step. This dense formulation enables DTMR to deform only the parts of the motion that need a change in timing. The problem is solved with sampling-based model predictive control (MPC) in parallel on a GPU. We evaluate DTMR against baselines on two hours of human motion with four humanoid
robots, where the results show that DTMR outperforms baseline methods, particularly on dynamic motions. We also show that allowing more temporal deformation yields more precise retargeting. We further compare DTMR with a baseline that optimizes the temporal dimension, where the result shows that DTMR retargets more precisely under the same deformation budget while being ~19x faster. Lastly, policies trained on our references transfer to a real humanoid
robot.
Read the full abstract ▾
All authors Jaeryeong Kim, Taerim Yoon, Jin Cheng, Sungjoon Choi, Stelian Coros
Robotics
Correcting WHERE, Preserving HOW: Compositional Generalization for Vision-Language-Action Models via Referential Guidance
Yanyan Zhang, Disheng Liu, Xinpeng Li et al. (+8 more)
While
Vision-
Language-Action (VLA) models enable flexible action generation, their generalization across diverse environmental elements, including manipulated objects, destinations, and backgrounds, is limited by the lack of diversity in robotic training data. Trained end-to-end on such data, VLAs tend to exploit visual shortcuts, associating actions with task-irrelevant visual features rather than the intended task semantics. These shortcuts block recomposition of elements already seen by the policy, that is, compositional generalization. Existing approaches mitigate such
entanglement through task-relevant perception or targeted data diversification, but offer no explicit mechanism for unseen recomposition and require backbone-specific modifications with retraining. We observe that under such recomposition, VLAs often fail at global grounding while retaining local manipulation skills that recover near the correct target in familiar configurations. Therefore, we propose Referential Guidance (ReGuide), a training-free wrapper that, given object poses from a grounding module, combines semantic and
geometric rebinding to guide the end-effector into demonstration-supported configurations of the instructed referent, where the frozen policy can resume execution. Experiments in simulation across multiple VLA backbones as well as on a real
robot show that ReGuide improves success rates under compositional shifts by up to 56.8 and 75.0 percentage points, respectively, while preserving standard-task performance.
Read the full abstract ▾
All authors Yanyan Zhang, Disheng Liu, Xinpeng Li, Chaoda Song, Mohsen Hariri, Debargha Ganguly, Wang Yang, Kai Ye, Bryce Grant, Vipin Chaudhary, Yu Yin
Robotics
FAST-Sync: Fast Group Synchronization for any Matrix Lie Group
Shane Holmes, Yiran Luo, Firat Taxpulat et al. (+2 more)
Group synchronization (GS) is the problem of estimating a set of $N$ unknown elements $g_1,\ldots, g_N \in \mathcal{G}$ in a group $\mathcal{G}$, given noisy measurements of a subset of their pairwise ratios $g_i^{-1} g_j$. GS problems lie at the core of many state estimation tasks in robotics and computer
vision, including 3D
vision, robotic mapping, inertial navigation, and molecular reconstruction. Unfortunately, GS problems are typically both high-dimensional and non-convex, and therefore hard to solve in general. In this paper, we present Fast-Sync, a fast linear approximation method for GS that is suitable for initializing local
manifold-based optimizers or certifiable global methods. Our approach generalizes chordal initialization to arbitrary
matrix Lie groups, and additionally proposes two new key algorithmic enhancements: we show how to exploit both the Kronecker-product structure in the problem data
matrix and the
topology of the synchronization graph to improve speed, scalability, and accuracy. Experimental evaluation across several GS tasks demonstrates that Fast-Sync provides high-quality initializations that enable local optimizers to efficiently recover globally optimal GS solutions, achieving high success rates even with considerable measurement noise.
Read the full abstract ▾
All authors Shane Holmes, Yiran Luo, Firat Taxpulat, David M. Rosen, Frank Dellaert
Robotics
Drone Soccer: Learning to Manipulate with Multicopter Downwash
Neelay Joglekar, Bavin Saravanan, Yutong Wang et al. (+3 more)
Although multicopter drones are traditionally designed for "perception-only" tasks, like mapping and exploration, recent work has sought to develop Unmanned Aerial Manipulators (UAMs) to solve mobile manipulation tasks. Aerial manipulation performance can be impacted by "downwash," the airflow produced by propellers, but current state-of-the-art UAMs either ignore downwash or treat it as a disturbance. Instead, is it possible to actively use downwash as a tool during manipulation? We design a drone soccer task to explore the feasibility of downwash-based manipulation. Specifically, we develop a simplified downwash dynamics model which we use to train an RL policy to dribble a soccer ball. We further demonstrate that our policy transfers to real world deployment. This work provides key insights into novel manipulation capabilities for multicopters.
Read the full abstract ▾
All authors Neelay Joglekar, Bavin Saravanan, Yutong Wang, Varun Kandiyappan, Junyi Geng, Sebastian Scherer
Robotics
Data-Efficient Adaptation of a Driving VLA to Class 8 Trucks
Satyajeet Das, Aaron Buxbaum, Niels Joubert et al. (+1 more)
Class 8 trucks differ from passenger cars in geometry, dynamics, and maneuvering requirements. As a result,
vision-
language-action (VLA) models trained for passenger vehicles do not readily transfer to Class 8 trucks, particularly in unstructured scenarios such as accident scenes and construction zones. Rather than training a truck-driving VLA from scratch, we propose an adapt-then-steer strategy that adapts an off-the-shelf VLA to generate trajectories for Class-8 trucks in these challenging scenarios. In the adapt stage, we use NVIDIA's Alpamayo 1.5 as the base model, fine-tuning only its action-generation stack on a few hundred real-world construction and accident-related highway scenarios. In the steer stage, we introduce Flow Velocity Steering (FVS) to further refine the model's predictions while holding the adapted VLA fixed. FVS is a compact, flow-time-conditioned residual module that adds learned corrections to the action-space flow velocity used to update the action sequence at each generation step. In open-loop evaluation on a scenario-disjoint held-out set, targeted fine-tuning more than halves single-candidate average displacement error (ADE) and final displacement error (FDE) over the entire 6.4 s horizon compared to the base model. Using the same targeted demonstrations, FVS further reduces the fine-tuned model's full-horizon ADE and FDE by 13.9% and 16.5%, respectively. At matched data budgets, targeted supervision yields 19-26% lower full-horizon ADE than general truck-driving supervision, while the targeted model remains competitive with a model fine-tuned on approximately 65 times as many general truck-driving scenarios. These results support adapt-then-steer for data-efficient vehicle-domain transfer to Class 8 trucks. Our project website is available at https://truckvla.github.io.
Read the full abstract ▾
All authors Satyajeet Das, Aaron Buxbaum, Niels Joubert, Gaurav S. Sukhatme
Robotics
Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies
Galbot Team: Xuchuan Chen, Xiaoqian Cheng, Yu Deng et al. (+30 more)
GPT-6 Astra exhibits a remarkable ability to generate numerical
robot actions, extending its role beyond high-level planning. To assess Astra's capabilities as general-purpose embodied policies, we conduct comprehensive evaluations across six domains, examining direct control, cooperation with learned policies, and feedback-driven adaptation. In gripper manipulation, Astra can correct task targets and prepare contact conditions for subsequent policy execution; hybrid control with {\pi}0.5 achieves 48% success on the evaluated RoboDojo subset. In dexterous manipulation, hybrid control achieves 50% success in ten experience-guided DexJoCo trials, while direct in-hand control struggles to coordinate finger contacts. In mobile manipulation, hybrid control reaches 38.7% success on the evaluated RoboCasa365. In navigation, Astra leads our local comparisons, reaching 92% success on RxR instruction following and 82% on HM3D object search, although search incurs substantial detours. In locomotion, dense motion-reference generation remains unreliable: none of five sequential attempts on a single obstacle course reaches the goal, despite improvements in stability and forward progress. In humanoid loco-manipulation, Astra exceeds baseline methods on 13 of 30 HumanoidBench tasks with pretrained whole-body controllers. These findings reveal a gap between useful task decisions and reliable physical control. Inference latency further constrains practical control: across 50 RoboDojo instances per condition, policy-assisted and direct control consume 624.8 million and 1.132 billion
tokens. A 30-second locomotion run requires 250 model calls averaging 39.86 seconds each, with physics paused during inference.
Read the full abstract ▾
All authors Galbot Team: Xuchuan Chen, Xiaoqian Cheng, Yu Deng, Lihe Ding, Shaocong Dong, Xiangjun Gao, Haozhe Jia, Zekai Li, Zhoujian Li, Yunrui Lian, Sikai Liang, Chenghuai Lin, Dairu Liu, Jiahang Liu, Qingtao Liu, Yuxuan Ma, Zekun Qi, Jiayi Su, He Wang, Ruochen Xu, Tianyu Xu, Xudong Xu, Zhe Xu, Mi Yan, Siming Yan, Li Yi, Ruixi Yu, Jinlu Zhang, Yintianrun Zhang, Zhikai Zhang, Zhizheng Zhang, Yixin Zheng, Weiyi Zhu
Robotics
Behavioral Persistence and Incomplete Functional Transfer of Co-evolved Communication in Evolutionary Robotics
Fernando Montes-Gonzalez
This work evaluates the direct transfer of a co-evolved communication protocol from a 2D simulation to a 3D physical environment, without retraining the network weights. Two e-puck-type
robots, controlled by a GRU network with residual connection, were evaluated in a food-seeking task with social signaling. The sensory and motor translation layer required three corrections for stable physical operation, including the calibration of a hunger term based on a measurable asymmetry in the trained residual weights. Even with these corrections, the transfer was partial and asymmetric: one
agent reached the food source in one of thirty tested seeds, while the other did not reach it in any. Task success was measured by both
agents reaching the food area. An additional experiment incorporating explicit directional information in the social channel produced observable changes in the trajectory of the receiving
agent and improvements in several specific cases. However, these improvements were not enough to allow the second
agent to reach the food source, suggesting that the limitation may not be explained solely by signal translation, but also by the ability to navigate under the new physical constraints. The results suggest that successful transfer of emergent communication may depend not only on preserving the signaling process itself, but also on preserving the ecological and navigational conditions under which the protocol evolved.
Read the full abstract ▾
Robotics
Onboard Vision and MPC Navigation for Underwater Robots: An Open BlueROV2 Platform for Multi-Robot Experiments & Docking
Victor Nan Fernandez-Ayala, Wiktor Kowalczyk, Cezary Banaszek et al. (+1 more)
Autonomous underwater
robots require robust perception, estimation and control to operate in confined environments. This paper presents an open-source BlueROV2 platform combining onboard
vision with nonlinear Model Predictive Control (NMPC) for autonomous navigation and docking. The platform integrates an NVIDIA Jetson Orin NX and an Intel RealSense D435i stereo camera in a modular pressure housing. Underwater-calibrated stereo depth and realtime object detection provide relative position measurements of nearby BlueROV2 vehicles in the camera and body frames. A quaternion-based
estimator fuses external pose and inertial measurements, while an NMPC controller based on a nonlinear six-degree-of-freedom model tracks planned navigation and docking trajectories. To support reproducible development, we also provide open-source physics-based PX4 SITL and Gazebo environments, multi-
robot simulation tools and a lowcost physical docking station. Experiments evaluate underwater perception, onboard computational performance, state estimation, trajectory tracking and autonomous docking.
Read the full abstract ▾
All authors Victor Nan Fernandez-Ayala, Wiktor Kowalczyk, Cezary Banaszek, Dimos V. Dimarogonas
Robotics
What to Attend, What to Keep: Skill-Conditioned Visuotactile Representation with Progress-Guided Event Memory
Amir-Hossein Shahidzadeh, Seungjae Lee, Eadom Dessalene et al. (+5 more)
Robotic manipulation integrates
vision, touch, and
language, whose importance shifts across stages:
vision guides reaching, while touch, through its
evolution over time, decides grasping,
alignment, and contact. Yet existing multi-modal manipulation policies typically use fixed temporal contexts and fusion strategies, despite shifts in what each modality contributes across different skills. We study how
vision and touch should be combined at the level of primitive skills, asking what each skill needs from each sensor, and propose a skill-conditioned representation in which the queried skill conditions fusion over modality-specific short-term observation
tokens while attending to a sparse event
memory that retains terminal observations from the last $K$ executed skills. Evaluated by skill progress estimation on three contact-rich tasks, it reduces slip-detection delay by 87% against fine-tuned SOTA progress models, twist-completion delay by 67.5% against a
vision-only ablation, and progress error on a blind search task by 92% through sparse event
memory. Gains concentrate exactly where completion is defined by contact or task history. More broadly, our results suggest that observation formation not only policy architecture is a central challenge in multi-modal representation. Project Website: http://what-to-attend-what-to-keep.github.io/
Read the full abstract ▾
All authors Amir-Hossein Shahidzadeh, Seungjae Lee, Eadom Dessalene, Shanthosh Raaj Mohanram Mageswari, Soroush Etemad, Furong Huang, Cornelia Fermüller, Yiannis Aloimonos
Robotics
CADeT: Causal-Aware Deformation Transmission for Indirect Robotic Manipulation of Soft Tissue
Junlei Hu, Dominic Jones and Pietro Valdastri
Indirect manipulation of deep-seated deformable anatomy inaccessible to the
robot is challenging in
robot-assisted minimally invasive surgery (RAMIS) because intervening tissues spatially filter deformation transmission. Passive observations can be ambiguous because the Decoupled and Blocked modes may produce similar motion responses. We propose CADeT, a causal-aware deformation transmission framework that integrates structural causal model (SCM) with active sensing to infer a latent transmission mode and estimate a state-dependent adhesion Jacobian online. During normal manipulation, control actions update the mode belief; when ambiguity persists, an additional probing action is selected to improve mode distinguishability. The mode belief and learned Jacobian are incorporated into a belief-aware model predictive controller for indirect target-shape control. Validation in simulation and on the da Vinci research kit (dVRK), using phantom and ex vivo porcine tissues, shows higher mode-identification accuracy and faster shape-error convergence than the evaluated model-free and model-based baselines. These results show that active sensing improves mode identification and indirect deformation control under the evaluated conditions.
Read the full abstract ▾
Robotics
Diffusion-2BC: Hybrid Diffusion and Regression Training for Offline Behavior Cloning in Autonomous Driving
Bruno Maciel Machado and Eric Aislan Antonelo
Behavior cloning provides an offline route to autonomous-driving policy learning, but mean-squared-error regression is poorly matched to demonstrations in which one observation admits several valid actions.
Diffusion policies can represent conditional
multimodal action distributions, yet their closed-loop performance may be unstable when visual features and control are learned from limited data. This paper presents
Diffusion-2BC, which combines a
diffusion denoising objective with an auxiliary deterministic behavior-cloning loss over a shared visual encoder. The auxiliary branch is used only during training; inference remains
diffusion-based. The proposed method is evaluated in the controlled Claw environment and in bird's-eye-view CARLA navigation, including route-conditioned driving, route-free navigation through multiple intersections, and cross-map evaluation from Town01 to Town02. In the Claw task,
Diffusion-2BC reduced the mean mask-distance error by approximately 10% relative to a
diffusion-based behavior-cloning baseline and by 85% relative to standard deterministic behavior cloning. In route-free CARLA,
Diffusion-2BC traveled substantially farther before termination under the evaluation protocol than both baselines in Town01 and Town02. Additional qualitative rollouts revealed distinct route choices, showing the
multimodal behavior of the proposed
diffusion-based
agent. The results indicate that an auxiliary regression signal can improve the closed-loop reliability of
diffusion behavior cloning while preserving
multimodal prediction in the controlled
benchmark.
Read the full abstract ▾