Robotics
Interdisciplinary Workshop on Mechanical Intelligence: Summary Report
Victoria A. Webster-Wood, Nicholas Gravish, Amir Alavi et al. (+37 more)
This report provides a summary of the outcomes of the Interdisciplinary Workshop on Mechanical Intelligence held in 2024. Mechanical Intelligence (MI) represents the phenomenon that novel structural features of material/biological/robotic systems can encode intelligence through responsiveness, adaptivity,
memory, and learning in the mechanical structure itself. This is in contrast to computational intelligence, wherein the intelligence functions occur through electrical signaling and computer code. The two-day workshop was held at NSF headquarters on May 30-31 and included 38 invited academic researcher participants, and 8 program officers from the NSF. The workshop was structured around active small and large group discussions in groups of 4-5 and 9-10 with the goal of addressing topical questions on MI. Working groups entered notes into shared presentation slides for each discussion session and presented their outcomes in a final presentation on the last day. Here we summarize the overall outcomes of the workshop.
Read the full abstract ▾
All authors Victoria A. Webster-Wood, Nicholas Gravish, Amir Alavi, Andres F Arrieta, Sarah Bergbreiter, Anthony Bloch, Laura Blumenschein, C. Chase Cao, Aja Mia Carter, Paolo Celli, Tony Chen, Margaret Coad, Mark Cutkosky, Michael Dickey, Brian Do, Robert Full, Mahdi Haghshenas-Jaryani, Kaushik Jayaram, Aaron Johnson, Eva Kanso, Emma Lejeune, Chen Li, Suyi Li, Jeffrey Lipton, Rob MacCurdy, Matt McHenry, Jean-Michel Mongeau, Todd Murphey, Mark Plecnik, Jordan Raney, Ryan D. Sochol, Hannah Stuart, Zeynep Temel, Michael Tolley, Barry Trimmer, T.J. Wallin, Kon-Well Wang, Wenzhong Yan, Mark Yim, Wenlong Zhang
Robotics
ReMemNav: Memory-Based Decision Correction and Target Verification for Zero-Shot Object Navigation
Feng Wu, Wei Zuo, Wenliang Yang et al. (+3 more)
Zero-shot object
navigation requires
agents to locate unseen targets in unfamiliar environments without prior maps or task-specific training. Despite the commonsense
reasoning ability of
vision-
language models (VLMs), existing mapless navigators often suffer from repeated exploration and premature stopping caused by limited historical context and false-positive target predictions. We propose ReMemNav, a training-free framework that combines lightweight semantic grounding,
memory-based decision correction, and target verification. RAM-derived semantic priors guide panoramic direction selection, while a geometry-triggered correction mechanism retrieves historical scene descriptions when the selected direction points toward previously visited regions. When a target is predicted to be visible, ReMemNav verifies the corresponding single-view observation before the final approach. Depth-based action sampling then converts the high-level decision into collision-free motion. Experiments on HM3D and MP3D show that ReMemNav achieves higher success rates and path efficiency than existing training-free zero-shot baselines. Specifically, ReMemNav achieves absolute SR/SPL gains of 1.7%/5.3% on HM3D v0.1, 18.2%/11.1% on HM3D v0.2, and 8.7%/6.9% on MP3D.
Read the full abstract ▾
All authors Feng Wu, Wei Zuo, Wenliang Yang, Jun Xiao, Yang Liu, Xinhua Zeng
Robotics · Cross-listed from cs.CV
Towards automatic smoke detector inspection: Recognition of the smoke detectors in industrial facilities and preparation for future drone integration
Lukas Kratochvila, Jakub Stefansky, Simon Bilik et al. (+6 more)
Fire safety consists of a complex pipeline, and it is a very important topic of concern. One of its frontal parts are the smoke detectors, which are supposed to provide an alarm prior to a massive fire appears. As they are often difficult to reach due to high ceilings or problematic locations, an automatic inspection system would be very beneficial as it could allow faster revisions, prevent workers from dangerous work in heights, and make the whole process cheaper. In this study, we present the smoke detector recognition part of the automatic inspection system, which could easily be integrated to the drone system. As part of our research, we compare two popular convolutional-based object detectors YOLOv11 and SSD widely used on embedded devices together with the state-of-the-art
transformer-based RT-DETRv2 with the backbones of different sizes. Due to a complicated way of collecting a sufficient amount of data for training in the real-world environment, we also compare several training strategies using the real and semi-synthetic data together with various augmentation methods. To achieve a robust testing, all models were evaluated on two test
datasets with an expected and difficult appearance of the smoke detectors including motion blur, small resolution, or not complete objects. The best performing detector is the YOLOv11n, which reaches the average mAP@0.5 score of 0.884. Our code, pretrained models and
dataset are publicly available.
Read the full abstract ▾
All authors Lukas Kratochvila, Jakub Stefansky, Simon Bilik, Robert Rous, Tomas Zemcik, Michal Wolny, Frantisek Rusnak, Ondrej Cech, Karel Horak
Robotics
Characterization of Constraints in Flexible Unknown Environments
Samrat Bhattacharyya and Nabil Simaan
This paper presents an online path planning
algorithm for safe autonomous manipulation of a flexibly constrained object in an unknown environment. Methods for real time identification and characterization of perceived flexible constraints and global stiffness are presented. Used in tandem, these methods allow a
robot to simultaneously explore, characterize, and manipulate an elastic system safely.
Navigation without a-priori knowledge of the system is achieved using constraint exploration based on local force and position information. The perceived constraint stiffness is considered at multiple poses along an object's (system) trajectory. Using stiffness eigenvector information, global stiffness behavior is characterized and identified using an atlas of simple mechanical constraints, such as hinges and planar constraints. Validation of these
algorithms is carried out by simulation and experimentally. The ability to recognize several common simple mechanical constraints (such as a flexible hinge) in real time, and to subsequently identify relevant screw parameters is demonstrated. These results suggest the feasibility of simultaneous global constrain/stiffness exploration and safe manipulation of flexibly constrained objects. We believe that this approach will eventually enable safe cooperative manipulation in applications such as organ retraction and manipulation during surgery
Read the full abstract ▾
Robotics
Switchable-Polarity Electropermanent Magnet: Reconfigurable Magnetic Fields for Scalable Fluidic Control
Bingchao Wang, Jonah Mack, Francesco Giorgio-Serchi et al. (+1 more)
Scalable control of pneumatic and fluidic networks remains fundamentally constrained by architectures that require continuous power input, dense external control hardware, and fixed routing
topologies. Current valve arrays rely on such continuous actuation and mechanically fixed routing, imposing substantial thermal and architectural overhead. Here, we introduce the Switchable-polarity Electropermanent Magnet (S-EPM), a bistable
magnetic architecture that spatially redistributes its external
magnetic field through transient electrical excitation. By reconfiguring internal flux pathways within a composite magnet assembly, the S-EPM exchanges reinforced and weakened
magnetic-field regions while maintaining bistable, zero-power state retention. We integrate this architecture into a compact pinch-valve to robustly control pneumatic and
liquid media. This state-encoded
magnetic control enables programmable fluidic networks, including decoders, hierarchical distribution modules, and a nonvolatile six-port routing array. By
embedding functionality in persistent
magnetic states rather than continuous power or static plumbing, this work establishes a scalable foundation for programmable fluidics and self-driving laboratories.
Read the full abstract ▾
All authors Bingchao Wang, Jonah Mack, Francesco Giorgio-Serchi, Adam A. Stokes
Robotics
FODMP: Fast One-Step Diffusion of Movement Primitives Generation for Time-Dependent Robot Actions
Xirui Shi, Arya Ebrahimi, Yi Hu et al. (+1 more)
Diffusion models are increasingly used for
robot learning, but current designs face a clear trade-off. Action-chunking
diffusion policies like ManiCM are fast to run, yet they only predict short segments of motion. This makes them reactive, but unable to capture time-dependent motion primitives, such as following a spring-damper-like behavior with built-in dynamic profiles of acceleration and deceleration. Recently, Movement Primitive
Diffusion (MPD) partially addresses this limitation by parameterizing full trajectories using Probabilistic Dynamic Movement Primitives (ProDMPs), thereby enabling the generation of temporally structured motions. Nevertheless, MPD integrates the motion decoder directly into a multi-step
diffusion process, resulting in prohibitively high inference latency that limits its applicability in real-time control settings. We propose FODMP (Fast One-step
Diffusion of Movement Primitives), a new framework that distills
diffusion models into the ProDMPs trajectory parameter space and generates motion using a single-step decoder. FODMP retains the temporal structure of movement primitives while eliminating the inference bottleneck through single-step consistency
distillation. This enables
robots to execute time-dependent primitives at high inference speed, suitable for closed-loop
vision-based control. On standard manipulation
benchmarks (MetaWorld, ManiSkill), FODMP runs up to 10 times faster than MPD and 7 times faster than action-chunking
diffusion policies, while matching or exceeding their success rates. Beyond speed, by generating fast acceleration-deceleration motion primitives, FODMP allows the
robot to intercept and securely catch a fast-flying ball, whereas action-chunking
diffusion policy and MPD respond too slowly for real-time interception.
Read the full abstract ▾
All authors Xirui Shi, Arya Ebrahimi, Yi Hu, Jun Jin
Robotics · Cross-listed from cs.SE
IndustriConnect: MCP Adapters and Mock-First Evaluation for AI-Assisted Industrial Operations
Melwin Xavier, Melveena Jolly, Vaisakh M A et al. (+1 more)
AI assistants can decompose multi-step workflows, but they do not natively speak industrial protocols such as Modbus, MQTT/Sparkplug B, or OPC UA, so this paper presents INDUSTRICONNECT, a prototype suite of Model Context Protocol (MCP) adapters that expose industrial operations as schema-discoverable AI tools while preserving protocol-specific connectivity and safety controls; the system uses a common response envelope and a mock-first workflow so adapter behavior can be exercised locally before connecting to plant equipment, and a deterministic
benchmark covering normal, fault-injected, stress, and recovery scenarios evaluates the flagship adapters, comprising 870 runs (480 normal, 210 fault-injected, 120 stress, 60 recovery trials) and 2820 tool calls across 7 fault scenarios and 12 stress scenarios, where the normal suite achieved full success, the fault suite confirmed structured error handling with adapter-level uint16 range validation, the stress suite identified concurrency boundaries, and same-session recovery after endpoint restart is demonstrated for all three protocols, with results providing evidence spanning adapter correctness, concurrency behavior, and structured error handling for AI-assisted industrial operations.
Read the full abstract ▾
All authors Melwin Xavier, Melveena Jolly, Vaisakh M A, Midhun Xavier
Robotics
Saranga: MilliWatt Ultrasound for Navigation in Visually Degraded Environments on Palm-Sized Aerial Robots
Manoj Velmurugan, Phillip Brush, Colin Balfour et al. (+2 more)
Tiny palm-sized aerial
robots possess exceptional agility and cost-effectiveness in navigating confined and cluttered environments. However, their limited payload capacity directly constrains the sensing suite on-board the
robot, thereby limiting critical navigational tasks in Global Positioning System (GPS)-denied wild scenes. Common methods for obstacle avoidance use cameras and LIght Detection And Ranging (
LIDAR), which become ineffective in visually degraded conditions such as low visibility,
dust, fog or darkness. Other sensors, such as
RAdio Detection And Ranging (RADAR), have high power consumption, making them unsuitable for tiny aerial
robots. Inspired by bats, we propose Saranga, a low-power ultrasound-based perception stack that localizes obstacles using a dual sonar array. We present two key solutions to combat the low Peak Signal-to-Noise Ratio of $-4.9$ decibels: physical noise reduction and a deep learning based denoising method. Firstly, we present a practical way to block propeller induced ultrasound noise on the weak echoes. The second solution is to train a neural network to utilize the \textcolor{black}{long horizon of ultrasound echoes} for finding signal patterns under high amounts of uncorrelated noise where classical methods were insufficient. We generalize to the real world by using a synthetic data generation pipeline and limited real noise data for training. We enable a palm-sized aerial
robot to navigate in visually degraded conditions of dense fog, darkness, and snow in a cluttered environment with thin and transparent obstacles using only on-board sensing and computation. We provide extensive real world results to demonstrate the efficacy of our approach.
Read the full abstract ▾
All authors Manoj Velmurugan, Phillip Brush, Colin Balfour, Richard J. Przybyla, Nitin J. Sanket
Robotics · Cross-listed from cs.LG
DreamerAD: Efficient Reinforcement Learning via Latent World Model for Autonomous Driving
Pengxuan Yang, Yupeng Zheng, Deheng Qian et al. (+11 more)
We introduce DreamerAD, the first latent world model framework that enables efficient reinforcement learning for autonomous driving by compressing
diffusion sampling from 100 steps to 1 - achieving 80x speedup while maintaining visual interpretability. Training RL policies on real-world driving data incurs prohibitive costs and safety risks. While existing pixel-level
diffusion world models enable safe imagination-based training, they suffer from multi-step
diffusion inference latency (2s/frame) that prevents high-frequency RL interaction. Our approach leverages denoised latent features from
video generation models through three key mechanisms: (1) shortcut forcing that reduces sampling complexity via recursive multi-resolution step compression, (2) an autoregressive dense reward model operating directly on latent representations for fine-grained credit assignment, and (3) Gaussian vocabulary sampling for GRPO that constrains exploration to physically plausible trajectories. DreamerAD achieves 87.7 EPDMS on NavSim v2, establishing state-of-the-art performance and demonstrating that latent-space RL is effective for autonomous driving.
Read the full abstract ▾
All authors Pengxuan Yang, Yupeng Zheng, Deheng Qian, Zebin Xing, Qichao Zhang, Linbo Wang, Yichen Zhang, Shaoyu Guo, Zhongpu Xia, Qiang Chen, Junyu Han, Lingyun Xu, Yifeng Pan, Dongbin Zhao
Robotics · Cross-listed from cs.CV
TAG: Target-Agnostic Guidance for Stable Object-Centric Inference in Vision-Language-Action Models
Jiaying Zhou, Zhihao Zhan, Ruifeng Zhai et al. (+5 more)
Vision–
Language–Action (VLA) policies have shown strong progress in mapping
language instructions and visual observations to robotic actions, yet their reliability degrades in cluttered scenes with distractors. By analyzing failure cases, we find that many errors do not arise from infeasible motions, but from instance-level grounding failures: the policy often produces a plausible grasp trajectory that lands slightly off-target or even on the wrong object instance. To address this issue, we propose TAG (Target-Agnostic Guidance), a simple inference-time guidance mechanism that explicitly reduces distractor- and appearance-induced bias in VLA policies. Inspired by classifier-free guidance (CFG), TAG contrasts policy predictions under the original observation and an object-erased observation, and uses their difference as a residual steering signal that strengthens the influence of object evidence in the decision process. TAG does not require modifying the policy architecture and can be integrated with existing VLA policies with minimal training and inference changes. We evaluate TAG on standard manipulation
benchmarks, including LIBERO, LIBERO-Plus, and VLABench, where it consistently improves robustness under clutter and reduces near-miss and wrong-object executions.
Read the full abstract ▾
All authors Jiaying Zhou, Zhihao Zhan, Ruifeng Zhai, Qinhan Lyu, Hao Liu, Keze Wang, Liang Lin, Guangrun Wang
Robotics · Cross-listed from cs.CV
Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Driving
Linbo Wang, Yupeng Zheng, Qiang Chen et al. (+13 more)
We introduce Latent-WAM, an efficient end-to-end autonomous driving framework that achieves strong trajectory planning through spatially-aware and dynamics-informed latent world representations. Existing world-model-based planners suffer from inadequately compressed representations, limited spatial understanding, and underutilized temporal dynamics, resulting in sub-optimal planning under constrained data and compute budgets. Latent-WAM addresses these limitations with two core modules: a Spatial-Aware Compressive World Encoder (SCWE) that distills
geometric knowledge from a foundation model and compresses multi-view images into compact scene
tokens via learnable queries, and a Dynamic Latent World Model (DLWM) that employs a causal
Transformer to autoregressively predict future world status conditioned on historical visual and ego status representations. Extensive experiments on NAVSIM v2 and HUGSIM demonstrate new state-of-the-art results: 89.3 EPDMS on NAVSIM v2 and 28.9 HD-Score on HUGSIM, surpassing the best prior perception-free method by 3.2 EPDMS with significantly less training data and a compact 104M-parameter model. The code and model weights have been open-sourced at https://github.com/yukki555/Latent-WAM.
Read the full abstract ▾
All authors Linbo Wang, Yupeng Zheng, Qiang Chen, Shiwei Li, Yichen Zhang, Zebin Xing, Qichao Zhang, Xiang Li, Deheng Qian, Pengxuan Yang, Yihang Dong, Ce Hao, Xiaoqing Ye, Junyu Han, Yifeng Pan, Dongbin Zhao
Robotics
Chameleon: Control-Indexed Prospective Memory for Visuomotor Manipulation
Xinying Guo, Chenxi Jiang, Hyun Bin Kim et al. (+4 more)
Robots often observe information that determines a future action long before that action is executed. In a shell game, for example, a
robot first sees which cup hides the ball, watches the cups move, and only later needs to choose the correct cup. The final observation alone is not enough for a decision: the correct action depends on an earlier event. We refer to this temporal gap as observation-action delay. It makes
memory a policy-facing problem: a policy must keep similar histories distinct, retrieve the past event relevant to the current decision, and convert that recall into an action-ready state. We call these requirements separability, addressability, and prospectiveness. We introduce Chameleon, a ~60M visuomotor policy for control-indexed prospective
memory. Chameleon writes
embodied event
memory, preserves separable histories, retrieves control-relevant traces, and trains the resulting working state to be prospective. We also introduce Camo-
Dataset, a real-
robot benchmark that isolates observation-action delay by making the decision scene visually ambiguous, so the correct action must be inferred from earlier observations. Chameleon improves decision/end-to-end success on Camo-
Dataset from 22.5%/21.3% to 80.8%/71.3%. On public long-horizon
memory benchmarks, it achieves 87.1% +/- 0.8% on LIBERO-10, 97.3% +/- 4.5% on MemoryBench, and 75.1% +/- 1.4% on MIKASA-Robo, setting the state of the art for same-size models and exceeding multiple larger VLA baselines under the reported protocols. Probes and ablations show that Chameleon learns separable, addressable, and prospective
memory, and that these properties drive its performance gains.
Read the full abstract ▾
All authors Xinying Guo, Chenxi Jiang, Hyun Bin Kim, Yuhang Han, Ying Sun, Yang Xiao, Jianfei Yang
Robotics · Cross-listed from cs.LG
Towards Safe Learning-Based Non-Linear Model Predictive Control through Recurrent Neural Network Modeling
Mihaela-Larisa Clement, Mónika Farsang, Agnes Poks et al. (+4 more)
The practical deployment of nonlinear model predictive control (NMPC) is often limited by online computation: solving a nonlinear program at high control rates can be expensive on embedded hardware, especially when models are complex or horizons are long. Learning-based NMPC approximations shift this computation offline but typically demand large expert
datasets and costly training. We propose Sequential-AMPC, a sequential neural policy that generates MPC candidate control sequences by sharing parameters across the prediction horizon. For deployment, we wrap the policy in a safety-augmented online evaluation and fallback mechanism, yielding Safe Sequential-AMPC. Compared to a naive feedforward policy baseline across several
benchmarks, Sequential-AMPC requires substantially fewer expert MPC rollouts and yields candidate sequences with higher feasibility rates and improved closed-loop safety. On high-dimensional systems, it also exhibits better learning dynamics and performance in fewer epochs while maintaining stable validation improvement where the feedforward baseline can stagnate.
Read the full abstract ▾
All authors Mihaela-Larisa Clement, Mónika Farsang, Agnes Poks, Johannes Edelmann, Manfred Plöchl, Radu Grosu, Ezio Bartocci
Robotics
Design, Modelling and Characterisation of a Miniature Fibre-Reinforced Soft Bending Actuator for Endoluminal Interventions
Xiangyi Tan, Aoife McDonald-Bowyer, Danail Stoyanov et al. (+1 more)
Miniaturised soft pneumatic actuators are crucial for robotic intervention within highly constrained anatomical pathways. This work presents the design and validation of a fibre-reinforced soft actuator at the centimetre scale for inte- gration into an endoluminal robotic platform for natural-orifice interventional and diagnostic applications. A single-chamber geometry reinforced with embedded Kevlar fibre was de- signed to maximise
curvature while preserving sealing integrity, fabricated using a multi-stage multi-stiffness silicone casting process, and validated against a high-fidelity Abaqus FEM using experimentally parametrised hyperelastic material models and embedded beam reinforcement. The semi-cylindrical actuator has an outer diameter of 18,mm and a length of 37.5,mm. Single and double helix winding configurations, fibre pitch, and fibre density were investigated. The optimal 100 SH configuration achieved a bending angle of 202.9{\deg} experimentally and 297.6{\deg} in simulation, with structural robustness maintained up to 100,kPa and radial expansion effectively constrained by the fibre reinforcement. Workspace evaluation confirmed suitability for integration into the target device envelope, demonstrating that fibre-reinforcement strategies can be effectively translated to the centimetre regime while retaining actuator performance.
Read the full abstract ▾
All authors Xiangyi Tan, Aoife McDonald-Bowyer, Danail Stoyanov, Agostino Stilli
Robotics
Enhancing Drone Light Shows Performances: Optimal Allocation and Trajectories for Swarm Drone Formations
Yunes Alqudsi
Drone light shows (DLShows) represent a rapidly growing application of swarm robotics, creating captivating aerial displays through the synchronized flight of hundreds or thousands of unmanned aerial vehicles (
UAVs) as environmentally friendly and reusable alternatives to traditional pyrotechnics. This domain presents unique challenges in optimally assigning drones to visual waypoints and generating smooth, collision-free trajectories at a very large scale. This article introduces the Unified Assignment and Trajectory Generation (UATG) framework. The proposed approach concurrently solves two core problems: the optimal assignment of drones to designated goal locations and the generation of dynamically feasible, collision-free, time-parameterized trajectories. The UATG framework is specifically designed for DLShows, ensuring minimal transition times between formations and guaranteeing inter-drone collision avoidance. A key innovation is its exceptional computational efficiency, enabling the coordination of large-scale in real-time; for instance, it computes the optimal assignment and trajectories for 1008 drones in approximately one second on a standard laptop. Extensive simulations in realistic environments validate the framework's performance, demonstrating its capability to orchestrate complex formations, from alphanumeric characters to intricate 3D shapes, with precision and visual smoothness. This work provides a critical advancement for the DLShow industry, offering a practical and scalable solution for generating complex aerial choreography and establishing a valuable
benchmark for ground control station software designed for the efficient coordination of multiple
UAVs. A supplemental animated simulation of this work is available at https://youtu.be/-Fjrhw03594.
Read the full abstract ▾
Robotics
3D-Mix for VLA: A Plug-and-Play Module for Integrating VGGT-based 3D Information into Vision-Language-Action Models
Bin Yu, Shijie Lian, Xiaopeng Lin et al. (+8 more)
Vision-
Language-Action (VLA) models leverage
Multimodal Large
Language Models (MLLMs) for robotic control, but recent studies reveal that MLLMs exhibit limited spatial intelligence due to training predominantly on 2D data, resulting in inadequate 3D perception for manipulation tasks. While recent approaches incorporate specialized 3D
vision models such as VGGT to enhance spatial understanding, they employ diverse integration mechanisms without systematic investigation, leaving the optimal fusion strategy unclear. We conduct a comprehensive pilot study comparing nine VGGT integration schemes on standardized
benchmarks and find that semantic-conditioned gated fusion, which adaptively balances 2D semantic and 3D
geometric features based on task context, achieved the strongest performance among all nine evaluated fusion schemes in our pilot study. We present 3D-Mix, a plug-and-play module that integrates into diverse VLA architectures (GR00T-style and $\pi$-style) without modifying existing MLLM or action expert components. Experiments across six MLLM series (nine model variants, 2B–8B parameters) on SIMPLER and LIBERO show that 3D-Mix delivers consistent performance gains, averaging +7.0% on the out-of-domain (OOD) SIMPLER
benchmark across all nine GR00T-style variants, establishing a principled approach for enhancing spatial intelligence in VLA systems.
Read the full abstract ▾
All authors Bin Yu, Shijie Lian, Xiaopeng Lin, Zhaolong Shen, Yuliang Wei, Haishan Liu, Changti Wu, Hang Yuan, Bailing Wang, Cong Huang, Kai Chen
Robotics · Cross-listed from cs.LG
CoordLight: Learning Decentralized Coordination for Network-Wide Traffic Signal Control
Yifeng Zhang, Harsh Goel, Peizhuo Li et al. (+3 more)
Adaptive traffic signal control (ATSC) is crucial in alleviating congestion, maximizing throughput and promoting sustainable mobility in ever-expanding cities. Multi-
Agent Reinforcement Learning (MARL) has recently shown significant potential in addressing complex traffic dynamics, but the intricacies of partial observability and coordination in decentralized environments still remain key challenges in formulating scalable and efficient control strategies. To address these challenges, we present CoordLight, a MARL-based framework designed to improve intra-neighborhood traffic by enhancing decision-making at individual junctions (
agents), as well as coordination with neighboring
agents, thereby scaling up to network-level traffic optimization. Specifically, we introduce the Queue Dynamic State Encoding (QDSE), a novel state representation based on vehicle queuing models, which strengthens the
agents' capability to analyze, predict, and respond to local traffic dynamics. We further propose an advanced MARL
algorithm, named Neighbor-aware Policy Optimization (NAPO). It integrates an
attention mechanism that discerns the state and action dependencies among adjacent
agents, aiming to facilitate more coordinated decision-making, and to improve policy learning updates through robust advantage calculation. This enables
agents to identify and prioritize crucial interactions with influential neighbors, thus enhancing the targeted coordination and collaboration among
agents. Through comprehensive evaluations against state-of-the-art traffic signal control methods over three real-world traffic
datasets composed of up to 196 intersections, we empirically show that CoordLight consistently exhibits superior performance across diverse traffic networks with varying traffic flows. The code is available at https://github.com/marmotlab/CoordLight
Read the full abstract ▾
All authors Yifeng Zhang, Harsh Goel, Peizhuo Li, Mehul Damani, Sandeep Chinchali, Guillaume Sartoretti
Robotics
LATS: Large Language Model Assisted Teacher-Student Framework for Multi-Agent Reinforcement Learning in Traffic Signal Control
Yifeng Zhang, Peizhuo Li, Tingguang Zhou et al. (+2 more)
Adaptive Traffic Signal Control (ATSC) aims to optimize traffic flow and minimize delays by adjusting traffic lights in real time. Recent advances in Multi-
agent Reinforcement Learning (MARL) have shown promise for ATSC, yet existing approaches still suffer from limited representational capacity, often leading to suboptimal performance and poor generalization in complex and dynamic traffic environments. On the other hand, Large
Language Models (LLMs) excel at semantic representation,
reasoning, and analysis, yet their propensity for
hallucination and slow inference speeds often hinder their direct application to decision-making tasks. To address these challenges, we propose a novel learning paradigm named LATS that integrates LLMs and MARL, leveraging the former's strong prior knowledge and inductive abilities to enhance the latter's decision-making process. Specifically, we introduce a plug-and-play teacher-student learning module, where a trained
embedding LLM serves as a teacher to generate rich semantic features that capture each intersection's
topology structures and traffic dynamics. A much simpler (student) neural network then learns to emulate these features through knowledge
distillation in the latent space, enabling the final model to operate independently from the LLM for downstream use in the RL decision-making process. This integration significantly enhances the overall model's representational capacity across diverse traffic scenarios, thus leading to more efficient and generalizable control strategies. Extensive experiments across diverse traffic
datasets empirically demonstrate that our method enhances the representation learning capability of RL models, thereby leading to improved overall performance and generalization over both traditional RL and LLM-only approaches. [...]
Read the full abstract ▾
All authors Yifeng Zhang, Peizhuo Li, Tingguang Zhou, Mingfeng Fan, Guillaume Sartoretti
Robotics
A Sensorless, Inherently Compliant Anthropomorphic Musculoskeletal Hand Driven by Electrohydraulic Actuators
Misato Sonoda, Ronan Hinchet, Amirhossein Kazemipour et al. (+2 more)
Robotic manipulation in unstructured environments requires end-effectors that combine kinematic dexterity with physical compliance. While traditional rigid hands rely on complex external sensors for safe interaction, electrohydraulic actuators offer a promising alternative by combining muscle-like compliance with self-sensing capability. This paper presents the design, control, and evaluation of a musculoskeletal robotic hand architecture powered entirely by remote Peano-HASEL actuators, optimized for safe manipulation. By relocating the actuators to the forearm, we isolate the grasping interface from electrical hazards while maintaining a slim, human-like profile. To address the inherently limited linear contraction of these soft actuators, we integrate a 1:2 pulley routing mechanism that mechanically amplifies tendon displacement. The resulting system prioritizes compliant interaction over high payload capacity, leveraging the intrinsic force-limiting characteristics of the actuators to provide inherent safety. This physical safety is augmented by the self-sensing nature of the HASEL actuators: by monitoring the operating current alone, we achieve real-time grasp detection and closed-loop contact-aware control without external force transducers or encoders. Experimental results demonstrate the system's dexterity and safety through the execution of grasp types from standard taxonomies and the non-destructive grasping of highly fragile objects such as a paper balloon. These findings represent a step toward simplified, inherently compliant soft robotic manipulation.
Read the full abstract ▾
All authors Misato Sonoda, Ronan Hinchet, Amirhossein Kazemipour, Yasunori Toshimitsu, Robert K. Katzschmann
Robotics
Evidence of an Emergent "Self" in Continual Robot Learning
Adidev Jhunjhunwala, Judah Goldfeder and Hod Lipson
A key challenge to understanding self-awareness has been a principled way of quantifying whether an intelligent system has a concept of a "self", and if so how to differentiate the "self" from other cognitive structures. We propose that the "self" can be isolated by seeking the invariant portion of cognitive process that changes relatively little compared to more rapidly acquired cognitive skills - because our self is the most persistent aspect of our experiences. We used this principle to analyze the cognitive structure of
robots under two conditions: One
robot learns a constant task, while a second undergoes continual learning under variable tasks. We find that
robots subjected to continual learning develop an invariant subnetwork that is significantly more stable (p < 0.001) compared to the control, and that this subnetwork is also functionally important: preserving it aids adaptation while damaging it impairs performance. We validate this pattern across three different
robots spanning locomotion and manipulation.
Read the full abstract ▾
Robotics
Toward Generalist Neural Motion Planners for Robotic Manipulators: Challenges and Opportunities
Davood Soleymanzadeh, Ivan Lopez-Sanchez, Hao Su et al. (+3 more)
State-of-the-art generalist manipulation policies have enabled the deployment of robotic manipulators in unstructured human environments. However, these frameworks struggle in cluttered environments primarily because they utilize auxiliary modules for low-level motion planning and control. Motion planning remains challenging due to the high dimensionality of the
robot's configuration space and the presence of workspace obstacles. Neural motion planners have enhanced motion planning efficiency by offering fast inference and effectively handling the inherent multi-modality of the motion planning problem. Despite such benefits, current neural motion planners often struggle to generalize to unseen, out-of-distribution planning settings. This paper reviews and analyzes the state-of-the-art neural motion planners, highlighting both their benefits and limitations. It also outlines a path toward establishing generalist neural motion planners capable of handling domain-specific challenges. For a list of the reviewed papers, please refer to https://davoodsz.github.io/planning-manip-survey.github.io/.
Read the full abstract ▾
All authors Davood Soleymanzadeh, Ivan Lopez-Sanchez, Hao Su, Yunzhu Li, Xiao Liang, Minghui Zheng
Robotics
Decentralized End-to-End Multi-AAV Pursuit Using Predictive Spatio-Temporal Observation via Deep Reinforcement Learning
Yude Li, Zhexuan Zhou, Huizhe Li et al. (+6 more)
Decentralized cooperative pursuit in cluttered environments is challenging for autonomous aerial swarms, especially under partial and noisy perception. Existing methods often rely on abstracted
geometric features or privileged ground-truth states, and therefore sidestep perceptual uncertainty in real-world settings. We propose a decentralized end-to-end multi-
agent reinforcement learning (MARL) framework that maps raw
LiDAR observations directly to continuous control commands. Central to the framework is the Predictive Spatio-Temporal Observation (PSTO), an
egocentric grid representation that aligns obstacle geometry with predictive
adversarial intent and teammate motion in a unified, fixed-resolution projection. Built on PSTO, a single decentralized policy enables
agents to navigate static obstacles, intercept dynamic targets, and maintain cooperative encirclement. Simulations demonstrate that the proposed method achieves superior capture efficiency and competitive success rates compared to state-of-the-art learning-based approaches relying on privileged obstacle information. Furthermore, the unified policy scales seamlessly across different team sizes without retraining. Finally, fully autonomous outdoor experiments validate the framework on a quadrotor swarm relying on only onboard sensing and computing.
Read the full abstract ▾
All authors Yude Li, Zhexuan Zhou, Huizhe Li, Yanke Sun, Yenan Wu, Yichen Lai, Yiming Wang, Youmin Gong, Jie Mei
Robotics
Environment-Grounded Multi-Agent Workflow for Autonomous Penetration Testing
Michael Somma, Markus Großpointner, Paul Zabalegui et al. (+2 more)
The increasing complexity and interconnectivity of digital infrastructures make scalable and reliable security assessment methods essential. Robotic systems represent a particularly important class of operational technology, as modern
robots are highly networked cyber-physical systems deployed in domains such as industrial automation, logistics, and autonomous services. This paper explores the use of large
language models for automated penetration testing in robotic environments. We propose an environment-grounded multi-
agent architecture tailored to Robotics-based systems. The approach dynamically constructs a shared graph-based
memory during execution that captures the observable system state, including network
topology, communication channels, vulnerabilities, and attempted exploits. This enables structured automation while maintaining traceability and effective context management throughout the testing process. Evaluated across multiple iterations within a specialized robotics Capture-the-Flag scenario (ROS/ROS2), the system demonstrated high reliability, successfully completing the challenge in 100% of test runs (n=5). This performance significantly exceeds literature
benchmarks while maintaining the traceability and human oversight required by frameworks like the EU AI Act.
Read the full abstract ▾
All authors Michael Somma, Markus Großpointner, Paul Zabalegui, Eppu Heilimo, Branka Stojanović
Robotics
Goal-Oriented Reactive Simulation for Closed-Loop Trajectory Prediction
Harsh Yadav and Tobias Meisen
Current trajectory prediction models are primarily trained in an open-loop manner, which often leads to
covariate shift and compounding errors when deployed in real-world, closed-loop settings. Furthermore, relying on static
datasets or non-reactive log-replay simulators severs the interactive loop, preventing the ego
agent from learning to actively negotiate surrounding traffic. In this work, we propose an on-policy closed-loop training paradigm optimized for high-frequency, receding horizon ego prediction. To ground the ego prediction in a realistic representation of traffic interactions and to achieve reactive consistency, we introduce a goal-oriented,
transformer-based scene decoder, resulting in an inherently reactive training simulation. By exposing the ego
agent to a mixture of open-loop data and simulated, self-induced states, the model learns recovery behaviors to correct its own execution errors. Extensive evaluation demonstrates that closed-loop training significantly enhances collision avoidance capabilities at high replanning frequencies, yielding relative collision rate reductions of up to 27.0% on nuScenes and 79.5% in dense DeepScenario intersections compared to open-loop baselines. Additionally, we show that a hybrid simulation combining reactive with non-reactive surrounding
agents achieves optimal balance between immediate interactivity and long-term behavioral stability.
Read the full abstract ▾
Robotics
Accelerated Spline-Based Time-Optimal Motion Planning with Continuous Safety Guarantees for Non-Differentially Flat Systems
Dries Dirckx, Jan Swevers and Wilm Decré
Generating time-optimal, collision-free trajectories for autonomous mobile
robots involves a fundamental trade-off between guaranteeing safety and managing computational complexity. State-of-the-art approaches formulate spline-based motion planning as a single Optimal Control Problem (OCP) but often suffer from high computational cost because they include separating hyperplane parameters as decision variables to enforce continuous collision avoidance. This paper presents a novel method that alleviates this bottleneck by decoupling the determination of separating hyperplanes from the OCP. By treating the separation
theorem as an independent classification problem solvable via a linear system or quadratic program, the proposed method eliminates hyperplane parameters from the optimisation variables, effectively transforming non-convex constraints into linear ones. Experimental validation demonstrates that this decoupled approach reduces trajectory computation times up to almost 60% compared to fully coupled methods in obstacle-rich environments, while maintaining rigorous continuous safety guarantees.
Read the full abstract ▾
Robotics
Equivariant Filter Transformations for Consistent and Efficient Visual–Inertial Navigation
Chungeng Tian, Fenghua He and Ning Hao
This paper presents an equivariant filter (EqF) transformation approach for visual–inertial
navigation. By establishing analytical links between EqFs with different
symmetries, the proposed approach enables systematic consistency design and efficient implementation. First, we formalize the mapping from the global system state to the local error-state and prove that it induces a nonsingular linear transformation between the error-states of any two EqFs. Second, we derive transformation laws for the associated linearized error-state systems and unobservable subspaces. These results yield a general consistency design principle: for any unobservable system, a consistent EqF with a state-independent unobservable subspace can be synthesized by transforming the local coordinate chart, thereby avoiding ad hoc
symmetry analysis. Third, to mitigate the computational burden arising from the non-block-diagonal Jacobians required for consistency, we propose two efficient implementation strategies. These strategies exploit the Jacobians of a simpler EqF with block-diagonal structure to accelerate covariance operations while preserving consistency. Extensive Monte Carlo simulations and real-world experiments validate the proposed approach in terms of both accuracy and runtime.
Read the full abstract ▾
Robotics
Knowledge-Guided Manipulation Using Multi-Task Reinforcement Learning
Aditya Narendra, Mukhammadrizo Maribjonov, Dmitry Makarov et al. (+2 more)
This paper introduces Knowledge Graph based Massively Multi-task Model-based Policy Optimization (KG-M3PO), a framework for multi-task robotic manipulation in partially observable settings that unifies Perception, Knowledge, and Policy. The method augments
egocentric vision with an online 3D scene graph that grounds open-vocabulary detections into a metric, relational representation. A dynamic-relation mechanism updates spatial, containment, and affordance edges at every step, and a graph neural encoder is trained end-to-end through the RL objective so that relational features are shaped directly by control performance. Multiple observation modalities (visual, proprioceptive, linguistic, and graph-based) are encoded into a shared latent space, upon which the RL
agent operates to drive the control loop. The policy conditions on lightweight graph queries alongside visual and proprioceptive inputs, yielding a compact, semantically informed state for decision making. Experiments on a suite of manipulation tasks with occlusions, distractors, and layout shifts demonstrate consistent gains over strong baselines: the knowledge-conditioned
agent achieves higher success rates, improved sample efficiency, and stronger generalization to novel objects and unseen scene configurations. These results support the premise that structured, continuously maintained world knowledge is a powerful inductive bias for scalable, generalizable manipulation: when the knowledge module participates in the RL computation graph, relational representations align with control, enabling robust long-horizon behavior under partial observability.
Read the full abstract ▾
All authors Aditya Narendra, Mukhammadrizo Maribjonov, Dmitry Makarov, Dmitry Yudin, Aleksandr Panov
Robotics
RoboHarness: A Memory-Augmented Policy Harness for Vision-Language-Action Model Robustness via In-Context Adaptation
Zhuoran Li, Zhiyang Li, Kaijun Zhou et al. (+1 more)
Despite the promise of
Vision-
Language-Action (VLA) models as generalist robotic controllers, their robustness against perceptual noise and environmental variations in out-of-distribution (OOD) tasks remains fundamentally limited by the absence of long-term
memory, causal failure attribution, and dynamic intervention capability. To address this, we propose RoboHarness, a
memory-augmented policy harness that upgrades frozen VLA policies for robust in-context adaptation without parameter fine-tuning. Specifically, RoboHarness operates through an online pipeline of contrastive Dual-
Memory Retrieval-Augmented Generation (RAG), an attribution-driven
vision-
language orchestrator implemented with a
multimodal large
language model, and extensible Model Context Protocol (MCP) interventions, while an offline
Memory Consolidation module continuously distills the execution traces into reliable priors. Experimental evaluations across three backbone models ($\pi_0$, $\pi_{0.5}$, and SmolVLA) on LIBERO-PRO and our proposed LIBERO-RoboHarness
benchmarks demonstrate that RoboHarness achieves an average absolute success rate gain of 56.6%. This includes a significant absolute improvement of 89.1% in long-horizon task chaining. The project page and source code are available at https://github.com/LZY-1021/RoboHarness.
Read the full abstract ▾
All authors Zhuoran Li, Zhiyang Li, Kaijun Zhou, Jinyu Gu
Robotics
PCHC: Enabling Preference Conditioned Humanoid Control via Multi-Objective Reinforcement Learning
Huanyu Li, Dewei Wang, Xinmiao Wang et al. (+4 more)
Humanoid robots often need to balance competing objectives, such as maximizing speed while minimizing energy consumption. While current reinforcement learning (RL) methods can master complex skills like fall recovery and perceptive locomotion, they are constrained by fixed weighting strategies that produce a single suboptimal policy, rather than providing a diverse set of solutions for sophisticated multi-objective control. In this paper, we propose a novel framework leveraging Multi-Objective Reinforcement Learning (MORL) to achieve Preference-Conditioned
Humanoid Control (PCHC). Unlike conventional methods that require training a series of policies to approximate the Pareto front, our framework enables a single, preference-conditioned policy to exhibit a wide spectrum of diverse behaviors. To effectively integrate these requirements, we introduce a Beta distribution-based
alignment mechanism based on preference vectors modulating a Mixture-of-Experts (MoE) module. We validated our approach on two representative
humanoid tasks. Extensive simulations and real-world experiments demonstrate that the proposed framework allows the
robot to adaptively shift its objective priorities in real-time based on the input preference condition.
Read the full abstract ▾
All authors Huanyu Li, Dewei Wang, Xinmiao Wang, Xinzhe Liu, Peng Liu, Chenjia Bai, Xuelong Li
Robotics
QuadFM: Foundational Text-Driven Quadruped Motion Dataset for Generation and Control
Li Gao, Fuzhi Yang, Jianhui Chen et al. (+4 more)
Despite significant advances in quadrupedal robotics, a critical gap persists in foundational motion resources that holistically integrate diverse locomotion, emotionally expressive behaviors, and rich
language semantics-essential for agile, intuitive human-
robot interaction. Current quadruped motion
datasets are limited to a few mocap primitives (e.g., walk, trot, sit) and lack diverse behaviors with rich
language grounding. To bridge this gap, we introduce Quadruped Foundational Motion (QuadFM) , the first large-scale, ultra-high-fidelity
dataset designed for text-to-motion generation and general motion control. QuadFM contains 11,784 curated motion clips spanning locomotion, interactive, and
emotion-expressive behaviors (e.g., dancing, stretching, peeing), each with three-layer annotation-fine-grained action labels, interaction scenarios, and natural
language commands-totaling 35,352 descriptions to support
language-conditioned understanding and command execution. We further propose Gen2Control RL, a unified framework that jointly trains a general motion controller and a text-to-motion generator, enabling efficient end-to-end inference on edge hardware. On a real quadruped
robot with an NVIDIA Orin, our system achieves real-time motion synthesis (<500 ms latency). Simulation and real-world results show realistic, diverse motions while maintaining robust physical interaction. The
dataset will be released at https://github.com/GaoLii/QuadFM.
Read the full abstract ▾
All authors Li Gao, Fuzhi Yang, Jianhui Chen, Liu Liu, Yao Zheng, Yang Cai, Ziqiao Li