Robotics
Dynamic Targeting of Satellite Observations Using Supplemental Geostationary Satellite Data and Hierarchical Planning
Akseli Kangaslahti, Itai Zilberstein, Alberto Candela et al. (+1 more)
The Dynamic Targeting (DT) mission concept is an approach to satellite observation in which a lookahead sensor gathers information about the upcoming environment and uses this information to intelligently plan observations. Previous work has shown that DT has the potential to increase the science return across applications. However, DT mission concepts must address challenges, such as the limited spatial extent of onboard lookahead data and instrument mobility, data throughput, and onboard computation constraints. In this work, we show how the performance of DT systems can be improved by using supplementary data streamed from geostationary satellites that provide lookahead information up to 35 minutes ahead of time rather than the 1 minute latency from an onboard lookahead sensor. While there is a greater volume of geostationary data, the search space for observation planning explodes exponentially with the size of the horizon. To address this, we introduce a hierarchical planning approach in which the geostationary data is used to plan a long-term observation blueprint in
polynomial time, then the onboard lookahead data is leveraged to refine that plan over short-term horizons. We compare the performance of our approach to that of traditional DT planners relying on onboard lookahead data across four different problem instances: three cloud avoidance variations and a storm hunting scenario. We show that our hierarchical planner outperforms the traditional DT planners by up to 41% and examine the features of the scenarios that affect the performance of our approach. We demonstrate that incorporating geostationary satellite data is most effective for dynamic problem instances in which the targets of interest are sparsely distributed throughout the overflight.
Read the full abstract ▾
All authors Akseli Kangaslahti, Itai Zilberstein, Alberto Candela, Steve Chien
Robotics
A Pivot-Based Kirigami Utensil for Hand-Held and Robot-Assisted Feeding
Keone Leao, Grace Brotherson, Iain Mischel et al. (+2 more)
Eating is a daily challenge for over 60 million adults with essential tremors and other mobility limitations. For these users, traditional utensils like forks or spoons are difficult to manipulate – resulting in accidental spills and restricting the types of food that can be consumed. Prior work has developed rigid, hand-held utensils that often fail to secure food, as well as soft, shape-changing utensils made strictly for
robot-assisted feeding. To assist a broader range of users, we introduce a re-designed kiri-spoon that can be leveraged as either a hand-held utensil or a
robot-mounted attachment. Our key idea – developed in collaboration with stakeholders – is a pivot-based design. With this design the kiri-spoon behaves like a pair of pliers: users squeeze the handles to change the shape of the utensil and enclose food morsels. In practice, users can apply this kiri-spoon as either a spoon (that scoops food) or as a fork (that pinches food); when the handles are closed, the utensil wraps around the morsel and prevents it from accidentally falling. We characterize the amount of force required to open or close the kiri-spoon, and show how designers can modify this force through kinematic or material changes. A highlight of our design is its accessibility: the hand-held version consists of just four 3D printed parts that snap together. By adding a servo motor, we can extend this same kinematic structure to
robot-assisted feeding. Across our user studies, adults with disabilities and elderly adults with Parkinson's reported that the kiri-spoon better met their needs and provided a more effective means of spill prevention than existing alternatives. See a
video of our kiri-spoon here: https://youtu.be/FFIomm5RL98
Read the full abstract ▾
All authors Keone Leao, Grace Brotherson, Iain Mischel, Sagar Parekh, Dylan P. Losey
Robotics
EmboAlign: Aligning Video Generation with Compositional Constraints for Zero-Shot Manipulation
Gehao Zhang, Zhenyang Ni, Payal Mohapatra et al. (+3 more)
Video generative models (VGMs) pretrained on large-scale internet data can produce temporally coherent rollout
videos that capture rich object dynamics, offering a compelling foundation for zero-shot robotic manipulation. However, VGMs often produce physically implausible rollouts, and converting their pixel-space motion into
robot actions through
geometric retargeting further introduces cumulative errors from imperfect depth estimation and keypoint tracking. To address these challenges, we present EmboAlign, a data-free framework that aligns VGM outputs with compositional constraints generated by
vision-
language models (VLMs) at inference time. The key insight is that VLMs offer a capability complementary to VGMs: structured spatial
reasoning that can identify the physical constraints critical to the success and safety of manipulation execution. Given a
language instruction, EmboAlign uses a VLM to automatically extract a set of compositional constraints capturing task-specific requirements, which are then applied at two stages: (1) constraint-guided rollout selection, which scores and filters a batch of VGM rollouts to retain the most physically plausible candidate, and (2) constraint-based trajectory optimization, which uses the selected rollout as initialization and refines the
robot trajectory under the same constraint set to correct retargeting errors. We evaluate EmboAlign on six real-
robot manipulation tasks requiring precise, constraint-sensitive execution, improving the overall success rate by 43.3 percentage points over the strongest baseline without any task-specific training data.
Read the full abstract ▾
All authors Gehao Zhang, Zhenyang Ni, Payal Mohapatra, Han Liu, Ruohan Zhang, Qi Zhu
Robotics
Safe-Night VLA: Seeing the Unseen via Thermal-Perceptive Vision-Language-Action Models for Safety-Critical Manipulation
Dian Yu, Qingchuan Zhou, Bingkun Huang et al. (+2 more)
Current
Vision-
Language-Action (VLA) models rely primarily on RGB perception, preventing them from capturing modalities such as thermal signals that are imperceptible to conventional visual sensors. Moreover, end-to-end generative policies lack explicit safety constraints, making them fragile when encountering obstacles and novel scenarios outside the training distribution. To address these limitations, we propose Safe-Night VLA, a
multimodal manipulation framework that enables
robots to see the unseen while enforcing rigorous safety constraints for thermal-aware manipulation in unstructured environments. Specifically, Safe-Night VLA integrates long-wave infrared thermal perception into a pre-trained
vision-
language backbone, enabling semantic
reasoning grounded in thermodynamic properties. To ensure safe execution under out-of-distribution conditions, we incorporate a safety filter via control barrier functions, which provide deterministic workspace constraint enforcement during policy execution. We validate our framework through real-world experiments on a Franka manipulator, introducing a novel evaluation paradigm featuring temperature-conditioned manipulation, subsurface target localization, and reflection disambiguation, while maintaining constrained execution at inference time. Results demonstrate that Safe-Night VLA outperforms RGB-only baselines and provide empirical evidence that foundation models can effectively leverage non-visible physical modalities for robust manipulation.
Read the full abstract ▾
All authors Dian Yu, Qingchuan Zhou, Bingkun Huang, Majid Khadiv, Zewen Yang
Robotics
Vision-Language System using Open-Source LLMs for Gestures in Medical Interpreter Robots
Thanh-Tung Ngo, Emma Murphy and Robert J. Ross
Effective communication is vital in healthcare, especially across
language barriers, where non-verbal cues and gestures are critical. This paper presents a
privacy-preserving
vision-
language framework for medical interpreter
robots that detects specific
speech acts (consent and instruction) and generates corresponding robotic gestures. Built on locally deployed open-source models, the system utilizes a Large
Language Model (LLM) with few-shot prompting for intent detection. We also introduce a novel
dataset of clinical conversations annotated for
speech acts and paired with gesture clips. Our identification module achieved 0.90 accuracy, 0.93 weighted precision, and a 0.91 weighted F1-Score. Our approach significantly improves computational efficiency and, in user studies, outperforms the
speech-gesture generation baseline in human-likeness while maintaining comparable appropriateness.
Read the full abstract ▾
Robotics
Environment-Aware Path Generation for Robotic Additive Manufacturing of Structures
Mahsa Rabiei and Reza Moini
Robotic Additive Manufacturing (AM) has emerged as a scalable and customizable construction method in the last decade. However, current AM design methods rely on pre-conceived (A priori) toolpath of the structure, often developed via offline slicing software. Moreover, considering the dynamic construction environments involving obstacles on terrestrial and extraterrestrial environments, there is a need for online path generation methods. Here, an environment-aware path generation framework (PGF) is proposed for the first time in which structures are designed in an online fashion by utilizing four path planning (PP)
algorithms (two search-based and two sampling-based). To evaluate the performance of the proposed PGF in different obstacle arrangements (periodic, random) for two types of structures (closed and open), structural (path roughness, turns, offset, Root Mean Square Error (RMSE), deviation) and computational (run time) performance metrics are developed. Most challenging environments (i.e., dense with high number of obstacles) are considered to saturate the feasibility limits of PP
algorithms. The capability of each of the four path planners used in the PGF in finding a feasible path is assessed. Finally, the effectiveness of the proposed structural performance metrics is evaluated individually and comparatively, and most essential metrics necessary for evaluation of toolpath of the resulting structures are prescribed. Consequently, the most promising path planners in challenging environments are identified for robotic additive manufacturing applications.
Read the full abstract ▾
Robotics
Introducing the transitional autonomous vehicle lane-changing dataset: Empirical Experiments
Abhinav Sharma, Zijun He and Danjue Chen
Transitional autonomous vehicles (tAVs), which operate beyond SAE Level 1-2 automation but short of full autonomy, are increasingly sharing the road with human-driven vehicles (HDVs). As these systems interact during complex maneuvers such as lane changes, new patterns may emerge with implications for traffic stability and safety. Assessing these dynamics, particularly during mandatory lane changes, requires high-resolution trajectory data, yet
datasets capturing tAV lane-changing behavior are scarce. This study introduces the North Carolina Transitional Autonomous Vehicle Lane-Changing (NC-tALC)
Dataset, a high-fidelity trajectory
dataset designed to characterize tAV interactions during lane-changing maneuvers. The
dataset includes two controlled experimental series. In the first, tAV lane-changing experiments, a tAV executes lane changes in the presence of adaptive cruise control (ACC) equipped target vehicles, enabling analysis of lane-changing execution. In the second, tAV responding experiments, two tAVs act as followers and respond to cut-in maneuvers initiated by another tAV, enabling analysis of follower response dynamics. The
dataset contains 152 trials (72 lane-changing and 80 responding trials) sampled at 20 Hz with centimeter-level RTK-GPS accuracy. The NC-tALC
dataset provides a rigorous empirical foundation for evaluating tAV decision-making and interaction dynamics in controlled mandatory lane-changing scenarios.
Read the full abstract ▾
Robotics
Contact-Grounded Policy: Dexterous Visuotactile Policy with Generative Contact Grounding
Zhengtong Xu, Yeping Wang, Ben Abbatematteo et al. (+4 more)
Contact-rich dexterous manipulation with multi-finger hands remains an open challenge in robotics because task success depends on multi-point contacts that continuously evolve and are highly sensitive to object geometry, frictional transitions, and slip. Recently, tactile-informed manipulation policies have shown promise. However, most use tactile signals as additional observations rather than modeling contact state or how their action outputs interact with low-level controller dynamics. We present Contact-Grounded Policy (CGP), a visuotactile policy that grounds multi-point contacts by predicting coupled trajectories of actual
robot state and tactile feedback, and using a learned contact-consistency mapping to convert these predictions into executable target
robot states for a compliance controller. CGP consists of two components: (i) a conditional
diffusion model that forecasts future
robot state and tactile feedback in a compressed latent space, and (ii) a learned contact-consistency mapping that converts the predicted
robot state-tactile pair into executable targets for a compliance controller, enabling it to realize the intended contacts. We evaluate CGP using a physical four-finger Allegro V5 hand with Digit360 fingertip tactile sensors, and a simulated five-finger Tesollo DG-5F hand with dense whole-hand tactile arrays. Across a range of dexterous tasks including in-hand manipulation, delicate grasping, and tool use, CGP outperforms visuomotor and visuotactile
diffusion-policy baselines.
Read the full abstract ▾
All authors Zhengtong Xu, Yeping Wang, Ben Abbatematteo, Jom Preechayasomboon, Sonny Chan, Nick Colonnese, Amirhossein H. Memar
Robotics
TransMASK: Masked State Representation through Learned Transformation
Sagar Parekh, Preston Culbertson and Dylan P. Losey
When humans learn new manipulation skills, they are able to generalize these skills to new contexts and environments. In particular, when learning, humans can easily separate task-relevant aspects of the environment (e.g., object location) and distractors (e.g., the table color). Ideally,
robot policies should be able to learn similar, generalizable representations, but instead they often fail under small environment shifts such as changes in lighting, object instance, or initial configuration. In this paper, we propose a self-supervised method that learns a mask which, when multiplied by the observed image features, attempts to transform these features to retain only those which are relevant to the task. Our method — which we call TransMASK — can be combined with a variety of imitation learning frameworks (such as
diffusion policies) without any additional labels or alterations to the loss function. By introducing a learned mask to the network during training, we aim to induce competitive pressure among the image features during training to force the policy to only attend to features which are consistently task-relevant. We find empirically that our masks are interpretable and can reject known spurious features such as the position of distractor objects or the background color. When compared to other representation learning methods for imitation learning, we find that TransMASK results in policies that are more robust to distribution shifts for irrelevant features, achieving at least 30 % improvement over the baselines when tested on out-of-distribution environments. See our project website: https://transmask.github.io/TransMASK/
Read the full abstract ▾
Robotics
Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Search
Imen Mahdi, Matteo Cassinelli, Fabien Despinoy et al. (+2 more)
Open-world interactive object search in household environments requires understanding semantic relationships between objects and their surrounding context to guide exploration efficiently. Prior methods either rely on
vision-
language embeddings similarity, which does not reliably capture task-relevant relational semantics, or large
language models (LLMs), which are too slow and costly for real-time deployment. We introduce SCOUT: Scene Graph-Based Exploration with Learned Utility for Open-World Interactive Object Search, a novel method that searches directly over 3D scene graphs by assigning utility scores to rooms, frontiers, and objects using relational exploration heuristics such as room-object containment and object-object co-occurrence. To make this practical without sacrificing open-vocabulary generalization, we propose an offline procedural
distillation framework that extracts structured relational knowledge from LLMs into lightweight models for on-
robot inference. Furthermore, we present SymSearch, a scalable symbolic
benchmark for evaluating semantic
reasoning in interactive object search tasks. Extensive evaluations across symbolic and simulation environments show that SCOUT outperforms
embedding similarity-based methods and matches LLM-level performance while remaining computationally efficient. Finally, real-world experiments demonstrate effective transfer to physical environments, enabling open-world interactive object search under realistic sensing and
navigation constraints.
Read the full abstract ▾
All authors Imen Mahdi, Matteo Cassinelli, Fabien Despinoy, Tim Welschehold, Abhinav Valada
Robotics
RFM-HRI : A Multimodal Dataset of Medical Robot Failure, User Reaction and Recovery Preferences for Item Retrieval Tasks
Yashika Batra, Giuliano Pioldi, Promise Ekpo et al. (+5 more)
While
robots deployed in real-world environments inevitably experience interaction failures, understanding how users respond through verbal and non-verbal behaviors remains under-explored in human-
robot interaction (HRI). This gap is particularly significant in healthcare-inspired settings, where interaction failures can directly affect task performance and user trust. We present the
Robot Failures in Medical HRI (RFM-HRI)
Dataset, a
multimodal dataset capturing dyadic interactions between humans and
robots embodied in crash carts, where communication failures are systematically induced during item
retrieval tasks. Through Wizard-of-Oz studies with 41 participants across laboratory and hospital settings, we recorded responses to four failure types (
speech, timing, comprehension, and search) derived from three years of crash-cart
robot interaction data. The
dataset contains 214 interaction samples including facial action units, head pose,
speech transcripts, and post-interaction self-reports. Our analysis shows that failures significantly degrade affective valence and reduce perceived control compared to successful interactions. Failures are strongly associated with confusion, annoyance, and frustration, while successful interactions are characterized by surprise, relief, and confidence in task completion. Emotional responses also evolve across repeated failures, with confusion decreasing and frustration increasing over time. This work contributes (1) a publicly available
multimodal dataset (RFM-HRI), (2) analysis of user responses to different failure types and preferred recovery strategies, and (3) a crash-cart
retrieval scenario enabling systematic comparison of recovery strategies with implications for safety-critical failure recovery. Our findings provide a foundation for failure detection and recovery methods in
embodied HRI.
Read the full abstract ▾
All authors Yashika Batra, Giuliano Pioldi, Promise Ekpo, Arman Sayatqyzy, Purnjay Maruur, Shalom Otieno, Kevin Ching, Angelique Taylor
Robotics
Control Lyapunov Functions for Underactuated Soft Robots
Huy Pham and Zach J. Patterson
Soft and soft-rigid hybrid
robots are inherently underactuated and operate under tight actuator limits, making task-space control with stability guarantees challenging. Common nonlinear strategies for soft
robots (e.g., those based on PD control) often rely on the assumption of full actuation with no actuator limits. This paper presents a general control framework for task-space regulation and tracking of underactuated soft
robots under bounded inputs. The method enforces a rapidly exponentially stabilizing control Lyapunov function as a convex inequality constraint while simultaneously satisfying underactuated full-body dynamics and actuator bounds. We validate the approach in simulation on several platforms spanning increasing underactuation: a simple two link tendon-driven "finger", a trimmed helicoid manipulator, and a highly underactuated spiral
robot. We compare against a number of baseline methods from the literature. Results show improved task-space accuracy and consistent Lyapunov convergence under input limits, achieving superior set-point and trajectory-tracking performance.
Read the full abstract ▾
Robotics
RACAS: Controlling Diverse Robots With a Single Agentic System
Dylan R. Ashley, Jan Przepióra, Yimeng Chen et al. (+5 more)
Many robotic platforms expose an API through which external software can command their actuators and read their sensors. However, transitioning from these low-level interfaces to high-level autonomous behaviour requires a complicated pipeline, whose components demand distinct areas of expertise. Existing approaches to bridging this gap either require retraining for every new embodiment or have only been validated across structurally similar platforms. We introduce RACAS (
Robot-Agnostic Control via Agentic Systems), a cooperative agentic architecture in which three LLM/VLM-based modules (Monitors, a Controller, and a
Memory Curator) communicate exclusively through natural
language to provide closed-loop
robot control. RACAS requires only a natural
language description of the
robot, a definition of available actions, and a task specification; no source code, model weights, or reward functions need to be modified to move between platforms. We evaluate RACAS on several tasks using a wheeled ground
robot, a recently published novel multi-jointed robotic limb, and an
underwater vehicle. RACAS consistently solved all assigned tasks across these radically different platforms, demonstrating the potential of agentic AI to substantially reduce the barrier to prototyping robotic solutions.
Read the full abstract ▾
All authors Dylan R. Ashley, Jan Przepióra, Yimeng Chen, Ali Abualsaud, Nurzhan Yesmagambet, Shinkyu Park, Eric Feron, Jürgen Schmidhuber
Robotics · Cross-listed from cs.CV
From Decoupled to Coupled: Robustness Verification for Learning-based Keypoint Detection with Joint Specifications
Xusheng Luo and Changliu Liu
Keypoint detection underpins many
vision tasks, including pose estimation, viewpoint recovery, and 3D reconstruction, yet modern neural models remain vulnerable to small input perturbations. Despite its importance, formal robustness verification for keypoint detectors is largely unexplored due to high-dimensional inputs and continuous coordinate outputs. We propose the first coupled robustness verification framework for heatmap-based keypoint detectors that bounds the joint deviation across all keypoints, capturing their interdependencies and downstream task requirements. Unlike prior decoupled, classification-style approaches that verify each keypoint independently and yield conservative guarantees, our method verifies collective behavior. We formulate verification as a falsification problem using a mixed-integer linear program (MILP) that combines reachable heatmap sets with a polytope encoding joint deviation constraints. Infeasibility certifies robustness, while feasibility provides counterexamples, and we prove the method is sound: if it certifies the model as robust, then the keypoint detection model is guaranteed to be robust. Experiments show that our coupled approach achieves high verified rates and remains effective under strict error thresholds where decoupled methods fail.
Read the full abstract ▾
Robotics
Task Parameter Extrapolation via Learning Inverse Tasks from Forward Demonstrations
Serdar Bahar, Fatih Dogangun, Matteo Saveriano et al. (+2 more)
Generalizing skill policies to novel conditions remains a key challenge in
robot learning. Imitation learning methods, while data-efficient, are largely confined to the training region and consistently fail on input data outside it, leading to unpredictable policy failures. Alternatively, transfer learning approaches offer methods for trajectory generation robust to both changes in environment and tasks, but they remain data-hungry and lack accuracy in zero-shot generalization. We address these challenges in the context of task inversion learning and propose a novel joint learning approach to achieve accurate and efficient knowledge transfer. Our method constructs a common representation of the forward and inverse tasks, and leverages auxiliary forward demonstrations from novel configurations to successfully execute the corresponding inverse tasks, without any direct supervision. We demonstrate the extrapolation capabilities of our framework through ablation studies and experiments in simulated and real-world environments that require complex manipulation skills with a diverse set of objects and tools, where we outperform
diffusion-based and
multimodal VAE alternatives.
Read the full abstract ▾
All authors Serdar Bahar, Fatih Dogangun, Matteo Saveriano, Yukie Nagai, Emre Ugur
Robotics
PRISM: Personalized Refinement of Imitation Skills for Manipulation via Human Instructions
Arnau Boix-Granell, Alberto San-Miguel-Tello, Magí Dalmau-Moreno et al. (+1 more)
This paper presents PRISM: an instruction-conditioned refinement method for imitation policies in robotic manipulation. This approach bridges Imitation Learning (IL) and Reinforcement Learning (RL) frameworks into a seamless pipeline, such that an imitation policy on a broad generic task, generated from a set of user-guided demonstrations, can be refined through reinforcement to generate new unseen fine-grain behaviours. The refinement process follows the Eureka paradigm, where reward functions for RL are iteratively generated from an initial natural-
language task description. Presented approach, builds on top of this mechanism to adapt a refined IL policy of a generic task to new goal configurations and the introduction of constraints by adding also human feedback correction on intermediate rollouts, enabling policy reusability and therefore data efficiency. Results for a pick-and-place task in a simulated scenario show that proposed method outperforms policies without human feedback, improving robustness on deployment and reducing computational burden.
Read the full abstract ▾
All authors Arnau Boix-Granell, Alberto San-Miguel-Tello, Magí Dalmau-Moreno, Néstor García
Robotics
TEGA: A Tactile-Enhanced Grasping Assistant for Assistive Robotics via Sensor Fusion and Closed-Loop Haptic Feedback
Hengxu You, Tianyu Zhou, Fang Xu et al. (+2 more)
Recent advances in teleoperation have enabled sophisticated manipulation of dexterous robotic hands, with most systems concentrating on guiding finger positions to achieve desired grasp configurations. However, while accurate finger positioning is essential, it often overlooks the equally critical task of grasp force modulation, vital for handling objects of diverse hardness, texture, and shape. This limitation poses a significant challenge for users, especially individuals with upper limb disabilities who lack natural tactile feedback and rely on indirect cues to infer appropriate force levels. To address this gap, We present the tactile enhanced grasping assistant (TEGA), a closed loop assistive teleoperation framework that fuses EMG based intent2force inference with visuotactile sensing mapped into real time vibrotactile feedback via a wearable haptic vest, enabling intuitive, proportional force adjustment during manipulation. A wearable haptic vest delivers real time tactile feedback, allowing users to dynamically refine grasp force during manipulation. User studies confirm that the system substantially improves grasp stability and task success, underscoring its potential for assistive robotic applications.
Read the full abstract ▾
All authors Hengxu You, Tianyu Zhou, Fang Xu, Kaleb Smith, Eric Jing Du
Robotics
RoboPocket: Improve Robot Policies Instantly with Your Phone
Junjie Fang, Wendi Chen, Han Xue et al. (+7 more)
Scaling
robot learning is bottlenecked not only by data volume, but also by whether the data targets the policy's failures. Existing
robot-free handheld interfaces enable portable in-the-wild data collection, yet remain open-loop: users record demonstrations without feedback from the current policy. Interactive methods such as DAgger provide such feedback by collecting corrective on-policy data, but require deploying imperfect policies on physical
robots, making them costly, unsafe, and difficult to scale. We present RoboPocket, a smartphone-based system for
robot-free policy iteration. By combining on-device sensing, display, and communication with cloud-side policy inference, RoboPocket provides real-time collection guidance and visualizes the policy's predicted trajectory in augmented reality as AR Visual Foresight. This allows users to identify policy weaknesses and collect targeted corrective data without deploying a physical
robot. An asynchronous finetuning pipeline continuously incorporates new corrections and updates the policy within minutes, realizing a DAgger-inspired
robot-free correction loop. Across diverse real-world manipulation tasks, we validate RoboPocket as a scalable data engine and show that online corrective data collection improves sample efficiency by up to $2\times$ over scaling offline demonstrations. Project page and
videos: https://robo-pocket.github.io.
Read the full abstract ▾
All authors Junjie Fang, Wendi Chen, Han Xue, Fangyuan Zhou, Tian Le, Yi Wang, Yuting Zhang, Jun Lv, Chuan Wen, Cewu Lu
Robotics
Safe-SAGE: Social-Semantic Adaptive Guidance for Safe Engagement through Laplace-Modulated Poisson Safety Functions
Lizhi Yang, Ryan M. Bena, Meg Wilkinson et al. (+4 more)
Traditional safety-critical control methods, such as control barrier functions, suffer from semantic blindness, exhibiting the same behavior around obstacles regardless of contextual significance. This limitation leads to the uniform treatment of all obstacles, despite their differing semantic meanings. We present Safe-SAGE (Social-Semantic Adaptive Guidance for Safe Engagement), a unified framework that bridges the gap between high-level semantic understanding and low-level safety-critical control through a Poisson safety function (PSF) modulated using a Laplace guidance field. Our approach perceives the environment by fusing multi-sensor point clouds with
vision-based instance
segmentation and persistent object tracking to maintain up-to-date semantics beyond the camera's field of view. A multi-layer safety filter is then used to modulate system inputs to achieve safe
navigation using this semantic understanding of the environment. This safety filter consists of both a model predictive control layer and a control barrier function layer. Both layers utilize the PSF and flux modulation of the guidance field to introduce varying levels of conservatism and multi-
agent passing norms for different obstacles in the environment. Our framework enables legged
robots to safely navigate semantically rich, dynamic environments with context-dependent safety margins.
Read the full abstract ▾
All authors Lizhi Yang, Ryan M. Bena, Meg Wilkinson, Gilbert Bahati, Andy Navarro Brenes, Ryan K. Cosner, Aaron D. Ames
Robotics
cuRoboV2: Dynamics-Aware Motion Generation with Depth-Fused Distance Fields for High-DoF Robots
Balakumar Sundaralingam, Adithyavairavan Murali and Stan Birchfield
Effective
robot autonomy requires motion generation that is safe, feasible, and reactive. Current methods are fragmented: fast planners output physically unexecutable trajectories, reactive controllers struggle with high-fidelity perception, and existing solvers fail on high-DoF systems. We present cuRoboV2, a unified framework with three key innovations: (1) B-spline trajectory optimization that enforces smoothness and torque limits; (2) a GPU-native TSDF/ESDF perception pipeline that generates dense signed distance fields covering the full workspace, unlike existing methods that only provide distances within sparsely allocated blocks, up to 10x faster and in 8x less
memory than the state-of-the-art at manipulation scale, with up to 99% collision recall; and (3) scalable GPU-native whole-body computation, namely
topology-aware kinematics, differentiable inverse dynamics, and map-reduce self-collision, that achieves up to 61x speedup while also extending to high-DoF
humanoids (where previous GPU implementations fail). On
benchmarks, cuRoboV2 achieves 99.7% success under 3kg payload (where baselines achieve only 72–77%), 99.6% collision-free IK on a 48-DoF
humanoid (where prior methods fail entirely), and 89.5% retargeting constraint satisfaction (vs. 61% for PyRoki); these collision-free motions yield locomotion policies with 21% lower tracking error than PyRoki and 12x lower cross-seed variance than GMR. A ground-up codebase redesign for discoverability enabled LLM coding assistants to author up to 73% of new modules, including hand-optimized CUDA kernels, demonstrating that well-structured robotics code can unlock productive human-LLM collaboration. Together, these advances provide a unified, dynamics-aware motion generation stack that scales from single-arm manipulators to full
humanoids. Code is available at https://github.com/NVlabs/curobo.
Read the full abstract ▾
Robotics
Observing and Controlling Features in Vision-Language-Action Models
Hugo Buurmeijer, Carmen Amo Alonso, Aiden Swann et al. (+1 more)
Vision-
Language-Action models (VLAs) have shown remarkable progress towards
embodied intelligence. While their architecture partially resembles that of Large
Language Models (LLMs), VLAs exhibit higher complexity due to their multi-modal inputs/outputs and often hybrid nature of
transformer and
diffusion heads. This is part of the reason why insights from mechanistic interpretability in LLMs, which explain how the internal model representations relate to their output behavior, do not trivially transfer to VLA counterparts. In this work, we investigate whether VLA internal representations support lightweight behavioral steering without retraining. Across four frontier VLA models, linear observers recover state- and action-relevant information in both autoregressive and
transformer–flow-matching architectures, and provide robust directions to causally alter VLA outputs. Building on this, we introduce a controller that minimally modifies representations to place observer predictions within prescribed target intervals. Closed-loop experiments with $\pi_{0.5}$ in the LIBERO simulator and on DROID hardware demonstrate improved constraint satisfaction while retaining task performance, with only approximately $1\%$ inference overhead. Together, these experiments show that lightweight linear interventions can reliably steer VLA behavior while preserving closed-loop capabilities, enabling
alignment with user preferences and task requirements without fine-tuning.
Read the full abstract ▾
All authors Hugo Buurmeijer, Carmen Amo Alonso, Aiden Swann, Marco Pavone
Robotics
Residual RL-MPC for Robust Microrobotic Cell Pushing Under Time-Varying Flow
Yanda Yang and Sambeeta Das
Contact-rich micromanipulation in microfluidic flow is challenging because small disturbances can break pushing contact and induce large lateral drift. We study planar
cell pushing with a
magnetic rolling microrobot that tracks a waypoint-sampled reference curve under time-varying Poiseuille flow in simulation. We propose a hybrid controller that augments a nominal MPC with a learned residual policy trained by SAC. The policy outputs a bounded 2D velocity correction that is contact-gated, so residual actions are applied only during
robot-
cell contact, preserving reliable approach behavior and stabilizing learning. All methods share the same actuation interface and speed envelope for fair comparisons. Simulation results show improved robustness and tracking accuracy over pure MPC and PID under nonstationary flow, with generalization from a clover training curve to unseen circle and square trajectories. A residual-bound sweep identifies an intermediate correction limit as the best trade-off, which we use in all
benchmarks.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Planning in 8 Tokens: A Compact Discrete Tokenizer for Latent World Model
Dongwon Kim, Gawon Seo, Jinsung Lee et al. (+2 more)
World models provide a powerful framework for simulating environment dynamics conditioned on actions or instructions, enabling downstream tasks such as action planning or policy learning. Recent approaches leverage world models as learned simulators, but its application to decision-time planning remains computationally prohibitive for real-time control. A key bottleneck lies in latent representations: conventional tokenizers encode each observation into hundreds of
tokens, making planning both slow and resource-intensive. To address this, we propose CompACT, a discrete tokenizer that compresses each observation into as few as 8
tokens, drastically reducing computational cost while preserving essential information for planning. An action-conditioned world model that occupies CompACT tokenizer achieves competitive planning performance with orders-of-magnitude faster planning, offering a practical step toward real-world deployment of world models.
Read the full abstract ▾
All authors Dongwon Kim, Gawon Seo, Jinsung Lee, Minsu Cho, Suha Kwak
Robotics
PhysiFlow: Physics-Aware Humanoid Whole-Body VLA via Multi-Brain Latent Flow Matching and Robust Tracking
Weikai Qin, Sichen Wu, Ci Chen et al. (+5 more)
In the domain of
humanoid robot control, the fusion of
Vision-
Language-Action (VLA) with whole-body control is essential for semantically guided execution of real-world tasks. However, existing methods encounter challenges in terms of low VLA inference efficiency or an absence of effective semantic guidance for whole-body control, resulting in instability in dynamic limb-coordinated tasks. To bridge this gap, we present a semantic-motion intent guided, physics-aware multi-
brain VLA framework for
humanoid whole-body control. A series of experiments was conducted to evaluate the performance of the proposed framework. The experimental results demonstrated that the framework enabled reliable
vision-
language-guided full-body coordination for
humanoid robots.
Read the full abstract ▾
All authors Weikai Qin, Sichen Wu, Ci Chen, Mengfan Liu, Linxi Feng, Xinru Cui, Haoqi Han, Hesheng Wang
Robotics
ROScopter: A Multirotor Autopilot based on ROSflight 2.0
Jacob Moore, Ian Reid, Phil Tokumaru et al. (+2 more)
ROScopter is a lean multirotor autopilot built for researchers. ROScopter seeks to accelerate simulation and hardware testing of research code with an architecture that is both easy to understand and simple to modify. ROScopter is designed to interface with ROSflight 2.0 and runs entirely on an onboard flight computer, leveraging the features of ROS 2 to improve modularity. This work describes the architecture of ROScopter and how it can be used to test application code in both simulated and hardware environments. Hardware results of the default ROScopter behavior are presented, showing that ROScopter achieves similar performance to another state-of-the-art autopilot for basic waypoint-following maneuvers, but with a significantly reduced and more modular code-base.
Read the full abstract ▾
All authors Jacob Moore, Ian Reid, Phil Tokumaru, Randy Beard, Tim McLain
Robotics
Loop Closure via Maximal Cliques in 3D LiDAR-Based SLAM
Javier Laserna, Saurabh Gupta, Oscar Martinez Mozos et al. (+2 more)
Reliable loop closure detection remains a critical challenge in 3D
LiDAR-based SLAM, especially under sensor noise, environmental ambiguity, and viewpoint variation conditions. RANSAC is often used in the context of loop closures for
geometric model fitting in the presence of outliers. However, this approach may fail, leading to map inconsistency. We introduce a novel deterministic
algorithm, CliReg, for loop closure validation that replaces RANSAC verification with a maximal clique search over a compatibility graph of feature correspondences. This formulation avoids random sampling and increases robustness in the presence of noise and outliers. We integrated our approach into a real- time pipeline employing binary 3D descriptors and a Hamming distance
embedding binary search tree-based matching. We evaluated it on multiple real-world
datasets featuring diverse
LiDAR sensors. The results demonstrate that our proposed technique consistently achieves a lower pose error and more reliable loop closures than RANSAC, especially in sparse or ambiguous conditions. Additional experiments on 2D projection-based maps confirm its generality across spatial domains, making our approach a robust and efficient alternative for loop closure detection.
Read the full abstract ▾
All authors Javier Laserna, Saurabh Gupta, Oscar Martinez Mozos, Cyrill Stachniss, Pablo San Segundo
Robotics
Accelerating Sampling-Based Control via Learned Linear Koopman Dynamics
Wenjian Hao, Yuxuan Fang, Zehui Lu et al. (+1 more)
This paper presents an efficient model predictive path integral (MPPI) control framework for systems with complex nonlinear dynamics. To improve the computational efficiency of classic MPPI while preserving control performance, we replace the nonlinear dynamics used for trajectory propagation with a learned linear deep Koopman operator (DKO) model, enabling faster rollout and more efficient trajectory sampling. The DKO dynamics are learned directly from interaction data, eliminating the need for analytical system models. The resulting controller, termed MPPI-DK, is evaluated in simulation on pendulum balancing and surface vehicle
navigation tasks, and validated on hardware through reference-tracking experiments on a quadruped
robot. Experimental results demonstrate that MPPI-DK achieves control performance close to MPPI with true dynamics while substantially reducing computational cost, enabling efficient real-time control on robotic platforms.
Read the full abstract ▾
All authors Wenjian Hao, Yuxuan Fang, Zehui Lu, Shaoshuai Mou
Robotics
OpenFrontier: General Navigation with Visual-Language Grounded Frontiers
Esteban Padilla-Cerdio, Boyang Sun, Marc Pollefeys et al. (+1 more)
Open-world
navigation requires
robots to make decisions in complex everyday environments while adapting to flexible task requirements. Conventional
navigation approaches often rely on dense 3D reconstruction and hand-crafted goal metrics, which limits their generalization across tasks and environments. Recent advances in
vision-
language navigation (VLN) and
vision-
language-action (VLA) models enable end-to-end policies conditioned on natural
language, but typically require interactive training, large-scale data collection, or task-specific fine-tuning with a mobile
agent. We formulate
navigation as a sparse subgoal identification and reaching problem and observe that providing visual anchoring targets for high-level semantic priors enables highly efficient goal-conditioned
navigation. Based on this insight, we select visual frontiers as semantic anchors and propose OpenFrontier, a
navigation framework that requires no task-specific training or fine-tuning and seamlessly integrates diverse
vision-
language prior models. OpenFrontier enables efficient
navigation with a lightweight system design, without dense 3D semantic mapping, task-specific policy training, or model fine-tuning. We evaluate OpenFrontier across multiple
navigation benchmarks and demonstrate strong zero-shot performance, as well as effective real-world deployment on a mobile
robot.
Read the full abstract ▾
All authors Esteban Padilla-Cerdio, Boyang Sun, Marc Pollefeys, Hermann Blum
Robotics
OmniDP: Beyond-FOV Large-Workspace Humanoid Manipulation with Omnidirectional 3D Perception
Pei Qu, Zheng Li, Yufei Jia et al. (+5 more)
The deployment of
humanoid robots for dexterous manipulation in unstructured environments remains challenging due to perceptual limitations that constrain the effective workspace. In scenarios where physical constraints prevent the
robot from repositioning itself, maintaining omnidirectional awareness becomes far more critical than color or semantic information.While recent advances in visuomotor policy learning have improved manipulation capabilities, conventional RGB-D solutions suffer from narrow fields of view (FOV) and self-occlusion, requiring frequent base movements that introduce motion uncertainty and safety risks. Existing approaches to expanding perception, including active
vision systems and third-view cameras, introduce mechanical complexity, calibration dependencies, and latency that hinder reliable real-time performance. In this work, We propose OmniDP, an end-to-end
LiDAR-driven 3D visuomotor policy that enables robust manipulation in large workspaces. Our method processes panoramic point clouds through a Time-Aware
Attention Pooling mechanism, efficiently encoding sparse 3D data while capturing temporal dependencies. This 360{\deg} perception allows the
robot to interact with objects across wide areas without frequent repositioning. To support policy learning, we develop a whole-body teleoperation system for efficient data collection on full-body coordination. Extensive experiments in simulation and real-world environments show that OmniDP achieves robust performance in large-workspace and cluttered scenarios, outperforming baselines that rely on
egocentric depth cameras.
Read the full abstract ▾
All authors Pei Qu, Zheng Li, Yufei Jia, Ziyun Liu, Liang Zhu, Haoang Li, Jinni Zhou, Jun Ma
Robotics
CT-Enabled Patient-Specific Simulation and Contact-Aware Robotic Planning for Cochlear Implantation
Lingxiao Xun, Gang Zheng, Alexandre Kruszewski et al. (+1 more)
Robotic cochlear-implant (CI) insertion requires precise prediction and regulation of contact forces to minimize intracochlear trauma and prevent failure modes such as locking and buckling. Aligned with the integration of advanced medical imaging and robotics for autonomous, precision interventions, this paper presents a unified CT-to-simulation pipeline for contact-aware insertion planning and validation. We develop a low-dimensional, differentiable Cosserat-rod model of the electrode array coupled with frictional contact and pseudo-dynamics regularization to ensure continuous stick-slip transitions. Patient-specific cochlear anatomy is reconstructed from CT imaging and encoded via an analytic parametrization of the scala-tympani lumen, enabling efficient and differentiable contact queries through closest-point projection. Based on a differentiated
equilibrium-constraint formulation, we derive an online direction-update law under an RCM-like constraint that suppresses lateral insertion forces while maintaining axial advancement. Simulations and benchtop experiments validate deformation and force trends, demonstrating reduced locking/buckling risk and improved insertion depth. The study highlights how CT-based imaging enhances modeling, planning, and safety capabilities in
robot-assisted inner-ear procedures.
Read the full abstract ▾
All authors Lingxiao Xun, Gang Zheng, Alexandre Kruszewski, Renato Torres