Robotics
LeWAM: End-to-End World and Action Modeling with JEPAs
Minghao Fu, Tavis Siebert, Eryk Halicki et al. (+1 more)
Joint-
embedding predictive architectures (JEPAs) learn compact latent world models without pixel reconstruction. Existing JEPA pipelines typically train visual representations and action-conditioned dynamics before adding a control policy or solver. We introduce LeWAM, an end-to-end JEPA for control with: (1) a joint training objective over next latent state prediction and action flow matching, and (2) an isotropic Gaussian constraint on the latent state, enabling a single model to support direct policy execution and efficient planning by evaluating sampled actions with its learned latent dynamics. Compared with prior JEPAs in visual planning, LeWAM improves success rate from 28.6% to 89.7% on contact-rich tasks and from 10.9% to 46.2% on long-horizon tasks, and plans 32.3 times faster. On behavior cloning tasks in simulation and physical
robots, LeWAM is also competitive with established methods. Interpretability analyses show that LeWAM learns control-relevant representations and action distributions effectively guiding test-time planning and control.
Read the full abstract ▾
All authors Minghao Fu, Tavis Siebert, Eryk Halicki, Randall Balestriero
Robotics
Gamma: Grounded Multi-Agent Memory with Mechanical Audit for Memory-Intensive Manipulation
Yudong Liu, Yueqian Lin, Zijia Tang et al. (+11 more)
Pretrained
vision-
language-action (VLA) policies fail on
memory tasks, behaviour that depends on events observed long before the current frame, because their training data is overwhelmingly Markov and mapping episodic
memory directly to action chunks is hard. Yet the same policy, finetuned once to follow a privileged oracle's grounded-subgoal text, nearly solves these tasks: the difficulty lies entirely in mapping
memory to a grounded symbolic subgoal. Existing work entrusts this stage to a single VLM that selects keyframes and reasons over raw frames, and falls far short of the oracle: key identities and dynamics are buried in noisy pixels, the
reasoning spans hundreds of frames in an unbounded context, and
hallucination corrupts the
memory itself. We instead present GAMMA (Grounded multi-
Agent Memory with Mechanical Audit), which builds the stage as a collaborating two-
agent memory system: a detector tool (SAM-3) for spatial grounding, a VLM writer that distils each replan window into one grounded text line, an append-only text
memory bank, and a VLM reasoner that reads the bank and emits the next grounded subgoal at the oracle's cadence, wrapped in a propose-verify harness that admits, defers, corrects, or rejects every
agent claim against mechanical visual evidence, at training-corpus construction and at deployment alike. On a sixteen-task
memory-manipulation suite GAMMA recovers a large fraction of the oracle ceiling with the policy held fixed, reaching the ceiling on several tasks, and transfers with unchanged contracts to a second
benchmark.
Read the full abstract ▾
All authors Yudong Liu, Yueqian Lin, Zijia Tang, Yuxi Zheng, Yuan Li, Shuai Zhang, Yi Li, Taotao Jing, Shuangjun Liu, Dashan Gao, Ning Bi, Jingwei Sun, Yiran Chen, Hai Helen Li
Robotics
Tactile Sensing-based Secure and Gentle Grasping of Unknown Objects
Feng Han
Reliable robotic manipulation requires maintaining a firm grasp while strictly avoiding both contact slip and structural damage to the object. Satisfying these inherent secure-and-gentle grasping constraints is challenging for unknown objects. We present a novel online data-driven predictive control framework for safe dynamic robotic grasping of unknown objects. The proposed system features a pair of compact, single-channel tactile-sensing modules that characterize the gripper-object dynamic interaction in real time, entirely bypassing the need for prior knowledge or offline training. A finite-state-machine-based grasping force probing mechanism is developed to establish the initial grasping force and explicitly identify grasping force constraints. Through online dynamics learning, a Gaussian Process (GP) model captures gripper-object interaction dynamics. The model uncertainties are further used to form robust grasping constraints, which are embedded into a model predictive control scheme. We demonstrate error convergence in a probabilistic sense. The proposed control framework runs at 100~Hz in real time to dynamically manipulate and handle objects rather than static holding. Extensive experimental evaluations across diverse contact surface
topologies and various types of objects validate superior performance and efficiency compared to baseline strategies. The proposed method is able to manipulate objects ranging from delicate items, such as raw tofu and marshmallows, to rough objects, such as rocks.
Read the full abstract ▾
Robotics
Multimodal Image-based Robotic Assistance and Guidance Environment (MIRAGE): A Novel Unified Framework for Surgical Navigation and Visualization
Ehsan Nasiri, Yi Zheng, Michael A. Kokko et al. (+4 more)
This paper presents the computational architecture and quantitative preclinical evaluation of MIRAGE, a
multimodal image-guided robotic surgery framework for real-time visualization and navigation. It integrates stereovision scene reconstruction, augmented-reality (AR)
video overlay, quantified heads-up display (HUD) navigation, and intraoperative ultrasound within the da Vinci console's TilePro display (Intuitive Surgical). Additional modules include optical/electromagnetic tracker registration, stereo calibration, fiducial rigid registration with RMSE feedback, and 3D point-cloud reconstruction.
Video I/O and rendering run in a C++ backend, while a Python GUI coordinates calibration, registration, reconstruction, visualization, and user interaction through an in-process zero-copy binding layer. The framework formalizes spatial transforms, module states, and synchronization of heterogeneous
video and tracking streams. Preoperative calibration uses synchronized checkerboard image pairs and optimization, with estimated intrinsics, extrinsics, and rectification reused intraoperatively. MIRAGE was validated in phantom experiments across da Vinci S, Si, Xi, and SP platforms. Across clinically relevant working distances, mean 3D reprojection error was 0.6mm, with best 0.1mm RMSE at 100mm on Xi. Results support MIRAGE as a preclinical technical-validation platform for accurate, responsive, and extensible
multimodal surgical guidance.
Read the full abstract ▾
All authors Ehsan Nasiri, Yi Zheng, Michael A. Kokko, Xiaoyao Fan, William R. Warner, Keith D. Paulsen, Ryan J. Halter
Robotics · Cross-listed from eess.SY
Distributionally Robust Adaptive Iterative Covariance Steering for Small-Body Proximity Operations
Vivek Khatana, Aditya Gahlawat, Naira Hovakimyan et al. (+1 more)
Autonomous small-body proximity operations require continuous-thrust guidance under uncertain
gravity, solar-radiation pressure (SRP), and
stochastic disturbances. We combine iterative covariance steering (iCS) with $L_1$ distributionally robust adaptive control ($L_1$-DRAC) to provide risk-aware collision avoidance despite mismatch between the nominal planning model and the true dynamics. Iterative covariance steering plans a nominal feedback policy satisfying distributionally robust conditional value-at-risk keep-out constraints, while $L_1$-DRAC compensates for dynamics uncertainty and bounds the residual distributional mismatch through a Wasserstein radius used by the planner. In $\mathbf{1000}$ paired Monte Carlo trials with
gravity, SRP, and
diffusion mismatch, $L_1$-DRAC adaptive control reduces the keep-out violation rate from $\mathbf{32.7\%}$ to $\mathbf{0.2\%}$ and the physical collision rate from $\mathbf{16.4\%}$ to zero, without actuator saturation.
Read the full abstract ▾
All authors Vivek Khatana, Aditya Gahlawat, Naira Hovakimyan, Petros Voulgaris
Robotics
Reflex: Visually Grounded Reactive Humanoid Control
Taoyang Jia, Weikai Huang, Linxin Song et al. (+6 more)
Humans effortlessly react to the world; they adeptly return a tennis shot, catch a falling cup, step aside from a passing cart, or pass a box tossed during truck unloading. These everyday actions require anticipating a moving target and reflexively responding with the whole body within about a second. Such visual reactive whole-body control remains challenging for humanoids, which must (1) coordinate whole-body control, (2) react to object dynamics in real time, and (3) ground perception in egocentric
vision. We study catching thrown boxes as a representative task that demands all three capabilities; existing systems cover only part of it, relying on end-effectors, motion capture, or marked objects. We present Reflex, a three-stage framework that learns these capabilities stage by stage while connecting them through a compact representation of object dynamics. First, a state-based policy learns whole-body catching with reinforcement learning. Second, the policy learns to infer the dynamics of a moving box from delayed and incomplete observations. Third, an egocentric RGB-D encoder learns to recover this dynamics representation directly from visual observations while keeping the whole-body controller fixed. This decomposition provides each capability with a direct training signal and makes visual learning substantially more scalable. In simulation, Reflex achieves 85.7% catch success from egocentric RGB-D, compared with 87.9% when given privileged box observations. Temporal visual history is critical: removing it reduces success by 35.6 and 39.8 points on centered and laterally offset throws. On a real Unitree G1, Reflex catches human-thrown boxes with 65% and 40% success at 2.5 and 5 m, reaching 76% and 59% of human catchers' success at the same distances.
Read the full abstract ▾
All authors Taoyang Jia, Weikai Huang, Linxin Song, Jared Darlington, Jieyu Zhao, Yue Wang, Jiafei Duan, Zhongzheng Ren, Ranjay Krishna
Robotics
Verifier-Mediated Multi-LLM Coordination for Multi-Robot Path Planning
Vivek Khatana, Zijian Song, Naira Hovakimyan et al. (+1 more)
Foundation models are increasingly used for high-level multi-
robot task planning, yet independently generated symbolic plans can become jointly infeasible after
geometric grounding because of shared spatial and temporal constraints. We study execution-aware coordination of large-
language-model (LLM) planners for multi-
robot navigation. We compare a centralized architecture, in which one LLM jointly reasons over the
robot team, with a decentralized multi-LLM architecture in which each
robot independently proposes a task plan. A
geometric verifier grounds the candidate plans into
robot trajectories, detects predicted inter-
robot conflicts, and converts conflict information into structured feedback for targeted replanning. This verifier-mediated architecture separates semantic task
reasoning from execution-level feasibility while enabling iterative coordination among LLM planners. Experiments in CARLA and on Crazyflie aerial
robots evaluate coordination performance as task complexity increases and characterize the trade-offs between centralized
reasoning, decentralized verifier-mediated coordination, and physical executability.
Read the full abstract ▾
All authors Vivek Khatana, Zijian Song, Naira Hovakimyan, Petros Voulgaris
Robotics
Trajectory Prediction for Autonomous Driving in Road Construction Zones Using Human-in-the-Loop Driving Data
Shuo Zhang, Aman Sunesh, Semiha Ergan et al. (+1 more)
Ego-trajectory prediction in urban roadway work zones remains an underexplored topic in autonomous-driving research, with limited dedicated data and systematic evaluation. Roadway work zones involve temporary changes in the lane layouts and connectivity of the roads, complicating interactions with surrounding vehicles and workers. These changes make future vehicle trajectories difficult to predict. To address these challenges, we collect WZDrive-Data, a
multimodal dataset comprising 1,183 work-zone driving segments from 33 drivers, using a human-in-the-loop driving simulator with prerecorded worker animations and VR-controlled workers. Driver control and gaze signals are recorded from human participants, while vehicle trajectories and scene information are obtained from the simulator. Our proposed
multimodal ego-trajectory predictor, WZDrive, achieves a mean minADE (K=6) of 1.09 m and minFDE (K=6) of 2.12 m. Compared with the best-performing state-of-the-art baseline for each metric in our
benchmark, WZDrive reduces minADE (K=6) by 7.18% and minFDE (K=6) by 13.18%. This research addresses a gap in autonomous-driving research on urban work zones by combining human-in-the-loop data collection with
multimodal learning from human driving signals and in the context of work zones.
Read the full abstract ▾
All authors Shuo Zhang, Aman Sunesh, Semiha Ergan, Kaan Ozbay
Robotics
StereoEngine: Scaling Stereo Bimanual Robot Data without Stereo Cameras
Jason Chen, I-Chun Arthur Liu, Gaurav Sukhatme et al. (+1 more)
Stereo
vision is increasingly used for robotic manipulation, with a growing body of work incorporating it into
Vision-
Language-Action (VLA) models. However, the scarcity of large-scale, real-world stereo
datasets remains a fundamental bottleneck for scaling such policies. Recent generative monocular-to-stereo methods target settings such as VR content and autonomous driving. No prior work produces realistic synthetic stereo data for robotic manipulation, where stereo supervision has come predominantly from simulation. We propose StereoEngine, which introduces two complementary methods to synthesize stereo data for robotic manipulation. In the first, we warp a monocular (left)
video into the right view using estimated depth and inpaint the disoccluded regions with a
video diffusion model, lifting monocular demonstrations into stereo pairs without training on real stereo. In the second, we render a stereo pair in simulation and jointly generate both photorealistic views, producing consistent left-right data without any real
video as input. StereoEngine outperforms existing baselines in generated stereo quality, and using its synthetic data for policy mid-training or task-specific fine-tuning substantially improves downstream bimanual manipulation. Project website: https://stereoengine.github.io
Read the full abstract ▾
All authors Jason Chen, I-Chun Arthur Liu, Gaurav Sukhatme, Daniel Seita
Robotics
Visual-Wi-Fi Navigation to Signal-Emitting Devices
Lihao Wang, Tianjian Xu, Tianmin Shu et al. (+1 more)
Indoor navigation to task-relevant targets remains difficult when the target is small, occluded, or visually indistinct. Meanwhile, smart appliances, mobile devices, and IoT tags emit Wi-Fi signals during everyday communication, providing a pervasive but underused spatial cue. We study visual-Wi-Fi navigation, where a
robot reaches a signal-emitting device in an unmapped environment using onboard visual observations and Wi-Fi packets overheard from the device, without connecting to the device or modifying infrastructure. Realizing this requires pose-dependent Wi-Fi observations during closed-loop policy learning and reliable spatial cues from bearing measurements corrupted by indoor multipath propagation on a moving
robot. We present WiFiSpaces, which extends the Habitat embodied simulator with online ray-traced Wi-Fi sensing, rendering Wi-Fi observations for a
robot-mounted
antenna array at continuous poses during policy rollouts. On top of it, we build ViFiNav, a localization-aware navigation framework: an online source-localization
algorithm exploits
robot motion to disambiguate and robustly aggregate bearings from multiple viewpoints into a confidence-aware source estimate, and a learned policy fuses this evolving estimate with depth observations and learns to act on it as it is refined. Experiments show that exploiting
robot motion is key to accurate source estimates, and that fine-tuning on ray-traced Wi-Fi observations in WiFiSpaces improves the success rate of ViFiNav. The policy transfers zero-shot to unseen scene
datasets, and we deploy it on a real
robot with a Wi-Fi sensor built from a commodity router, without any real-world training.
Read the full abstract ▾
All authors Lihao Wang, Tianjian Xu, Tianmin Shu, Renjie Zhao
Robotics
REMM: Revisable Embodied Memory for Manipulation
Revanth Krishna Senthilkumaran, Ruwan Wickramarachchi, Alessandro Oltramari et al. (+1 more)
Long-horizon manipulation requires
robots to retain task-relevant history while revising conclusions no longer supported by physical evidence. Retaining observations alone does not specify which completed conditions should be withdrawn after a disturbance or which obligations require repair. We introduce REMM, Revisable Embodied
Memory for Manipulation, a supervisory
memory layer for a frozen G0.5
vision-
language-action executor. REMM separates an append-only evidence history from revisable task state and explicitly links derived relations and completed conditions to their supporting facts. When accepted evidence changes a supporting fact, it withdraws dependent conclusions and schedules targeted recovery at verified skill boundaries without updating the executor. Across 51 physical trials, including operator-assisted runs, REMM improves mean recorded task progress over the strongest baseline from 57% to 91% in intervention recovery and from 43% to 70% in sequence recall. Complementary
video diagnostics show improved grounding of retained history into the next instruction, including comparisons with $\pi_{0.5}$-compatible high-level planner adaptations. These results support revisable
memory as a practical interface for continued manipulation without policy retraining. Project: https://revanthsenthil.github.io/remm/.
Read the full abstract ▾
All authors Revanth Krishna Senthilkumaran, Ruwan Wickramarachchi, Alessandro Oltramari, Jonathan Francis
Robotics
FallGown: A Soft Pneumatic Wearable Device for Fall Prevention
Megan C. Coram, Haoran Lei, Steven H. Collins et al. (+4 more)
Concerns about falling and potential resulting injuries prevent many older adults from participating in forms of health-improving exercise such as walking, dance, and tai chi. Existing passive devices like walkers require use of the arms for support, and active devices like exoskeletons and robotic frames are often rigid, bulky, and stigmatized, which limits their use. The FallGown is a soft wearable support structure consisting of pressurized pneumatic pouches within a fabric shell; it can support the full weight of a user's body and aims to prevent falls. It is portable (6.9 kg and 0.051 m3 when stored), minimally burdensome (7.24-20.8 N drag force depending on floor type and whether an optional rolling base frame is used), and convenient (less than 2.5 minutes on average to don and less than 0.5 minutes on average to doff). The optional rolling base lowers friction with the floor, but reduces portability and skirt deformation (which can be useful to fit through doors and other tight spaces). Fall prevention testing with a single user revealed a tradeoff between device size and stability against tipping; a skirt with 12 pneumatic pouches and a floor diameter of 1.4 meters with a base frame prevented all falls except in one case with a mechanical failure. On-board pressure sensors can sense the user's weight distribution, providing an opportunity for monitored exercise and adaptive support.
Read the full abstract ▾
All authors Megan C. Coram, Haoran Lei, Steven H. Collins, Steve B. Cousins, Michelle Baldonado, Cynthia Sung, Allison M. Okamura
Robotics
Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models
Justin Tien-Smith, Matthew Pidden and Rolandos Alexandros Potamias
Vision-
language-action models (VLAs) offer a promising route toward general-purpose robotics by transferring semantic knowledge from large-scale
vision-
language pretraining into
language-conditioned control. However, most VLA evaluations emphasize aggregate manipulation success under changes in objects, instructions, or environments, leaving unclear how reliably these policies ground spatial descriptions in the physical scenes where they act. We therefore ask a sharper question: how reliably can current VLAs translate spatial descriptions into appropriate real-
robot behavior? To investigate this, we introduce Spa-Bench, a real-
robot benchmark that evaluates spatial grounding across six tabletop tasks covering Object State, Relative Size, Referential Description, Relational Placement, Ordinal Position, and Counting. Our primary evaluation compares in-distribution (ID) and out-of-distribution (OOD) spatial configurations while retaining training support for the relevant objects, concepts, and actions, reducing object and motor unfamiliarity as alternative explanations for failure. Four diagnostic conditions further probe physical execution, sensitivity to wording, object-task associations, and recognition that no action is required. We release 4,092 episodes collected on the SO-101
robot arm: 1,200 human-teleoperated demonstrations and 2,892 real-
robot evaluation rollouts from five state-of-the-art VLA policies. Across the three fully evaluated policies, success falls from 67.5-73.3% on ID spatial configurations to 40.7-52.0% on OOD spatial configurations, while matched direct instructions improve success by 28.3-40.0 percentage points. These results show that the evaluated VLAs can often execute the required manipulations but lack robust spatial grounding. Project materials, including
videos, code, and
benchmark data, are available at https://justintiensmith.github.io/Spa-Bench/.
Read the full abstract ▾
Robotics
Non-Prehensile Multi-Object Placement with Mobile Robots via Diffusion-Based Trajectory Planning
Lance Pharand, Megnath Ramesh, Mark Chiu et al. (+2 more)
Non-prehensile manipulation with mobile
robots has primarily targeted placement in coarse goal regions. Precision placement has been studied with manipulators on
benchmarks such as Push-T, where an omnidirectional end effector can readily re-approach an object from any side. For a mobile
robot, correcting the same placement errors may require wide repositioning maneuvers around the object. We formalize multi-object pushing with a mobile
robot as a multi-target assignment problem at tight tolerances. We propose a
diffusion planner with a causal factorization: each
robot denoising step is followed by an object step conditioned on the current clean
robot estimate. This factorization allows us to sample multiple object outcomes for a single
robot plan, capturing the multi-modality of pushing. We also design a set of pushing-specific costs for mobile
robots, used for both guidance and best-of-$N$ sampling at inference. On Push-T, our planner attains the highest success rate of the methods we compare, while having the lowest rate of implausible object motion. Across three multi-box pushing environments resembling warehouse scenarios, it raises the fraction of boxes placed by $21.5$ percentage points over an unguided baseline, and replans $1.8\times$ faster than a planner that denoises the two trajectories together. Lastly, we deploy the guided planner on a TurtleBot 3 Burger with object shapes absent from the training data, where it places $85.0\%$ of objects within tolerance against
Diffusion Policy's $37.5\%$.
Read the full abstract ▾
All authors Lance Pharand, Megnath Ramesh, Mark Chiu, Yash Vardhan Pant, Stephen L. Smith
Robotics
Robust and Scalable UAV-UGV Planning for Aerial Monitoring under Energy and Precedence Constraints
Cahit Ikbal Er and Yasin Yazicioglu
We consider planning for unmanned aerial vehicle (
UAV) and unmanned ground vehicle (UGV) teams in aerial monitoring missions with ordering requirements on the monitoring points to be visited by the
UAV. The system consists of multiple
UAV-UGV teams, each comprising one
UAV and one UGV.
UAV energy limitations are modeled through endurance, defined as the maximum flight time on a full battery. The UGVs serve as mobile charging stations and transport the
UAVs between consecutive aerial tours. Given a set of monitoring points subject to ordering constraints and the teams' initial and terminal positions, the objective is to minimize mission completion time while satisfying
UAV energy limits and the required visit order under bounded travel-time disturbances that may arise during execution. Unlike prior work, the plan jointly determines vehicle routes and their timing, allowing teams to slow down as needed to satisfy the ordering constraints. We develop a
polynomial-time
algorithm that generates efficient solutions to the underlying NP-hard planning problem. We establish feasibility guarantees, characterize the
algorithm's computational complexity, and evaluate its performance via numerical comparisons.
Read the full abstract ▾
Robotics
PriMoRetarget: Humanoid Motion Transfer Through Prioritized Motion Retargeting
Nicholas Ioannidis, Dun Yang, Max Smith et al. (+4 more)
Motion retargeting is a critical preprocessing step for transferring human motion data to humanoid
robots. While existing methods are effective for single-character motions, interactions involving multiple characters and objects introduce additional contacts and spatial relationships that must be preserved. Retargeting is commonly formulated as a weighted multi-objective optimization problem, where conflicting objectives can require careful tuning of relative weights and can still compromise important motion characteristics. We introduce PriMoRetarget, a prioritized motion-retargeting framework that replaces relative objective weighting with an explicit priority hierarchy. Lower-priority updates are constrained to preserve higher-priority objectives, allowing critical characteristics such as foot placement and interaction contacts to take precedence over lower-priority pose objectives. The method is evaluated on four
datasets spanning single-character motion, character-object manipulation, and character-character interactions. The resulting motions better preserve task-relevant interactions and provide reference trajectories that can be tracked more reliably than those produced by widely used retargeting methods.
Read the full abstract ▾
All authors Nicholas Ioannidis, Dun Yang, Max Smith, Tue Vu, Ziyu Zhang, Shayegan Omidshafiei, Xue Bin Peng
Robotics
Distributed Harmonic-Covariance Feedback for Hole Closure in Planar Robot Swarms
Jitian Liu and Subhrajit Bhattacharya
We consider hole closure in a planar multi-
agent system whose sensing disks form a time-varying Čech complex. We propose a harmonic-covariance feedback that uses harmonic edge signals to locate holes and converts those signals into
agent forces. For distributed evaluation, we use a fixed-degree
polynomial of the regularized Hodge
Laplacian. This
polynomial is evaluated through local operator applications. The closed loop has a unique global trajectory from every safe initial state. The model data and initial state are supplied with procedures for approximation to any requested accuracy with explicit error bounds. We assume that hole area is measurable and bounded, holes evolve continuously without splitting or merging, and only finitely many temporary new holes appear. Under these conditions, offline finite calculations produce rational lower bounds for the cumulative hole area on the time horizon without querying trajectory data. These bounds converge to that cumulative area, whether finite or infinite. If it is finite, holes close in the time-averaged sense: the average normalized hole area over every fixed-length time window tends to zero as time tends to infinity. In a 15-second numerical demo, all holes close by 9.9s and remain closed through the end.
Read the full abstract ▾
Robotics
Selectively Actuated Fiberbots for High-precision Minimally Invasive Surgery
Jinshi Zhao, Libaihe Tian, Alex Ranne et al. (+9 more)
Long, slender instruments are essential for minimally invasive surgery (MIS), but achieving precise distal manipulation becomes increasingly difficult when actuation extends along the entire instrument. Here, we present an electrothermally actuated fiber-based
robot designed for MIS procedures, in which Joule heating is concentrated in the distal active segment. The fiber is fabricated using a thermal drawing process and uses embedded metallic wires for thermal actuation. This robotic fiber, 1.9 mm in diameter and 1.2 m in length, can precisely manipulate surgical
laser fibers with positional precision below 100 $\mu$m. The compact size of the fiber
robot allows it to pass through the working channel of a flexible endoscope, while keeping both devices mechanically uncoupled to allow independent motion. A phantom demonstration combining the robotic fiber with a da Vinci instrument conceptually demonstrated its compatibility with existing endoscopic systems for clinical applications. Our robotic fiber enables precise distal motion and holds the potential to improve clinical outcomes by enhancing precision in MIS procedures.
Read the full abstract ▾
All authors Jinshi Zhao, Libaihe Tian, Alex Ranne, Qindong Zheng, Yue Li, Xiaotong Guo, Ziqi Yang, Ali Anil Demircali, Mohamed E. M. K. Abdelaziz, Bruno Gil Rosa, Ferdinando Rodriguez y Baena, Burak Temelkuran
Robotics
Image-Based Path-Planning for Navigation of Soft-Growing Robots in the Spinal Sub-Arachnoid Space
George Dimas, Panagiotis G. Kalozoumis, Panagiotis Vartholomeos et al. (+1 more)
Recently, soft robotic procedures are being developed aiming at achieving targeted minimally invasive operations or drug administration in specific locations in the human body. Accordingly, specialized path-planning
algorithms need to be investigated for the estimation of efficient paths that would protect the delicate tissue structures and simultaneously satisfy a set of requirements for the efficient navigation of the
robot. In this context, we propose an image-based 3D pathplanning
algorithm, which is an extension of the A*
algorithm, for the navigation of a soft-growing
robot inside the spinal subarachnoid space (SSS). The proposed
algorithm is capable of estimating a safe pathway towards the goal location, while ensuring the establishment of anchor points that facilitate the efficient steering of the soft
robot. The
algorithm is evaluated using a highly detailed model of the SSS with respect to its capacity to satisfy the
robot movement requirements and path efficiency.
Read the full abstract ▾
All authors George Dimas, Panagiotis G. Kalozoumis, Panagiotis Vartholomeos, Dimitris K. Iakovidis
Robotics
Graph-Based Heuristics for Minimum Constraint Removal in Cluttered Robot Motion Planning
Antony Thomas, Fulvio Mastrogiovanni and Marco Baglietto
Humanoid and service
robots operating in clut- tered human environments may encounter motion planning problems that are infeasible when all surrounding objects are treated as fixed obstacles. The Minimum Constraint Removal problem, which we refer to as MCR, addresses this situation by identifying the minimum number of constraints, such as movable objects, whose removal enables a collision-free path. However, computing an optimal MCR solution is NP-hard. In this work, we propose a graph-based heuristic pipeline that combines bidirectional feasible-set generation with fixed- length searches over reduced MCR graphs. We further derive an extended feasible-set generator for instances in which the candidates produced by the first generator do not contain a complete optimal removal set. Across six
benchmarks, the evaluated pipeline matches the exact solution in three cases and, in the remaining cases, selects at most 31.25% more obstacles than the optimum, while being between 3.4 and 33.7 times faster than exact search. In one of the
benchmarks, involving a manipulation task amidst clutter, selecting two rather than seven obstructing objects reduces the combined MCR, inverse- kinematics, and motion planning time from 84.72 s to 30.59 s. These results indicate that the proposed approach provides a practical compromise between computational efficiency and solution quality for motion planning among movable obstacles.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Mind the Temporal Gap: Evaluating Video Models for Temporal Understanding in Driving
Shiva Sreeram, Erfan Aasi, Tsun-Hsuan Wang et al. (+2 more)
Modern
video-
language models are increasingly treated as general-purpose
video reasoners, yet it remains unclear whether they robustly capture temporal dynamics in safety-critical settings such as driving. We study this question through controlled ego-motion tasks that require identifying actions from driving
videos. Across
video-
language models, we find that standard prompting, visual prompting, few-shot prompting, and chain-of-thought prompting remain unreliable and highly task- and model-dependent, with persistent class biases, although newer models show substantial task-specific few-shot gains. In contrast, lightweight LoRA finetuning and simple probes trained on intermediate visual features recover strong motion-discriminative performance, indicating that motion-relevant cues are often present inside the models but are not reliably used by the final
video-to-
language pathway. These results suggest that, in driving, current
video-
language models remain unreliable temporal reasoners and often behave more like frame-level, bias-driven recognizers than dependable temporal world models.
Read the full abstract ▾
All authors Shiva Sreeram, Erfan Aasi, Tsun-Hsuan Wang, Alaa Maalouf, Daniela Rus
Robotics
CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained Optimization
Yinlong Dai, Homanga Bharadhwaj and Dylan P. Losey
Prior research on learning from humans often requires real human demonstrations. But advances in
video generation models offer a new paradigm: we can now
prompt these generators to make synthetic
videos of the task. On the plus side, these synthetic
videos take place in the
robot's current environment, and do not require actual human teaching. However, while the generated
videos appear to show plausible motions, they lack both physical consistency and
robot grounding, and thus the
robot cannot directly roll out the trajectory shown by the
video. In this paper we seek to formalize the process of going from synthetic
video generation to real-world
robot trajectories for zero-shot manipulation tasks. We recognize that this problem is an instance of constrained optimization. The object trajectories extracted from the synthetic
video provide an initial guess at the correct motion, and the
robot must refine these trajectories (for instance, within a digital twin) while enforcing physical feasibility and semantic consistency. Our optimization-based formalism provides insights into the
video generation properties necessary for success. Specifically, we show that the
video generation must probabilistically seed the search within a semantic basin of attraction where the trajectory can be locally modified without losing its task
alignment. Our analysis further reveals that state-of-the-art approaches can be viewed as instances of our unified formalism with key simplifications. In our experiments we remove these simplifications to more closely match the underlying optimization, leading to more robust zero-shot trajectories from synthetic
videos and improving average task success by 34.1% compared to the best-performing baseline. See
videos at our project website: https://collab.me.vt.edu/CASTER
Read the full abstract ▾
Robotics
What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents
Junxiang Wang, Yuchen Liu, Yi Wu et al. (+6 more)
Memory allows large
language model (LLM)
agents to use past experience, often stored as textual records, when responding to requests. For embodied
agents that control
robots, visual and spatial observations introduce a broader choice of
memory modalities, affecting both the information available for subsequent tasks and the model inference costs of constructing and using
memory. We introduce MeMoREval, a
benchmark for studying this problem through a controlled experience-request protocol.
Agents first experience a sequence of events without knowing the subsequent tasks, and subsequently they use
memory to respond to information and action requests without receiving new observations. The
benchmark comprises 53 tasks across four simulated indoor environments and measures task success alongside the inference costs of
memory construction and use. Our reference evaluations reveal that storing frames outperforms storing generated descriptions at lower write-time inference cost for the primary model, while storing frames with scene graphs achieves the highest aggregate task success for every tested model. Extensions to accumulated experience, selective frame retention, and visual input to
memory writers demonstrate how the
benchmark accommodates further
memory-design investigations. Two physical-
robot demonstrations establish the feasibility of applying its protocol beyond simulation.
Read the full abstract ▾
All authors Junxiang Wang, Yuchen Liu, Yi Wu, Guofei Chen, Nir Pechuk, Tao Chen, Zackory Erickson, Yuzhe Qin, Tianyu Li
Robotics
PanTact: Teaching Whole-Body Dexterity with a Wearable Contact Interface
Xiaomeng Xu, Han Zhang, Zhian Ruan et al. (+6 more)
Humanoid
robots have many degrees of freedom and contact surfaces, yet most loco-manipulation research restricts manipulation to the hands and avoids other body contacts with the environment. To extend manipulation beyond the hands, we present PanTact, a system that learns humanoid whole-body dexterity, the capability to manipulate with any suitable body surface, directly from
robot-free human demonstrations. Our key idea is to use contact as manipulation-centric information that transfers across embodiments. Contact locations and timing specify the interactions needed to accomplish a task, while the
robot chooses how to coordinate its body to realize them within its own physical constraints. Specifically, PanTact captures, learns, and executes whole-body contact via: 1) a wearable,
robot-free data collection interface that augments egocentric
vision and bimanual UMI grippers with whole-body modular contact patches to record body contact locations and states; 2) a visuomotor policy that mitigates the egocentric observation gap and predicts whole-body contact goals; and 3) contact-aware whole-body control formulations that realize active contact goals using suitable
robot surfaces. We deploy PanTact on a Unitree G1 humanoid without additional body contact sensors, demonstrating whole-body dexterity involving body support, pushing, sweeping, and load-sharing. Website: https://pantact.github.io
Read the full abstract ▾
All authors Xiaomeng Xu, Han Zhang, Zhian Ruan, Xiaofeng Guo, Jinkun Cao, Ruihan Yang, Carmelo Sferrazza, Shuran Song, Guanya Shi
Robotics
Fly-by-Code: Embodied Coding Agents for Aerial Manipulation with Active Visual and Physical Feedback
Jaewoo Lee, Jeongyeon Seo, Sihyun Cho et al. (+10 more)
Embodied coding
agents construct
robot policies from natural-
language instructions by generating executable programs and refining them from execution feedback. This refinement assumes that execution feedback reveals whether the task succeeded and, after a failure, provides sufficient evidence to guide program revision. Aerial manipulation violates this assumption in two ways. Payload constraints restrict onboard camera coverage and leave task-critical outcomes outside the
robot's view (partial observability), while strong dynamic coupling and limited control margins make physical interactions fragile (limited interaction robustness) and leave failure causes unresolved. Re-examining the same feedback cannot recover observations that were never collected or separate failure causes that the available evidence does not distinguish. To acquire this missing evidence, we introduce Active Feedback (AF), an active "View and Probe" framework. When execution feedback leaves unresolved questions, the
agent synthesizes a separate diagnostic program that runs in simulation to inspect unseen outcomes from informative viewpoints (View) or test hypotheses about failure causes through targeted physical interactions (Probe). Across six simulated aerial manipulation tasks with 60 trials per condition, AF reduces false completion declarations from 25 to 5 and raises final task success from 31.7% to 75% compared with terminal feedback under the same coding-
agent configuration and task-execution limit. Using shared
robot APIs, task programs refined with AF in simulation run on a physical aerial manipulator without task-program source-code modification and complete multi-stage manipulation tasks in the real world. These results show that executable diagnosis fills gaps in execution feedback for embodied coding
agents, improving both completion assessment and program refinement in aerial manipulation.
Read the full abstract ▾
All authors Jaewoo Lee, Jeongyeon Seo, Sihyun Cho, Gyeongrak Choe, Yutong Wang, Bavin Saravanan, Jia-Bin Huang, Furong Huang, Sebastian Scherer, Guanya Shi, H. Jin Kim, Seungjae Lee, Dongjae Lee
Robotics
Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration
Jusuk Lee, Sungha Kim, Yeonsoo Park et al. (+8 more)
While learning dexterous manipulation from a single human
video offers a promising alternative to costly
robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the
video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human
video. Our key insight is to abstract the
video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the
video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.
Read the full abstract ▾
All authors Jusuk Lee, Sungha Kim, Yeonsoo Park, Jonguk Cheon, Yoonkyo Jung, Yongjun You, H. Jin Kim, Jia-Bin Huang, Furong Huang, Youngseok Jang, Seungjae Lee
Robotics
DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training
Junyan Li, Ruizhi Li, Yu Liu et al. (+6 more)
We present DreamTrue, a multi-view, cross-embodiment
robot world model for action-faithful and physically plausible
video prediction. Training such a model on existing
robot datasets faces two obstacles: imprecise calibration can impair action following, while limited coverage of unsuccessful interactions can bias predictions toward successful outcomes. To improve action following across embodiments, we render action trajectories into image-space conditions and introduce offline
geometric calibration to align these conditions with the target
videos. To broaden interaction coverage, we introduce counterfactual post-training, modifying recorded action trajectories and generating future
videos under a wider range of actions and contact configurations. To provide feedback on these predictions without paired ground-truth futures, we construct a human-annotated
video dataset covering
robot, object, and interaction defects and use it to train an embodied
video reward model. Its scores guide reinforcement-learning post-training toward more physically plausible interaction outcomes. On AgiBot, DreamTrue attains state-of-the-art action following, while reducing the human-assessed interaction defect rate from from 48.12% to 6.25%. Notably, our model ranks first in the world model track of the AgiBot World Challenge 2026. The project page can be found at https://brave-eai.github.io/DreamTrue.
Read the full abstract ▾
All authors Junyan Li, Ruizhi Li, Yu Liu, Xiangshuo Liu, Mingchao Sun, Hongyu Pan, Mu Xu, Lue Fan, Zhaoxiang Zhang
Robotics
CSF: Contextual Safety Filtering for Motion Generators
Lizhi Yang, Yiling Hou, Yao Tang et al. (+4 more)
Text-conditioned motion generators produce trackable whole-body motion, but they have no notion of scene-dependent safety: the same action may target an object or a person. Existing safeguards either inspect the
prompt, require labeled motion data, or enforce
geometric constraints; therefore, they do not directly account for how scene context changes a motion's meaning. We introduce contextual safety filtering (CSF), a training-free filter that grounds natural-
language safety rules in safe and unsafe reference trajectories produced by the generator. For each active rule, safe and unsafe reference trajectories define an affine safety value that a safe reference tracking CBF-QP enforces. Across four pretrained generators with different architectures, CSF activates the intended rules in all explicit and scene-triggered unsafe cases and reduces the danger-event rate by up to 90%, while preserving 88-100% of benign motions. We demonstrate the complete system on a real-world Unitree G1, where it successfully prevents unsafe motions in a variety of scenarios, including interactions with humans and objects.
Read the full abstract ▾
All authors Lizhi Yang, Yiling Hou, Yao Tang, Junheng Li, Daniel Weng, Blake Werner, Aaron D. Ames
Robotics
A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control
Octi Zhang, Mateo Guaman Castro, Patrick Yin et al. (+4 more)
General-purpose
robots must perform a wide range of tasks from agile locomotion to dexterous manipulation. While sim-to-real reinforcement learning (RL) has proven to be a useful tool for this goal, current RL pipelines depend on engineering-heavy, per-task structural priors such as shaped rewards and demonstrations. Recent work has shown that diverse simulator resets, combined with massively parallel simulation, can alleviate much of this engineering burden on several manipulation problems. However, we find that naively scaling this paradigm to more precise or dynamic problems remains non-trivial. While simulator resets can help with exploration, uniformly sampling over this distribution wastes a growing fraction of learning experience on task configurations the policy has already mastered or cannot yet attempt. This makes it challenging to see the expected benefits of scaling parallel environments for RL, since much of the learning signal in a batch is wasted during learning. To mitigate this, we introduce Success Guided Sampling (SGS), a simple adaptive sampler that concentrates RL training on task configurations around the frontier of the policy's capabilities. Doing so allows large-scale simulated RL to make the most out of the experience in a batch, enabling much more effective scaling to large-scale parallel simulation. Across experiments using up to $2^{20}$ (over one million) parallel environments, SGS enables RL to solve challenging multi-terrain quadruped locomotion and contact-rich assembly tasks that prior methods fail to solve. Finally, we distill the learned manipulation policies into RGB-based policies and demonstrate zero-shot transfer to several challenging assembly tasks on real hardware. Project website: https://sgs-rl.github.io/.
Read the full abstract ▾
All authors Octi Zhang, Mateo Guaman Castro, Patrick Yin, Ignacio Dagnino, Abhishek Gupta, Rosario Scalise, Byron Boots
Robotics
SpatialHarness: Test-Time Spatial Scaffolding for Fine Robotic Manipulation
Jiayu Wang, Yue Yu, Bin Zhu et al. (+2 more)
Frontier
multimodal foundation models (e.g., GPT-6 Astra) have recently shown strong potential for direct robotic control, yet their performance on fine manipulation remains limited. We argue that an important source of failure is not necessarily insufficient policy capability, but insufficient spatial observability, where task-critical spatial relationships may be poorly revealed by the existing physical camera setup. We introduce SpatialHarness, a test-time embodied harness that provides test-time spatial scaffolding for fine robotic manipulation without policy fine-tuning or changes to the physical sensing setup. SpatialHarness maintains an online simulated scene synchronized with real-world execution, identifies task-critical spatial relationships, and renders complementary virtual views that expose them to a frozen
multimodal policy. To keep the simulated scene aligned during interaction, we develop interaction-aware scene synchronization that distinguishes static, held, and transition modes. We evaluate SpatialHarness on four real-
robot manipulation tasks spanning precise
geometric alignment, object-relative placement, and articulated-object interaction. Using the same frozen GPT-6 Astra policy, SpatialHarness substantially improves task success, including from 26.7% to 66.7% on plug insertion and from 0% to 100% on Tower of Hanoi. These results indicate that improving spatial observability at test time can unlock fine-manipulation capabilities already present in strong
multimodal foundation models. Project website: https://emilia113.github.io/SpatialHarness/.
Read the full abstract ▾
All authors Jiayu Wang, Yue Yu, Bin Zhu, Zhiyao Yang, Jingjing Chen