Robotics
Cognitively-Grounded On-Device Runtime Learning for Ground Robots in Unknown Physical Environments
Yihao Cai, Yanbing Mao and Christian Lebiere
This paper presents \ul{CogRun}, a framework that enables safety-critical ground
robots to perform cognitively-grounded runtime learning entirely on edge-AI devices in unknown physical environments, without prior maps or perceptual knowledge. CogRun consists of three components: a Learning-
Agent, a Rational-
Agent, and a Coordinator. The Learning-
Agent is novel in cognitive-neural learning architecture, which featurs dedicated replay buffers, cognition-driven experience sampling, and a safety-aware action blending of actor-critic reinforcement learning (RL) with instance-based learning (IBL). The Rational-
Agent is a non-learning module that complements the Learning-
Agent by exclusively handling safety-critical functions, while the Coordinator manages interactions between the two
agents to promote safe and efficient runtime learning. CogRun's full autonomy stack (i.e., perception, learning, and control) on edge-AI devices eliminates dependence on wireless communications, enabling broader applications in challenging environments with limited or no connectivity. Experiments on a quadruped
robot in real-world wild forests and on an off-road autonomous vehicle in a simulated wild forest demonstrate that CogRun enables safe and efficient runtime learning, allowing
robots to safely and continuously interact with the physical world for enhancing task performance in complex, unknown environments.
Read the full abstract ▾
Robotics
Development of a Voice-Controlled Tendon-Driven Bionic Hand
Urja Kohli, Shagata Chanda, Kritika Gandhi et al. (+3 more)
The impairment of the hands can seriously affect the abilities of every individual to perform the every-day activity, so the design of stable and controllable support devices is a significant field of study. This paper is about the design and implementation of an automated bionic hand which is dedicated to the coordinated finger movement through the simplified and efficient actuation mechanism. The method that the proposed system was designed on is the tendon-based method whereby the servo motors generate the movement of the fingers, with assistance of the angular control which is calibrated. An actuation is controlled by a microcontroller that will be programmed by use of an Arduino-based microcontroller to carry out programmed gestures that include open hand, fist, pinch and half flexion. It has an interface that is voice command enabled to make it easy to interact with a Bluetooth based sender receiver architecture which offers an option of executing trained commands which are immediately converted to finger actions. To explore the motions behavior, finger coordination and control response to the input, the behavior of the experiment system is tested. The actuation of the fingers was found to take a total of about 7-8 seconds to achieve full flexion of all fingers in a sequence. The system showed repetitive and constant motion throughout several actuation cycles without loss of any apparent tension or precision of control. There was a stable grasp of objects of different shapes and sizes, which implied consistent coordination between the fingers. These findings indicate that the proposed system offers predictable and steady control behavior and has a simple and efficient mechanical and control architecture.
Read the full abstract ▾
All authors Urja Kohli, Shagata Chanda, Kritika Gandhi, Charu Nigam, Aditi Surya Kamal, Pooja Bhati
Robotics
Longitudinal Robot Learning from Demonstration with Care Providers in a Home Environment
Nina Moorman, Julianna Schalkwyk, Vriksha Srihari et al. (+6 more)
Learning from demonstration (LfD) methods enable non-expert end users to teach
robots novel skills without explicit programming. However most evaluations of the usability of LfD with non-experts has been conducted in controlled laboratory environments with a robotics experimenter present. In this work we identify non-expert end users' key barriers when teaching
robots via demonstration without live robotics expert feedback in a home environment. In our human subjects experiment we support the non-expert end users through two forms of demonstrator guidance developed in prior work: pre-training and adaptive feedback. Towards the ecological validity of the evaluation, we conduct this experimentation over multiple visits, with a population of care providers. Finally, we propose to open source the resulting LfD
dataset of care providers teaching a
robot assistive tasks over multiple visits to a home environment.
Read the full abstract ▾
All authors Nina Moorman, Julianna Schalkwyk, Vriksha Srihari, Qingyu Xiao, Kamel Alrashedy, Hongseok Jeong, Kiersten Lange, Matthew B. Luebbers, Matthew Gombolay
Robotics
CRESSim-Neo: A Batched GPU Simulation Engine for Surgical Robotics and Robot Learning
Yafei Ou, Ahnaf Naheen, Tleukhan Mussin et al. (+3 more)
We introduce CRESSim-Neo, a batched GPU simulation engine for surgical robotics and
robot learning. CRESSim-Neo combines position-based simulation of rigid bodies, deformable tissues, fluids, and strands with batched rendering, surgery-specific sensing, and a GPU-resident data pipeline. The engine supports applications including tissue manipulation, fluid suction, suturing, cable-driven
robots, and ultrasound image synthesis. Direct access to physics and rendering buffers enables GPU-resident
robot learning and zero-copy PyTorch integration using DLPack. We demonstrate CRESSim-Neo across rigid-body, deformable-body, and fluid simulation tasks, including
vision-based and surgical
robot-learning scenarios. On an NVIDIA RTX 4090, the engine achieves up to 2.03 million environment steps per second for 8192 parallel CartPole environments, and scales to batched surgical scenarios involving tissue deformation, fluid interaction, and ultrasound sensing. Overall, CRESSim-Neo provides a unified and scalable platform for surgical simulation, synthetic data generation, and surgical
robot learning.
Read the full abstract ▾
All authors Yafei Ou, Ahnaf Naheen, Tleukhan Mussin, Hans Jarales, Melwin Chacko Moncy, Mahdi Tavakoli
Robotics
Control-Oriented Learning for Dynamic Tracking and Stability Analysis of Soft Pneumatic Actuators
Nithin S. Kumar and Eric J. Barth
Soft pneumatic actuators offer inherent compliance and safe interaction but remain difficult to model and control because of their highly nonlinear, distributed dynamics. We present a control-oriented data-driven modeling and control framework that decomposes actuator behavior into a nonlinear static
equilibrium model and a linear residual dynamics model identified using Extended Dynamic Mode Decomposition with control (EDMDc). This representation enables feedforward compensation, task-space feedback control, and local closed-loop stability analysis through an augmented linear model. Experiments achieve approximately 1 mm root mean square error (RMSE) during low-speed (approximately 10 mm/s) trajectory tracking and below 10 mm RMSE at higher speeds (approximately 100 mm/s). The framework further achieves stable tracking of highly dynamic user-generated references with peak accelerations exceeding 25 m/s^2 while simultaneously performing real-time obstacle avoidance. Finally, the proposed stability analysis is experimentally validated by accurately predicting stable, marginal, and unstable operating regimes. These results demonstrate that structured, control-oriented learning provides an accurate and practical framework for soft actuator control.
Read the full abstract ▾
Robotics
Sequential Object Placement Optimization with Convex Decomposition
Yuezhe Zhang, Xiangyu Lyu, Sohan Rudra et al. (+2 more)
Robotic object packing has been a core challenge for robotic deployment in logistics, industry, etc., due to the curse of dimensionality in combinatorial search and the difficulty of dealing with dynamic collision constraints for irregularly shaped objects. Current heuristic and learning-based methods mainly assume a limited spatial discretization resolution of space, and computation becomes extremely inefficient as discretization accuracy increases. In this work, we eliminate this assumption by introducing SOPO-CD, which frames sequential object placement as a differentiable nonlinear optimization problem with hard constraints in a decomposed free space. We formulate the constraints of placing a convex object inside a convex hull as constraining the vertices of the object to lie inside the convex hull. The constraints and their derivatives can be written in closed form and calculated efficiently. We implement a custom solver that achieves local-optimal placements within tightly constrained space in milliseconds; a $50 \times$ speedup compared to a fine-grained grid search method. We evaluate our framework on 2D Tangram, 2D Tetris, and 3D Bin Packing, and have demonstrated strong computational performance and packing utility. We also demonstrate its real-world applicability for solving the Tangram puzzle using a
robot equipped with a dexterous hand.
Read the full abstract ▾
All authors Yuezhe Zhang, Xiangyu Lyu, Sohan Rudra, Davide Tateo, Georgia Chalvatzaki
Robotics
SkyDrive: Learning to Drive in a New City from Aerial Traffic Monitoring
Weijiang Xiong, Lan Feng, Alexandre Alahi et al. (+1 more)
Autonomous driving has made remarkable progress through imitation learning with massive human demonstration data. However, a trained planner often degrades severely when applied to a new environment zero-shot, because of domain shifts in traffic regulations, road layout and driving behaviors. Therefore, adapting a trajectory planner to a new city typically requires resource-demanding local data collection with a vehicle sensor suite. In this work, we show that driving behavior can be learned from a scalable and efficient alternative. We introduce SkyDrive, a framework that utilizes drone-based traffic monitoring to provide efficient supervision for autonomous driving
agents in a new environment. While vehicle-based data collection logs the ego and its surroundings, an aerial platform naturally observes many road users simultaneously over an extended field of view. As a result, every vehicle can be a data source with grounded driving behavior, effectively scaling up the amount of supervision. Based on 137 hours of aerial traffic monitoring footage, we extract 650K driving samples and construct a
benchmark for trajectory planners and motion predictors. Zero-shot experiments with multiple models reveal significant cross-city domain gaps, but many of them can be alleviated by limited supervision from the sky, e.g., 30 minutes of monitoring per location. Our findings show that aerial traffic monitoring is an efficient and scalable data source for adapting autonomous driving systems in new cities. Data and code will be made publicly available.
Read the full abstract ▾
All authors Weijiang Xiong, Lan Feng, Alexandre Alahi, Nikolas Geroliminis
Robotics
Extending Ground-Constraint LiDAR-IMU Calibration to Tilted Surfaces in a Continuous-Time Framework
Vassili Korotkine, Pierre Chamoun, Mohammed Ayman Shalaby et al. (+1 more)
This paper presents a novel method that extends targetless
LiDAR-IMU calibration for ground vehicles to non- flat environments. Calibration typically necessitates full exci- tation of the sensor rig, a requirement that is not fulfilled by ground vehicles in normal operation. To address the degenerate planar motion, state-of-the-art methods propose residuals that assume the colinearity of the
gravity and physical surface normal vectors, restricting usage to cases where the ground is assumed flat. This paper proposes ground-plane residuals that do not require this assumption, and are applicable for planar motion on a tilted surface. Results are demonstrated on a
dataset collected from a Husky ground vehicle, on the M2DGR
dataset, as well as on an offroad vehicle
dataset. Repeatability is shown to be improved both in tilted and flat-ground scenarios, with strong improvement demonstrated for the tilted case. The implementation and experiments are open-sourced at https://github.com/vkorotkine/licalib_tilted_ground.
Read the full abstract ▾
All authors Vassili Korotkine, Pierre Chamoun, Mohammed Ayman Shalaby, James Richard Forbes
Robotics
ROS2 Connect: A new ROS2 over WAN Solution
Daniel Schott, Lakshminarasimhan Srinivasan, Christian Herrmann et al. (+1 more)
The
Robot Operating System 2 (ROS2) has become a widely adopted framework for the development of distributed robotic systems. However, its communication architecture, based on DDS and RTPS, relies on multicast discovery mechanisms that are typically unavailable in wide-area network (WAN) environments, making remote operation challenging. This work presents ROS2 Connect, a WebSocket-based communication framework that enables transparent and secure ROS2 interaction across routed networks without requiring modifications to network infrastructure or DDS configurations. The proposed client-server architecture supports bidirectional exchange of topics, services, actions, and system data while integrating authentication and access control mechanisms. Experimental evaluation over a real WAN connection demonstrates significantly lower latency, higher stability, and improved scalability compared to existing solutions, including DDS Router, rosbridge and Zenoh. Initial results show that ROS2 Connect provides a reliable foundation for teleoperation and distributed robotics applications over wide-area networks.
Read the full abstract ▾
All authors Daniel Schott, Lakshminarasimhan Srinivasan, Christian Herrmann, Andreas Nüchter
Robotics
GaussVLA: Geometry-Aware Spatial Reasoning for Vision-Language-Action Model
Md Selim Sarowar, Md Tanvir Islam, Sungho Kim et al. (+1 more)
Vision-
Language-Action (VLA) models encode visual observations as flat 2D patch
tokens that carry no intrinsic
geometric structure, and augmenting them with dense monocular depth injects per-pixel scalar values that encode neither surface orientation nor
geometric confidence. This leaves the policy with limited structured spatial
reasoning for action prediction. We propose GaussVLA, a Mamba-based VLA that incorporates two custom modules: Gaussian Spatial Tokenizer (GST) to lift frozen semantic and depth features into compact 3D Gaussian
tokens, pools geometrically salient regions with learned queries, and Depth-Aware Chain-of-Thought (DA-CoT) that performs structured, non-autoregressive
geometric reasoning under
language and flow-time conditioning. Across both simulation and real-world evaluations, GaussVLA demonstrates strong spatial-manipulation performance while remaining parameter-efficient. On LIBERO, it achieves 93.5% average success and 100.0% success on the Spatial suite with only 200M parameters, improving over SpatialVLA by 19.7% relative average success while remaining significantly more parameter-efficient.
Read the full abstract ▾
All authors Md Selim Sarowar, Md Tanvir Islam, Sungho Kim, Sangtae Ahn
Robotics
Do Robotic World Models Really Follow Actions? Diagnosing and Aligning Action-Conditioned Generation for Policy Learning
Sixiang Chen, Jiaming Liu, Jixian Wu et al. (+7 more)
Action-conditioned world models are increasingly used as learned simulators for policy evaluation and improvement, yet their effectiveness rests on an unverified assumption: generated futures faithfully reflect arbitrary valid actions. Existing
benchmarks are typically confined to expert demonstrations, leaving off-expert action following inadequately evaluated. To address this gap, we introduce WorldEcho, which probes action following over a broader action distribution using visual integrity and SE(3) trajectory
alignment. Our diagnosis shows that current world models reasonably execute expert actions but struggle with diverse off-expert trajectories, either ignoring the commanded actions or producing visually invalid rollouts. We further propose WorldSync, which strengthens action following along three complementary axes: distributional coverage, representational grounding, and intervention-effect
alignment. It broadens the training distribution over action consequences, grounds intermediate
video representations in action-induced
robot dynamics through an Action-Forcing Expert, and aligns predicted changes under action interventions with the corresponding changes in ground-truth futures. Experiments on RoboTwin
benchmarks and real-
robot tasks show that WorldSync improves WorldEcho metrics and serves as a more reliable simulator for iterative policy improvement, enabling policies to achieve higher success rates.
Read the full abstract ▾
All authors Sixiang Chen, Jiaming Liu, Jixian Wu, Yichen Guo, Tinghao Wang, Siyuan Qian, Hao Chen, Jiajun Cao, Jian Tang, Shanghang Zhang
Robotics
Latent Action as Intention Enables Efficient Future Imagination for World Action Models
Xiang Li, Yupeng Zheng, Songen Gu et al. (+11 more)
World action models (WAMs) improve
robot control by modeling how observations evolve, but generating future observations at test time incurs substantial latency. Fast-WAM removes this process for efficiency; however, our matched implementations show lower generalization for Fast-WAM than for future-aware alternatives, especially with scarce
robot demonstrations and in out-of-distribution scenarios. To bridge this gap, we introduce **LAWA**, a WAM architecture that uses compact latent actions as an operational representation of future intentions, enabling efficient test-time future imagination without generating future observations. Specifically, a discrete tokenizer enhanced by action-free pre-training produces manipulation-centric codebook targets. LAWA jointly denoises a continuous latent state anchored to these targets with executable action chunks while omitting the future-
video branch at inference. On RoboCasa, LAWA achieves state-of-the-art average success rates of 65.6% and 80.8% in the few-shot and full data settings, improving over the matched Fast-WAM baseline by 9.6 and 4.5 points, respectively. It also preserves the performance level of the matched Joint-WAM variant while requiring 42.9% lower inference latency. LAWA also demonstrates competitive zero-shot robustness on LIBERO-Plus and superior performance on real-world tasks. These results show that future imagination need not be discarded: retaining it with compact latent actions yields an effective trade-off among performance, generalization, and latency. Code and models will be released.
Read the full abstract ▾
All authors Xiang Li, Yupeng Zheng, Songen Gu, Huailiang Ma, Feng Yu, Yuhang Zheng, Xian Nie, Shanshuai Yuan, Yujie Zang, Weize Li, Shuai Tian, Moyang Liu, Ya-Qin Zhang, Wenchao Ding
Robotics · Cross-listed from cs.LG
$(\text{DNN})^2$: Doubly Non-Negative Relaxations for Deep Neural Networks
Hanna Jiamei Zhang, Alan Papalia, Michael Everett et al. (+1 more)
Existing linear program (LP) and semidefinite program (SDP) relaxations for rectified linear unit (ReLU) neural network (NN) verification yield overly-conservative safety guarantees due to significant relaxation gaps. While the completely positive program (CPP) formulation closes this gap, it is NP-hard to solve. Its cheapest tractable relaxation, the doubly non-negative program (DNN), retains critical constraints as an SDP, but one whose size exceeds the reach of interior-point methods at practical scale. While Burer-Monteiro (BM) factorization has been applied to make SDP-based verification scalable, no such result exists for the strictly tighter DNN formulation. A key obstacle is that additional non-negativity constraints in the DNN cause dual multipliers for optimality certification to be non-unique, making standard certification methods inapplicable. We propose a novel
eigenvalue maximization procedure that searches the non-unique multiplier space for a valid certificate, i.e. a global optimality guarantee. Experiments demonstrate that our approach $(\text{DNN})^2$ produces bounds consistently tighter than the standard SDP method, often matching the exact solution, and that our certification procedure confirms global optimality when a valid certificate exists. These results are a key step toward providing tight, certifiable, and computationally scalable verification guarantees needed to deploy neural network controllers and perception modules in safety-critical autonomous systems.
Read the full abstract ▾
All authors Hanna Jiamei Zhang, Alan Papalia, Michael Everett, David M. Rosen
Robotics
One-Shot Learning from Demonstration of Contact-Rich Robotic Manipulation by Identifying Physical Interactions
A. H. G. Overbeek, H. van der Kooij and M. Vlutters
Learning from Demonstration (LfD) allows
robots to learn manipulation tasks directly from humans, thereby supporting the versatile application of
robots. Most LfD methods do not explicitly model the physical interactions between a
robot and its environment, such as the making and breaking of contact, while these are crucial during manipulation tasks. Because the same basic physical interactions recur often, they can be a basis for robust, generalizable, and adaptive task reproduction. We propose an LfD method that explicitly uses what physical interactions take place where and when. Using that information, a hybrid position-force controller tracks demonstrated trajectories until contact-based transition conditions from the demonstrations are met. We evaluate our method in real
robot experiments consisting of opening doors and locks, bolt picking and screwing, dislodging, and surface contouring. We show that explicitly modeling physical interactions benefits LfD in four ways. First, by allowing reproduction of complex, sequential, and contact-rich manipulation tasks using only a single demonstration and no prior knowledge of the task. Second, by facilitating robustness to unknown
geometric variations in the environment. Third, by facilitating generalization when
geometric variations are known. Fourth, by facilitating online adaptation using
geometric information explored during task reproduction. We discuss how robustness, generalization, and adaptivity can be explicitly implemented, which is generally lacking in the LfD literature. Thereby, our work aims to close a gap in interpretable few-shot LfD of robotic manipulation.
Read the full abstract ▾
Robotics
Fiber Bragg Grating Whiskers for Bioinspired Hydrodynamic Perception on Underwater Robots
Hao Li, Tianyu Tu, Siyue Yao et al. (+7 more)
Harbor seals track hydrodynamic trails with their vibrissae, enabling passive perception of moving targets in dark or turbid water. Inspired by this capability, we present compact fiber Bragg grating (FBG) whiskers for underwater
robots. Like seal whiskers, they have a non-uniform taper and elliptical cross-section. Controlled towing experiments show a monotonic relative-flow response from 0.1 to 0.6 m/s, a strong reduction of self-induced oscillation relative to a cylindrical baseline, and a pronounced dependence on angle of attack. Experiments with a pitching foil show that the whiskers can detect the characteristic
vortices shed by a stationary or moving source, detectable several seconds after the source has passed. Using this information, a single front-mounted whisker enabled a small underwater
robot to distinguish between continuing straight and executing a turn, selecting the correct branch in 17 of 20 trials (85.0%) from whisker signals alone. These results connect bioinspired hydrodynamic sensing to
robot action and suggest the utility of whiskers for tracking underwater objects.
Read the full abstract ▾
All authors Hao Li, Tianyu Tu, Siyue Yao, Ziyang Chang, Juhyun Jung, Xiaochi Xie, Long Yin Chung, Tian-Ao Ren, Genliang Chen, Mark Cutkosky
Robotics
GaussianWAM: Distilling Geometry and Semantics from 3D Gaussian Fields into World-Action Models
Zijian Zhang, Yuqing Jiang, Weitao Zhou et al. (+6 more)
World-Action Models (WAMs) jointly learn future visual prediction and action generation, using
video dynamics as a representation-learning signal for robotic manipulation. However, their
video latents are primarily optimized for visual prediction and are not explicitly encouraged to preserve cross-view
geometric structure or spatially localized, object-relevant semantics. We propose GaussianWAM, a training-time representation-enhancement framework that organizes
geometric and semantic supervision through a 3D Gaussian field. Given synchronized multi-view observations, frozen geometry and
vision foundation models provide depth, camera parameters, and dense semantic features. GaussianWAM binds these heterogeneous signals to shared Gaussian primitives and renders spatially aligned semantic, depth, and coverage targets, which are distilled into the current-observation representations of the WAM. All teacher models, Gaussian components, and auxiliary prediction heads are removed after training, leaving the original WAM inference path without additional modules or forward computation. On LIBERO-Plus, GaussianWAM improves FastWAM from 52.05% to 71.29% and Cosmos Policy from 71.52% to 77.30%. Direct CLIP and VGGT
distillation already establishes a strong FastWAM baseline of 69.37%, while Gaussian-field unification further improves it to 71.29%, supporting the benefit of spatially organizing heterogeneous teacher signals. GaussianWAM also improves performance on standard LIBERO and shows positive transfer trends on RoboTwin and real-world manipulation. These results suggest that training-time Gaussian
distillation provides a practical way to inject geometry- and semantics-related supervision into WAM representations without changing their deployment architecture.
Read the full abstract ▾
All authors Zijian Zhang, Yuqing Jiang, Weitao Zhou, Minglei Li, Jinhao Zhang, Yao Mu, Xiaofan Li, Hao Zhao, Haibao Yu
Robotics
VIP: Variation-based Iterative-learning Planning for Robotic Navigation
Shuli Lv, Pengda Mao, Chen Min et al. (+4 more)
Over the past decade, autonomous robotic systems have been increasingly deployed in applications such as surveying, search and rescue, and last-mile delivery. These applications require
robots to generate safe and efficient motion plans in large, complex, and obstacle-dense environments, often under limited onboard computing resources. However, conventional planning methods commonly rely on finite-dimensional trajectory parameterization or increasingly long prediction horizons, leading to rapidly growing computational costs, particularly in multi-
robot scenarios. This paper presents a novel variation-based iterative-learning planning (VIP) framework for efficient motion planning of both single
robots and robotic swarms. Instead of optimizing a large number of discrete trajectory variables, VIP directly updates the planning command as a continuous function in an infinite-dimensional function space. The same variation-based update can be implemented in a model-in-the-loop manner for offline planning or in a
robot-in-the-loop manner between online physical executions. By avoiding the computational burden associated with horizon expansion and high-dimensional trajectory discretization, VIP maintains a per-iteration computational complexity of $\mathcal{O}(n)$, where $n$ denotes the number of spatial discretization points. Extensive simulations and real-world experiments demonstrate that the proposed framework can efficiently generate and iteratively improve motion plans for different planning objectives, robotic platforms, and swarm configurations, highlighting its effectiveness, computational efficiency, and scalability as a general planning methodology.
Read the full abstract ▾
All authors Shuli Lv, Pengda Mao, Chen Min, Li Hong, Runxiao Liu, Shuai Wang, Quan Quan
Robotics
Gripper-aware Vision Language Action Models
Hanyi Zhang, Zihong Luo, Tianyu Li et al. (+16 more)
Vision language action models (VLAs) have advanced general purpose robotic grasping and manipulation by enabling
robots to interpret visual observations and natural
language instructions to generate executable action sequences. However, existing VLAs often implicitly assume gripper invariance, despite grasping strategies being inherently embodiment-dependent. Different gripper types, such as parallel-jaw and suction, usually require distinct interaction strategies to achieve the same grasping objective. Moreover, current
datasets for VLAs predominantly rely on parallel-jaw grippers, limiting gripper-aware learning. To address this gap, we introduce MiGA, a multi-gripper-aware
dataset spanning five distinct gripper types across multiple
robots with 103,000 demonstrations, explicitly capturing strategy divergence under shared task objectives. We further propose GVLA, which combines a new multi-gripper tokenizer with adapter-based policy routing. Our new gripper encoding induces structured
embedding information that balances parameter sharing and strategy differentiation, while layer-wise probing confirms meaningful gripper-conditioned representations for VLAs. Intensive experiments in both simulation and real-world
robots show that our GVLA outperforms the current baselines across evaluated settings. Our method also improves zero-shot generalization or few-shot adaptation to new objects or unseen tasks, and enable more efficient gripper adaptation.
Read the full abstract ▾
All authors Hanyi Zhang, Zihong Luo, Tianyu Li, Khang Nguyen, Basu Hela, Shreyas Kumar, Ngoc Duy Tran, Feng Dai, Charith Munasinghe, Jorge Peña Queralta, Giovanni Toffetti, Khoa Vo, Ngan Le, Ravi Prakash, Quan Vuong, Tung D. Ta, Long Hu, Anh Nguyen, Baoru Huang
Robotics
Fiber Optic Sensing Glove for High Performance Dexterous Manipulation Capture
J.D. Peiffer, Taylor Niehues, Li Guan et al. (+2 more)
Capturing hand pose during dexterous manipulation remains difficult:
vision-based methods degrade under occlusion and challenging lighting, while sensorized gloves, though occlusion-free, are prone to drift and
magnetic interference and rarely match motion-capture accuracy. We introduce a fiber optic sensing glove for full hand pose tracking that targets these failure modes, using multi-core shape-sensing fibers that capture each fiber's full 3D shape rather than
curvature alone. A novel pipeline registers each reconstructed fiber shape to a common hand reference frame, and a new inverse-kinematics solver reconstructs full hand pose at 60 Hz using curve constraints. Benchmarked on a 2-hour
dataset of dexterous object manipulation tasks across 5 subjects, the glove achieves 7.2 mm mean fingertip position error against motion capture ground truth, reduced to 4.9 mm by a one-time factory calibration of the fiber routing hub that transfers across users and sessions. These capabilities enable high-fidelity data capture and bimanual virtual teleoperation - both essential to advancing the robotics field.
Read the full abstract ▾
All authors J.D. Peiffer, Taylor Niehues, Li Guan, Ziyi Kou, Ergys Ristani