Robotics
RoboShape: Information-Theoretic Point Cloud Representations for Privacy-Aware Robot Perception
Oguzhan Baser, Mirac Sozen, Kaan Kale et al. (+2 more)
With the increased adoption of robotic
agents operating in human environments by scanning and sharing 3D representations (e.g., for fleet learning, cloud-based planning, or collaborative mapping), collected point clouds reveal not just the objects in a scene but also sensitive spatial context, such as room function or information that occupants never consented to disclose. Traditional point cloud encoders offer no principled control over this: either all is preserved, or none. Hence, we introduce RoboShape, an information theory guided compression head following the frozen Sonata encoder. We project voxel-level
embeddings using the Donsker-Varadhan formulation of mutual information (MI). Specifically, we maximize the MI between
embeddings and object-level understanding while minimizing it for private attributes. RoboShape leads to 87.5% smaller
embeddings that retain 98.7% of object classification utility while collapsing sensitive attribute predictions by 39.3% across the three real-world indoor
LiDAR datasets. Its
privacy-preserving
embeddings are cheaper to transmit over the network or to train a model for any downstream tasks. We release the RoboShape codebase to give the robotics community a practical, encoder-agnostic tool for building perception pipelines that are compact,
privacy-aware, and deployment-ready.
Read the full abstract ▾
All authors Oguzhan Baser, Mirac Sozen, Kaan Kale, Sandeep Chinchali, Sriram Vishwanath
Robotics
S.E.A.G.R: A Socially and Emotionally Aware Greeting Robot Framework with Dual-Layer Cultural and Affective Modulation
Sajjad Hussain, Ranveer Bhura, Amandip Dutta et al. (+1 more)
This paper presents SEAGR (Socially and Emotionally Aware Greeting
Robot), a robotic greeting framework designed for human-
robot interaction environments involving users from diverse cultural backgrounds and different emotional states. Since greeting behaviour strongly influences first impressions, user comfort, and trust,
robots operating in public spaces must be able to interact in a socially appropriate and adaptive manner. However, many existing systems still rely on static greeting routines that do not account for cultural variation, emotional context, or interpersonal distance. SEAGR introduces a dual-layer modulation framework in which cultural identity determines the appropriate greeting type, while affective cues influence how that greeting is executed. The system combines context-aware cultural mapping,
emotion-based gesture modulation, and proxemic regulation within a unified Sense-Think-Act architecture. A low-cost prototype is implemented using a USB camera, ultrasonic sensor, Arduino-controlled servos, and a laptop-based Python processing system. This work is presented as a system design and proof-of-concept; empirical validation through user studies is explicitly acknowledged as a current limitation and is identified as the primary direction for future work.
Read the full abstract ▾
All authors Sajjad Hussain, Ranveer Bhura, Amandip Dutta, Shwetangshu Biswas
Robotics
Multi-Objective Kinodynamic Motion Planning with Asymptotic Pareto Optimality
Yusif Razzaq, Anne Theurkauf, Nisar Ahmed et al. (+1 more)
In this paper, we address the challenge of multi-objective motion planning for systems under kinodynamic constraints. We consider three problem classes: (i) lexicographic optimization, in which objectives are minimized according to a strict priority ordering, (ii) constrained optimization, in which a primary objective is minimized subject to bounds on the remaining costs, and (iii) Pareto front optimization, in which the goal is to approximate the full set of optimal trade-offs among competing objectives. We first show that established cost scalarization methods for multi-objective problems cannot be extended to continuous-domain systems with correctness guarantees. Then, we propose a unified algorithmic framework built upon the Stable Sparse-RRT (SST)
algorithm, in which the single representative maintained at each witness neighborhood is replaced by a representative set of locally Pareto-optimal nodes. This structure gives rise to three distinct
algorithms: lexSST for lexicographic minimization, coSST for constrained optimization, and poSST for Pareto-front approximation. We provide theoretical guarantees for the completeness and optimality of our
algorithms and demonstrate their effectiveness through extensive empirical evaluations.
Read the full abstract ▾
All authors Yusif Razzaq, Anne Theurkauf, Nisar Ahmed, Morteza Lahijanian
Robotics
Environment Design for Reliable Shared Autonomy with Probabilistic Guarantees
Yi-Shiuan Tung, Himanshu Gupta, Gyanig Kumar et al. (+3 more)
Shared autonomy enables humans and
robots to collaboratively perform tasks by combining human input with autonomous assistance. Most prior work focuses on improving intent inference under a fixed environment, overlooking how workspace design itself affects inference difficulty. We observe that the physical arrangement of objects directly influences the separability of candidate goals under noisy user inputs. We formulate workspace design as an optimization problem and derive a probabilistic correctness guarantee under a bounded noise model. Through simulation experiments across multiple tabletop scenarios, we show that optimized layouts improve goal inference reliability and reduce ambiguity compared to baseline arrangements. We further demonstrate a real-world shared autonomy system that integrates the proposed inference framework. This highlights the role of environment design as a complementary axis for improving shared autonomy systems.
Read the full abstract ▾
All authors Yi-Shiuan Tung, Himanshu Gupta, Gyanig Kumar, Heyang Huang, Bradley Hayes, Alessandro Roncone
Robotics
Risk-Aware Preference Learning for Stochastic Outcomes
Yi-Shiuan Tung, Yuni Wu, Wei Jiang et al. (+2 more)
Learning reward functions from human preferences is a widely used approach for aligning
robot behavior with user expectations in human-
robot interaction. Most existing approaches assume that humans evaluate uncertain outcomes using expected utility (EU), aggregating outcome utilities linearly with their probabilities. However, behavioral evidence shows that humans are systematically risk-sensitive, overweighting rare negative events and exhibiting loss aversion. We study the consequences of this mismatch in social
robot navigation, where safety-critical outcomes (e.g., collisions) are rare but highly consequential. We compare EU with Cumulative Prospect Theory (CPT), a nonlinear model of human decision-making, within a Bradley-Terry preference learning framework. Our preliminary experiments show that when preferences are generated by risk-sensitive users, CPT-based learners recover reward functions with substantially lower regret compared to EU-based learners. Our results highlight the importance of modeling human risk sensitivity when learning rewards from preferences over
stochastic robot outcomes.
Read the full abstract ▾
All authors Yi-Shiuan Tung, Yuni Wu, Wei Jiang, Alessandro Roncone, Bradley Hayes
Robotics
VTAP Gripper: Synergizing Fingertip Sensing and a Visuo-Tactile Active Palm for Dexterous In-Hand Manipulation
Yuhao Zhou, Sheeraz Athar, Zhixian Hu et al. (+4 more)
This paper presents a tactile-reactive gripper that integrates a Visuo-Tactile Active Palm (VTAP) and compliant, reconfigurable fingers equipped with tactile array sensors. The design exploits structured finger-palm synergy and multi-modal perception to achieve both robust grasping and fine manipulation. The actuated bi-modal palm seamlessly combines long-range visual localization with contact-rich tactile feedback, substantially extending the system's manipulation capability. To bridge the embodiment gap between human hand motion and the heterogeneous three-finger structure, we further propose a staged, gesture-conditioned retargeting framework for dexterous teleoperation. Extensive experiments validate the system across a range of challenging tasks: reactive grasping of YCB and fragile objects, in-hand syringe reorientation and plunger actuation, singulation of clustered objects down to 3 mm in diameter, and
vision-tactile peg-in-hole insertion. Results demonstrate that high manipulation performance can be achieved through coordinated finger-palm interaction and multi-modal sensing, without resorting to high degrees of freedom anthropomorphic designs. The VTAP gripper and its retargeting framework offer a practical reference architecture for dexterous gripper design, manipulation, and contact-rich data collection in support of learning-based approaches. Project webpage: https://yuhochau.github.io/vtap/.
Read the full abstract ▾
All authors Yuhao Zhou, Sheeraz Athar, Zhixian Hu, Binghao Huang, Yunzhu Li, Juan Wachs, Yu She
Robotics
Robust Silicone Pour Casting and Sensor Embedding Procedures for Soft Robotic Actuators
Harshit Thakker, Paul Dela Cruz, Mostafa Mo. Massoud et al. (+1 more)
Soft
robots are well-suited for applications such as rehabilitation and surgery that require adaptable and safe interaction with their environment. However, the challenges of reproducible and scalable fabrication of soft
robots limit their real-world deployment. Various fabrication methods have been introduced, but many are labor-intensive and prone to human error. Therefore, traditional two-part pour casting remains an attractive option. This paper presents procedures for robust, repeatable, and scalable fabrication of soft pneumatic actuators using two-part pour casting. The presented methods prevent internal cavity clogging and ensure air-tight sealing. Additionally, a robust sensor
embedding procedure for thin-film flex sensors is presented, which allows for accurate and repeatable data acquisition. Finite Element Modeling (FEM) of the soft actuator is performed to analyze stress and deformation from internal pressure loadings. Pneumatic actuation experiments with PID pressure control are performed. Automated image processing is used to calibrate the embedded flex sensor to bending angle measurements. Staircase and sinusoidal profile actuation experiments validate the performance of the fabricated actuator. Angle response experiments for the staircase input show repeatable performance, and the sinusoidal input shows a small amount of hysteresis consistent with viscoelastic response to pneumatic actuation of soft actuators. Simulated and real-world bending angles show comparable response. These methods provide a repeatable and robust fabrication procedure, validated across two operators and 24 successful fabrications, along with
benchmark simulations and experimental testing. These
benchmarks will enable more widespread adoption of soft robotics.
Read the full abstract ▾
All authors Harshit Thakker, Paul Dela Cruz, Mostafa Mo. Massoud, Jacqueline Libby
Robotics
NeuroCommitSSM: Decision-Centric Shared Autonomy for Safe Assistive Manipulation via EEG-EMG-ET Commit Readiness
Tipu Sultan, Param Sangani, Kody Cool et al. (+4 more)
We present NeuroCommitSSM, a decision-centric framework that models when to execute, not just what to do, for safe commit-to-execute control in assistive robotic manipulation. NeuroCommitSSM predicts a continuous commit-readiness score c_t in [0,1] from synchronized electroencephalography (
EEG), electromyography (EMG), and eye-tracking (ET), and converts it into discrete commit events through dwell and hysteresis filtering. A three-state finite-state supervisor, HOLD-ASSIST-COMMIT (HAC), gates execution by requiring both a sustained commit-readiness signal from the neural model and real-time perception and
robot-state feasibility, including target visibility, inverse kinematics solvability, and collision-free planning, before initiating motion. We evaluate the framework on N=32 subjects performing five activities of daily living (ADL) tasks aligned with the International Classification of Functioning, Disability and Health (ICF), using leave-one-subject-out (LOSO) cross-validation and seven sensor-dropout scenarios (S0-S6). NeuroCommitSSM achieves 0.950 action-balanced accuracy with 0.75 false commit events per 1000 REST windows (FP/1k REST), and maintains low false commits and stable state transitions under sensor loss. For example, in the
EEG-only condition, it achieves 0.785 balanced accuracy and 0.29 FP/1k REST, whereas the Temporal Convolutional Network baseline produces 99.95 FP/1k REST under the same condition. Hardware-in-the-loop (HIL) validation on a Kinova Gen3 arm shows that feasibility-checked execution reduces false starts and decision instability without sacrificing task success. Supplementary materials, including code,
datasets,
videos, and additional analyses, are available at https://madibabaiasl.github.io/NeuroCommitSSM/.
Read the full abstract ▾
All authors Tipu Sultan, Param Sangani, Kody Cool, Pascal Sikorski, Guangping Liu, Hadi Akbarpour, Madi Babaiasl
Robotics
Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectories
Xiaomi Robotics Team: Jun Guo, Piaopiao Jin, Jason Li et al. (+30 more)
We present Xiaomi-Robotics-1, a foundational
vision-
language-action (VLA) model capable of (1) following diverse
language instructions to perform a wide range of mobile manipulation tasks in unseen environments out-of-the-box, and (2) efficiently adapting to novel downstream tasks with minimal fine-tuning data. We propose a two-stage training recipe consisting of pre-training and post-training. During pre-training, we imbue the model with broad and generalizable action-generation capabilities by training on over 100k hours of real-world manipulation trajectories collected via UMI devices. Crucially, we develop a scalable auto-labeling pipeline that annotates trajectory clips with natural
languages describing scene state transitions, providing rich and precise conditioning for action learning. During post-training, we aim to align these capabilities with
robot embodiments and imperative instructions that humans naturally use to
prompt robots. Extensive experiments demonstrate strong scaling behavior. Xiaomi-Robotics-1 consistently improves with increased data scales and model sizes during pre-training. This scaling behavior directly transfers to post-training, where a stronger pre-training model yields better out-of-the-box real-
robot performance in unseen environments. Furthermore, Xiaomi-Robotics-1 serves as a strong
robot foundation policy that can be efficiently fine-tuned on complex, dexterous tasks with high data efficiency. Across multiple simulation
benchmarks, Xiaomi-Robotics-1 outperforms state-of-the-art methods. Notably, it establishes a new state-of-the-art with a 57.4% success rate on RoboCasa365, surpassing the previous best of 46.6%. Furthermore, it achieves an average score of 20.07 on RoboDojo, significantly outperforming the prior state-of-the-art (13.07). Code and model checkpoints will be released. Project page: https://robotics.xiaomi.com/xiaomi-robotics-1.html
Read the full abstract ▾
All authors Xiaomi Robotics Team: Jun Guo, Piaopiao Jin, Jason Li, Peiyan Li, Yingyan Li, Futeng Liu, Wanli Peng, Optimus Qin, Yifei Su, Nan Sun, Qiao Sun, Runze Suo, Heyun Wang, Yunhong Wang, Rujie Wu, Caoyu Xia, Lina Zhang, Jack Zhao, Guoliang Chen, Wenlong Chen, Xinze He, Bin Li, Qing Li, Zhuorong Li, Heng Qu, Wenxuan Song, Diyun Xiang, Yifan Xie, Peiran Xu, Hangjun Ye, Wen Ye, Han Zhao, Quanyun Zhou
Robotics · Cross-listed from cs.HC
Interactive 3D Tangible Display with a High-Speed Stiffness-Variable Jamming Module
Chanyoung Ahn, Jaesung Lee and Donhyun Hwang
Multisensory integration, particularly through visual and tactile feedback, plays a crucial role in enhancing audience engagement with artworks. Although recent research has increasingly explored tactile experiences in art, existing systems often lack real-time variable stiffness modulation and depend on bulky mechanical infrastructures. In this work, we propose a novel tangible display based on a
magnetic jamming mechanism, enabling real-time, low-noise, and low-voltage stiffness modulation integrated into traditional sculptural artworks. Our system combines visual motion and dynamic tactile feedback within a compact standalone module, allowing audiences to interactively experience variations in the
rigidity and form of features such as those found in the traditional Korean mask Hahoetal. This approach offers a new paradigm for interactive art, enabling more immersive, multisensory engagement through the fusion of cultural artifacts and modern technology. Our project page is available at https://cold-young.github.io/jamming_tangible/.
Read the full abstract ▾
Robotics
RoboTTT: Context Scaling for Robot Policies
Yunfan Jiang, Yevgen Chebotar, Ruijie Zheng et al. (+8 more)
Recent
robot foundation models operate with single-step or short-history visuomotor context. We introduce Test-Time-Training
Robot Policies (RoboTTT), a
robot model and training recipe that scale visuomotor context to 8K timesteps, three orders of magnitude beyond state-of-the-art policies, without growing inference latency. At this context length, we unlock new
robot capabilities: one-shot in-context imitation from human
video demonstrations, on-the-fly policy improvement, robustness to perturbations, and stronger performance on multi-stage, long-horizon tasks. We also observe, for the first time, steady gains in closed-loop performance as pretraining context length scales. At its core, RoboTTT integrates Test-Time Training into
robot foundation models such as
Vision-
Language-Action policies, yielding a sequence model whose recurrent state consists of fast weights, parameters updated by gradient descent during both training and inference, compressing histories into weight space and retrieving contextual information for long-context conditioning. To scale training context length, the recipe combines sequence action forcing with truncated backpropagation through time. On challenging real-
robot manipulation tasks, RoboTTT improves overall performance by 87% over the single-step context baseline and fully completes a five-minute, ten-stage assembly task, which no baseline ever does. RoboTTT trained with 8K-timestep context outperforms the same model pretrained with 1K timesteps by 62%, suggesting context length as a new scaling axis for
robot foundation models.
Videos are available at https://research.nvidia.com/labs/gear/robottt/
Read the full abstract ▾
All authors Yunfan Jiang, Yevgen Chebotar, Ruijie Zheng, Fengyuan Hu, Yunhao Ge, Jimmy Wu, Tianyuan Dai, Scott Reed, Li Fei-Fei, Yuke Zhu, Linxi "Jim" Fan
Robotics · Cross-listed from cs.LG
BadWAM: When World-Action Models Dream Right but Act Wrong
Qi Li, Xingyi Yang and Xinchao Wang
World-action models (WAMs) are emerging as a promising foundation for
embodied control: rather than predicting actions alone, they learn representations that couple action generation with future world prediction. This coupling is often viewed as a source of robustness, interpretability, and safety, as a
robot's action can in principle be checked against its imagined future. In this paper, we show that this assumption is fragile. We introduce BadWAM, a unified framework for modeling and evaluating World-Action Drift Attacks: a new class of WAM-specific
adversarial attacks that use small visual perturbations to break the
alignment between what a WAM imagines and what it executes. BadWAM characterizes this attack surface along two natural criteria: attack strength and stealthiness. When the adversary prioritizes disruption, BadWAM instantiates an action-only
adversarial attack, which directly drives the model toward task-failing actions. When the adversary additionally prioritizes stealth, BadWAM instantiates an imagination-preserving
adversarial attack, which seeks to induce harmful action shifts while keeping the model's predicted future close to its clean imagination. Together, these two attacks capture a spectrum of WAM-specific failures: from overt action hijacking to stealthier cases where the model appears to imagine a plausible future but executes a desynchronized action. We evaluate BadWAM across different variants of WAMs. Results show that our attacks substantially reduce task success rates under closed-loop execution. For example, our action-only attack reduces the model performance from 96.5% to 43.1% success. The results of our imagination-preserving attack further exposes a WAM-specific vulnerability: moderate future-preserving regularization can maintain strong attack performance while reducing future imagination drift.
Read the full abstract ▾
Robotics · Cross-listed from cs.MA
Stigmergic Graph Memory: An Environment-Aware Approach for Many-to-Many Multi-Agent Pickup and Delivery
Aditya Dutta and Joon-Seok Kim
Automated fulfillment warehouses must continuously assign and execute pickup-and-delivery work while avoiding congestion. In many-to-many Multi-
Agent Pickup and Delivery (MAPD), a request specifies a stock-keeping unit rather than fixed endpoints, requiring the controller to select an
agent, source, and destination before path planning. Existing graph-guidance methods primarily influence routing after goals are fixed, leaving endpoint instantiation uninformed by recent traffic. We introduce Stigmergic Graph
Memory (SGM), a bounded, decaying
memory layer that records recent execution signals on warehouse nodes and directed edges to rank feasible endpoints and route preferences without altering collision constraints or planner validity. Across paired request streams on five layouts, three load levels, and 25 seeds per condition, SGM outperforms two reconstructed many-to-many allocation baselines in all 15 map-load conditions, with paired throughput gains of 20.5-36.7%. These results show that recent execution
memory can improve warehouse throughput by shaping which feasible goals enter the planner, not only how
agents travel to already fixed goals.
Read the full abstract ▾
Robotics
AHEAD: Anticipatory Hand-Driven Teleoperation via Human Intent Prediction
Seok Joon Kim, Junho Lee, Federica Spinola et al. (+2 more)
Direct hand-driven teleoperation maps an operator's hand motion to
robot end-effector commands at every frame, enabling precise control, but it requires constant monitoring and correction during approach, grasp, and placement, which can be slow and fatiguing. For repetitive pick-and-place tasks, supervisory (goal-based) teleoperation simplifies this process: the operator specifies goals/waypoints, and the
robot executes the motion using planning
algorithms. Yet, this introduces latency, as the
robot must wait for the next command before it can plan and act. "How can we reduce
robot reaction time while lowering operator workload?" To tackle this question, we present AHEAD, a real-time VR teleoperation system that anticipates operator intent to enable proactive, hand-driven control. In a digital twin, the operator performs pick-and-place naturally, using hand motion to convey high-level commands rather than a continuous
robot trajectory. AHEAD processes a short window of 3D hand and head signals together with scene context through an
attention-based classifier to predict the intended grasp object and placement slot. A state machine converts intent predictions into stable
robot goals, enabling early motion while remaining stable under noisy predictions and corrective hand movements. AHEAD's intent prediction module achieves Top1 accuracy: 76% for grasp objects and 76% for target slots. Moreover, our user study shows AHEAD reduces
robot reaction latency by 0.6 s (object) and 1.4 s (slot) relative to baselines. Participants also reported lower operator load, indicating faster
robot responses while maintaining low operator effort in practice.
Read the full abstract ▾
All authors Seok Joon Kim, Junho Lee, Federica Spinola, Taein Kwon, Mohsen Moghaddam
Robotics
Scaling Behavior Foundation Model for Humanoid Robots
Weishuai Zeng, Kangning Yin, Xiaojie Niu et al. (+15 more)
Humanoid control requires natural whole-body coordination, precise real-time responses to control signals, and robust generalization across diverse environmental contexts, making it a cornerstone for generalist
embodied agents. Behavior Foundation Models (BFMs) have recently emerged as a promising solution to address these challenges by leveraging large-scale behavioral data to achieve superior expressiveness, versatility and generalization. However, despite growing interest in scaling BFMs to further improve their capabilities, it remains unclear how key factors, including the learning paradigm, behavioral data and model architecture should be coordinated to enable effective scaling. In this work, we revisit the scaling recipe for BFMs and demonstrate that substantial performance gains can be achieved through the coordination of three core components: 1) the learning paradigm of motion tracking that reformulates diverse
humanoid control problems as the reproduction of integrated whole-body behaviors in the global frame; 2) the strategic synergy between on-policy rollout quantity and reference motion diversity; and 3) the expressive and scalable model architecture termed
Humanoid Transformer that facilitates the natural emergence of structured behavioral representations. Through extensive experiments in both simulation and real-world deployment, we demonstrate that our approach yields significant improvements in control fidelity and task generalization, reducing Mean Per-Keypoint Position Error (MPKPE) on the test set by over 10% in local mode and 82% in global mode compared with existing
humanoid controllers. These results establish BFM as a principled and effective foundation for scalable and general-purpose
humanoid control.
Read the full abstract ▾
All authors Weishuai Zeng, Kangning Yin, Xiaojie Niu, Shunlin Lu, Weixiang Zhong, Jiahe Chen, Feiyu Jia, Xiao Chen, Zirui Wang, Furui Xu, Ming Zhou, Kailin Li, Weinan Zhang, He Wang, Li Yi, Dahua Lin, Jiangmiao Pang, Jingbo Wang
Robotics
Assessing Physical Frailty and Fall-Risk Indicators with Social Robots: An in situ Evaluation with Older Adults
Aniol Civit, Antonio Andriella, Alba Martínez et al. (+4 more)
Frailty assessments are crucial to evaluate the risk of adverse events and the health and social care needs of older adults, yet their administration remains resource-intensive and typically relies on coarse clinical outcomes, such as task completion times, which may overlook biomechanical indicators of functional decline. To address this, we present a robotic framework that guides older adults through standardised frailty and fall-risk tests while capturing clinical scores and additional frailty-related metrics, offering a deeper insight into a user's condition. The system uses a Behaviour Tree architecture that coordinates perception, decision-making, interaction, and measurement modules. Using
vision-based skeleton tracking, the
robot evaluates established clinical tests, including the Short Physical Performance Battery (SPPB) and the Timed Up and Go (TUG). The framework was co-designed with healthcare professionals and evaluated in situ during six months in a rehabilitation centre's research lab with N=81 older adults.
Robot-derived measurements were compared against therapist assessments and clinical reference instruments, including a gait analysis walkway and an inertial measurement unit (IMU). Results showed excellent agreement for most test completion times and gait-related parameters ($ICC > 0.9$). And, substantial agreement for the overall SPPB score comparing the
robot and the therapist ($k = 0.67$) and moderate agreement comparing the
robot and the IMU ($k=0.55$). The findings highlight that social
robots can provide reliable and objective frailty assessments in healthcare settings while enabling the collection of relevant mobility indicators beyond conventional outcomes.
Read the full abstract ▾
All authors Aniol Civit, Antonio Andriella, Alba Martínez, Joan Ars, Aida Ribera, Cristian Barrué, Guillem Alenyà
Robotics
Catch, Throw, Repeat: Planning for Human-Robot Partner Juggling
Jonathan Rainer Lippert, Kai Ploeger, Abir Chowdhury et al. (+3 more)
Dynamic object exchange between humans and
robots remains a challenging problem due to uncertainty in perception, timing, and contact-rich interaction. Human-
robot juggling represents a particularly demanding instance of this problem, requiring precise real-time coordination, predictive motion planning with feedback control, and robustness to variability in human motion. Enabling such skills is of interest for advancing physical human-
robot interaction and shared autonomy. We present a real-time planning and control architecture for human-
robot partner juggling that enables a
robot to reliably catch and throw balls in synchronized multi-ball patterns with a human partner. The system integrates predictive ball tracking, adaptive online trajectory optimization using a multiple-shooting formulation, and a state-machine-based coordination logic to enable synchronized multi-ball human-
robot partner juggling. In a user study with 8 participants of varying juggling skill from beginner to expert, we demonstrate that our system can achieve three-ball cascades shared between the
robot and the human. All participants exceeded previously reported best-case results within a 10-minute test session, with one participant extending the previous record for shared three-ball cascade juggling fivefold to 20 consecutive
robot catches, and another participant achieving a 100% success rate with 40 consecutive catches in a single-ball catch-and-return setting.
Video documentation can be found at https://kai-ploeger.com/partner-juggling
Read the full abstract ▾
All authors Jonathan Rainer Lippert, Kai Ploeger, Abir Chowdhury, Hermann Müller, Jan Peters, Alap Kshirsagar
Robotics
Goal-Oriented Semantic Communication for Distributed ISAC-Enabled Vehicle Coordination
Wenjie Liu and Yansha Deng
Vehicle coordination at unsignalized intersections relies on accurate real-time vehicle state acquisition and reliable command-and-control (C&C) signal delivery. However, existing studies typically treat sensing, communication, and control separately, which may lead to redundant transmissions, outdated state information, and unreliable vehicle coordination. In this paper, we investigate a new scenario of distributed integrated sensing and communication (
ISAC)-enabled vehicle coordination at intersections, where multiple roadside units (RSUs) collaboratively transmit sensing signals for vehicle state acquisition and C&C signals for vehicle movement control under the management of a central base station (BS). To improve signaling efficiency, we propose a unified goal-oriented semantic communication (GSC) framework, which transmits sensing and C&C signals only when they are semantically important for improving intersection traffic throughput. Specifically, an extended Kalman filter (EKF) is adopted to predict vehicle states and fuse distributed sensing measurements. A masked hybrid proximal policy optimization (MHPPO) framework is then developed to jointly determine sensing transmission decisions, C&C transmission decisions, and C&C signal contents based on a value-of-information (VoI) reward. Furthermore, we propose an uncertainty-aware transmission design (UTD), including robust beamforming and VoI-based time-division power allocation, to improve sensing and communication reliability under vehicle state uncertainty and inter-RSU interference. Simulation results show that our proposed framework achieves 100% collision-free vehicle coordination with significantly reduced signaling overhead compared with predictive
ISAC baselines adapted from state-of-the-art related studies and several ablation baselines.
Read the full abstract ▾
Robotics
DriftWorld: Fast World Modeling through Drifting
Susie Lu, Haonan Chen, Weirui Ye et al. (+1 more)
Predictive world models enable
robots to simulate the visual outcomes of their actions, but state-of-the-art
diffusion-based models remain costly because generating each rollout requires multi-step iterative denoising. We introduce DriftWorld, an action-conditioned world model based on drifting generative models. DriftWorld learns a conditional drift during training, enabling it to generate future observations for a given action sequence in a single forward pass during inference. Across Bridge-V2, RT-1,
Language Table, Push-T, and Robomimic, DriftWorld runs at over 40 fps and is 12+ times faster than
diffusion-based baselines, while matching or improving their visual generation quality. This makes DriftWorld an efficient world model for
robot simulation and further enables downstream applications including inference-time action search and offline policy evaluation.
Read the full abstract ▾
All authors Susie Lu, Haonan Chen, Weirui Ye, Yilun Du
Robotics · Cross-listed from cs.CV
SUFLECA: Scaling Up Feature Learning for CAD-to-image Alignment
Saad Ejaz, Miguel Fernandez-Cortizas, Javier Civera et al. (+2 more)
CAD-to-image
alignment aims to estimate an object's 9D pose (rotation, translation, and anisotropic scale) from a single RGB image, with applications in robotics and augmented reality. Recent zero-shot methods use
vision foundation models to match image regions to CAD models; yet their correspondences are typically appearance-driven or unreliable under occlusion or synthetic-to-real domain shift. To address these limitations, we introduce SUFLECA (Scaling Up Feature LEarning for CAD-to-image
Alignment), a weakly supervised framework for zero-shot CAD
alignment with two key contributions. First, SUFLECA scales up geometry-grounded feature learning from pretrained visual representations through Normalized Object Coordinates (NOCs) supervision on images spanning up to 12 real and synthetic
datasets, learning compact geometry-aware features that generalize across domains. Second, we propose a geometrically consistent matching
algorithm that establishes reliable CAD-to-image correspondences. Together, these contributions enable accurate, sub-second
alignment per object instance without iterative pose refinement. On ScanNet25k, SUFLECA achieves 32.8%/42.6% category/instance accuracy, outperforming the strongest zero-shot baseline by 9.7/12.5 percentage points with a smaller computational footprint, and for the first time on this
benchmark, even surpassing existing pose-supervised methods. Code is available at: https://github.com/snt-arg/SUFLECA
Read the full abstract ▾
All authors Saad Ejaz, Miguel Fernandez-Cortizas, Javier Civera, Holger Voos, Jose Luis Sanchez-Lopez
Robotics
Learning Agile Navigation in Crowded Environments for Quadruped Robots
Shuyu Wu, Zeyu Liu, Tianbao Zhang et al. (+6 more)
Navigating dynamic and crowded environments presents significant challenges for quadruped
robots due to severe sensor occlusion and unpredictable human motion. Existing approaches face a trade-off: model-based methods, such as Velocity Obstacles (VO), theoretically guarantee safety but rely on accurate obstacle motion estimates that often fail in dense crowds, while end-to-end learning methods offer robustness but lack motion prediction capability of obstacles, leading to collisions or conservative behaviors. To solve this, we propose VOP-Nav, a novel
navigation system that combines the
geometric safety of VO with the agile adaptability of end-to-end learning. Using only local onboard observations, our system avoids explicit obstacle detection and tracking pipelines. The VOP-Net processes multi-frame
LiDAR data to implicitly encode dynamic constraints and predict a safe velocity region derived from Velocity Obstacle theory. Importantly, the VO predictions serve a dual role: they are used as input to the
navigation policy during inference and as a reward signal during training to encourage safe motion. Evaluations in Isaac Gym demonstrate that VOP-Nav achieves higher success rates than all baselines while balancing locomotion speed and collision avoidance. Real-world deployment on a Unitree Go2 quadruped
robot further validates the system's robustness and efficiency in complex indoor and outdoor dynamic environments.
Read the full abstract ▾
All authors Shuyu Wu, Zeyu Liu, Tianbao Zhang, Fanxing Li, Fangyu Sun, Mingkang Xiong, Wei Xi, Wenxian Yu, Danping Zou
Robotics
Risk-Aware Belief Control Barrier Functions over Random Finite Sets
Shaohang Han, Gang Chen, Yixi Cai et al. (+5 more)
Ensuring
robot safety in unknown, dynamic environments is a fundamental requirement. It involves inferring the states of an unknown and time-varying number of moving objects from noisy, incomplete measurements. We address safe control under the induced multi-object state uncertainty with a risk-aware belief control barrier function (BCBF) framework. The uncertainty is captured by a random finite set (RFS) belief, estimated by a sequential Monte Carlo probability hypothesis density (SMC-PHD) filter that represents it with a set of particles. Building directly on these particles, we construct a nonsmooth BCBF, establish forward invariance of the safe set under continuous prediction, and derive an explicit condition under which discrete updates preserve safety. Simulation and real-world
underwater experiments demonstrate the effectiveness and efficiency of the proposed approach.
Read the full abstract ▾
All authors Shaohang Han, Gang Chen, Yixi Cai, Ignacio Torroba, Ivan Stenius, Patric Jensfelt, Javier Alonso-Mora, Jana Tumova
Robotics
CosFly-VLA: A Spatially Aware Vision-Language-Action Model for UAV Tracking
Ruilong Ren, Songsheng Cheng, Yunpeng Zhou et al. (+9 more)
Dynamic target tracking is essential for Unmanned Aerial Vehicles (
UAVs) operating in complex urban environments, where both the target and the camera viewpoint change continuously. Existing
Vision-
Language-Action (VLA) policies can track visible targets effectively, but their performance often degrades when buildings, vegetation, or roadside objects block the line of sight. During sustained occlusion, a policy may lose the target state, execute actions toward an incorrect region, and amplify this error through subsequent observations until re-acquisition becomes impossible. To this end, we present CosFly-VLA, a spatially aware VLA model that jointly grounds the target, estimates its visibility, and generates continuous flight actions through a structured prediction interface. To train this policy, we use a large-scale recipe over diverse data sources. Spatially Grounded Continued Pretraining (CPT) on a 500k mixed pool injects
UAV-view depth, distance, and 3-D spatial
reasoning. A three-stage Curriculum-based Supervised Fine-Tuning (SFT) process then specializes the tracker through multi-head warm-up followed by two-stage curriculum learning over natural and hard / long-occlusion data. Chain-of-Thought (CoT) training subsequently teaches recovery-oriented
reasoning traces before structured answers. Finally, a closed-loop Reinforcement Learning (RL) stage optimizes tracking behavior with a multi-component reward covering stand-off tracking, grounding quality, collision avoidance, and task success. Relative to OpenVLA, CosFly-VLA-0.8B reduces open-loop Average Displacement Error (ADE) by 34.1% on seen-test and 35.3% on unseen-test. Closed-loop optimization improves Success Rate (SR) by 29.8% and 2.5%, respectively. These results demonstrate progress from visible-frame imitation toward spatially grounded action-closed-loop control, evaluated under a shared oracle state history.
Read the full abstract ▾
All authors Ruilong Ren, Songsheng Cheng, Yunpeng Zhou, Hanxuan Chen, Xiangyue Wang, Tianle Zeng, Shuai Yuan, Binbo Li, Hanzhong Guo, Ji Pei, Da Zhang, Kangli Wang
Robotics
AeroAct: Action-Centered World-Action Models for Language-Conditioned Quadrotor Flight
Xinhong Zhang, Qiyuan Zhu, Yubo Huang et al. (+8 more)
Language-conditioned quadrotor flight requires a policy to ground semantic goals, anticipate the visual consequences of ego-motion, and output control references that remain smooth and dynamically executable under rapidly changing first-person views. Existing aerial
vision-
language navigation and
vision-
language-action methods commonly use discrete actions, high-level waypoints, or instantaneous velocity commands, which provide limited supervision about how flight actions change future observations. We present AeroAct, an action-centered world-action model (WAM) for quadrotor
navigation. To the best of our knowledge, AeroAct is the first WAM instantiated and demonstrated for real-world aerial flight. The model adapts a pretrained
video diffusion Transformer to predict local trajectory-action chunks from
egocentric visual history, proprioception, and
language. Future first-person frames are used during training as dense consequence supervision, while deployment directly decodes actions without generating future
video. To obtain aligned visual, state,
language, and dynamically feasible action data, we build a DiffAero-based pipeline with complementary Isaac Lab and 3D Gaussian splatting renderers. We further introduce a low-cost handheld collection device that couples camera observations with motion estimates to recreate flight-like
egocentric trajectories, and a self-guidance procedure that improves temporal consistency across overlapping trajectory chunks. Closed-loop simulation and real-world experiments show that temporal visual context improves target tracking and object-search performance, and that WAM-based policies can be executed on a physical quadrotor.
Read the full abstract ▾
All authors Xinhong Zhang, Qiyuan Zhu, Yubo Huang, Haolin Chen, Runqing Wang, Yuhao Mo, Zhongxin Chen, Yu Hu, Xinjiang Wang, Jian Sun, Gang Wang
Robotics
Steering Robustness into World Action Models via Mechanistic Interpretability and Optimal Control
Jihoon Hong, Julian Skifstad, Qiyue Dai et al. (+2 more)
World Action Models (WAMs) enable semantically- and physically-informed control but are brittle under distribution shift. In this work, we use mechanistic interpretability to study how robustness-relevant perturbations are represented in WAM activation space. Comparing activations across successful and unsuccessful rollouts, we find some WAM architectures exhibit low-dimensional linear separability for robustness-critical features, while others do not. This motivates the use of contrastive activation directions for training-free WAM steering. We also show that local linearity in WAM activation dynamics enables efficient feedback steering via model-based optimal control, yielding World-Action Linear Quadratic Regulator (WA-LQR), a minimally-invasive reduced-order LQR controller. Via mechanistic evaluations, we predict strong steerability in the Cosmos-Policy and DiT4DiT models but weak steerability in LingBot-VA, consistent with steering intervention results. On Cosmos-Policy and DiT4DiT, WA-LQR generalizes contrastive directions to new tasks and improves robustness to camera, gripper, and visual-noise perturbations over unsteered and
prompt steering baselines.
Read the full abstract ▾
All authors Jihoon Hong, Julian Skifstad, Qiyue Dai, Alice Chan, Glen Chou
Robotics
Human-Robot Interaction in GenAI Architectures via the Agent-Client Protocol
Jesus Moncada-Ramirez, Jose-Raul Ruiz-Sarmiento and Javier Gonzalez-Jimenez
Recent advances in Generative Artificial Intelligence (GenAI), particularly Large
Language Models (LLMs), are driving robotic architectures toward
agent-based high-level orchestration, in which natural-
language instructions can be translated into context-aware action sequences. While the integration of these
agents and robotic capabilities is increasingly converging toward standardization through the Model Context Protocol (MCP), the upper Human-
Robot Interaction (HRI) layer remains fragmented by proprietary, ad hoc interfaces that hinder real-time human-in-the-loop collaboration. To address this fragmentation, this paper proposes the adoption of the
Agent-Client Protocol (ACP) – a communication standard originally introduced for coding
agents in software engineering – as a unified communication contract for the HRI layer in
agent-based robotic systems. By combining ACP at the interface-
agent link and MCP at the
agent-execution link, we formulate a fully decoupled three-layer architecture that separates human interaction, deliberative orchestration, and physical execution. This
topology removes rigid architectural dependencies, enabling heterogeneous user interfaces to connect to the same robotic system and allowing the underlying robotic platform to be replaced without requiring client-specific integration changes. Moreover, it provides native support for collaborative HRI capabilities such as real-time observability, explicit human authorization, and immediate task interruption. We experimentally evaluate the proposed architecture on a physical mobile
robot, demonstrating interoperability across three heterogeneous user interfaces and validating real-time human-in-the-loop workflows with negligible latency overhead.
Read the full abstract ▾
Robotics
OASIS-Map: Object-Level Change Detection in Multi-Session Mapping using Semantic Correspondence Matching
Haedam Oh, Yifu Tao, Nived Chebrolu et al. (+1 more)
Map representations which are consistent across repeated visits to a real-world semi-static environment are very useful for long-term robotic inspection. In such settings, the scene may evolve while the
robot is absent, with objects appearing, disappearing, moving, or being replaced, quickly making a static map outdated. Existing change-detection methods reason through geometry, category-level semantics, or object persistence. However, achieving reliable object association across revisits remains a key challenge, especially under partial views, occlusion, and imperfect
segmentation. In this work, we propose OASIS-Map, a multi-session mapping system that maintains a spatio-temporally consistent object-level map by establishing dense patch-level semantic correspondences between temporal observations. These correspondences detect where the scene has changed and incrementally associate objects across revisits as the
robot re-observes the environment. We demonstrate OASIS-Map on three challenging real-world scenarios: object rearrangements in 3RScan, visually similar car replacements in a car park, and large-scale scene changes in an outdoor
market. We achieve 0.783 F1 on change detection in a car replacement scenario in a car park and 0.667 F1 on moved object association in 3RScan. https://dynamic.
robots.ox.ac.uk/projects/oasis-map/
Read the full abstract ▾
All authors Haedam Oh, Yifu Tao, Nived Chebrolu, Maurice Fallon
Robotics
Modeling and Validation of Quality of Control for Edge-Offloaded Collaborative Navigation
Neelabhro Roy, Mikael Hammarling, Victor Nan Fernandez-Ayala et al. (+4 more)
Collaborative control in complex environments is severely challenged by
stochastic wireless delay and reliability variations, which can degrade
navigation, tracking, and collision avoidance. These network-induced uncertainties complicate the maintenance of energy efficiency during collaborative tasks, and can potentially lead to over-provisioning of resources. In this paper, for a
navigation setup with dynamic collision avoidance, we address this challenge by expanding the quality of control (QoC) framework from prior works to practical robotic models. Our approach (i) models end-to-end network effects on closed-loop performance, (ii) systematically explores the impact of various control parameters dictating robotic motion on network latency-reliability (iii) validates these models through experiments on a private 5G testbed across varying delay, reliability and control configurations. Our analysis indicates the optimal control-communication co-design operating regimes for practical
robots and also compares the QoC performance of standard ROS~2 quality of service (QoS) policies under real-world conditions and showing how RELIABLE QoS offers 51.5% better QoC than BEST-EFFORT under certain experimental settings.
Read the full abstract ▾
All authors Neelabhro Roy, Mikael Hammarling, Victor Nan Fernandez-Ayala, Gourav Prateek Sharma, Mani H. Dhullipalla, Dimos V. Dimarogonas, James Gross