Robotics
Differentiable Inverse Graphics for Zero-shot Scene Reconstruction and Robot Grasping
Octavio Arriaga, Proneet Sharma, Jichen Guo et al. (+3 more)
Operating effectively in novel real-world environments requires robotic systems to estimate and interact with previously unseen objects. Current state-of-the-art models address this challenge by using large amounts of training data and test-time samples to build black-box scene representations. In this work, we introduce a differentiable neuro-graphics model that combines neural foundation models with physics-based differentiable rendering to perform zero-shot scene reconstruction and
robot grasping without relying on any additional 3D data or test-time samples. Our model solves a series of constrained optimization problems to estimate physically consistent scene parameters, such as meshes, lighting conditions, material properties, and 6D poses of previously unseen objects from a single RGBD image and bounding boxes. We evaluated our approach on standard model-free few-shot
benchmarks and demonstrated that it outperforms existing
algorithms for model-free few-shot pose estimation. Furthermore, we validated the accuracy of our scene reconstructions by applying our
algorithm to a zero-shot grasping task. By enabling zero-shot, physically-consistent scene reconstruction and grasping without reliance on extensive
datasets or test-time sampling, our approach offers a pathway towards more data efficient, interpretable and generalizable
robot autonomy in novel environments.
Read the full abstract ▾
All authors Octavio Arriaga, Proneet Sharma, Jichen Guo, Marc Otto, Siddhant Kadwe, Rebecca Adam
Robotics
Signal or 'Noise': Human Reactions to Robot Errors in the Wild
Maia Stiber, Sameer Khan, Russell Taylor et al. (+1 more)
In the real world,
robots frequently make errors, yet little is known about people's social responses to errors outside of lab settings. Prior work has shown that social signals are reliable and useful for error management in constrained interactions, but it is unclear if this holds in the real world - especially with a non-social
robot in repeated and group interactions with successive or propagated errors. To explore this, we built a coffee
robot and conducted a public field deployment ($N = 49$). We found that participants consistently expressed varied social signals in response to errors and other stimuli, particularly during group interactions. Our findings suggest that social signals in the wild are rich (with participants volunteering information about the interaction), but "noisy." We discuss lessons, benefits, and challenges for using social signals in real-world HRI.
Read the full abstract ▾
All authors Maia Stiber, Sameer Khan, Russell Taylor, Chien-Ming Huang
Robotics · Cross-listed from cs.HC
Applying Ground Robot Fleets in Urban Search: Understanding Professionals' Operational Challenges and Design Opportunities
Puqi Zhou, Charles R. Twardy, Cynthia Lum et al. (+6 more)
Urban searches demand rapid, defensible decisions and sustained physical effort under high cognitive and situational load. Incident commanders must plan, coordinate, and document time-critical operations, while field searchers execute evolving tasks in uncertain environments. With recent advances in technology, ground-
robot fleets paired with computer-
vision-based situational awareness and LLM-powered interfaces offer the potential to ease these operational burdens. However, no dedicated studies have examined how public safety professionals perceive such technologies or envision their integration into existing practices, risking building technically sophisticated yet impractical solutions. To address this gap, we conducted focus-group sessions with eight police officers across five local departments in Virginia. Our findings show that ground
robots could reduce professionals' reliance on paper references, mental calculations, and ad-hoc coordination, alleviating cognitive and physical strain in four key challenge areas: (1) partitioning the workforce across multiple search hypotheses, (2) retaining group awareness and situational awareness, (3) building route planning that fits the lost-person profile, and (4) managing cognitive and physical fatigue under uncertainty. We further identify four design opportunities and requirements for future ground-
robot fleet integration in public-safety operations: (1) scalable multi-
robot planning and control interfaces, (2) agency-specific route optimization, (3) real-time replanning informed by debrief updates, and (4)
vision-assisted cueing that preserves operational trust while reducing cognitive workload. We conclude with design implications for deployable, accountable, and human-centered urban-search support systems
Read the full abstract ▾
All authors Puqi Zhou, Charles R. Twardy, Cynthia Lum, Myeong Lee, David J. Porfirio, Michael R. Hieb, Chris Thomas, Xuesu Xiao, Sungsoo Ray Hong
Robotics
Capturing Visual Environment Structure Correlates with Control Performance
Jiahua Dong, Yunze Man, Pavel Tokmakov et al. (+1 more)
The choice of visual representation is key to scaling generalist
robot policies. However, direct evaluation via policy rollouts is expensive, even in simulation. Existing proxy metrics focus on the representation's capacity to capture narrow aspects of the visual world, like object shape, limiting generalization across environments. In this paper, we take an analytical perspective: we probe pretrained visual encoders by measuring how well they support decoding of environment state – including geometry, object structure, and physical attributes – from images. Leveraging simulation environments with access to ground-truth state, we show that this probing accuracy strongly correlates with downstream policy performance across diverse environments and learning settings, significantly outperforming prior metrics and enabling efficient representation selection. More broadly, our study provides insight into the representational properties that support generalizable manipulation, suggesting that learning to encode the latent physical state of the environment is a promising objective for control.
Read the full abstract ▾
All authors Jiahua Dong, Yunze Man, Pavel Tokmakov, Yu-Xiong Wang
Robotics
PDF-HR: Pose Distance Fields for Humanoid Robots
Yi Gu, Yukang Gao, Yangchen Zhou et al. (+7 more)
Pose and motion priors play a crucial role in
humanoid robotics. Although such priors have been widely studied in human motion recovery (HMR) domain with a range of models, their adoption for
humanoid robots remains limited, largely due to the scarcity of high-quality
humanoid motion data. In this work, we introduce Pose Distance Fields for
Humanoid Robots (PDF-HR), a lightweight prior that represents the
robot pose distribution as a continuous and differentiable
manifold. Given an arbitrary pose, PDF-HR predicts its distance to a large corpus of retargeted
robot poses, yielding a smooth measure of pose plausibility that is well suited for optimization and control. PDF-HR can be integrated as a reward shaping term, a regularizer, or a standalone plausibility scorer across diverse pipelines. We evaluate PDF-HR on various
humanoid tasks, including single-trajectory motion tracking, general motion tracking, style-based motion mimicry, and general motion retargeting. Experiments show that this plug-and-play prior consistently and substantially strengthens strong baselines. Code and models will be released.
Read the full abstract ▾
All authors Yi Gu, Yukang Gao, Yangchen Zhou, Xingyu Chen, Yixiao Feng, Mingle Zhao, Yunyang Mo, Zhaorui Wang, Lixin Xu, Renjing Xu
Robotics · Cross-listed from cs.SE
Beyond the Control Equations: An Artifact Study of Implementation Quality in Robot Control Software
Nils Chur, Thorsten Berger, Einar Broch Johnsen et al. (+1 more)
A controller – a software module managing hardware behavior – is a key component of a typical
robot system. While control theory gives safety guarantees for standard controller designs, the practical implementation of controllers in software introduces complexities that are often overlooked. Controllers are often designed in continuous space, while the software is executed in discrete space, undermining some of the theoretical guarantees. Despite extensive research on control theory and control modeling, little
attention has been paid to the implementations of controllers and how their theoretical guarantees are ensured in real-world software systems. We investigate 184 real-world controller implementations in open-source
robot software. We examine their application context, the implementation characteristics, and the testing methods employed to ensure correctness. We find that the implementations often handle discretization in an ad hoc manner, leading to potential issues with real-time reliability. Challenges such as timing inconsistencies, lack of proper error handling, and inadequate consideration of real-time constraints further complicate matters. Testing practices are superficial, no systematic verification of theoretical guarantees is used, leaving possible inconsistencies between expected and actual behavior. Our findings highlight the need for improved implementation guidelines and rigorous verification techniques to ensure the reliability and safety of robotic controllers in practice.
Read the full abstract ▾
All authors Nils Chur, Thorsten Berger, Einar Broch Johnsen, Andrzej Wąsowski
Robotics · Cross-listed from cs.HC
PuppetAI: A Customizable Platform for Designing Tactile-Rich Affective Robot Interaction
Jiaye Li, Tongshun Chen, Siyi Ma et al. (+2 more)
We introduce PuppetAI, a modular soft
robot interaction platform. This platform offers a scalable cable-driven actuation system and a customizable, puppet-inspired
robot gesture framework, supporting a multitude of interaction gesture
robot design formats. The platform comprises a four-layer decoupled software architecture that includes perceptual processing, affective modeling, motion scheduling, and low-level actuation. We also implemented an affective expression loop that connects human input to the
robot platform by producing real-time emotional gestural responses to human vocal input. For our own designs, we have worked with nuanced gestures enacted by "soft
robots" with enhanced dexterity and "pleasant-to-touch" plush exteriors. By reducing operational complexity and production costs while enhancing customizability, our work creates an adaptable and accessible foundation for future tactile-based expressive
robot research. Our goal is to provide a platform that allows researchers to independently construct or refine highly specific gestures and movements performed by social
robots.
Read the full abstract ▾
All authors Jiaye Li, Tongshun Chen, Siyi Ma, Elizabeth Churchill, Ke Wu
Robotics
Dull, Dirty, Dangerous: Understanding the Past, Present, and Future of a Key Motivation for Robotics
Nozomi Nakajima, Pedro Reynolds-Cuéllar, Caitrin Lynch et al. (+1 more)
In robotics, the concept of "dull, dirty, and dangerous" (DDD) work has been used to motivate where
robots might be useful. In this paper, we conduct an empirical analysis of robotics publications between 1980 and 2024 that mention DDD, and find that only 2.7% of publications define DDD and 8.7% of publications provide concrete examples of tasks or jobs that are DDD. We then review the social science literature on "dull," "dirty," and "dangerous" work to provide definitions and guidance on how to conceptualize DDD for robotics. Finally, we propose a framework that helps the robotics community consider the job context for our technology, encouraging a more informed perspective on how robotics may impact human labor.
Read the full abstract ▾
All authors Nozomi Nakajima, Pedro Reynolds-Cuéllar, Caitrin Lynch, Kate Darling
Robotics · Cross-listed from cs.CV
AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generation
Jin-Chuan Shi, Binhong Ye, Tao Liu et al. (+6 more)
Reconstructing dynamic hand-object interactions from
monocular videos is critical for dexterous manipulation data collection and creating realistic digital twins for robotics and VR. However, current methods face two prohibitive barriers: (1) reliance on neural rendering often yields fragmented, non-simulation-ready geometries under heavy occlusion, and (2) dependence on brittle Structure-from-Motion (SfM) initialization leads to frequent failures on in-the-wild footage. To overcome these limitations, we introduce AGILE, a robust framework that shifts the paradigm from reconstruction to agentic generation for interaction learning. First, we employ an agentic pipeline where a
Vision-
Language Model (VLM) guides a generative model to synthesize a complete, watertight object mesh with high-fidelity texture, independent of
video occlusions. Second, bypassing fragile SfM entirely, we propose a robust anchor-and-track strategy. We initialize the object pose at a single interaction onset frame using a foundation model and propagate it temporally by leveraging the strong visual similarity between our generated asset and
video observations. Finally, a contact-aware optimization integrates semantic,
geometric, and interaction stability constraints to enforce physical plausibility. Extensive experiments on HO3D, DexYCB, ARCTIC, and in-the-wild
videos reveal that AGILE outperforms baselines in global
geometric accuracy while demonstrating exceptional robustness on challenging sequences where prior arts frequently collapse. By prioritizing physical validity, our method produces simulation-ready assets validated via real-to-sim retargeting for robotic applications. Project page: https://agile-hoi.github.io.
Read the full abstract ▾
All authors Jin-Chuan Shi, Binhong Ye, Tao Liu, Junzhe He, Yangjinhui Xu, Xiaoyang Liu, Zeju Li, Hao Chen, Chunhua Shen
Robotics
From Vision to Assistance: Gaze and Vision-Enabled Adaptive Control for a Back-Support Exoskeleton
Alessandro Leanza, Paolo Franceschi, Blerina Spahiu et al. (+1 more)
Back-support exoskeletons have been proposed to mitigate spinal loading in industrial handling, yet their effectiveness critically depends on timely and context-aware assistance. Most existing approaches rely either on load-estimation techniques (e.g., EMG, IMU) or on
vision systems that do not directly inform control. In this work, we present a
vision-gated control framework for an active lumbar occupational exoskeleton that leverages
egocentric vision with wearable gaze tracking. The proposed system integrates real-time grasp detection from a first-person YOLO-based perception system, a finite-state machine (FSM) for task progression, and a variable admittance controller to adapt torque delivery to both posture and object state. A user study with 15 participants performing stooping load lifting trials under three conditions (no exoskeleton, exoskeleton without
vision, exoskeleton with
vision) shows that
vision-gated assistance significantly reduces perceived physical demand and improves fluency, trust, and comfort. Quantitative analysis reveals earlier and stronger assistance when
vision is enabled, while questionnaire results confirm user preference for the
vision-gated mode. These findings highlight the potential of
egocentric vision to enhance the responsiveness, ergonomics, safety, and acceptance of back-support exoskeletons.
Read the full abstract ▾
All authors Alessandro Leanza, Paolo Franceschi, Blerina Spahiu, Loris Roveda
Robotics
Relational Scene Graphs for Object Grounding of Natural Language Commands
Julia Kuhn, Francesco Verdoja, Tsvetomila Mihaylova et al. (+1 more)
Robots are finding wider adoption in human environments, increasing the need for natural human-
robot interaction. However, understanding a natural
language command requires the
robot to infer the intended task and how to decompose it into executable actions, and to ground those actions in the
robot's knowledge of the environment, including relevant objects,
agents, and locations. This challenge can be addressed by combining the capabilities of large
language models (LLMs) to understand natural
language with 3D scene graphs (3DSGs) for grounding inferred actions in a semantic representation of the environment. However, many 3DSGs lack explicit spatial relations between objects, even though humans often rely on these relations to describe an environment. This paper investigates whether incorporating open- or closed-vocabulary spatial relations into 3DSGs can improve the ability of LLMs to interpret natural
language commands. To address this, we implement two pipelines using off-the-shelf models: an LLM-based pipeline for target object grounding from open-vocabulary
language commands and a
vision language model-based pipeline to add open-vocabulary spatial edges to 3DSGs from images captured while mapping. Finally, we evaluate two LLMs across 14 scenes using 905 natural
language statements (786 procedurally-generated, 119 human-authored) to assess performance on the downstream task of target object grounding. Our study demonstrates that explicit spatial relations improve the ability of LLMs to ground objects, and while open-vocabulary relation generation with
vision language models proves feasible from
robot-captured images, our analysis did not yield evidence favoring either open- or closed-vocabulary relations.
Read the full abstract ▾
All authors Julia Kuhn, Francesco Verdoja, Tsvetomila Mihaylova, Ville Kyrki
Robotics
Radar-Inertial Odometry For Computationally Constrained Aerial Navigation
Jan Michalczyk
Recently, the progress in the radar sensing technology consisting in the miniaturization of the packages and increase in measuring precision has drawn the interest of the robotics research community. Indeed, a crucial task enabling autonomy in robotics is to precisely determine the pose of the
robot in space. To fulfill this task sensor fusion
algorithms are often used, in which data from one or several exteroceptive sensors like, for example,
LiDAR, camera,
laser ranging sensor or GNSS are fused together with the Inertial Measurement Unit (IMU) measurements to obtain an estimate of the
navigation states of the
robot. Nonetheless, owing to their particular sensing principles, some exteroceptive sensors are often incapacitated in extreme environmental conditions, like extreme illumination or presence of fine particles in the environment like smoke or fog. Radars are largely immune to aforementioned factors thanks to the characteristics of electromagnetic waves they use. In this thesis, we present Radar-Inertial Odometry (RIO)
algorithms to fuse the information from IMU and radar in order to estimate the
navigation states of a (Uncrewed Aerial Vehicle)
UAV capable of running on a portable resource-constrained embedded computer in real-time and making use of inexpensive, consumer-grade sensors. We present novel RIO approaches relying on the multi-state tightly-coupled Extended Kalman Filter (EKF) and Factor Graphs (FG) fusing instantaneous velocities of and distances to 3D points delivered by a lightweight, low-cost, off-the-shelf Frequency Modulated Continuous Wave (FMCW) radar with IMU readings. We also show a novel way to exploit advances in deep learning to retrieve 3D point correspondences in sparse and noisy radar point clouds.
Read the full abstract ▾
Robotics
Can We Redesign a Shoulder Exosuit to Enhance Comfort and Usability Without Losing Assistance?
Roberto Ferroni, Daniele Filippo Mauceri, Jacopo Carpaneto et al. (+2 more)
Reduced shoulder mobility limits upper-limb function and the performance of activities of daily living across a wide range of conditions. Wearable exosuits have shown promise in assisting arm elevation, reducing muscle effort, and supporting functional movements; however, comfort is rarely prioritized as an explicit design objective, despite it strongly affects real-life, long-term usage. This study presents a redesigned soft shoulder exosuit (Soft Shoulder v2) developed to address comfort-related limitations identified in our previous version, while preserving assistive performance. In parallel, assistance was also improved, shifting from the coronal plane to the sagittal plane to better support functionally relevant hand positioning. A controlled comparison between the previous (v1) and redesigned (v2) modules was conducted in eight healthy participants, who performed static holding, dynamic lifting, and a functional pick and place task. Muscle activity, kinematics, and user-reported outcomes were assessed. Both versions increased endurance time, reduced deltoid activation, and preserved transparency during unpowered shoulder elevation. However, the difference between them emerged most clearly during functional tasks and comfort evaluation. The redesigned module facilitated forward arm positioning and increased transverse plane mobility by up to 30 deg, without increasing muscular demand. User-reported outcomes further indicated a substantial improvement in wearability, with markedly lower perceived pressure and higher ratings in effectiveness, ease of use, and comfort compared to the previous design. Taken together, these findings show that targeted, user-centered design refinements can improve comfort and functional interaction without compromising assistive performance, advancing the development of soft exosuits suitable for prolonged and daily use.
Read the full abstract ▾
All authors Roberto Ferroni, Daniele Filippo Mauceri, Jacopo Carpaneto, Alessandra Pedrocchi, Tommaso Proietti
Robotics
Act, Sense, Act: Learning Active Perception from Large-Scale Egocentric Human Data
Jialiang Li, Yi Qiao, Yunhan Guo et al. (+2 more)
Achieving generalizable manipulation in unconstrained environments requires the
robot to proactively resolve information uncertainty, i.e., the capability of active perception. However, existing methods are often confined in limited types of sensing behaviors, restricting their applicability to complex environments. In this work, we formalize active perception as a history-dependent perception-action loop driven by information-seeking action and decision branching, providing a structured categorization of visual active perception paradigms. Building on this perspective, we introduce CoMe-VLA, a cognitive and
memory-aware
vision-
language-action (VLA) framework that leverages large-scale human
egocentric data to learn versatile exploration and manipulation priors. Our framework integrates a cognitive auxiliary head for autonomous sub-task transitions and a dual-track
memory system to maintain consistent self and environmental awareness by fusing proprioceptive and visual temporal contexts. By aligning human and
robot hand-eye coordination behaviors in a unified
egocentric action space, we train the model progressively in three stages. Extensive experiments on a wheel-based
humanoid have demonstrated strong robustness and adaptability of our proposed method across diverse long-horizon tasks spanning multiple active perception scenarios.
Read the full abstract ▾
All authors Jialiang Li, Yi Qiao, Yunhan Guo, Changwen Chen, Wenzhao Lian
Robotics
A Unified Complementarity-based Approach for Rigid-Body Manipulation and Motion Prediction
Bingkun Huang, Xin Ma, Nilanjan Chakraborty et al. (+1 more)
Robotic manipulation in unstructured environments requires planners to reason jointly about free-space motion and sustained, frictional contact with the environment. Existing (local) planning and simulation frameworks typically separate these regimes or rely on simplified contact representations, particularly when modeling non-convex or distributed contact patches. Such approximations limit the fidelity of contact-mode transitions and hinder the robust execution of contact-rich behaviors in real time. This paper presents a unified discrete-time modeling framework for robotic manipulation that consistently captures both free motion and frictional contact within a single mathematical formalism (Unicomp). Building on complementarity-based rigid-body dynamics, we formulate free-space motion and contact interactions as coupled linear and nonlinear complementarity problems, enabling principled transitions between contact modes without enforcing fixed-contact assumptions. For planar patch contact, we derive a frictional contact model from the maximum power dissipation principle in which the set of admissible contact wrenches is represented by an ellipsoidal limit surface. This representation captures coupled force-moment effects, including torsional friction, while remaining agnostic to the underlying pressure distribution across the contact patch. The resulting formulation yields a discrete-time predictive model that relates generalized velocities and contact wrenches through quadratic constraints and is suitable for real-time optimization-based planning. Experimental results show that the proposed approach enables stable, physically consistent behavior at interactive speeds across tasks, from planar pushing to contact-rich whole-body maneuvers.
Read the full abstract ▾
All authors Bingkun Huang, Xin Ma, Nilanjan Chakraborty, Riddhiman Laha
Robotics · Cross-listed from cs.CV
S-MUSt3R: Sliding Multi-view 3D Reconstruction
Leonid Antsfeld, Boris Chidlovskii, Yohann Cabon et al. (+2 more)
The recent paradigm shift in 3D
vision led to the rise of foundation models with remarkable capabilities in 3D perception from uncalibrated images. However, extending these models to large-scale RGB stream 3D reconstruction remains challenging due to
memory limitations. This work proposes S-MUSt3R, a simple and efficient pipeline that extends the limits of foundation models for
monocular 3D reconstruction. Our approach addresses the scalability bottleneck of foundation models through a simple strategy of sequence
segmentation followed by segment
alignment and lightweight loop closure optimization. Without model retraining, we benefit from remarkable 3D reconstruction capacities of MUSt3R model and achieve trajectory and reconstruction performance comparable to traditional methods with more complex architecture. We evaluate S-MUSt3R on TUM, 7-Scenes and proprietary
robot navigation datasets and show that S-MUSt3R runs successfully on long RGB sequences and produces accurate and consistent 3D reconstruction. Our results highlight the potential of leveraging the MUSt3R model for scalable
monocular 3D scene in real-world settings, with an important advantage of making predictions directly in the metric space.
Read the full abstract ▾
All authors Leonid Antsfeld, Boris Chidlovskii, Yohann Cabon, Vincent Leroy, Jerome Revaud
Robotics
TACO: Temporal Consensus Optimization for Continual Neural Mapping
Xunlan Zhou, Hongrui Zhao and Negar Mehr
Neural implicit mapping has emerged as a powerful paradigm for robotic
navigation and scene understanding. However, real-world robotic deployment requires continual adaptation to changing environments under strict
memory and computation constraints, which existing mapping systems fail to support. Most prior methods rely on replaying historical observations to preserve consistency and assume static scenes. As a result, they cannot adapt to continual learning in dynamic robotic settings. To address these challenges, we propose TACO (TemporAl Consensus Optimization), a replay-free framework for continual neural mapping. We reformulate mapping as a temporal consensus optimization problem, where we treat past model snapshots as temporal neighbors. Intuitively, our approach resembles a model consulting its own past knowledge. We update the current map by enforcing weighted consensus with historical representations. Our method allows reliable past geometry to constrain optimization while permitting unreliable or outdated regions to be revised in response to new observations. TACO achieves a balance between
memory efficiency and adaptability without storing or replaying previous data. Through extensive simulated and real-world experiments, we show that TACO robustly adapts to scene changes, and consistently outperforms other continual learning baselines. Code is available at https://iconlab.negarmehr.com/TACO
Read the full abstract ▾
Robotics
EgoActor: Grounding Task Planning into Spatial-aware Egocentric Actions for Humanoid Robots via Visual-Language Models
Yu Bai, MingMing Yu, Chaojie Li et al. (+3 more)
Deploying
humanoid robots in real-world settings is fundamentally challenging, as it demands tight integration of perception, locomotion, and manipulation under partial-information observations and dynamically changing environments. As well as transitioning robustly between sub-tasks of different types. Towards addressing these challenges, we propose a novel task - EgoActing, which requires directly grounding high-level instructions into various, precise, spatially aware
humanoid actions. We further instantiate this task by introducing EgoActor, a unified and scalable
vision-
language model (VLM) that can predict locomotion primitives (e.g., walk, turn, move sideways, change height), head movements, manipulation commands, and human-
robot interactions to coordinate perception and execution in real-time. We leverage broad supervision over
egocentric RGB-only data from real-world demonstrations, spatial
reasoning question-answering, and simulated environment demonstrations, enabling EgoActor to make robust, context-aware decisions and perform fluent action inference (under 1s) with both 8B and 4B parameter models. Extensive evaluations in both simulated and real-world environments demonstrate that EgoActor effectively bridges abstract task planning and concrete motor execution, while generalizing across diverse tasks and unseen environments.
Read the full abstract ▾
All authors Yu Bai, MingMing Yu, Chaojie Li, Ziyi Bai, Xinlong Wang, Börje F. Karlsson
Robotics · Cross-listed from cs.HC
The Supportiveness-Safety Tradeoff in LLM Well-Being Agents
Himanshi Lalwani and Hanan Salam
Large
language models (LLMs) are being integrated into socially assistive
robots (SARs) and other conversational
agents providing mental health and well-being support. These
agents are often designed to sound empathic and supportive in order to maximize user's engagement, yet it remains unclear how increasing the level of supportive framing in system
prompts influences safety relevant behavior. We evaluated 6 LLMs across 3 system
prompts with varying levels of supportiveness on 80 synthetic queries spanning 4 well-being domains (1440 responses). An LLM judge framework, validated against human ratings, assessed safety and care quality. Moderately supportive
prompts improved empathy and constructive support while maintaining safety. In contrast, strongly validating
prompts significantly degraded safety and, in some cases, care across all domains, with substantial variation across models. We discuss implications for
prompt design, model selection, and domain specific safeguards in SARs deployment.
Read the full abstract ▾
Robotics · Cross-listed from cs.HC
Robot-Assisted Group Tours for Blind People
Yaxin Hu, Masaki Kuribayashi, Allan Wang et al. (+5 more)
Group interactions are essential to social functioning, yet effective engagement relies on the ability to recognize and interpret visual cues, making such engagement a significant challenge for blind people. In this paper, we investigate how a mobile
robot can support group interactions for blind people. We used the scenario of a guided tour with mixed-visual groups involving blind and sighted visitors. Based on insights from an interview study with blind people (n=5) and museum experts (n=5), we designed and prototyped a robotic system that supported blind visitors to join group tours. We conducted a field study in a science museum where each blind participant (n=8) joined a group tour with one guide and two sighted participants (n=8). Findings indicated users' sense of safety from the
robot's navigational support, concerns in the group participation, and preferences for obtaining environmental information. We present design implications for future robotic systems to support blind people's mixed-visual group participation.
Read the full abstract ▾
All authors Yaxin Hu, Masaki Kuribayashi, Allan Wang, Seita Kayukawa, Daisuke Sato, Bilge Mutlu, Hironobu Takagi, Chieko Asakawa
Robotics
Gust Estimation and Rejection with a Disturbance Observer for Proprioceptive Underwater Soft Morphing Wings
Tobias Cook, Leo Micklem, Huazhi Dong et al. (+3 more)
Unmanned
underwater vehicles are increasingly employed for maintenance and surveying tasks at sea, but their operation in shallow waters is often hindered by hydrodynamic disturbances such as waves, currents, and
turbulence. These unsteady flows can induce rapid changes in direction and speed, compromising vehicle stability and manoeuvrability. Marine organisms contend with such conditions by combining proprioceptive feedback with flexible fins and tails to reject disturbances. Inspired by this strategy, we propose soft morphing wings endowed with proprioceptive sensing to mitigate environmental perturbations. The wing's continuous deformation provides a natural means to infer dynamic disturbances: sudden changes in camber directly reflect variations in the oncoming flow. By interpreting this proprioceptive signal, a disturbance observer can reconstruct flow parameters in real time. To enable this, we develop and experimentally validate a dynamic model of a hydraulically actuated soft wing with controllable camber. We then show that
curvature-based sensing allows accurate estimation of disturbances in the angle of attack. Finally, we demonstrate that a controller leveraging these proprioceptive estimates can reject disturbances in the lift response of the soft wing. By combining proprioceptive sensing with a disturbance observer, this technique mirrors biological strategies and provides a pathway for soft
underwater vehicles to maintain stability in hazardous environments.
Read the full abstract ▾
All authors Tobias Cook, Leo Micklem, Huazhi Dong, Yunjie Yang, Michael Mistry, Francesco Giorgio-Serchi
Robotics
Integrated Exploration and Sequential Manipulation on Scene Graph with LLM-based Situated Replanning
Heqing Yang, Ziyuan Jiao, Shu Wang et al. (+3 more)
In partially known environments,
robots must combine exploration to gather information with task planning for efficient execution. To address this challenge, we propose EPoG, an Exploration-based sequential manipulation Planning framework on Scene Graphs. EPoG integrates a graph-based global planner with a Large
Language Model (LLM)-based situated local planner, continuously updating a belief graph using observations and LLM predictions to represent known and unknown objects. Action sequences are generated by computing graph edit operations between the goal and belief graphs, ordered by temporal dependencies and movement costs. This approach seamlessly combines exploration and sequential manipulation planning. In ablation studies across 46 realistic household scenes and 5 long-horizon daily object transportation tasks, EPoG achieved a success rate of 91.3%, reducing travel distance by 36.1% on average. Furthermore, a physical mobile manipulator successfully executed complex tasks in unknown and dynamic environments, demonstrating EPoG's potential for real-world applications.
Read the full abstract ▾
All authors Heqing Yang, Ziyuan Jiao, Shu Wang, Yida Niu, Si Liu, Hangxin Liu
Robotics
HoRD: Robust Humanoid Control via History-Conditioned Reinforcement Learning and Online Distillation
Puyue Wang, Jiawei Hu, Yan Gao et al. (+7 more)
Humanoid robots can suffer significant performance drops under small changes in dynamics, task specifications, or environment setup. We propose HoRD, a two-stage learning framework for robust
humanoid control under domain shift. First, we train a high-performance teacher policy via history-conditioned reinforcement learning, where the policy infers latent dynamics context from recent state–action trajectories to adapt online to diverse randomized dynamics. Second, we perform online
distillation to transfer the teacher's robust control capabilities into a
transformer-based student policy that operates on sparse root-relative 3D joint keypoint trajectories. By combining history-conditioned adaptation with online
distillation, HoRD enables a single policy to adapt zero-shot to unseen domains without per-domain retraining. Extensive experiments show HoRD outperforms strong baselines in robustness and transfer, especially under unseen domains and external perturbations. Code and project page are available at https://tonywang-0517.github.io/hord/.
Read the full abstract ▾
All authors Puyue Wang, Jiawei Hu, Yan Gao, Junyan Wang, Yu Zhang, Gillian Dobbie, Tao Gu, Wafa Johal, Ting Dang, Hong Jia
Robotics
Quantile Transfer for Reliable Operating Point Selection in Visual Place Recognition
Dhyey Manish Rajani, Michael Milford and Tobias Fischer
Visual Place Recognition (VPR) is a key component for localization in Global
Navigation Satellite System (GNSS)-denied environments, but its performance critically depends on selecting an image matching threshold (operating point) that balances precision and recall. Thresholds are typically hand-tuned offline for a specific environment and fixed during deployment, leading to degraded performance under environmental change. We propose a method that automatically estimates the operating point of a VPR system to maximize recall whilst aiming to achieve 100% precision. The method uses a small calibration traversal with known correspondences and transfers thresholds to deployment via
quantile normalization of similarity score distributions. This
quantile transfer ensures that thresholds remain stable across calibration sizes and query subsets. Experiments with seven state-of-the-art VPR techniques across five
benchmark datasets demonstrate that our proposed approach consistently outperforms existing baselines, enabling the underlying VPR technique to operate at 100% precision in approximately twice as many deployment scenarios (median improvement), while retrieving up to 29% more correct matches at that precision. The method eliminates manual tuning by adapting to new environments and generalizing across operating conditions. Our code is available at https://github.com/DhyeyR-007/
Quantile-Transfer-for-Reliable-VPR.
Read the full abstract ▾
Robotics
Safe and Stylized Trajectory Planning for Autonomous Driving via Diffusion Model
Shuo Pei, Yong Wang, Yuanchen Zhu et al. (+4 more)
Achieving safe and stylized trajectory planning in complex real-world scenarios remains a critical challenge for autonomous driving systems. This paper proposes the SDD Planner, a
diffusion-based framework designed to effectively reconcile safety constraints with driving styles in real time. The framework integrates two core modules: a Multi-Source Style-Aware Encoder, which employs distance-sensitive
attention to fuse dynamic
agent data and environmental contexts for heterogeneous safety-style perception; and a Style-Guided Dynamic Trajectory Generator, which adaptively modulates priority weights within the
diffusion denoising process to generate user-preferred yet safe trajectories. Extensive experiments demonstrate that SDD Planner achieves state-of-the-art performance. On the StyleDrive
benchmark, it improves the SM-PDMS metric by 3.9% over WoTE, the strongest baseline. Furthermore, on the NuPlan Test14 and Test14-hard
benchmarks, SDD Planner ranks first with overall scores of 91.76 and 80.32, respectively, outperforming leading methods such as PLUTO. Real-vehicle closed-loop tests further confirm that SDD Planner maintains high safety standards while aligning with preset driving styles, validating its practical applicability for real-world deployment.
Read the full abstract ▾
All authors Shuo Pei, Yong Wang, Yuanchen Zhu, Chen Sun, Qin Li, Yanan Zhao, Huachun Tan
Robotics
GeneralVLA: Generalizable Vision-Language-Action Models with Knowledge-Guided Trajectory Planning
Guoqing Ma, Siheng Wang, Zeyu Zhang et al. (+2 more)
Large foundation models have shown strong open-world generalization to complex problems in
vision and
language, but similar levels of generalization have yet to be achieved in robotics. One fundamental challenge is that the models exhibit limited zero-shot capability, which hampers their ability to generalize effectively to unseen scenarios. In this work, we propose GeneralVLA (Generalizable
Vision-
Language-Action Models with Knowledge-Guided Trajectory Planning), a hierarchical
vision-
language-action (VLA) model that can be more effective in utilizing the generalization of foundation models, enabling zero-shot manipulation and automatically generating data for robotics. In particular, we study a class of hierarchical VLA model where the high-level ASM (Affordance
Segmentation Module) is finetuned to perceive image keypoint affordances of the scene; the mid-level 3DAgent carries out task understanding, skill knowledge, and trajectory planning to produce a 3D path indicating the desired
robot end-effector trajectory. The intermediate 3D path prediction is then served as guidance to the low-level, 3D-aware control policy capable of precise manipulation. Compared to alternative approaches, our method requires no real-world robotic data collection or human demonstration, making it much more scalable to diverse tasks and viewpoints. Empirically, GeneralVLA successfully generates trajectories for 14 tasks, significantly outperforming state-of-the-art methods such as VoxPoser. The generated demonstrations can train more robust behavior cloning policies than training with human demonstrations or from data generated by VoxPoser, Scaling-up, and Code-As-Policies. We believe GeneralVLA can be the scalable method for both generating data for robotics and solving novel tasks in a zero-shot setting. Code: https://github.com/AIGeeksGroup/GeneralVLA. Website: https://aigeeksgroup.github.io/GeneralVLA.
Read the full abstract ▾
All authors Guoqing Ma, Siheng Wang, Zeyu Zhang, Shan Yu, Hao Tang
Robotics
AppleVLM: End-to-end Autonomous Driving with Advanced Perception and Planning-Enhanced Vision-Language Models
Yuxuan Han, Kunyuan Wu, Qianyi Shao et al. (+6 more)
End-to-end autonomous driving has emerged as a promising paradigm integrating perception, decision-making, and control within a unified learning framework. Recently,
Vision-
Language Models (VLMs) have gained significant
attention for their potential to enhance the robustness and generalization of end-to-end driving models in diverse and unseen scenarios. However, existing VLM-based approaches still face challenges, including suboptimal lane perception,
language understanding biases, and difficulties in handling corner cases. To address these issues, we propose AppleVLM, an advanced perception and planning-enhanced VLM model for robust end-to-end driving. AppleVLM introduces a novel
vision encoder and a planning strategy encoder to improve perception and decision-making. Firstly, the
vision encoder fuses spatial-temporal information from multi-view images across multiple timesteps using a deformable
transformer mechanism, enhancing robustness to camera variations and facilitating scalable deployment across different vehicle platforms. Secondly, unlike traditional VLM-based approaches, AppleVLM introduces a dedicated planning modality that encodes explicit Bird's-Eye-View spatial information, mitigating
language biases in
navigation instructions. Finally, a VLM decoder fine-tuned by a hierarchical Chain-of-Thought integrates
vision,
language, and planning features to output robust driving waypoints. We evaluate AppleVLM in closed-loop experiments on two CARLA
benchmarks, achieving state-of-the-art driving performance. Furthermore, we deploy AppleVLM on an AGV platform and successfully showcase real-world end-to-end autonomous driving in complex outdoor environments.
Read the full abstract ▾
All authors Yuxuan Han, Kunyuan Wu, Qianyi Shao, Renxiang Xiao, Zilu Wang, Cansen Jiang, Yi Xiao, Liang Hu, Yunjiang Lou
Robotics
Towards Next-Generation SLAM: A Survey on 3DGS-SLAM Focusing on Performance, Robustness, and Future Directions
Li Wang, Ruixuan Gong, Yumo Han et al. (+6 more)
Traditional Simultaneous Localization and Mapping (SLAM) systems often face limitations including coarse rendering quality, insufficient recovery of scene details, and poor robustness in dynamic environments. 3D Gaussian Splatting (3DGS), with its efficient explicit representation and high-quality rendering capabilities, offers a new reconstruction paradigm for SLAM. This survey comprehensively reviews key technical approaches for integrating 3DGS with SLAM. We analyze performance optimization of representative methods across four critical dimensions: rendering quality, tracking accuracy, reconstruction speed, and
memory consumption, delving into their design principles and breakthroughs. Furthermore, we examine methods for enhancing the robustness of 3DGS-SLAM in complex environments such as motion blur and dynamic environments. Finally, we discuss future challenges and development trends in this area. This survey aims to provide a technical reference for researchers and foster the development of next-generation SLAM systems characterized by high fidelity, efficiency, and robustness.
Read the full abstract ▾
All authors Li Wang, Ruixuan Gong, Yumo Han, Lei Yang, Lu Yang, Ying Li, Bin Xu, Huaping Liu, Rong Fu
Robotics
Viewpoint Matters: Dynamically Optimizing Viewpoints with Masked Autoencoder for Visual Manipulation
Pengfei Yi, Yifan Han, Junyan Li et al. (+2 more)
Robotic manipulation continues to be a challenge, and imitation learning (IL) enables
robots to learn tasks from expert demonstrations. Current IL methods typically rely on fixed camera setups, where cameras are manually positioned in static locations, imposing significant limitations on adaptability and coverage. Inspired by human active perception, where humans dynamically adjust their viewpoint to capture the most relevant and least noisy information, we propose MAE-Select, a novel framework for active viewpoint selection in single-camera robotic systems. MAE-Select fully leverages pre-trained multi-view masked autoencoder representations and dynamically selects the next most informative viewpoint at each time chunk without requiring labeled viewpoints. Extensive experiments demonstrate that MAE-Select improves the capabilities of single-camera systems and, in some cases, even surpasses multi-camera setups. The project will be available at https://mae-select.github.io.
Read the full abstract ▾
All authors Pengfei Yi, Yifan Han, Junyan Li, Litao Liu, Wenzhao Lian
Robotics · Cross-listed from cs.CV
SPOT-Occ: Sparse Prototype-guided Transformer for Camera-based 3D Occupancy Prediction
Suzeyu Chen, Leheng Li and Ying-Cong Chen
Achieving highly accurate and real-time 3D occupancy prediction from cameras is a critical requirement for the safe and practical deployment of autonomous vehicles. While this shift to sparse 3D representations solves the encoding bottleneck, it creates a new challenge for the decoder: how to efficiently aggregate information from a sparse, non-uniformly distributed set of voxel features without resorting to computationally prohibitive dense
attention. In this paper, we propose a novel Prototype-based Sparse
Transformer Decoder that replaces this costly interaction with an efficient, two-stage process of guided feature selection and focused aggregation. Our core idea is to make the decoder's
attention prototype-guided. We achieve this through a sparse prototype selection mechanism, where each query adaptively identifies a compact set of the most salient voxel features, termed prototypes, for focused feature aggregation. To ensure this dynamic selection is stable and effective, we introduce a complementary denoising paradigm. This approach leverages ground-truth masks to provide explicit guidance, guaranteeing a consistent query-prototype association across decoder layers. Our model, dubbed SPOT-Occ, outperforms previous methods with a significant margin in speed while also improving accuracy. Source code is released at https://github.com/chensuzeyu/SpotOcc.
Read the full abstract ▾