Peer Revenue Fully funded

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Computer Science Human-Computer Interaction

cs.HC Watch Human-Computer Interaction

Wednesday 2 September

Human-Computer Interaction · Cross-listed from cs.CV

Beauty is in the AI of the beholder: MLLMs systematically overrate facial attractiveness

Santiago Grandas, Juan Sebastian Cely-Acosta, Mohit Mendiratta et al. (+2 more)

Beauty assessments from Multimodal Large Language Models (MLLMs) are increasingly popular amongst users, companies, and aestheticians. This raises the question of whether these AI models can accurately reflect human judgments of attractiveness. In a pre- registered exploratory study, we compared the attractiveness ratings of 2,513 human participants to four widely used commercial AI models: Claude, Gemini, GPT, and Grok. Results showed that MLLMs systematically rate faces more favourably and within a narrower range than humans and, at the time of study, do not reproduce human ratings in absolute terms. However, MLLMs exhibit strong correlations with human attractiveness judgments, accurately tracking the rank-ordering of faces. MLLMs may judge faces by different cues than humans; only face age was a predictor of facial attractiveness in both humans and MLLMs, with inconsistent patterns across models for ethnicity and gender. AI models strongly agree with one another, except for Grok, which also showed the lowest agreement with humans. Our findings suggest that while they may be able to approximate rank-orderings of human attractiveness, current off-the-shelf commercial MLLMs systematically overrate the beauty of human faces.
Read the full abstract

All authors Santiago Grandas, Juan Sebastian Cely-Acosta, Mohit Mendiratta, Shafee Hassan, Macken Murphy

Human-Computer Interaction · Cross-listed from cs.CV

Orthogonal Ensembles and Tested Explanations for Performer-Independent Body-Motion Emotion Recognition

Naoto Nishida and Yoshio Ishiguro

We study body-only, 12-class acted-emotion classification from skeleton motion under leave-performer-out (LPO) evaluation, a hard, underdetermined setting: chance is 8.3%, and a protocol-matched reproduced STGCN++ baseline reaches only 25.73 +/- 4.03% Macro-F1. We show that reliable gains come not from a new architecture but from combining eleven models with orthogonal error modes: under 10-fold LPO cross-validation on the labeled training performers, an equal-weight logit-mean ensemble reaches 36.80 +/- 4.00% per-fold Macro-F1, a protocol-matched +11.07 pp (+43% relative) over the same-split reproduced baseline. Our central contribution is a tested explanation suite: for a strong ensemble member, part-masking and counterfactual edits show (rather than assert) that its decisions depend on motion-grounded body-region evidence, and this region saliency aligns with rule-based Laban Movement Analysis (LMA) attributes far more than with classical kinematics: region-level saliency-LMA Spearman rho = +0.500 versus +0.033, roughly 15x, and the alignment holds for the submitted 11-way ensemble itself at rho = +0.517; the audit is post hoc and needs no retraining. The same suite faithfully reports a negative: within-window temporal saliency is diffuse rather than localized. On the hidden challenge test set the submitted ensemble scored 37.23 % Macro-F1 and received the Best Performance Award of the MMAC Challenge 2026 (Human score is 39 %). Code is available at https://github.com/nawta/diema-challenge and the presentation at https://nawta.github.io/mmac2026/.
Read the full abstract

Human-Computer Interaction

Decoding Decision Correctness from EEG Under High Cognitive Workload in Virtual Reality: Implications for Collaborative Brain-Computer Interface Teams

Christopher Baker, Stephen Hinton, Tom Reed et al. (+1 more)

Collaborative Brain-Computer Interfaces (cBCIs) offer a promising mechanism to augment team decision-making, but existing approaches rely exclusively on evidence available only after a decision has been made and reported, such as reaction time or stated confidence. This limits their use to explaining or discounting a decision after the fact, rather than informing a team's response before it is finalised. We tested whether spatial-covariance EEG features could instead provide a genuinely pre-emptive signal of an operator's decision correctness, available within the response window itself, and whether such a signal depends on cognitive workload. Using a continuous virtual reality target-detection task, participants (N = 23) completed a within-subject workload manipulation (High vs. Low). At the team level, weighting votes by this pre-emptive neural signal, available before a response is committed, produced substantial accuracy gains on contested (evenly-split) trials under High Workload (57% to 88% as team size increased from 2 to 16), but was actively detrimental under Low Workload. Critically, this advantage held even against post-hoc behavioural signals: confidence was the strongest single team-level signal overall, but by definition cannot inform a decision still in progress, whereas the neural signal can. These findings indicate that EEG-based decision-reliability signals are not a general-purpose team augmentation tool, but a workload-conditional one, with clear implications for when and how cBCI systems should be deployed in operational teams.
Read the full abstract

All authors Christopher Baker, Stephen Hinton, Tom Reed, Stephen Fairclough

Human-Computer Interaction · Cross-listed from cs.SD

Removing Speech, Keeping Activities: A Privacy Firewall for Acoustic Sensing in Assisted Living

Pavlos Nicolaou and Christos Efstratiou

Acoustic sensing offers a promising non-intrusive approach for monitoring daily activities of older adults, yet speech privacy concerns remain a critical barrier to real-world deployment. We present a privacy firewall pipeline based on a U-Net encoder-decoder, trained entirely on synthetic data, that removes speech from ambient audio while preserving environmental sounds indicative of daily activities. Activity recognition is performed using VGGish transfer learning with an SVM classifier. Evaluated on the ESC-50 and SINS datasets across multiple speech content levels, the proposed model reduced residual speech to 0% VAD-detectable speech (Silero Voice Activity Detection) under all tested conditions, outperforming Facebook Denoiser (6.55% residual), SepFormer (36.34%) and ConvTasNet (47.21%) on ESC-50 at the 100% speech level. On ESC-50 at 40% speech level, classification performance recovers to 85% precision and 85% recall after speech removal, compared with 81%/75% before removal and an 84%/83% speech-free baseline. Evaluation on real-world participant home recordings collected with the AudioHive app showed 0% VAD-detectable speech after processing while maintaining 76% precision and recall. The pipeline enables privacy-preserving acoustic sensing without sacrificing activity recognition performance, addressing a key obstacle to the adoption of ambient monitoring in elderly care.
Read the full abstract

Human-Computer Interaction

Towards a Foundational Ontology for Identifying and Resolving Contradictions in Dialogue-based Human-Robot Interactions

Maitreyee Tewari and Michele Persiani

Existing Human-Robot Interaction (HRI) literature has focused on identifying and structuring errors, failures, conflicts, and knowledge issues (called in this work as contradictions) in domain-specific dialogue-based interactions. However, there is still lack of a formal computational framework to represent and define these contradictions, interoperable and usable across HRI and human-agent interaction (HAI) domains. Thus, this research project aims to capture, represent, and evaluate the notion of (1) dialogue-based collaborative interaction and (2) related contradictions in a foundational ontology. METHONTOLOGY, a systematic approach to build domain-independent ontologies was applied. In the conceptualisation stage of the presented ontology, concepts and models from Activity Theory were used. Preliminary results presented in this short article are: (i) Natural language definitions of dialogues and related contradictions in HRI, (ii) Set Theoretic definitions of dialogues and contradictions, and (iii) First Order Logic (FoL) formulation of the contradiction concepts and three novel principles guiding dialogue-based interactions between humans and robots. In summary, we report on ongoing work to develop a foundational ontology based on Activity Theory called Activity Theory-based foundational ontology (ATFOt) to capture and represent the notion of contradictions in HRI.
Read the full abstract

Human-Computer Interaction · Cross-listed from cs.AI

Propose to Learn, Learn to Propose: Evaluability-Aware Assistance under Bounded Rationality

Yifan Zhu, Sammie Katt and Samuel Kaski

AI assistants often collaborate by proposing candidate edits, plans, or designs that users evaluate before adoption. Existing assistance methods focus on proposal quality or user-goal inference, often assuming that the user can reliably evaluate any proposal, which can fail in practice because of bounded rationality. We study evaluability-aware proposal planning, where proposals serve both as task interventions and as probes for learning latent preferences and evaluation constraints, where the resulting belief updates then guide later proposals. We formalise this setting as ProSE, a hidden-parameter sequential assistance problem, and instantiate it with a KL-regularised bounded-rational binary response model in which acceptance trades off value gain against a distance-dependent evaluability penalty. Analysing the planning consequence of this likelihood reveals that likely accepted proposals and informative probes need not coincide, which explains why planners that only pursue acceptance systematically underperform. We operationalise ProSE with ProSE-Plan, a depth-2 Bayes-adaptive planner that scores proposals by possible responses and response-induced posterior beliefs. In controlled graph simulations, ProSE-Plan improves over evaluability-unaware and myopic baselines when evaluation cost is the bottleneck, and a probe-commit ablation confirms that our approach selects informative proposals that simpler methods miss. Our results thus identify user evaluability as a planning-relevant dimension of AI assistance, complementary to generation quality and preference inference.
Read the full abstract

Human-Computer Interaction

OmegaUse-SOP: SOP Engineering for Professional Computer Use from Human Demonstrations

Yixiong Xiao, Lang An, Hucheng Yang et al. (+9 more)

Large language models (LLMs) are increasingly evolving from conversational assistants into agents capable of operating external digital environments. Graphical user interface (GUI) agents play an important role in this transition, as many real-world workflows remain accessible only through user-facing software interfaces. However, despite recent progress on general computer-use benchmarks, domain-specific professional standard operating procedures (SOPs) remain challenging for GUI agents because they often involve implicit domain knowledge, software-specific conventions, and task-level verification requirements. We introduce OmegaUse-SOP, a human-in-the-loop SOP Engineering system for transforming human demonstrations of professional computer use into reusable SOP skills for GUI agents. Analogous to prompt engineering, SOP Engineering iteratively refines demonstrations, execution rules, and domain knowledge to convert professional SOPs into reusable GUI-agent skills. OmegaUse-SOP consists of four modules: Observe, Reason, Configure, and Execute. Together, these modules record expert operations as multimodal GUI traces, abstract low-level events into semantic step-level instructions, incorporate domain rules and task-specific parameters, and execute the resulting skills in live GUI environments through step-wise grounding, action generation, and verification. To demonstrate its effectiveness, we collaborate with a power-sector client and test OmegaUse-SOP on photovoltaic simulation workflows in PVsyst 7.2. The results suggest that OmegaUse-SOP can improve GUI-agent reliability on professional SOP tasks, highlighting a practical path toward deploying GUI agents in domain-specific professional software environments.
Read the full abstract

All authors Yixiong Xiao, Lang An, Hucheng Yang, Pinxue Ma, Yongquan Chen, Jingjia Cao, Yusai Zhao, Ting Wang, Ting Liu, Siqi Bao, Jingbo Zhou, Hua Wu

Human-Computer Interaction

Reconciling Kinesthetic Mismatches: A Somatic Alignment Mindset for Musical Body Transformation

Ziyue Piao, Isabelle Cossette and Marcelo M. Wanderley

Mastering musical performance requires precise multisensory coordination, yet learners encounter a kinesthetic mismatch, which is a discrepancy between the internal perception of an action and the actual physiological state of the body. While multisensory Body Transformation Experiences (BTE) provide tools to bridge this gap, existing designs often focus on external correction rather than internal alignment. To address this, we propose the Somatic Alignment Mindset (SAM), a conceptual lens that integrates Taoist philosophy to shift the focus of HCI design from prescriptive feedback toward holistic embodied unity. By positioning technology as a reflective medium, SAM operationalizes the principles of Adaptation, Assessment, and Awareness to reconcile somatic discrepancies and foster deep, self-aligned musical mastery.
Read the full abstract

Human-Computer Interaction

Knowing Is Not Enough: Information Retrievability as a Precondition to Effective LLM Oversight

Xinyu Fu, Narayan Ramasubbu and Dennis Galletta

Large language models (LLMs) are increasingly embedded in organizational work, yet their errors often pass human review. Prior research locates such failures in users' capability to review LLM output or their engagement in doing so. We develop an alternative, retrieval-based account of human oversight and posit that error detection is more effective when oversight-relevant information is accessible to users at the moment of review. Across two randomized lab-in-the-field experiments with 640 customer-facing employees, we show that self-generated explanations improve error detection and strengthen recall of verification-relevant reasoning, while cues that reactivate such reasoning help sustain detection under repeated LLM use. Theoretically, we identify information retrievability as a distinct precondition for effective oversight and specify generative encoding and cue-supported reactivation as mechanisms that build and sustain it. Practically, lightweight onboarding self-explanations and daily retrieval cues can make human oversight more resilient as LLM use becomes routine.
Read the full abstract

Human-Computer Interaction

Exploring Breathing-Music Coupling: Using the Breathing Mirror for Somatic Reflection in Piano Performance

Ziyue Piao, Yohei Wada, Isabelle Cossette et al. (+2 more)

While breathing is essential to living and for sound production in some instruments, for pianists, it is often a hidden and automatic process, making it difficult to analyze or refine. A critical gap exists between data and awareness: while sensors record precise physical metrics, they fail to capture the performer's somatic experience. Conversely, the high cognitive load of performance makes it nearly impossible for musicians to recall their internal states with temporal precision. To address this, we present a system, Breathing Mirror, and associated methodology designed to externalize the pianist's internal somatic experience through three analytical lenses: a Baseline View (synchronized signals), a First-Person View (subjective recall), and an Interpersonal View (collaborative reflection). Through a four-week longitudinal study with a skilled amateur pianist (35 years of experience), we evaluated the system's effectiveness by recording respiratory data using textile-integrated strain sensor belts. The results show that the Breathing Mirror reveals some patterns of breathing-music coupling and identifies critical blind spots where objective data diverges from subjective perception. Furthermore, we propose four somatic themes regarding the link between breathing and musical elements, offering a foundation for future large-scale validation across a broader range of pianists. This work provides a new way to study body signals, transforming breathing from an internal biological function into an articulate expressive parameter.
Read the full abstract

All authors Ziyue Piao, Yohei Wada, Isabelle Cossette, Marcelo M. Wanderley, Akira Maezawa

Tuesday 1 September

Human-Computer Interaction

Targeted and Traceable Investigation of Multi-Agent LLM Dialogue via Semantic Bundling of Knowledge Graphs

Zeyu Hua, Adam Coscia and Alex Endert

Multi-agent LLM systems today are increasingly automated, logging LLM-LLM interactions as conversational transcripts. Yet analyzing such dialogue for insights remains challenging, including attributing behaviors to the correct actor and summarizing interactions across a long exchange. We present a targeted and traceable approach to investigating multi-agent LLM dialogue, applied to the VAST Challenge 2026 MC1 dataset. The challenge asks participants to reconstruct and explain which internal communications among AI agents at TenantThread, a property tech company, led to an inappropriate information release. We first convert the dialogue into a knowledge graph (KG) and then investigate it with AgentK, a visual analytics system for interactive Semantic Bundling of nodes and edges. We found that our approach directly addresses two main challenges: (1) the KG structure enables users to identify actors worth investigating faster; and (2) summarizing only the region surrounding an actor of interest better supports per-actor attribution than reading raw conversations.
Read the full abstract

Human-Computer Interaction · Cross-listed from q-bio.NC

Pulling Illusion in Individuals with Neurological Disorders

Takeshi Tanabe, Satoshi Yamamoto, Toru Yamada et al. (+2 more)

The pulling illusion induced by asymmetric vibration stimuli has attracted attention for its potential applications in rehabilitation and sensory assessment. However, the underlying mechanism of the pulling illusion remains unclear. This study addressed the central question of whether peripheral vibrotactile sensitivity alone is sufficient for the illusion to emerge or whether processing beyond basic vibration detection is also required. Neurological disorders can involve impairments at different levels of the nervous system, providing an opportunity to examine this question. Accordingly, we evaluated directional discrimination performance for the pulling illusion and fingertip vibration detection thresholds in 25 participants with diverse neurological disorders affecting different levels of the nervous system, from peripheral to central. Clustering analysis identified contrasting profiles, with high directional discrimination performance despite elevated vibration detection thresholds and chance-level performance despite relatively low-to-intermediate thresholds. In the generalized linear mixed model, motor-related signs, including hemiplegia and tremor, showed a robust negative association with directional discrimination performance, whereas vibration detection threshold was not robustly associated with performance. Furthermore, in participants with hemiplegia, directional discrimination performance was around chance level on the affected side and close to 100% on the unaffected side, despite stimulus amplitudes well above the measured vibration detection thresholds on both sides. Collectively, these findings suggest that the pulling illusion depends on perceptual processing beyond basic vibration detection, through which asymmetric vibration is experienced as directional pulling.
Read the full abstract

All authors Takeshi Tanabe, Satoshi Yamamoto, Toru Yamada, Daisuke Ishii, Yutaka Kohno

Human-Computer Interaction · Cross-listed from cs.AI

Beyond "AI Helps Humans": Decision-Targeted Evaluation Design for Human-Agent Teams in the Agentic Era

Hamed Khosravi and Xiaoming Huo

Wherever a coding agent works under engineer supervision, or a clinical model assists a radiologist, the deployment question is whether to keep the human-AI workflow or replace it with the human alone or the agent alone. The human-AI workflow is worth keeping only if it beats both of those alternatives. Yet once it is deployed, neither alternative outcome is observed: recovering one means replaying the task under that alternative, and every replay costs expert time or compute. Under a fixed replay budget, the design question is therefore which tasks should be more likely to receive a human-only replay, and which an agent-only replay. Existing methods do not directly target this decision. Agent benchmarks do not choose which missing baseline to measure, variance-based sampling ignores which of the two comparisons is closer to failing, and Bayesian information methods focus on learning model parameters instead of making the deployment decision. We propose TEAM-Design, a rule that gives every task two replay probabilities, one per baseline. It raises a probability where the missing baseline outcome is hard to predict from what is already known about the task and where that comparison is harder to establish, and lowers it where replay is expensive. We prove that the rule solves this budgeted design problem, and that drawing the replays at random from recorded probabilities still controls the chance of wrongly declaring that the workflow beats both. We reanalyze 6 clinical settings, where no human-AI workflow beats both alternatives, and a coding benchmark, where one does, then evaluate TEAM-Design on synthetic designs and on a semi-synthetic design built from a real chest X-ray reader study. TEAM-Design works best when one of the two comparisons is clearly harder to settle than the other, and can do worse than variance-based allocation when the two are similarly difficult.
Read the full abstract

Human-Computer Interaction

Beyond Instruction-Driven Editing: Source-Grounded Problem Discovery with User-Governed Repair for Scientific Posters

Xingda Lyu, Honglin Lu, Xinye Luo et al. (+1 more)

Interactive editors usually assume that users already know what to change. Yet an important interaction state comes earlier: a user may recognize that an artifact is not working without knowing what intervention to request. We call this the articulation gap. We introduce PROS (Proactive Refinement Of Scientific Posters), which separates epistemic initiative from behavioral authority: the system can surface source-grounded candidate problems, while users decide which become repair goals and whether resulting changes are committed. Accepted issues hand off to native-object PPTX editing with validation and reversible preview. We also introduce PROS-Bench, a source-linked collection of 120 papers and 320 editable PPTX posters, including a 120-poster matched primary core and a separate conference representation challenge. On the primary core, PROS achieves a mean VLM-rated stage-balanced diagnosis quality score of 67.2 on a 0-100 scale and 87.6% operator-verified target resolution among accepted diagnoses. Temporally blinded automated scoring yields a +22.7-point paper-macro accepted-target uplift, yet 14.8% of assessable accepted targets decline. This divergence shows why problem discovery, local resolution, and realized outcome should be evaluated separately. More broadly, intelligent editors can support problem discovery before a concrete edit request exists without taking authority over consequential change.
Read the full abstract

All authors Xingda Lyu, Honglin Lu, Xinye Luo, Shiqi Yang

Human-Computer Interaction · Cross-listed from q-bio.NC

Slow-Fast Brain-Computer Interfaces: Preventing Neuroadaptive Overfitting in AI-Mediated Neural Interfaces

Aarthy Nagarajan

Artificial intelligence (AI) is transforming brain-computer interfaces (BCIs) from task-specific neural decoders into adaptive systems that complete language, smooth movement, regulate rehabilitation support and adjust stimulation. These capabilities can increase speed, fluency, usability and clinical reach, yet conventional performance metrics may overlook losses in intent fidelity, authorship, agency, therapeutic challenge and durable clinical benefit. I define neuroadaptive overfitting as a closed-loop failure mode in which an AI-mediated BCI becomes over-optimized to short-term proxies of success, including reduced effort, rapid acceptance, lower workload or smooth task completion, while drifting from the user's durable goals. I then propose Slow-Fast BCI, a framework for pacing AI assistance according to decoder evidence, uncertainty, contextual and clinical stakes, fatigue, and user- or clinician-defined goals. The framework distinguishes fast assistance when intent is clear and stakes are low, guarded assistance under uncertainty and slow assistance when misalignment could compromise safety, agency, authorship, motor learning or therapeutic value. Across communication, motor-control, neurorehabilitation and closed-loop neuromodulation applications, I outline corresponding safeguards and evaluation measures. This Perspective argues that AI-mediated BCIs should be evaluated not only by decoding accuracy and task performance, but also by how AI assistance is deployed: when systems act autonomously, seek confirmation, preserve user effort or return control to the user.
Read the full abstract

Human-Computer Interaction

VirSqueezer: Generating Realistic Deformations and Squeezing Dynamics in VR from Fine-Grained Squeezing Controls

Qian Zhang, Xiaoming Chen, Xiaorui Ma et al. (+2 more)

Squeezing is one of the most natural forms of hand manipulation, inherently involving fine-grained, temporally evolving, per-finger flexion. In VR content creation, squeezing plays a unique role in enabling particular visual effects such as localized deformations and dynamic behaviors, e.g., bursting a Coke can or juicing a fruit, thereby expanding the expressive possibilities of VR content. However, existing techniques, such as 3D Gaussian splatting-based methods and diffusion-based video generation models, are limited in their ability to simulate fine-grained virtual squeezing effects. We introduce VirSqueezer, a framework designed to generate both localized deformations (primary effects) and complex squeezing dynamics, such as rupture and overflow (secondary effects). VirSqueezer captures squeezing control signals using a SenseGlove and provides the user with inferred resistance force feedback during the squeezing process. By estimating object contact areas, inferring physical properties, and simulating physical responses, VirSqueezer computes conditions that guide generation models for visual effect generation, ensuring both visual coherence and temporal synchronization with the simulation. Consequently, VirSqueezer enables the generation of physically realistic visual effects directly from continuous, fine-grained squeezing control signals. Our extensive evaluation demonstrates VirSqueezer's ability to reproduce realistic localized deformations, generate convincing visual dynamics, and maintain consistency in fine-grained squeezing controls.
Read the full abstract

All authors Qian Zhang, Xiaoming Chen, Xiaorui Ma, Haisheng Li, Weidong Cai

Human-Computer Interaction

Designing Proactive Thought Partners for Writing

Chao Zhang, Abe Davis, Chih-Wei Chen et al. (+1 more)

Writing involves diverse cognitive activities, from ideation to revision, and writers' needs vary across individuals and moments. Proactive AI promises to provide the right support at the right time, yet existing proactive tools largely focus on generic textual assistance, such as autocomplete. This paper studies the design space of proactive thought partners: AI agents that proactively offer customizable, higher-level cognitive support during writing. We instantiated this concept in a technology probe and deployed it with 16 participants for one week. The probe allows users to create partners by configuring their roles and proactivity. As users write, relevant partners take the initiative at appropriate moments to offer suggestions. Our findings show that participants configured proactive support through prospective planning, used suggestions for both idea generation and self-monitoring, and valued lightweight visual representations alongside non-directive rhetorical framing for non-intrusive interventions. We derive implications for designing proactive writing assistants around customization, timing, engagement, and representation.
Read the full abstract

All authors Chao Zhang, Abe Davis, Chih-Wei Chen, Chin-Chia Hsu

Human-Computer Interaction

Evaluating Usability in Biomedical Visualization: Rethinking Heuristic Evaluation for Spatial Omics and Multidisciplinary Research Platforms

Yulia A. Levites Strekalova, Rachel Liu Galvin, Jessica M. Ray et al. (+11 more)

Introduction: Clinical research informatics (CRI) platforms support biomedical discovery by integrating advanced computational tools into research workflows. Emerging technologies such as spatial omics and AI-enabled imaging expand research capabilities but introduce complex interfaces that increase cognitive burden and alter established analytical processes. Traditional usability frameworks identify general usability issues but often miss challenges specific to high-dimensional biomedical data. Methods: We conducted two complementary studies involving 39 participants to evaluate conventional usability heuristics and identify CRI-specific criteria. Study 1 included 19 undergraduates completing interactive tasks, and Study 2 involved 20 clinical professionals completing an asynchronous hierarchical task framework. Observational and interview data were analyzed using deductive coding based on standard usability heuristics and emerging CRI-specific themes. Results: Simultaneous presentation of complex data overlays and analytical tools overwhelmed users, particularly those with limited spatial-omics experience. Participants relied on trial-and-error exploration and struggled with unlabeled tools in data-rich environments. Feedback indicated that users benefit from phased onboarding, contextual guidance, and progressive feature introduction rather than immediate access to all functionality. Discussion: High-dimensional research platforms require domain-specific usability criteria beyond traditional frameworks. We propose three specialized heuristics: Active Parameter Transparency, Point-of-Use Guidance, and Phased Feature Disclosure. These heuristics help developers manage complexity, provide contextual support, and improve accessibility for multidisciplinary research teams.
Read the full abstract

All authors Yulia A. Levites Strekalova, Rachel Liu Galvin, Jessica M. Ray, Samuel P. Border, Mishal Khan, Samantha Hoffman, Christina D. Beharry, Katie Kloss, Philipp Haessner, David Manthey, Sanjay Jain, Michael T. Eadon, Laura Barisoni, Pinaki Sarder

Human-Computer Interaction

Better Situational Awareness in AR-HRC? A Comparative Study of Augmented Reality and Mobile Interfaces for Human-Robot Collaboration

Zhehan Qu, Christian Fronk, Jaewoong Jeong et al. (+2 more)

Augmented reality (AR) facilitates human-robot collaboration (HRC) by enabling in-situ spatial visualizations of the robot and the joint task. However, in safety-critical HRC scenarios such as search-and-rescue, spatial visualizations may also reshape visual attention in ways that create competing situational awareness (SA) demands, potentially introducing new safety concerns. While prior AR-HRC work suggests potential benefits for SA, rigorous evaluations that jointly consider robot and environmental awareness across multiple levels of SA remain limited. We address this through a between-subjects study with 30 participants comparing custom AR and mobile interfaces presenting equivalent information, measuring robot and environmental SA with the Situation Awareness Global Assessment Technique (SAGAT) across all three levels, with concurrent eye tracking to identify the attentional mechanisms underlying any SA differences. Both interfaces achieved high usability; relative to the mobile baseline, AR improved perception-level awareness of the robot but yielded no gains in higher-level robot awareness or in environmental awareness at any level. Gaze analysis explained this: AR freed attention from the map, but that attention was re-invested in the conformal visuals rather than the physical environment. Freeing the eyes from a screen is not the same as directing them to the world, a distinction AR interfaces for safety-critical HRC must design around.
Read the full abstract

All authors Zhehan Qu, Christian Fronk, Jaewoong Jeong, Pavel Manakhov, Maria Gorlatova

Human-Computer Interaction

Cross-Modal Guidance for Out-of-View Object Search in Simulated Prosthetic Vision

Adyah Rastogi, Apurv Varshney, Tobias Höllerer et al. (+1 more)

Out-of-view guidance is well established in virtual and augmented reality, but its effectiveness may depend on the visual bandwidth available to the user. We test this under simulated prosthetic vision (SPV), where visual guidance must share the same sparse representation used to inspect the scene. Nineteen participants performed object search under two SPV conditions differing in electrode density and phosphene spread (10x10 and 20x20) and four guidance conditions (no guidance, visual, haptic, audio) all driven by the same horizontal target-offset variable. All three modalities reduced search time and head movement. The tested auditory and haptic cues produced approximately 25% faster overall search and 11-13% faster target acquisition than the visual cue, despite similarly direct orienting trajectories. The tested haptic and auditory cues also shortened post-acquisition search. Final head-target angular offset was reduced substantially more in the 10x10 SPV condition; there, all three cues also reduced vertical localization error by approximately 45-58% despite providing no elevation information. Under severe visual constraints, guidance performance depended on cue implementation and search stage.
Read the full abstract

All authors Adyah Rastogi, Apurv Varshney, Tobias Höllerer, Michael Beyeler

Human-Computer Interaction · Cross-listed from cs.CL

InSight: A Benchmark for Agentic Claim Verification in Interactive Visualizations

Maeve Hutchinson, Syed Mahbubul Huq, Mohammad Albinhassan et al. (+3 more)

Vision Language Models have demonstrated remarkable proficiency in interpreting static visual artifacts, but modern data analysis is inherently dynamic, requiring the active interrogation of interactive environments. Existing benchmarks are predominantly constrained to static imagery and one-shot question answering and fail to capture the epistemic demands of this domain, where evidence is frequently occluded, distributed across linked views, or conditionally revealed through user agency. In this paper, we introduce InSight, a benchmark for agentic claim verification over interactive visualizations. The dataset consists of 21,349 claims derived from human-authored analytical narratives and grounded in fully interactive web-based environments. Agents must navigate these environments to determine whether a natural language claim is supported, refuted or not verifiable given the available evidence. Unlike traditional evaluations, InSight treats interaction traces as intrinsic proxies for reasoning, enabling a rigorous audit of how models seek and synthesize visual evidence. We evaluate state-of-the-art models, revealing that interactive verification remains a non-trivial challenge. We release InSight at https://github.com/maevehutch/insight.
Read the full abstract

All authors Maeve Hutchinson, Syed Mahbubul Huq, Mohammad Albinhassan, Radu Jianu, Aidan Slingsby, Pranava Madhyastha

Human-Computer Interaction · Cross-listed from eess.IV

GazeRefine: Expert Gaze as a Test-Time Prompt for Training-Free Medical Image Segmentation

Mohammed Oussama Benyahia, Marouane Tliba, Mohamed Amine Kerkouri et al. (+10 more)

Medical image segmentation remains difficult to scale because high-performing methods typically rely on dense expert annotations and task-specific training. We introduce GazeRefine, a training-free framework that uses gaze as an inference-time prompt for zero-shot medical image segmentation. Sparse, duration-weighted fixations are converted into foreground and background priors that initialize semantic prototypes in frozen DINOv3 feature space. These prototypes are iteratively refined through foreground-background discrimination, feature-space affinity propagation, and anchoring to the initial gaze guidance, allowing segmentation to extend beyond directly fixated regions while limiting semantic drift. GazeRefine requires no segmentation masks, fine-tuning, adapters, prompt encoders, or gradient updates. We evaluate the method on gaze-annotated polyp segmentation and prostate MRI segmentation. The results show strong performance on colonoscopy images and competitive performance on prostate MRI, supporting gaze-guided prototype refinement as a promising approach for segmentation-label-efficient, human-in-the-loop medical image segmentation. Our tools and code can be found in the following repository: https://github.com/MohammedOussamaBEN/GazeRefine.git
Read the full abstract

All authors Mohammed Oussama Benyahia, Marouane Tliba, Mohamed Amine Kerkouri, Taifour Yousra, Bin Wang, Max Bengtsson, Gorkem Durak, Elif Keles, Zuheng Ming, Marek Penhaker, Azeddine Beghdadi, Ulas Bagci, Aladine Chetouani

Human-Computer Interaction

Beyond Technological Solutionism: Rethinking XR in Healthcare

Md Haseen Akhtar, Cecilia Landa-Avila, Shital Desai et al. (+5 more)

The healthcare industry's enthusiastic adoption of Extended Reality (XR) technologies obscures a concerning reality: we were building increasingly sophisticated ways to perpetuate fundamentally broken healthcare systems. Through three deeply personal narratives - a rural patient cut off from care infrastructure, an urban professional navigating fragmented services, and a first-generation immigrant confronting cultural barriers - this provocation paper exposes how our obsession with technological innovation often worsens rather than resolves healthcare disparities. By applying the SEIPS 3.0 model to examine diabetes-CVD care coordination, we identify an "innovation paradox" where advanced technology creates new barriers to effective care. Our care interdependencies framework reveals that healthcare outcomes are shaped primarily by human relationships (50-60%), organizational coordination (25-30%), and sociocultural factors (15-20%), not technological sophistication. This research challenges the HCI community to confront its role in perpetuating healthcare inequities, demands a fundamental rethinking and proposes a new framework for healthcare innovation that prioritizes human relationships over technical capability, systemic change over feature sets, and actual care delivery over technological ambition. For healthcare providers, technology developers, and policymakers, our findings suggest that effective care coordination requires us to step back from our techno-solutionist mindset and engage
Read the full abstract

All authors Md Haseen Akhtar, Cecilia Landa-Avila, Shital Desai, Claire Warden, Andrew Morris, Mark Anderson, Thomas Cochrane, Janakarajan Ramkumar

2609.01028 · 1 Sep · 5 pages, 1 figure, Accepted paper at the Interactive Health workshop In Proceedings of CHI 25 Workshop on Envisioning the Future of Interactive Health, April 27th, 2025, Yokohama, Japan, ACM, New York, NY, USA, 2 pages

Abstract PDF

Human-Computer Interaction · Cross-listed from cs.CL

Disclosure-Gated User Simulation for Companion-Agent Evaluation

Yao Liu, Yu He, Jihao Huang et al. (+2 more)

Playing the user with a large language model is standard in scalable evaluation, and has a repeatedly diagnosed failure: the simulated user is excessively cooperative, so a system under test can score by the number of questions it asks rather than by making the user willing to speak. We answer with a disclosure gate conditioning information release on the companion agent's behaviour: its state is a ladder of five ordered gates, merged onto three observable depth layers. We specify, ablate, and audit it. The gate is a load-bearing component of the environment: on a published companion-agent benchmark's English corpus, withdrawing per-example gate statements from training displaces the ranking of the 12 systems beyond the noise band of a seed re-run, while per-system scores show no detectable change. We state two acceptance criteria: a ranking must be order-preserving, and absolute scores must be scale-stable. Only one candidate passes both – the simulator we release – correlating at 0.993 with the published leaderboard. Prompting a frontier model instead barely moves the ranking while shifting every score up, invisible to ranking-only checks; we supply the specification, ablations, and downstream sensitivity analysis the benchmark's environment lacked.
Read the full abstract

All authors Yao Liu, Yu He, Jihao Huang, Xinyang Liu, Feng Zhou

Human-Computer Interaction

Ctrl-F-Resist. Practices, Challenges, and Technical Needs of Civil Society Organizations Monitoring the Far-Right Online

Elisabeth Steffen and Helena Mihaljević

As far-right actors increasingly exploit online platforms to disseminate ideology and mobilize supporters, civil society organizations (CSOs) play a vital yet underrecognized role in monitoring antidemocratic dynamics online. Unlike fact-checkers or content moderators, CSOs engage in long-term, contextualized analysis, often in resource-constrained settings and under precarious conditions. Despite their critical societal role, CSOs face significant barriers to adopting or co-developing technical solutions, including legal uncertainty, limited platform access, and chronic underfunding. Existing research and tool development efforts have largely overlooked these actors in favor of more institutionally embedded stakeholders. This paper addresses this gap through a qualitative study with 15 practitioners from 12 Germany-based CSOs engaged in online monitoring, positioning them as key yet overlooked stakeholders in the governance of digital spaces. We explore their current practices, challenges, and expectations regarding technological support. Our findings show that monitoring remains largely manual due to the lack of tailored tools, with enhanced search capabilities emerging as the most pressing technical need. While participants express openness to AI-supported features such as media processing and content discovery, many remain skeptical of automated classification, citing concerns around trust, legal usability, and professional credibility. Grounded in these findings, we introduce a conceptual monitoring workflow and describe its implementation in an open-source Telegram monitoring prototype designed to flexibly support diverse monitoring goals. We outline concrete design, policy, and research recommendatios, and introduce the manual labor trap as an empirically grounded concept that explains why monitoring CSOs tend to remain locked into labor-intensive, low-capacity arrangements.
Read the full abstract

Human-Computer Interaction · Cross-listed from cs.CL

TEIDAN: A Multilingual Multiparty Dialogue Corpus

Taiga Mori, Koji Inoue, Mikey Elmers et al. (+2 more)

Multi-party interaction is a central setting for human communication and a necessary target for human-agent interaction systems that must participate in group conversation. Yet available corpora often focus on meetings, task-oriented interaction, text-based interaction, or acted scenarios, and fewer resources support cross-linguistic comparison of spontaneous face-to-face triadic discussion. This paper presents TEIDAN, a multilingual multimodal corpus that currently consists of Japanese and English three-party conversations. TEIDAN records groups of three participants discussing open-ended topics with individual pin microphones, a microphone array, and participant-facing cameras, and provides IPU-based transcripts for both language portions. Earlier studies used subsets of the Japanese portion for task-specific benchmarks in multi-party dialogue modeling; in contrast, this paper presents TEIDAN as a corpus resource spanning both Japanese and English, with planned expansion to additional languages. We describe the collection design, participants, recording setup, transcription format, and corpus statistics, and provide preliminary analyses to illustrate how TEIDAN can support research on turn-taking, addressee recognition, and multimodal grounding in human-human and human-agent interaction.
Read the full abstract

All authors Taiga Mori, Koji Inoue, Mikey Elmers, Divesh Lala, Tatsuya Kawahara

Human-Computer Interaction

No Pixel Left Behind: Filling Gaps in Anime Colorization

Masahiro Kono, Akinobu Maejima, Yuki Koyama et al. (+2 more)

Animation production workflows often involve digital colorization of line art, where small unpainted regions ("gaps") frequently occur and remain an underexplored challenge. We conducted a formative study in Japanese animation (anime) pipelines and found that while the paint bucket tool is widely used for base coloring, tiny enclosed areas are frequently overlooked, resulting in time-consuming manual detection and filling. We introduce GapFill, a tool grounded in professional practices that reduces the effort of gap detection, zooming, and color selection. Our deep-learning method suggests appropriate fill colors by referencing surrounding regions, leveraging the flat-color nature of anime-style images. In a user study with 13 professional colorists, our system improved performance and usability in gap-filling tasks over conventional methods. The study also suggested that prediction accuracy alone is not the primary factor for usability, that appropriate colors can be contextually ambiguous, and that GapFill can complement existing tools depending on users' trust in new AI-powered assistance.
Read the full abstract

All authors Masahiro Kono, Akinobu Maejima, Yuki Koyama, Yotam Sechayk, Takeo Igarashi

Human-Computer Interaction

GazeTune: Facilitating Precise Gaze-Driven Interactions with Cascaded Touch Input

Jina Kim, Eric J. Gonzalez, Yang Zhang et al. (+1 more)

Eye gaze has become an essential input for spatial computing, but its coarse targeting and saccadic nature limit precision and complicate continuous interactions such as dragging, especially under user motion. Gaze+pinch has also become standard in XR for its convenience, yet mid-air gestures remain imprecise, fatiguing, and socially unacceptable. These limitations underscore the need for an approach that preserves the speed of gaze while enabling stable, fine control. We present GazeTune, a cascaded multimodal interaction technique combining gaze and touch to refine gaze-based selection and manipulation. Touch serves as a refinement channel within gaze pointing, allowing precise cursor and target control. Our work investigates how gaze-and-touch enhances dragging and mitigates Motion-Induced instability. In a study (N=20), we compared GazeTune against gaze-only and gaze-pinch methods in 2D dragging. Results show that GazeTune achieves significantly lower error with comparable execution time, validating its effectiveness and balanced trade-off between time and accuracy.
Read the full abstract

All authors Jina Kim, Eric J. Gonzalez, Yang Zhang, Sang Ho Yoon

Human-Computer Interaction · Cross-listed from cs.CR

SoK: Motion Data Privacy in Extended Reality

Azim Ibragimov, Alina Vasina, Uliana Polshcha et al. (+1 more)

Extended Reality (XR) provides immersive, interactive 3D experiences. To enable these experiences, the devices must track user motion so the system can respond to actions such as grabbing, looking at, or moving an object. However, motion tracking has raised privacy concerns since it records a person's motion patterns. These motion patterns have been studied extensively across various fields (i.e., gait identification and profiling) and have been shown to reveal sensitive information. With the adoption of XR, these patterns became easier to record and obtain than ever. This creates a fundamental privacy tension: motion tracking enables core XR functionality yet requires users to compromise their privacy. Prior systematization-of-knowledge (SoK) studies on XR privacy have examined the field broadly, with motion-related research distributed across several privacy domains rather than treated as a distinct area of study. However, XR motion privacy has gained significant momentum since the prior SoK, with the literature nearly quadrupling in size and thereby warranting a dedicated systematization of this topic. This SoK examines 134 relevant papers on privacy concerns in motion patterns recorded by XR headsets, including how adversaries can obtain users' motion patterns, the inferences they can draw from them, and methods for protecting users. Based on this review, we synthesize a taxonomy of motion modalities, representations, and inference risks; develop an XR motion threat model; systematize the attack and defense approaches in the XR motion literature; identify gaps in the literature; and provide guidelines for future studies evaluating motion privacy mechanisms. Together, our SoK clarifies the state of XR motion privacy and provides recommendations for future evaluations.
Read the full abstract

All authors Azim Ibragimov, Alina Vasina, Uliana Polshcha, Eric D. Ragan

Human-Computer Interaction · Cross-listed from cs.RO

Human-robot conversation with multiple participants in noisy public spaces

Divesh Lala, Yogeeswaran Muthukumaran, Vincent Fernandes et al. (+10 more)

For noisy real-world environments such as those in open public spaces, spoken dialogue systems for both autonomous robots and avatars should be carefully designed to provide enhanced speech signals. These signals can be used either for speech recognition or, in the case of an avatar system, transmitted as clean speech to a remote operator. This work proposes an audio system that can be used for both these scenarios and was demonstrated as a proof-of-concept at the 2025 World Expo in Osaka. The first scenario is an attentive listening system with the android ERICA, and the second is a conversation support system with mobile Teleco robots, with one of them acting as an avatar for a remote operator. Both systems feature multi-party conversation and use a single multi-channel microphone array. We describe how our audio system not only enhances the speech of multiple speakers in a noisy environment, but provides a form of spatial audio which allows for more immersiveness in avatar-based conversational interactions.
Read the full abstract

All authors Divesh Lala, Yogeeswaran Muthukumaran, Vincent Fernandes, Kazushi Kato, Shota Fujiki, Zihao Chi, Masaya Iwasaki, Taiken Shintani, Megumi Kawata, Kazuki Sakai, Koji Inoue, Yuicihiro Yoshikawa, Tatsuya Kawahara