Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Computer Science Human-Computer Interaction

cs.HC Watch Human-Computer Interaction

Monday 5 October

Human-Computer Interaction

Supporting Couples' Social Well-being in Daily Life: A Needs Assessment and Co-design with Co-located Couples

Yuna Naito, Timothy Bickmore, Varun Mishra et al. (+1 more)

Romantic relationships shape physical, mental, and social well-being, and relationship quality is built through everyday interactions. Addressing minor issues early and reinforcing positive interactions help maintain relationship quality, and ubiquitous computing creates new opportunities to sense and support couples' relationships in daily life. However, existing systems remain largely developed with limited involvement from couples, raising questions about which types of sensing and support couples may accept in their private lives. To address this gap, we conducted exploratory needs-assessment interviews and co-design sessions with five co-located couples (10 participants) to identify their needs and preferences for relationship-attunement-enhancing technologies, including the use of LLMs. Couples' own designs converged on systems that detect meaningful moments—such as conflict onset—and offer subtle mediation rather than AI-generated advice, highlighting dyadic concerns around partners' agency, authenticity, and control over shared interaction data. We offer preliminary design implications for ubiquitous sensing and intervention systems that center the couple as a unit rather than the individual.
Read the full abstract

All authors Yuna Naito, Timothy Bickmore, Varun Mishra, Matthew S. Goodwin

Human-Computer Interaction

Smart Navigation for Visual Prostheses in Virtual Reality: An End-to-End Framework for Priority-Based Scene Translation and Path Guidance

Mohamed H. Abdellatif, Fatma S. Elsharkawy, Nouran H. Qassem et al. (+4 more)

Visual prosthetics provide a promising direction for partial restoration of functional vision for people with total retinal blindness. However, existing systems face significant challenges in translating complex visual scenes into meaningful perceptions due to limited spatial resolution, leading to difficulties in scene understanding. Furthermore, existing solutions don't adequately account for user requirements and concerns, and this creates a significant gap between user expectations and the developed solutions. To address these gaps, we conducted interviews with 10 blind human subjects. These interviews essentially indicated that the main key challenge for these people is outdoor navigation. In this paper, we present an end-to-end smart navigation system for visual prostheses users. Our approach employs a three-stage framework. First, an object detector is implemented to identify and localize points of interest for navigation tasks in pedestrian environments, and simultaneously generate the safest paths available to the users. Second, the detected objects are abstracted into simple geometric shapes suitable for low-spatial-resolution vision. The detected objects are filtered based on a multi-criteria priority scoring function. Finally, this information is encoded into optimized stimulation parameters, which are fed into the visual prosthesis implants to generate enhanced phosphene representations for obstacle avoidance and path planning. This whole system is validated with sighted participants using virtual reality to simulate outdoor navigation. Our smart navigation system improves user independence while taking into account the limitations of visual prosthetics.
Read the full abstract

All authors Mohamed H. Abdellatif, Fatma S. Elsharkawy, Nouran H. Qassem, Talal M. Emara, Nada K. Kotb, Muhammad Rushdi, Reham H. Elnabawy

Human-Computer Interaction · Cross-listed from cs.RO

End-to-End Safe Social Navigation via Multi-Task Reinforcement Learning and Probabilistic Perception

Tommaso Van Der Meer, Andrea Garulli, Antonio Giannitrapani et al. (+3 more)

Autonomous social navigation requires balancing efficiency, physical safety, and social compliance. Reinforcement Learning (RL) methods provide a viable and effective solution but often rely on unrealistic assumptions, such as the knowledge of humans' position and velocity. In this paper, we introduce JESSI (JAX-based E2E Safe Social Interpretable navigation), a lightweight end-to-end RL framework that maps raw LiDAR scans directly to kinematically feasible control commands. JESSI enhances safety via Dirichlet-parameterized continuous action spaces and deterministic bounding, while an integrated attention-based perception module extracts probabilistic human states for interpretable, socially aware decision-making. Through extensive simulations and real-world deployment on a differential-drive robot, we demonstrate that jointly optimizing the RL policy with a supervised perception signal in a multi-task paradigm enhances social behavior. Ultimately, JESSI is able to balance high navigation success rates and superior social behaviors compared to state-of-the-art baselines.
Read the full abstract

All authors Tommaso Van Der Meer, Andrea Garulli, Antonio Giannitrapani, Renato Quartullo, Alberto Vaglio, Alexandre Alahi

Sunday 4 October

Human-Computer Interaction · Cross-listed from cs.MM

An Edge-First Multimodal Asset Management Studio and Retrieval Engine for Heterogeneous AI-Generated Media

Chaoyu Lin

AI-generated image collections combine visual content with heterogeneous provenance such as prompts, model parameters, and workflow graphs. Managing these collections locally requires coordinated metadata extraction, responsive browsing, and retrieval without mandatory remote inference. We present Berry AI Studio, an edge-first asset management system built with Vue, Tauri, Rust, SQLite, and ONNX Runtime. Berry integrates format-specific metadata parsing with raw-payload retention, generation-aware cancellation of queued thumbnail work, a persistent multi-tier thumbnail manifest, cursor pagination, and local image-text retrieval. We evaluate these mechanisms in five independent repetitions on a pinned source revision. With 50,000 catalog records, the median warm first-page database call was 285.3 ms; at offset 40,000, keyset pagination reduced median raw-query latency from 57.9 to 0.46 ms. A controlled browser harness recorded a 0.3 ms median-of-run P95 update time during virtualized scrolling, while cancellation prevented all 1,200 stale speculative decodes in the tested trace. A versioned 10-case corpus produced 53/53 exact metadata-field matches, and controlled retrieval and grouping tests exercised the authentic storage and clustering paths. The implementation, benchmark scripts, raw logs, and checksummed result packages are publicly available.
Read the full abstract

Human-Computer Interaction

Educating future engineers about LLMs: A scalable workshop

R. Zhang, J. C. F. de Winter, T. Dicke et al. (+2 more)

As large language models (LLMs) are increasingly integrated into engineering workflows, students require hands-on experience to learn how to collaborate with them critically. This paper presents a scalable gamified workshop designed for engineering Master's students to practice human-AI collaboration in navigation planning. Using a mobile web interface across 10 workshop sessions, a total of 226 students wrote prompts for a non-reasoning and a reasoning LLM to solve grid-based navigation tasks of increasing complexity. The system returned robot-executable plans, trajectory visualizations, and automated scoring, culminating in a live demonstration on a Boston Dynamics Spot robot. In a post-workshop questionnaire, 81.5% reported substantial learning and 91.0% reported high engagement. Analysis of the submitted prompts revealed that students changed their strategies from step-by-step instructions for the non-reasoning LLM toward providing higher-level guidance for complex problem-solving tasks. We conclude that such interactive simulation-to-reality environments are viable for teaching the verification and collaboration skills necessary for responsible LLM use in engineering. Code is available at: https://github.com/renchizhhhh/LLM-robotics-workshop
Read the full abstract

All authors R. Zhang, J. C. F. de Winter, T. Dicke, D. Dodou, Y. B. Eisma

Human-Computer Interaction

Exploring the Effects of Personality in Human-Agent Interactions: A Study on User-Agent Synchrony with Human-based Vocalics

ai Alexander Hackney, Jhonathan Sora-Cardenas, Aibek Musaev et al. (+1 more)

User trust is paramount in human-agent interactions, as it allows users to feel comfortable being themselves around an agent. The process of building user rapport starts in how an agent was designed, from its modality to the setting, to any of the many features and characteristics that can be tailored. All these aspects can affect whether users will be able to properly interact with the virtual agent and achieve the intended purpose. One such feature that is critical in human-human interactions is personality. A person's personality can strongly influence whether those that interact with them perceive them as trustworthy. This study used virtual agents generated from human voices with known personality traits to evaluate user perceptions. We found that extraverted agents were deemed to be more likeable by users, and that there was no significant effect of user-agent synchrony on user perceptions of the agent. In addition, it was found that user personality, without regard for agent personality, affected user perceptions of the agents. Our observations suggest that user perceptions may depend more on agent-topic synchrony than user-agent synchrony, and contribute to the broader community with considerations for the design of effective human-agent interactions.
Read the full abstract

All authors ai Alexander Hackney, Jhonathan Sora-Cardenas, Aibek Musaev, Pedro Guillermo Feijóo-García

Human-Computer Interaction · Cross-listed from cs.AI

DelegationBench: Measuring When AI Agents Should Ask Before Acting

Shiva Pochampally

AI agents that send emails, edit files, and make purchases must decide when to act on their own and when to check with the user first. This decision is usually evaluated by showing a model a proposed action, asking whether it should proceed, and scoring agreement with human labels. We introduce DelegationBench to test whether such scores can be trusted. It has 156 scenarios with four possible responses (act, ask for permission, ask for missing information, refuse), and most scenarios come in matched pairs that change a single feature: whether the action was requested, what is at stake, whether it can be undone, or who will see it. Across ten models from five families, agreement scores mislead in three ways. A simple keyword rule, which we wrote after seeing the benchmark, agrees with our annotators more often than eight of the models, yet its decision changes in only 9 of 48 matched pairs. Equivalent ways of asking the same question change how often a model acts by up to 52.5 percentage points. And every model stops to ask the user less often when it must carry out the task with tools than when it judges a proposed action. When rules are stated explicitly, the same models follow them almost perfectly, so the gaps are not explained by a general inability to follow rules. We release the benchmark and evaluation tools and recommend reporting these properties separately rather than as one score.
Read the full abstract

Human-Computer Interaction

Reflecting on Creative-Boundaries with an AI Co-Doodler

Samia Menon, Samyukta Jayaram, Chetan Goenka et al. (+1 more)

In this pictorial, we consider how the negotiation of creative boundaries with a co-creative AI system can create moments for personal creative reflection. We ground this in our experiences with Froggi-Draw, a single-initiative co-doodling system that gives users power to decide when and how much an AI "collaborator" (Froggi) contributes to their drawing. From a 1-week pilot study where (n=8) novice and experienced artists doodled daily with the system, we report on ways participants navigated creative risk and uncertainty, and how their usage and perceptions of Froggi shifted over time. In moments of disruption, participants described the system as encroaching on their creative territory. We consider how the design of a supportive, co-creative AI "collaborator" might look like the design of a supportive power dynamic—and finding ways to offer users control to find, reflect upon, and flexibly negotiate the boundaries of that dynamic.
Read the full abstract

All authors Samia Menon, Samyukta Jayaram, Chetan Goenka, Shm Garanganao Almeda

Human-Computer Interaction · Cross-listed from cs.CY

Altruism as Infrastructure: Volunteer Moderation in a Bangladeshi Higher Education Facebook Group

Umme Jannat Taposhi, Md. Tawhid Anwar, Alvi Islam Ratul et al. (+1 more)

Aspiring international students across Asian countries increasingly depend on commercial education agents to navigate scholarships, documentation, and visas. Alongside this commercial infrastructure, volunteer-run Facebook groups have emerged. Unpaid admins and moderators, often under their real identities, vet information, screen scams, and guide members through scholarships, visas, and departure logistics. We study one such Bangladeshi group, HigherStudyAbroad: Global Hub of Bangladeshis, founded in 2010. Drawing on semi-structured interviews with 17 volunteer admins and moderators, we found that altruism becomes an organizing logic. It shapes moderators' identities, sustains invisible labor, and grants the group legitimacy against paid agents, even amid new technologies such as AI that could reshape this work. Human judgment remains central to how moderators sustain trust. We discuss implications for HCI's understanding of volunteer labor and migration infrastructure, along with design implications for platforms that depend on unpaid, long-term contribution.
Read the full abstract

All authors Umme Jannat Taposhi, Md. Tawhid Anwar, Alvi Islam Ratul, S M Taiabul Haque

Human-Computer Interaction

Human-Like Attention? A Psychophysical Comparison of Visual Search in Humans and MLLMs

Renchi Zhang, Joost C. F. de Winter, Dimitra Dodou et al. (+2 more)

Visual search is a fundamental cognitive ability. This study investigates whether Multimodal Large Language Models (MLLMs) exhibit human-like difficulty signatures in visual search tasks. We compared search performance of humans (n = 1,250) and MLLMs using identical 2D and 3D stimuli across different set sizes. Both groups showed efficient performance in feature searches, most clearly when the target had a unique color, but performance degradation in conjunction searches as set sizes increased. Additionally, we found strong correlations between human and MLLM error rates ($\rho = 0.82$), which suggests that MLLMs are sensitive to similar objective complexities, such as stimulus heterogeneity. However, differences were found as well: whereas humans invested extra search time to respond accurately on target-absent trials, MLLMs exhibited extreme present/absent response biases in complex searches. We conclude that MLLMs replicate high-level human performance signatures, yet their underlying computations differ significantly.
Read the full abstract

All authors Renchi Zhang, Joost C. F. de Winter, Dimitra Dodou, Harleigh C. Seyffert, Yke Bauke Eisma

Human-Computer Interaction · Cross-listed from cs.RO

Social Navigation for Tour-guide Robot

Vikram Shree and Jose Nino

We propose a force-based model for social navigation of a tour-guide robot. Social forces due to various factors like obstacles, user position and heading, have been accounted for in the model. We claim that each one of these forces makes the robot more sociable to the user and we design an experimental setup for evaluation. In the experiment, the user follows an autonomous robot to a destination in a known map, while undertaking a few simple sub-tasks in the middle, which serve as distractions. For each participant, we run several rounds of the experiment, each with different forces and a shortest path, A*-search baseline model. Using per-round subjective indicators, we propose to study the effect of our force model on constructs such as: follow-ability, perceived safety, and perceived intelligence.
Read the full abstract

Human-Computer Interaction

Imagining a Muslim Internet: Trust, Autonomy, and Segregation in a Faith-Aligned Browser

Umme Jannat Taposhi, Farhan Tanvir Niloy, Sabbir Bin Abdul Latif et al. (+2 more)

Religiously branded platforms raise important questions about trust, usability, and autonomy when technology is built around a specific faith. Prior HCI work on Islam and Muslim technology has focused on single-purpose tools such as prayer, scripture, and health apps, leaving infrastructures like browsers, which shape a user's relationship with the Internet, unexamined. We address this gap through semi-structured interviews with 16 users of Kahf Browser, a faith-aligned browser designed to support Muslims' online activities. Findings show religious identity motivates adoption but does not sustain it. Trust is not fixed by religious branding but shifts over time based on the browser's functionality, and users take pride in Muslim-built infrastructure. Building on these insights, we introduce porous digital segregation: a model where users seek not a sealed alternative internet but a protected default with selective exit mechanisms. We conclude by connecting findings to broader issues in faith-aligned technology and offering design recommendations.
Read the full abstract

All authors Umme Jannat Taposhi, Farhan Tanvir Niloy, Sabbir Bin Abdul Latif, Farida Chowdhury, S M Taiabul Haque

Human-Computer Interaction

Optimizing AI-Driven Messaging for Type 2 Diabetes Management: Insights from Patient Preference Elicitation

Angela Mastrianni, Defne Levine, Katerina Andreadis et al. (+4 more)

Generative AI (GenAI) allows for improved user experience within conversational agents for diabetes management by supporting dynamic, context-aware conversations. In this study, we elicited patient preferences for the communication style of a GenAI-based conversational agent (uMatter) developed to support diabetes management. We conducted an online survey with 125 individuals with type 2 diabetes. The survey included a discrete choice experiment to evaluate participant preferences for different types of messaging attributes. The survey also elicited participant perceptions and feedback on the messages from uMatter. We found significant preference heterogeneity for the inclusion of emojis within the messages. Additionally, qualitative findings indicated that participants had different desired personas and communication styles for the conversational agent. We propose strategies from recent human-computer interaction and natural language processing research that can be used to design GenAI-based conversational agents that align with the communication preferences of patients.
Read the full abstract

All authors Angela Mastrianni, Defne Levine, Katerina Andreadis, Lynn Xu, Priscilla D'Antico, Antoinette Schoenthaler, Devin Mann

Human-Computer Interaction

Rethinking Inline Citation Verification in Scholarly Communication

Xinrui Fang, Reese Fairchild, Nasi Wang et al. (+4 more)

Inline citations are central to scholarly communication, yet verifying their use is becoming increasingly challenging because of growing review pressures. We conducted a mixed-methods exploratory study to investigate how reviewers verify inline citations, the factors shaping their verification practices, and how they envision GenAI supporting this process. Across interviews (n=12) and a survey (n=203) of reviewers from HCI and AI venues, we found that reviewers' perceptions of citation importance, verification practices, and desired AI autonomy varied across citation types, reviewer characteristics, and research backgrounds. These findings highlight the need for adaptive support tailored to different citation types and reviewer practices, while revealing diverse preferences regarding the use of GenAI for this process. Moreover, effective citation verification should involve collaboration among reviewers, authors, and the broader research community, rather than relying solely on individual reviewers.
Read the full abstract

All authors Xinrui Fang, Reese Fairchild, Nasi Wang, Anran Xu, Simo Hosio, Sylvain Malacria, Koji Yatani

Human-Computer Interaction · Cross-listed from cs.CV

Answer with Evidence: Consistency-Aware Grounded Visual Question Answering for Roadside Traffic Scenes

Runwei Guan, Rongsheng Hu, Shangshu Chen et al. (+10 more)

Roadside traffic reasoning requires every free-form textual claim to be backed by visual evidence. Existing grounded multimodal large language models (MLLMs) frequently exhibit say-point mismatch, in which the textual answer contradicts the bounding boxes the model localizes. Evaluation metrics that score answers and boxes separately leave this failure unpenalized. We trace the mismatch to the conventional answer-then-ground factorization, which commits to a numerical claim before any object is enumerated. To measure it, we build RoadSceneVQA-G, a benchmark of 34.7K question-answer pairs in which every free-form answer is linked to the set of boxes that witnesses it, and we propose the Answer-Grounding Consistency (AGC) evaluation suite. To address it, we introduce Enumerate-then-Answer (EtA), which reverses the generation order so that answer-evidence agreement becomes a property of the output structure, and Enumeration-Consistent Policy Optimization (ECPO), a reinforcement learning stage that uses the union of multiple rollouts as a recall teacher without ground-truth boxes. EtA raises say-point consistency from 26.6% to 93.7% and grounding F1 from 52.2% to 73.0%, and ECPO further increases F1 to 75.6% without per-box supervision. On gRefCOCO, the same framework outperforms the strongest compared method, indicating that it transfers beyond traffic scenes. The project is available at https://github.com/GuanRunwei/RoadSceneVQA-G.
Read the full abstract

All authors Runwei Guan, Rongsheng Hu, Shangshu Chen, Ningwei Ouyang, Shaofeng Liang, Heyi Lin, Jinjing Zhu, Yang Shi, Dongming Wu, Jianfeng Dong, Daizong Liu, Henghui Ding, Hui Xiong

Human-Computer Interaction

Preference vs. Performance: EEG-Based Classification of Learner Engagement in Multimodal Instruction

Sri Jahnavi Adusumilli, Deepak Giri, Pallavi Vaswani et al. (+4 more)

Effective adaptive instructional systems require robust measures of learner engagement that go beyond static user profiles. This study employs Multimodal Learning Analytics (MMLA) to investigate the relationship between self-reported instructional modality preferences and objective neurophysiological markers of engagement. Thirty-seven participants engaged with learning content delivered via varying modalities (visual, auditory, reading/writing, kinesthetic). We captured real-time neural activity using two EEG devices: the Emotiv EpocX (14 channels, 128 Hz) and OpenBCI (16 channels, 125 Hz). Preferences were assessed using the VARK questionnaire. Consistent with literature challenging the "meshing hypothesis," aligning instructional modality with stated preferences did not significantly predict performance gains. However, spectral analysis of EEG data revealed divergent engagement patterns: when content aligned with preferences, distinct neural activity patterns emerged in theta and alpha frequency bands-markers associated with attention and cognitive processing. These signals were used to train a binary logistic regression classifier, achieving a mean accuracy of 83.21% with OpenBCI and 56.27% with Emotiv EpocX. These findings suggest that while self-reported preferences may not dictate learning outcomes, they significantly influence neurophysiological engagement, offering a viable, non-invasive input for adaptive educational algorithms.
Read the full abstract

All authors Sri Jahnavi Adusumilli, Deepak Giri, Pallavi Vaswani, Pallavi Singh, Megha Moncy, Lalitha Pranathi Pulavarthy, Saptarshi Purkayastha

Human-Computer Interaction

Proactive AI: From Turns to Replannable Dialogue Timelines

Zijie Yang

Conventional large-language-model chat interfaces typically follow user-initiated turns: each request elicits a response, after which the system waits for further input. Human asynchronous communication instead unfolds over time through message bursts, delays, silence, resumed topics, and self-initiated contact. Proactive dialogue therefore requires not only deciding when to speak, but also managing pending conversational actions and allowing them to be revised as context evolves. We introduce Proactive AI, a framework for proactive dialogue built around replannable temporal message queues. The framework treats interaction as a continuous event timeline and unsent conversational actions as revisable state. A user event, dialogue-pacemaker event, or scheduled decision may yield silence, one or more immediate messages, or future conversational actions. The system may also observe a user message without an immediate visible response and defer the response decision. Before delivery, every planned message must be reconsidered against the current context; reaching a scheduled time does not itself authorize delivery. We provide a formal semantics that characterizes when future conversational actions may be revised, withdrawn, or executed as context evolves. The framework extends AI participation beyond responses to current requests, enabling self-initiated exchanges without new requests, sustained follow-up over time, and revisions of subsequent actions as context evolves. It provides an executable basis for long-term human-AI interaction in tutoring, scientific collaboration, and everyday companionship.
Read the full abstract

Human-Computer Interaction · Cross-listed from cs.CR

Bounded Reasoning: Cognitive Hierarchy in Human-versus-AI Cyber Defense

Zahra Aref, Sheng Wei and Narayan B. Mandayam

Human-agent evaluations often compress interaction into a single performance score, even when human and automated policies adapt differently over time. We study this in a sequential cyber-defense game on an attack graph, where a human or reinforcement-learning defender protects cloud assets against a Deep Q-Network (DQN) attacker. We compare four defender settings: a human reward-only game operationalizing the DQN information structure, a human reward-plus-transition game operationalizing the Cognitive Hierarchy Theory-driven DQN (CHT-DQN) information structure, an automated DQN defender, and an automated CHT-DQN defender. In the reward-only game, participants receive payoff and reward feedback. In the reward-plus-transition game, they also see attacker-aware transition probabilities from the CHT-DQN model. Across 80 Mechanical Turk participants and matched automated simulations, human defenders adapted to outcomes in a way the automated defenders did not: they were more likely to reselect a node after a successful defense than after a failure, in both games, an asymmetry we interpret as consistent with Prospect Theory and Cumulative Prospect Theory. The 40-round average also mixes early rounds, where the DQN attacker acts mostly at random, with late rounds, where it mostly exploits; in the final stage, mean protection ranks the reward-plus-transition human game above the reward-only human game, then the automated CHT-DQN defender, then the automated DQN defender, an ordering the overall mean hides. By contrast, the reward-plus-transition game does not produce a statistically reliable overall gain in weighted data protection over the reward-only game. These results suggest that evaluating human-agent cyber-defense systems only by an averaged task score can miss behaviorally meaningful differences in adaptation, action allocation, and bounded human reasoning.
Read the full abstract

Saturday 3 October

Human-Computer Interaction

Refuses in Chat, Complies with Arms: LLM Refusal Does Not Survive Robot Control

Edward Sun, Christopher Leet, Sravanthi Machcha et al. (+4 more)

Frontier language models reliably refuse to explain how to cause physical harm. We ask whether this refusal survives when the same models control a robot. We introduce RoboHarm, a physical AI safety benchmark of harmful robot instructions, and test seven frontier models on it. We hold each model fixed and vary only the interface, from text to Code as Policies to vision agents that command end-effector poses or joint angles. In a MuJoCo simulation twin of a bimanual YAM arm robot, refusal falls from 73% of text requests to 18-25% once the model controls the robot, even when it is told what the objects are. On the physical robot, six models with end-effector control refused 18% of 612 trials against 19% in sim, and three with joint control refused 41% of 252 against 31%. GPT-5.6 Sol refused every text request and none of its trials on the real robot. Case studies of the robot transcripts show that models miss the hazard, name it without inferring its consequences, or act despite recognizing it, and their refusals often hinge on uncertainty about what an object is rather than on the harm of the action. Safety alignment learned in text thus fails to generalize to action, leaving current models with a poorly understood safety gap in the physical world.
Read the full abstract

All authors Edward Sun, Christopher Leet, Sravanthi Machcha, Achyutha Menon, Tzu Kit Chan, Yuchen Cui, Jay Chooi

Human-Computer Interaction · Cross-listed from cs.LG

EVFormer: An Egocentric Vision-EMG Bidirectional Attention Model for Bimanual Hand Pose Estimation

JiaCheng Ge, SiYu Zhang, ShengJie Li et al. (+1 more)

Egocentric bimanual hand pose estimation is important for virtual interaction, wearable control, and rehabilitation, but visual observations are often degraded by self-occlusion, hand-hand contact, and object manipulation. We propose EVFormer, a multimodal framework that combines the current RGB frame with the preceding 200 ms of bilateral wrist surface electromyography (sEMG) to estimate 44 finger and wrist joint angles. EVFormer separately encodes visual spatial features and sEMG temporal features, enables cross-modal information exchange through sequential bidirectional cross-attention, and integrates the two modalities using feature-wise gated fusion. We evaluate EVFormer in a single-participant feasibility study using one synchronized public EgoEMG recording with chronologically separated training, validation, and test splits. On 296 test samples, EVFormer achieves a mean absolute error of 11.482 degrees, compared with 13.228-13.610 degrees for vision-only, sEMG-only, late-fusion, and training-mean baselines. This corresponds to relative error reductions of 13.20% compared with the vision-only model and 14.23% compared with late fusion. EVFormer also achieves the lowest error in four of the five evaluated gesture classes. These results provide preliminary evidence that feature-level interaction between egocentric vision and sEMG can improve bimanual hand pose estimation. Further evaluation across participants, recording sessions, sensor placements, and real-world interaction conditions is required to establish the generalizability of the approach.
Read the full abstract

All authors JiaCheng Ge, SiYu Zhang, ShengJie Li, XinTong Yang

Human-Computer Interaction

Trends in EHR Satisfaction and Interoperability Among Family Physicians: A Five-Year Analysis of the ABFM Continuous Certification Questionnaire, 2022-2026

Carl Y. Zhang, Nathaniel Hendrix, Robert L. Phillips et al. (+1 more)

Objective: To assess trends in family physicians' (FPs) electronic health record (EHR) satisfaction and their experience of interoperability across organizations. Materials and Methods: Serial cross-sectional analysis of five waves of the American Board of Family Medicine's Continuous Certification Questionnaire (2022-26). Adjusted logistic regression compared odds of being very satisfied across 11 EHRs; Cochran-Armitage tests assessed vendor trends. Ease of using outside information was analyzed separately for same- and different-vendor sources. Results: Across 36,785 FPs, adjusted odds of being very satisfied relative to Epic ranged from 0.91 (95% CI, 0.74-1.11) for Elation Health to 0.17 (95% CI, 0.15-0.20) for Oracle Health/Cerner. Only Epic's users became significantly more satisfied (Benjamini-Hochberg adjusted P < .001), widening the vendor satisfaction gap. Within comparable periods, ease of using different-vendor information did not improve; fewer than 13% rated it very easy. Epic's interoperability advantage reversed by exchange type: non-Epic users had less than half the odds of Epic users of rating same-vendor interoperability very easy (adjusted OR, 0.41; 95% CI, 0.38-0.44) but 1.54 times the odds for different-vendor interoperability (95% CI, 1.35-1.75). Discussion: Rising national exchange volume has not yielded easier use of outside information. Epic's advantage is confined to its own network, so market concentration may shift, rather than solve, the different-vendor problem. Conclusion: The ABFM data are already informing EHR policy but also have utility for the EHR marketplace. While EHR satisfaction has known implications for burden and burnout, poor interoperability is a quality and safety threat that may be compounded by artificial intelligence.
Read the full abstract

All authors Carl Y. Zhang, Nathaniel Hendrix, Robert L. Phillips, Andrew W. Bazemore

Human-Computer Interaction

VoCa: Designing Speech-Canvas Interaction for Voice-Based Conversational Agents

Yate Ge, Run Yuan, Yueran Qi et al. (+7 more)

People write and sketch while speaking to explain, organize, and develop content together. Inspired by these practices, we investigate how voice agents can use a canvas alongside speech in multi-turn conversations with users. We conducted a two-part formative study: an observational study of how pairs coordinated speech and boardwork, followed by a design workshop that informed a design space for speech-canvas interaction with voice agents. Building on these insights, we developed VoCa, a voice agent that coordinates speech with visual object creation, annotation, and attention guidance. A five-day deployment with 18 participants examined usability, experiences of speech-canvas interaction, patterns of use, and desired improvements. Participants' experiences highlighted opportunities for speech-canvas interaction in learning, work, and daily life, alongside challenges in coordinating what agents say and show in ways users can follow and influence. These findings inform how voice agents can use a canvas alongside speech in conversation.
Read the full abstract

All authors Yate Ge, Run Yuan, Yueran Qi, Wenjie He, Jiaqi Mo, Yangshuo Chen, Wenbin Zuo, Xiaohua Sun, Weiwei Guo, Qi Wang

Human-Computer Interaction

Neurodiversity-Aware Multimodal Affective Computing for Neurodevelopmental Assessment: From Norm-Referenced Classification to Context-Sensitive Decision Support

Mateusz Pomianek and Anna Łężniak-Seruga

Automated neurodevelopmental assessment increasingly combines computer vision, speech, eye tracking, physiology, and machine learning, yet multimodality and discrimination do not establish construct validity, clinical usefulness, or appropriate interpretation of behavioral variation. We propose a testable architecture for neurodiversity-aware multimodal affective computing in which population-relative deviation is not treated as sufficient evidence of adverse functioning. This conceptual article reports neither a new participant-level study nor a trained predictive system, and no predictive or clinical superiority is claimed. The framework keeps observable signals, modality quality and availability, context, population-relative information, pooled within-person references and, when sufficiently supported, context-conditioned within person references, and predictive uncertainty distinguishable throughout inference. It defines structured decision-support outputs and auditable requirements for target validity, temporal integrity, contextual leakage, missing-modality robustness, calibration, subgroup evaluation, interpretability, privacy, and human decision boundaries. The contribution is architectural rather than algorithmic: it specifies empirically testable constraints that can be instantiated with different multimodal learning methods. Autism provides the principal motivating evidence base, without assuming unchanged transfer to other neurodevelopmental conditions.
Read the full abstract

Human-Computer Interaction · Cross-listed from cs.AI

Understanding Generative AI Use in Programming MOOCs: The Role of Course Context and Learner Characteristics

Marina Lepp

The increasing availability of generative artificial intelligence (GenAI) tools, such as ChatGPT and code-completion assistants, raises questions about how learners integrate these tools into learning activities, particularly in MOOCs that attract diverse participant populations. This study examines the use of GenAI in two programming MOOCs taught in Estonian that differ in duration, workload, topic complexity, and assignment volume: About Programming (4 weeks, 26 expected hours, n = 187) and Introduction to Programming (8 weeks, 78 expected hours, n = 182). Post-course questionnaire data were analyzed using non-parametric statistical methods to examine self-reported adoption, usage frequency, and purposes of GenAI use across courses and learner backgrounds. The results show that GenAI adoption was widespread in both MOOCs, with no statistically significant differences by gender, age, education level, or prior programming experience. However, participants in the longer, more extensive MOOC reported significantly higher usage frequency and were more likely to use GenAI for debugging and idea generation. Reported usage frequency for code explanation and debugging was also higher in the longer course. Exploratory analyses found limited relationships between GenAI use and learning-related outcomes. The findings suggest that course context may play a greater role than learner characteristics in shaping how GenAI tools are used. These results provide implications for instructional design and guidance in programming education.
Read the full abstract

Human-Computer Interaction · Cross-listed from cs.AI

MERCI Cards: An LLM Evaluation and Deployment Framework for High-Stakes Domains

Aparna Komarla and Annalisa Szymanski

As LLMs are increasingly deployed in high-stakes professional workflows, engineers and researchers require principled protocols to systematically track, monitor, and improve model performance across deployment cycles. We present a mathematical framework for iterative LLM evaluation and deployment, and demonstrate its application to AI systems used in criminal justice. Our framework formalizes LLM integration in high-stakes, high-risk, and resource-constrained domains across model selection, rubric design, evaluations and deployment via a weighted multi-objective optimization. We demonstrate that MERCI Cards can guide improvements of the system across deployment iterations, direct developer attention toward under-performing areas, and focus user attention on validation and error-correction in the LLM's outputs.
Read the full abstract

Human-Computer Interaction · Cross-listed from cs.CL

XTurnix: Large-Scale Self-Supervised Turn Control through Two-State Binary Decisions

Zhanxun Liu, Yifan Duan, Hengtao Wu et al. (+7 more)

General turn-taking behavior in real-time dialogue systems requires deciding whether to keep listening or start responding while listening, and whether to continue or stop while speaking. Existing turn detectors use heterogeneous, task-specific label spaces and are often trained on limited annotations or evaluated on isolated utterances, making them difficult to use as a unified causal controller with comprehensive context. We propose XTurnix, a compact text-based model that formulates turn control as two binary decisions conditioned on the AI's current listening or speaking state and predicts a single control token from the complete dialogue history. XTurnix is pretrained on 5.5 million causal action examples automatically derived from timestamped two-speaker transcripts, then fine-tuned on synthetic multi-turn examples with a flatter distribution across the four state-action labels. We evaluate XTurnix on four public benchmarks and a balanced self-curated benchmark. Across the public benchmarks, XTurnix achieves the best results on all SemanticVAD and LiveKit splits, ties the native Smart-Turn model on Smart-Turn Bench, and achieves the highest incomplete-turn accuracy on Easy-Turn. On the self-curated benchmark, it reaches 89.06% accuracy, more than 20 percentage points above the strongest third-party baseline at 68.75%, while maintaining F1 scores between 84.21% and 90.63% across all four categories. These results demonstrate unified listening- and speaking-state turn control in a single compact model. Code is available at https://github.com/xcc-zach/xturnix, with an interactive demo at https://huggingface.co/spaces/xcczach/xturnix-demo.
Read the full abstract

All authors Zhanxun Liu, Yifan Duan, Hengtao Wu, Chen Yang, Qinyuan Cheng, Kun Wang, Xingyu Zeng, Xipeng Qiu, Chaochao Lu, Xie Chen

Human-Computer Interaction

Benchmarking Psychological Dynamics in Generative Agents

Sumer S. Vaid and Ashley V. Whillans

Large language models (LLMs) are increasingly deployed to simulate human behavior, acting as computational replicas of human subjects. Yet the lived psychological experience of humans is difficult to benchmark, particularly as it unfolds over time. We introduce a psychometric benchmark for computational replicas: personas that carry a fixed identity through an evolving sequence of events. Built entirely from published norms and meta-analytic effects, the benchmark scores two dimensions of psychological realism. The first, internal validity, quantifies whether generated trajectories reproduce the internal structure of repeated human measurement: distributions, the between- versus within-person variance partition, temporal dependence, and range. The second, external validity, quantifies whether replicas recover established trait, state, and indicator relations. Across 36 open-weight and proprietary LLMs from nine developers (1B-671B parameters), most recover the direction of established relations (84.3% mean agreement) and the variance partition (26 of 34), yet the within-person correlation and distributional structure elude recovery. Internal validity is independent of scale and capability: a mid-size open LLM (Gemma-3-27B) strikes the best trade-off between the two dimensions. The benchmark is a precondition for using computational replicas in causal inference across domains (e.g., marketing, healthcare), and identifies within-person grounding as the central challenge ahead.
Read the full abstract

Human-Computer Interaction

Evaluating Just Noticeable Differences in Layered Opacity Visualizations

Caterina Ponti, Shano Liang, Lane Harrison et al. (+1 more)

Opacity is a widely used channel in data visualization, but it remains less well understood compared to channels such as color, length, size, etc. Recent work from Meng et al. investigated the impact of opacity across competing color schemes, finding that certain color schemes were associated with better participant accuracy. We examine these effects further in a controlled two-alternative forced-choice setup to determine whether opacity differences are truly equal across possible opacity comparison ranges. In a within-subjects study with 96 trials, including two competing color schemes (best and worst from Meng et al.) and 48 opacity pairs, we find little differences between color schemes but larger individual differences in accuracy. Further, results show stable performance in middle opacity ranges, with more errors occurring when comparing extreme values. We discuss potential implications for design guidelines and further study and make our study materials, analysis scripts, and data available at https://osf.io/zv9dx/overview?view_only=38d03cea1b3d42788593e3e6b1016cfd.
Read the full abstract

All authors Caterina Ponti, Shano Liang, Lane Harrison, Alark Joshi

Human-Computer Interaction

DimSteer: Steering LLM Authoring with Automatically Discovered Stylistic Controls

Ajit Mallavarapu and Ziwei Gu

Large language model writing interfaces often make users steer outputs by repeatedly articulating desired changes in natural language. Yet writers may recognize useful stylistic directions only after seeing alternatives, making revision recall-heavy. We present DimSteer, an authoring interface that samples prompt-local completions, discovers high-variance activation-space axes of variation, labels them, and exposes them as sliders with pole previews, diff comparison, and reset controls. Users can manipulate discovered dimensions, reducing the need to reformulate prompts for each stylistic adjustment. In a within-subjects study with 16 participants against a matched prompt-only baseline, DimSteer reduced mental demand, effort, and frustration while preserving comparable perceived success. Participants valued the surfaced dimensions, yet 15 of 16 disagreed that they would have thought to request the same changes in a prompt. Results suggest prompt-local controls can shift LLM authoring from recall-based prompting toward recognition-based exploration and direct manipulation, while preserving prompting for open-ended edits.
Read the full abstract

Friday 2 October

Human-Computer Interaction

LINK: A Corpus-Grounded Design Pattern Library for Educational Technologies

Lorena Lyon, Hyoungwook Jin, Vincent Cavez et al. (+3 more)

Designing learning interfaces is often an ill-structured problem: effective solutions depend on learning goals, subject matter, learner characteristics, and interaction context that are difficult to generalize. Designers must integrate knowledge from learning sciences, interaction design, and subject domains when translating learning goals into interface decisions. Yet, prior efforts to connect learning theory and design have largely focused on learning activities or specific technologies, leaving interface-level design knowledge fragmented across systems. In this work, we take a bottom-up approach to surface this knowledge from existing educational interfaces. We present LINK (Linking INstructional Knowledge to interfaces), a library of 27 recurring interface patterns, organized under nine higher-level design commitments, which describe broader learning-science-grounded goals for interfaces. As part of developing LINK, we conducted a user study (N=14) with designers to evaluate LINK's generative power. Our findings suggest that LINK helps designers generate interface ideas and articulate theory-grounded rationales for design decisions.
Read the full abstract

All authors Lorena Lyon, Hyoungwook Jin, Vincent Cavez, Roy Pea, Xu Wang, Hari Subramonyam