Human-Computer Interaction
Participant-Mediated Collection of Sensitive Digital Trace Data: The CANDOR Research Infrastructure
Andrew Zhao, Rijul Magu, Ekta Raj et al. (+2 more)
Digital trace data provide rich measures of behavior in everyday settings, but the research ecosystem supporting their collection is constrained by declining platform API access and a historical reliance on publicly observable data. Participant-mediated data donation offers a complementary approach in which individuals contribute selected portions of their own digital histories to research. Such data can include longitudinal and non-public behavior, span multiple platforms and modalities, and be linked to independently collected study measures, enabling study designs that are difficult to implement using public social media data alone. These opportunities also introduce methodological challenges around participant control, data minimization, heterogeneous platform exports,
privacy, and governance, particularly when semantic or
multimodal content is necessary to study the construct of interest. We present CANDOR (Collecting and Analyzing Networked Data for Open Research), an end-to-end infrastructure for participant-mediated collection and governance of sensitive digital trace data. CANDOR supports participant-directed selection of platforms, data types, and temporal ranges; modular platform- and modality-specific parsing and de-identification; linkage to independent study measures; and protected processing, storage, and access. We derive design requirements for this class of research and compare CANDOR with existing data donation infrastructures, identifying how different approaches support participant control, data minimization, scientifically necessary data richness, study-design flexibility, and governance. Together, this work provides a methodological and infrastructural framework for using participant-contributed digital traces in behavioral research, particularly when the data needed to address a scientific question are longitudinal, non-public,
multimodal, or sensitive.
Read the full abstract ▾
All authors Andrew Zhao, Rijul Magu, Ekta Raj, Teresa Elinjikkal, Munmun De Choudhury
Human-Computer Interaction
"We Are Tired of Explaining": Communication Practice and AI Roleplay Training for Community Health Workers in Rural India
Neil K. R. Sehgal, Sunny Rai, Sai Preethi Matam et al. (+4 more)
Community health workers (CHWs) in the Global South increasingly encounter AI-powered tools, yet the counseling work central to their role remains largely unsupported. We study communication practices among Accredited Social Health Activists (ASHAs) in rural Rajasthan, India, through simulated family-planning calls, semi-structured interviews, and an LLM chatbot roleplay design-probe with 20 participants. In calls, ASHAs often responded to social or material concerns by shifting to health-risk information, denying concerns, promising unspecified help, or listing medical solutions with limited explanation. A smaller set of responses instead engaged concerns, sought permission before involving family members, or left decisions with beneficiaries. We interpret these patterns through Motivational Interviewing, emphasizing restraint from correcting, persuading, or over-solving. Drawing across observed calls, interviews, and probe reactions, we derive design considerations for AI roleplay training: keep AI in a rehearsal role, provide descriptive rather than prescriptive feedback, and evaluate counseling process rather than agreement with prescribed responses.
Read the full abstract ▾
All authors Neil K. R. Sehgal, Sunny Rai, Sai Preethi Matam, Khushboo Gupta, Hamid Abdullah, Mohit Jain, Sharath Chandra Guntuku
Human-Computer Interaction
Quick-View Takeaways: How Does Title Framing Influences Pattern Identification in Line Charts?
Jasmine Lim, Tapendra Pandey, Arran Zeyu Wang et al. (+1 more)
Visual data communication in digital media is increasingly characterized by short
attention spans and snapshot-based viewing, often employing line charts to convey trends and patterns. Among all visual elements, titles are crucial ones that can shape how viewers interpret visual information and form chart takeaways. In this study, we examine how title characteristics, particularly title word count and intended message, influence people's pattern identification in single-class line charts. Participants viewed 50 line charts collected from online news media and identified the pattern they perceived. Our results demonstrate that both title word count and intended message significantly influence viewers' pattern identification. Our findings highlight the importance of title design in shaping chart takeaways and effective visualization communication.
Read the full abstract ▾
All authors Jasmine Lim, Tapendra Pandey, Arran Zeyu Wang, Ghulam Jilani Quadri
Human-Computer Interaction
How Does Title Framing Influence Pattern Identification in Line Charts?
Jasmine Lim, Tapendra Pandey, Arran Zeyu Wang et al. (+2 more)
Visual data communication in digital media is increasingly characterized by short
attention spans and snapshot-based viewing, often employing line charts to convey trends and patterns. Among all visual elements, titles are crucial elements that can shape how viewers interpret visual information and form chart takeaways. In this study, we examine how title characteristics, particularly title word count and intended message, influence people's pattern identification in single-class line charts. Participants viewed 50 line charts collected from online news media and identified the pattern they perceived. Our results demonstrate that both title word count and intended message significantly influence viewers' pattern identification. Our findings highlight the importance of title framing in shaping quick-view pattern takeaways and supporting effective visualization communication.
Read the full abstract ▾
All authors Jasmine Lim, Tapendra Pandey, Arran Zeyu Wang, Sungahn Ko, Ghulam Jilani Quadri
Human-Computer Interaction
CareMirror: Bringing Caregiver Wellbeing into the Dementia Care Ecosystem
Jiayue Melissa Shi, Ethan Nguyen, Drishti Goel et al. (+7 more)
Family caregivers of people living with dementia shoulder emotional and practical responsibilities, yet their own wellbeing often remains peripheral to dementia care. We built CareMirror, an envisioned caregiver wellbeing ecosystem with interconnected caregiver- and clinician-facing interfaces for longitudinal reflection, personalized support, and caregiver-controlled sharing with clinical care. We conducted semi-structured interviews with 14 caregivers, using CareMirror as a design probe to examine how they perceived this ecosystem and what expectations, concerns, and boundaries emerged around clinical connection. Caregivers valued
attention to their wellbeing, longitudinal awareness, context-sensitive support, and clinical visibility when it could lead to meaningful follow-up. However, repeated reflection could become burdensome or emotionally difficult, automatic clinical sharing could inhibit candid disclosure, and participants wanted control over what information entered clinical care. They also expected AI to support reflection and communication without replacing caregiver voice or clinician judgment. We contribute design considerations for proactive, clinically connected caregiver wellbeing support.
Read the full abstract ▾
All authors Jiayue Melissa Shi, Ethan Nguyen, Drishti Goel, Upasana Natarajan, Shashwat Srivatsa, Daniel S. Brown, Violeta J. Rodríguez, Dong Whi Yoo, Ravi Karkar, Koustuv Saha
Human-Computer Interaction
Lexplorer: Navigating the Complexity of Legal Document Landscapes
Daniel Fürst, Titus Pünder, Maximilian T. Fischer et al. (+1 more)
As technological and social innovations create novel regulatory challenges, legal systems grow in complexity - increasing the need for interfaces that enable effective interactions with legal document collections. Through interviews with legal scholars (n=15), we find that supporting legal work requires going beyond
retrieval-centered legal-information-system paradigms. Hence, we propose Lexplorer, a flexible interface for exploring, navigating, and analyzing legal documents, based on a taxonomy capturing user intents. Distinguishing text and data views for one, few, and many documents, Lexplorer enables context-sensitive interactions with evolving collections of interconnected legal texts, facilitating Adaptive Meaning Construction in law. We evaluate Lexplorer with legal scholars (n=20) in the context of European Union law, validating our elicited requirements, intent taxonomy, and prototype design. Resulting from a close collaboration between visual-analytics researchers and legal scholars, our work also provides nuanced insights into the process required to design interactive systems for expert domains driven by implicit methodological knowledge.
Read the full abstract ▾
All authors Daniel Fürst, Titus Pünder, Maximilian T. Fischer, Corinna Coupette
Human-Computer Interaction · Cross-listed from cs.CY
Evaluating Ambient Clinical Scribes in India: The Need for Multilingual Real-World Clinical Conversation Data
Siddharth D Jaiswal, Krithi S, Ashish Makani et al. (+3 more)
Ambient clinical scribes (ACS) are being rapidly deployed at scale across Global South healthcare settings, aiming to reduce clinician documentation time, especially in overburdened environments like India. These ACS are primarily developed or distilled from models built and validated on Global North
speech,
languages and consultation styles. Indian clinical encounters are brief, triadic, multilingual, code-mixed with low-resource
languages, and conducted in highly resource-constrained, noisy settings – increasing the likelihood of ASR and note-generation errors manyfold. We posit an urgent need to develop a standardized evaluation infrastructure to assess whether these systems are safe, reliable, and well-suited to the Indian healthcare setting. We substantiate our claims through a mixed-methods study – a systematic survey of publicly available patient-clinician conversational
datasets, a quantitative comparison of these
datasets against conversational and cultural markers drawn from the Indian clinical-communication literature, and semi-structured interviews with five organizations building and deploying ACS in India and Africa. Our survey shows that there are no publicly available, large-scale, real-world
benchmarks for ACS in India, with existing
datasets being overwhelmingly synthetic. We note that the available Global North
datasets diverge significantly from the expected conversational and cultural structures of Indian encounters. Finally, our interviews reveal that deploying organizations have each built proprietary, incomparable evaluation pipelines, creating a fragmented ecosystem with no independent and reliable basis for procurement. We call for the development of a publicly shared, real-world, multilingual
benchmark for ACS evaluation and outline the properties and policies such a
benchmark would require.
Read the full abstract ▾
All authors Siddharth D Jaiswal, Krithi S, Ashish Makani, Suvrankar Datta, Sunayana Sitaram, Mohit Jain
Human-Computer Interaction
LumiNote: LLM-Assisted Multimodal Instruction for VR Stage Lighting Education
Danxuan Liang, Chun Yin Li, Zheng Wei et al. (+4 more)
Stage lighting education requires instructors to bridge abstract concepts, technical operations, and learner-understandable representations. While Virtual Reality (VR) removes physical constraints, existing systems provide limited support for live instruction. We present LumiNote, an LLM-assisted VR system that transforms spoken pedagogical intent into instructor-reviewable spatial annotations, executable demonstrations, and linguistic support. In an exploratory study with 3 instructors and 24 students, we examined how instructors incorporated LumiNote into familiar lighting topics and how students received the resulting representations. We found LLM assistance most valuable for expressive, under-specified goals, but requiring greater expert intervention for fixture-specific or spatial configuration requests. Instructors engaged with generated suggestions as a controllable refinement process, shifting effort from manual setup toward pedagogical expression. However, representations that externalized expert
reasoning did not always align with novice comprehension. These findings characterize LLM-assisted VR instruction as a domain-grounded mediation process among expert expression, executable operations, and learner-facing representations.
Read the full abstract ▾
All authors Danxuan Liang, Chun Yin Li, Zheng Wei, Xian Xu, Meng Xia, Huamin Qu, Wai Tong
Human-Computer Interaction
When AI Becomes Hard to Understand: Cognitive Demands in Real-World Human-AI Conversations
Yingcan Carol Wang, Iman Munire Bilal and Qamar Zaman
Generative AI increasingly supports complex financial and health decisions, yet we know little about when its responses become difficult to process in real-world dialogue. We analyse more than 84,000 ChatGPT and Gemini conversations, using repeated prompting and clarification following misunderstanding as behavioural indicators of cognitive difficulty. We find that response characteristics such as length, readability and lexical diversity do not have fixed relationships with conversational difficulty; instead, their relationships depend on how they combine. Most notably, greater lexical diversity was associated with less repeated prompting in shorter responses, but this association weakened as response length increased, a pattern that replicated across financial and health conversations. We propose a conversational complexity budget to conceptualise these interdependencies: the demands associated with one response characteristic may depend on those accompanying it. The resulting design challenge is how to configure response complexity for the particular user, task and interaction.
Read the full abstract ▾
Human-Computer Interaction
"Piecing Data Connections Together Like a Puzzle": Effects of Increasing Task Complexity on the Effectiveness of Data Storytelling Enhanced Visualisations
Mikaela Elizabeth Milesi, Paola Mejia-Domenzain, Laura Brandl et al. (+6 more)
The emerging concept of data storytelling (DS) suggests that enhancing visualisations with annotations and narratives can make complex data more insightful than conventional visualisations. Previous works found that DS-enhanced visualisations are more effective than conventional visualisations for simple tasks like identifying key data points or the main message. However, no previous work has explored the extent to which DS enhancements influence task completion across different levels of cognitive complexity. We address this gap by presenting the results of a study where 128 participants completed tasks based on four visualisations (two line charts and two choropleth maps, either with or without DS elements) spanning a range of complexity based on Bloom's taxonomy, which has been applied in data visualisation to categorise tasks hierarchically from lower to higher-order thinking. Results suggest that while DS-enhanced visualisations effectively support lower-order tasks (finding data points and understanding insights), they don't necessarily aid the correct completion of higher-order tasks (application, analysis, evaluation and creation). However, DS enhancements improve how efficiently participants complete complex tasks.
Read the full abstract ▾
All authors Mikaela Elizabeth Milesi, Paola Mejia-Domenzain, Laura Brandl, Vanessa Echeverría, Yueqiao Jin, Dragan Gašević, Yi-Shan Tsai, Tanja Käser, Roberto Martínez-Maldonado
Human-Computer Interaction
[MM/AI] Mental Models in Human-AI Interaction: Methods and Challenges in the Generative and Agentic AI Era (Workshop)
Téo Sanchez, Bhada Yun, Prerna Ravi et al. (+6 more)
The mental model construct is widely used in HCI to refer to the knowledge structure people hold in order to reason about and interact with computing systems. Yet it is often operationalized intuitively: the construct is often used interchangeably with related concepts (e.g., folk theories, sensemaking) and methods of studying it (e.g., through elicitation) are many and diverse, with each method resting on distinct assumptions about what counts as a mental model. Generative and agentic AI systems may further complicate mental model formation and elicitation as such systems are opaque by design and increasingly act on users' behalf across files, applications, and on the web. Together, these challenges may hinder the commensurability of research on people's mental models of AI systems. The MM/AI workshop calls for a critical reassessment of how we understand and study mental models in human-AI interaction research. It aims to foster theoretical and methodological exchange on mental models in human-AI interaction, identify open challenges, and develop directions for future research. We invite short papers on users' or stakeholders' mental models of AI systems, particularly contributions that reflect on the conceptual and methodological foundations of the construct. The half-day workshop combines lightning talks, hands-on elicitation exercises, and structured discussions on key questions concerning the future of the mental model for human-AI interaction research.
Read the full abstract ▾
All authors Téo Sanchez, Bhada Yun, Prerna Ravi, Laura Schütz, Anna Neumann, Robin Shing Moon Chan, April Yi Wang, Qiaosi Wang, Sumit Asthana
Human-Computer Interaction
A Scenario-Knowledge-Driven Pipeline for Just-in-Time Assistance
Zhiyuan Li, Tatsunori Hara and Jun Ota
Detecting a silently struggling kiosk user is only the first step; deciding whether, when, and how to help depends on scenario knowledge usually buried in model weights and thresholds. We propose a scenario-knowledge-driven pipeline: a single scenario knowledge document, human-authored and version-controlled, configures sensing, constrains LLM
reasoning, and shapes a graded intervention proposal. Narration, assistance-need assessment, and proposal are kept separate for independent audit. As proof of concept, we replay two recorded kiosk sessions offline, chosen before the runs for their struggle evidence and retrospective detail. Both cases support what the design promises: checkable reporting and measured escalation. Across 95 updates, every sentence of the append-only narration cites the primitive events underlying it, and the rule layer detects 12 of 13 and 7 of 7 annotated struggle episodes under a strict criterion. The assessor de-escalates on recovery and reaches the top rung exactly once, under maximally converging evidence. At the decisive help-seeking turn, narration, assessment, and the participants' retrospective accounts converge. The appropriateness of these interventions, the pipeline's restraint on sessions without struggle, and the document's transfer to a new scenario frame the agenda.
Read the full abstract ▾
Human-Computer Interaction
Enhancing Procedural Writing Through Personalized Example Retrieval: A Case Study on Cooking Recipes
Paola Mejia-Domenzain, Jibril Frej, Seyed Parsa Neshaei et al. (+5 more)
Writing high-quality procedural texts is a challenging task for many learners. While example-based learning has shown promise as a feedback approach, a limitation arises when all learners receive the same content without considering their individual input or prior knowledge. Consequently, some learners struggle to grasp or relate to the feedback, finding it redundant and unhelpful. To address this issue, we present RELEX, an adaptive learning system designed to enhance procedural writing through personalized example-based learning. The core of our system is a multi-step example
retrieval pipeline that selects a higher quality and contextually relevant example for each learner based on their unique input. We instantiate our system in the domain of cooking recipes. Specifically, we leverage a fine-tuned Large
Language Model to predict the quality score of the learner's cooking recipe. Using this score, we retrieve recipes with higher quality from a vast database of over 180,000 recipes. Next, we apply BM25 to select the semantically most similar recipe in real-time. Finally, we use domain knowledge and regular expressions to enrich the selected example recipe with personalized instructional explanations. We evaluate RELEX in a 2 x 2 controlled study (personalized vs. non-personalized examples, reflective
prompts vs. none) with 200 participants. Our results show that providing tailored examples contributes to better writing performance and user experience.
Read the full abstract ▾
All authors Paola Mejia-Domenzain, Jibril Frej, Seyed Parsa Neshaei, Luca Mouchel, Tanya Nazaretsky, Thiemo Wambsganß, Antoine Bosselut, Tanja Käser
Human-Computer Interaction
Beyond "ChatGPT Can Make Mistakes": Designing Interventions to Support Metacognitive Monitoring in AI-Assisted Work
Manuel A. D. Santos, Paul Thiesse, Steeven Villa et al. (+4 more)
AI assistance places a metacognitive demand on users, who must judge their own competence and the system's. Yet designers lack comparative evidence on which interventions to choose, where to place them, and how to tell whether they worked. We elicited 30 interventions from 11 experts and, with prior work, organized them into a design space of time (when an intervention acts), level (whose competence is judged), and source (who supplies the monitoring cue). A between-subjects experiment (N = 917; 12 planning-and-organizing problems) compared a per-task reliability card, contrasting replies, pause points, and post-problem reflection against a baseline LLM assistant. Reliability cards and contrasting replies reduced estimation error and overconfidence and increased aggregate confidence discrimination. No task-performance improvement or average within-item discrimination gain was established. We contribute a shared vocabulary, a design space, and evidence that measured monitoring and task performance are separable design targets.
Read the full abstract ▾
All authors Manuel A. D. Santos, Paul Thiesse, Steeven Villa, Daniela Fernandes, Albrecht Schmidt, Verena Distler, Robin Welsch
Human-Computer Interaction · Cross-listed from cs.AI
Neuro-Symbolic Hierarchical Intention Anticipation in Human Behavior
Farnaz Soleimani, Abdelghani Chibani, Yacine Amirat et al. (+1 more)
Assistive autonomous systems must anticipate human goals before an observed behavior is complete. This article formulates anticipation as goal inference from a partially observed
multimodal episode together with structured prediction of the remaining behavior, rather than exact motor forecasting. A compact Hierarchical Planning Decoder (HPD) is attached to a frozen neuro-symbolic recognition encoder and predicts, at four ontological levels, the next actions, the remaining activities and low-level intentions, and the episode high-level intention(HLI). The decoder is trained with soft neuro-symbolic regularization combining transition-coherence and hierarchical continuity losses, and is decoded with hard reachability masks that enforce ontological validity at inference. On a compositional four-level
benchmark of 15,002
multimodal episodes built over NTU RGB+D 120 features, three headline properties are observed together. The advantage over the strongest sequential baseline grows with the anticipation horizon, from +1.7 points at step 1 to +7.3 points at step 3 (top-5). Under compositional generalization, where one parent association per multi-parent low level intention is held out, this advantage widens to +4.9 points at step 1. At the episode level, 96.8% of anticipated trajectories satisfy the joint logic constraints, above the 88.1% strongest-baseline value and the 73.9% ground-truth floor; soft logic terms alone account for a 59.8 to 71.1% relative reduction of HLI-reachability violations, and the hard masks then eliminate them entirely. Neural generation supplies predictive ranking, symbolic constraints supply onto logical validity, and their combination yields coherent hierarchical anticipation while exposing remaining challenges in compositional goal generalization and unordered set prediction.
Read the full abstract ▾
All authors Farnaz Soleimani, Abdelghani Chibani, Yacine Amirat, Ghazaleh Khodabandelou
Human-Computer Interaction · Cross-listed from cs.NI
Beyond Measurement Metrics: A Human-Centered Framework for Semantic Validation of Network Traffic Classification
Igor Cherepanov, David Sessler, Alex Ulmer et al. (+2 more)
Machine learning (ML) has become the dominant approach for network traffic classification, achieving very high predictive performance. However, a model is only valuable if it learns semantically meaningful and trustworthy patterns rather than exploiting spurious correlations. Conventional evaluation practices predominantly assess predictive performance. Consequently, whether the model relies on semantically meaningful patterns remains unknown. To address these challenges, we adapt the knowledge generation framework for network traffic classification. The adapted framework combines data, ML models, explainability, visualization, and expert
reasoning to support the iterative exploration, verification, and refinement of model behavior and data preprocessing. The framework is grounded in findings from the literature,
benchmark dataset analyses, practical experience with XAI-based traffic classification, and expert feedback, providing practical guidance for semantic model validation. By complementing predictive performance with semantic validation and human expertise, the proposed framework supports the development of network traffic classification models that are not only accurate but also robust and trustworthy.
Read the full abstract ▾
All authors Igor Cherepanov, David Sessler, Alex Ulmer, Thorsten May, Jörn Kohlhammer
Human-Computer Interaction
Disrupted Companionship: A Risk Assessment Framework and Cross-Platform Quantitative Analysis of Psychosocial Responses to AI Companion Disruptions
Chau Do, Yunhao Yuan, Koustuv Saha et al. (+2 more)
AI companions can provide meaningful relationships, yet these relationships remain vulnerable to platform-initiated changes. We study AI companion disruptions: platform changes that alter or terminate users' ongoing companionship with an AI. We compile 30 disruption events across major platforms, develop a taxonomy of six disruption types, identify three broad reasons for disruption, and propose a risk-assessment framework comprising four dimensions: relational discontinuity, population vulnerability, communication deficit, and transition-support deficit. Using longitudinal Reddit data, we estimate community-level psychosocial responses with a hierarchical
Bayesian interrupted time-series model incorporating predictive controls. Across events, disruption onset was associated with immediate increases in anxiety, stress, suicidal expression, and grief activation, with relational discontinuity and transition-support deficit being associated with more adverse immediate responses across several outcomes. Our findings provide a cross-platform characterization of AI companion disruptions, quantitative evidence of their psychosocial impacts, and a prospective framework for assessing their potential risks before implementation.
Read the full abstract ▾
All authors Chau Do, Yunhao Yuan, Koustuv Saha, Renwen Zhang, Talayeh Aledavood
Human-Computer Interaction
The Evolution of Coordination in a Collective Intelligence System: 25 Years of English Wikipedia and the Emergence of Generative AI
Neal Reeves, Maja Świeczkowska, Amy Rechkemmer et al. (+1 more)
English Wikipedia is one of the largest examples of collective intelligence on the Web, sustained not only by article production but also by volunteer coordination and governance. While prior research has examined coordination work in Wikipedia, less
attention has been paid to how participation in these spaces has evolved over time. Drawing on a longitudinal analysis spanning nearly 25 years of English Wikipedia, we examine editing patterns across five namespaces covering content, discussion, and governance. We find that participation in coordination spaces has declined relative to content production, particularly in governance areas, with a shrinking core of editors performing an increasing share of this work. Using Markov-based session metrics, we also find that editing has become more specialised, with editors moving less frequently between namespaces. Motivated by recent governance debates around generative AI, we conclude by investigating whether the availability of LLMs has altered these long-term trends. While short-term changes are visible, we find little evidence that generative AI fundamentally changed existing trajectories of coordination and participation.
Read the full abstract ▾
All authors Neal Reeves, Maja Świeczkowska, Amy Rechkemmer, Elena Simperl
Human-Computer Interaction
Available but Unclaimed: An Empirical Study of Human-AI Synergy
Robin Welsch, Michelle Rausch, Pascal Knierim et al. (+4 more)
People increasingly reason with large
language models (LLMs), yet complementary capabilities do not guarantee outperforming both components. In a between-subjects study, participants (N=535) solved a 40-item battery of
matrix reasoning, mental rotation, syllogisms, and letter-string analogies, unaided or with GPT-5.6-Luna, Claude Opus 4.8, Gemini 3.6 Flash, or Kimi K3. Each assisted trial required consultation with the model. Each model answered every item alone 100 times under matched elicitation. The assisted-unaided accuracy difference increased with item-level LLM competence. Deference varied across tasks and increased with competence within tasks. Post-advice confidence distinguished correct from incorrect answers less strongly than unaided confidence. In a reference comparison, about half the increase in LLM accuracy carried through to assisted accuracy. How much of that accuracy gain reached participants differed across the models. These findings motivate evaluating LLMs in interaction with humans and designing support for selective deference that preserves independent
reasoning.
Read the full abstract ▾
All authors Robin Welsch, Michelle Rausch, Pascal Knierim, Thomas Kosch, Jochen Kuhn, Albrecht Schmidt, Daniela Fernandes
Human-Computer Interaction
EgoAsk: Egocentric Teaching of Personalized Object Knowledge for Household Robots
Yuanda Hu, Wenbin Zuo, Yiting Shen et al. (+5 more)
Unlike users, who know their own belongings and routines, household
robots cannot easily acquire such personalized object knowledge automatically and depend on users to teach them. User-initiated teaching requires users to arrange dedicated teaching sessions and decide what to teach, even when they are unsure what the
robot needs to learn. We introduce EgoAsk, a smart-glasses-based system that proactively embeds personalized object teaching into everyday activities. EgoAsk shares the user's first-person view with the
robot, identifies gaps in personalized object knowledge, and analyzes ongoing activity to ask context-relevant questions that support future household assistance. To examine how teaching initiative and question timing affect users' teaching experiences, we conducted a within-subjects study with 18 participants and found lower reported knowledge-gap monitoring burden with
robot-initiated questioning and less need for context reconstruction with EgoAsk. These findings characterize teaching burdens and timing preferences, offering design implications for egocentric
robot-teaching systems.
Read the full abstract ▾
All authors Yuanda Hu, Wenbin Zuo, Yiting Shen, Tianle Chen, Hector Fabio Calero Tobar, Yate Ge, Xiaohua Sun, Weiwei Guo
Human-Computer Interaction
NephoCodex: Exploring Bounded Material Agency in Weather Data Physicalization
Yuxuan Weng and Yunge Wen
Weather is a complex, continuously changing system in which uncertainty is intrinsic. Physicalizing this uncertainty introduces further variation because computational outputs cannot fully determine material behavior. We distinguish computational uncertainty from material variability and introduce bounded material agency: computation constrains material realization without fixing its exact appearance. We present NephoCodex, a data physicalization system informed by a formative study that constructs five artistic weather states and predicts probability distributions over them. Probability-weighted mappings translate these distributions into material control proposals, while
entropy-based regulation, local sensing, and safety constraints bound their execution through mist, airflow, light, and transparent displays. A within-participant study found increased spatial presence and physical demand, while perceived data comprehensibility remained inconclusive after correction. These findings contribute to hybrid data physicalization by showing how variable material expression can be paired with stable digital annotations and how embodied experience can be evaluated separately from data comprehension.
Read the full abstract ▾
Human-Computer Interaction
Beyond Benefit or Risk: Perceived Impact Profiles of Human-AI Affective Interaction and Their Associations with Psychological Functioning
Lu Chen, Fenghua Tang, Jiayu Zhao et al. (+5 more)
Relational AI increasingly serves as an emotional shelter for humans, and its impact is mixed. Prior research has focused on either positive or negative impacts, leaving unclear how they are configured within individuals and relate to psychological functioning. To address these gaps, this study used a sequential mixed-methods design. Study 1 interviewed 52 users with emotional ties to AI and identified four positive impact domains (emotional relief, loneliness alleviation, enhanced interpersonal functioning, and personal growth) and four negative impact domains (virtual-real boundary blur, social replacement, cognitive-emotional reinforcement, and excessive use). Study 2 followed 673 Chinese AI users for six months and identified four profiles of individuals differently impacted by relational AI use: minimal impact, benefit-driven impact, mixed impact, and risk-driven impact. Users in the mixed impact and risk-driven impact profiles were both high in human-AI affective bonding, but those showing risk-driven impact had greater vulnerability, indicated by higher interpersonal need frustration and emotion-regulation difficulties, more depressive and anxiety symptoms, and lower self-esteem and flourishing. Users in the benefit-driven and mixed impact profiles showed more favorable psychological functioning. After controlling for baseline functioning and relevant
covariates, Wave 1 profiles did not predict five of the six Wave 2 indicators; only users in the mixed impact profile reported higher flourishing than those in the minimal impact profile. Overall, potential psychological harms associated with relational AI engagement appeared limited and selective. These findings portray relational AI as a heterogeneous socio-emotional context that may partly mirror users' states and traits, warranting individualized, adaptive safeguards.
Read the full abstract ▾
All authors Lu Chen, Fenghua Tang, Jiayu Zhao, Xuanying Li, Yanli Wang, Weijia Fang, Mengyu Miranda Gao, Zhuo Rachel Han
Human-Computer Interaction
CATVis: A Collaborative Multi-Agent Workflow for Turbomachinery Simulation Data Visualization
Zhe Wang, Zehao Lou, Guanghui Zhao et al. (+6 more)
Recent advances in AI for Science have enabled natural
language (NL) interfaces for scientific data analysis. In turbomachinery CFD post-processing, translating ambiguous high-level analytical goals (e.g.,
vortex identification) into precise visualization procedures supporting complex domain-specific analysis is challenging. We present CATVis, a Collaborative multi-
agent workflow system that bridges this gap by transforming NL intents into structured middle representation for visualization. Our approach reformulates domain-specific visualization procedures as composable workflow representations, and use multi
agent to generate workflow representations via intent planning, template generation, and error-aware refinement, where each stage incrementally updates a shared structured representation. We evaluate the impact of external knowledge and workflow structuring on generation accuracy, demonstrating that the proposed approach significantly improves complex workflow generation correctness while reducing
prompt complexity.
Read the full abstract ▾
All authors Zhe Wang, Zehao Lou, Guanghui Zhao, Yu Dong, Guan Li, Pengyi Xu, Gaorong Liang, Jun Liu, Guihua Shan
Human-Computer Interaction
EmoPhone: A Multi-Wave Dataset for In-the-Wild Mobile and Wearable Affect Sensing
Panyu Zhang, Minseo Park, Soowon Kang et al. (+7 more)
We introduce a three-wave, in-the-wild
multimodal dataset for affect sensing that integrates smartphone sensing, wearable sensing, and dense experience-sampling-method (ESM) labels collected annually from 2020 to 2022. The
dataset supports moment-level affect modeling through a shared dimensional label core across all waves, with additional affective descriptors available in the third wave (D-3). We describe the resource in terms of study design, temporal density of in-situ labels, and sensing and label coverage across waves. To support evaluation within this resource, we define an initial three-setting
benchmark spanning temporal prediction from within-user history, within-wave cross-user generalization, and cross-wave generalization in which each wave is treated as a separate
dataset. Our
benchmark results show that the strongest method family depends on the evaluation setting: supervised baselines perform best in the temporal setting, unsupervised domain adaptation is strongest overall in the within-wave cross-user setting, and domain generalization shows the strongest overall cross-wave performance, although its margin over strong baselines is modest. These findings indicate that robust mobile affective computing is constrained not only by label availability but also by substantial participant-level variability and realistic cross-wave differences inherent in longitudinal in-situ deployments.
Read the full abstract ▾
All authors Panyu Zhang, Minseo Park, Soowon Kang, Tomiris Ismatzoda, Azizbek Mustafakulov, Otabek Najimov, Woohyeok Choi, Jumabek Alikhanov, Surjya Ghosh, Uichin Lee
Human-Computer Interaction
Ptolemy: A Semantic Map of Exploratory Data Analysis
Dylan Wootton, Denny Bromley and Vidya Setlur
A central challenge in exploratory data analysis (EDA) is keeping track of what has already been examined in order to decide what to analyze next. In practice, analysts often run dozens of analyses while building an understanding of a
dataset. However, most tools provide little support for maintaining an overview of this evolving process, instead exposing only a linear history of analysis steps. These tools show sequence, what came before, but not position, how a current analysis relates to the broader space of possible analyses. As a result, analysts must mentally reconstruct which parts of the space they have explored and where gaps remain, increasing the risk of redundant work or overlooked patterns. We present Ptolemy, a navigational interface that externalizes analysis history as a semantic map. Each analytic step is represented as a point positioned by
embeddings derived from a structured description of its effective data view (e.g., columns, filters, transformations), allowing spatial distance to reflect analytic similarity. In a mixed-methods study comparing map, canvas, and tree representations, we find that maps improve global orientation and local comparison, while ordered layouts reduce decision cost. These findings surface a trade-off between orientation and actionability, and highlight design principles for supporting strategic exploration in EDA.
Read the full abstract ▾
Human-Computer Interaction
Beyond Gestures: Estimating Full Hand Pose and Contact Forces from Wrist-Worn Pressure Sensor Array
Svetoslav Kolev, Lingni Ma, Michael Goesele et al. (+3 more)
Capturing hand motion and interaction forces is critical for interactive computing, VR, and high-fidelity tactile demonstrations for
robot learning. We introduce a wrist-worn pressure-sensing wristband that recovers continuous full-hand pose and distributed contact force on a single wearable. The system consists of flexible capacitive sensor arrays around the wrist, which require no electrical skin contact, and a recurrent network that maps the resulting pressure signal to hand state. Our key insight is that muscle contraction and tendon displacement produce pressure patterns, which correlate strongly with hand pose and interaction force. To validate this, we collect synchronized recordings of wrist pressure, optical motion-capture hand pose, and tactile-glove interaction force, covering isolated finger motion, fingertip-force stress tests, and natural hand-object manipulation. On isolated single-user motion the wristband attains $4.6^\circ$ mean finger-joint MAE, and across four users manipulating everyday objects it estimates per-finger contact force at $R^2=0.57$, which an external pose signal brings up to $0.75$. We see the wristband as one node in a constellation of everyday wearables – e.g. paired with an egocentric camera – adding the contact force that
vision cannot observe and taking over when the hand is occluded.
Read the full abstract ▾
All authors Svetoslav Kolev, Lingni Ma, Michael Goesele, Renzo De Nardi, Jakob Engel, Richard Newcombe
Human-Computer Interaction
"ChatGPT, what am I missing?": Designing AI Workflows around Professional Task Structure to Shape Analytic AI Use
Zilin Ma, Suzi Jazmati, Marco Chimenton et al. (+4 more)
General-purpose AI lets users choose what support to request, but leaves them to structure the support a professional task requires. We examine how interactive workflows can embed professional task structure without prescribing how users engage with AI. We designed two scaffolded interfaces around the same negotiation scaffold: one presented a completed AI analysis, while the other supported user-directed, incremental development. A four-condition randomized experiment with 800 participants compared these interfaces with no-AI and an AI chat interface. AI-supported conditions improved preparation coverage over unaided work; the scaffolded workflows further improved coverage over chat. Although the scaffolded workflows produced similar coverage, the user-directed workflow elicited a broader repertoire of analytic requests and lower subjective effort. Professional scaffolding therefore depends not only on displayed structure but on how workflows organize users' engagement with it. Effective professional AI must structure how users and AI build analysis together.
Read the full abstract ▾
All authors Zilin Ma, Suzi Jazmati, Marco Chimenton, Yiyang Mei, Jacqueline Lane, Krzysztof Z. Gajos, Finale Doshi-Velez
Human-Computer Interaction · Cross-listed from cs.CV
A multimodal large language model for evidence-based autism spectrum disorder screening
Jun Chen, Qi Zhao, Yunliang Jiang et al. (+10 more)
The clinical management of autism spectrum disorder (ASD) faces a bottleneck in early screening, mainly because trained specialists are scarce and conventional assessment tools are subjective. Here, we introduce ASDchat, a
multimodal large
language model designed for evidence-based ASD screening, which takes
video, audio, and dialogue as input. ASDchat adopts a dual-branch architecture, where the decision branch generates screening probabilities and the evidence branch generates traceable, timestamped behavioral evidence aligned with the Autism Diagnostic Observation Schedule (ADOS-2). The model was trained and evaluated on a
dataset of 1,035 participants from 27 sites in China, which covered typically developing (TD) children, children with ASD, and children with other developmental disorders. For ASD versus TD, ASDchat reached an area under the receiver operating characteristic curve (AUC) of 0.953 $\pm$ 0.021. On 6 held-out sites that were not used for training, the mean AUC was 0.961. Furthermore, unsupervised clustering of the behavioral dimensions split the ASD cases into six subtypes with different phenotypic profiles, and ASDchat suggests an intervention for each subtype. ASDchat provides a feasible path for large-scale, evidence-based ASD screening in clinical practice.
Read the full abstract ▾
All authors Jun Chen, Qi Zhao, Yunliang Jiang, Shuqin Cao, Yunqiang Lin, Chenglong Jia, Qiang Guo, Guang Dai, Xiongtao Zhang, Mengmeng Wang, Xiaoyue Ma, Hui Wang, Wenguan Wang