Human-Computer Interaction · Cross-listed from cs.CL
Quantifying Hidden Salt for Precision Healthcare: Sodium Assessment via Joint-Factor Retrieval and Chain-of-Thought Inference
Mingyu Huang, Weiqing Min, Yuehui Fang et al. (+2 more)
Precision healthcare, particularly for conditions like hypertension and cardiovascular disease, necessitates monitoring of dietary sodium intake. However, tracking this is hindered by the prevalence of hidden salt in cooking, such as sodium in soy sauce and ketchup. While recipes offer a valuable data source for dietary analysis, sodium-rich seasonings are frequently omitted or described ambiguously in instructions. To solve this issue, we propose SALT, a Sodium Assessing & Level Tracking framework adopting an RAG framework to assess sodium content in recipes. Our framework first introduces a Joint-Factor
Embedding Retrieval module to locate similar recipes with specified sodium content for addressing the lack of contextual references. These retrieved samples provide contexts for subsequent inference. Then we design a structured 4-hop Chain-of-Thought inference module to refine the vague estimation from
language models through a multi-step sodium estimation. To facilitate our study, we further construct a recipe
dataset SALT54k with $54,151$ entries labeled with sodium quantities across $11$ common seasonings. Results on SALT54k demonstrate that our method achieves state-of-the-art performance in sodium estimation. Additional real-world validations confirm the effectiveness of our method, demonstrating its potential as a practical solution for AI-assisted precision healthcare.
Read the full abstract ▾
All authors Mingyu Huang, Weiqing Min, Yuehui Fang, Yuna He, Shuqiang Jiang
Human-Computer Interaction · Cross-listed from cs.PL
Direct Manipulation and Natural Language Programming, Together at Last?
Parker Ziegler, David Minh-Duy Cao, Justin Lubin et al. (+1 more)
Decades of programming
languages research has contributed novel approaches to program editing that go beyond modifying text, including direct manipulation programming, structure editing, and automated refactoring tools. However, the rapid growth of natural
language programming largely reinforces a view of programs as text and program editing as (unstructured) text transformation. How can we develop unified programming systems that bridge the gap between these approaches, supporting multiple editing paradigms in concert? We take a first step toward answering these questions by introducing a framework that enables program editing via both direct manipulation and natural
language, and instantiate this framework in a variant of the $\texttt{cartokit}$ direct manipulation programming system. Our key insight is to treat programs as sequences of structured edits and to use an edit
language as a shared interface for both direct manipulation and natural
language interactions, leveraging constrained decoding to support the latter. Using our instantiation, we conducted a within-subjects study ($N$=18) to understand how the combination of direct manipulation and natural
language as editing modalities changes the programming process compared to each modality alone. Perhaps surprisingly, we found that study participants overwhelmingly chose to edit via direct manipulation when both modalities were available, performing just 6.14% of edits via natural
language. Our thematic analysis of study sessions revealed that direct manipulation aided task decomposition, encouraged incremental editing, and helped mitigate known challenges in natural
language programming related to understanding model capabilities and model-generated code. Our edit-based framework and study findings lay out a possible pathway for future research on programming systems that blend natural
language with alternative editing modalities.
Read the full abstract ▾
All authors Parker Ziegler, David Minh-Duy Cao, Justin Lubin, Sarah E. Chasins
Human-Computer Interaction
Decolonial Discourse in Postcolonial Contexts: How YouTubers Negotiate Audience Tensions, Platform Governance, and State Influence
Dipto Das and Bryan Semaan
Decolonial discourse on online platforms is often framed in terms of creator motivations and expressive possibilities. In this paper, we examine what it takes to sustain such discourse under layered sociotechnical constraints. Drawing on semi-structured interviews with YouTubers engaging in Bengali decolonial discourse, we analyze how audience publics, platform governance, and state influences shape what becomes sayable, visible, and viable. We show how fragmented postcolonial identities among audiences produce legitimacy policing, harassment, and coordinated backlash, requiring ongoing relational labor from creators. At the platform level, differential monetization, opaque moderation, and copyright regimes reorganize which publics are economically viable and reinforce existing hierarchies. Further, intermediaries such as multi-channel networks mediate regulatory pressure, introducing political risks and constraints on participation. In response, content creators engage in strategies of negotiation, including boundary work, infrastructural improvisation, and multi-platform distribution. Overall, our findings highlight the layered dynamics of decolonial discourse in postcolonial contexts and the continuous work required to sustain it in platformed environments.
Read the full abstract ▾
Human-Computer Interaction
Kale: A Transformation-Safe Spreadsheet System
Michael Coblenz, Jacob Yim, Ajinkya Bokade et al. (+11 more)
Spreadsheet formulas can refer to rectangular ranges of arbitrary size. When a user changes the structure of a referenced table, the spreadsheet system updates the references to refer to a new range. Unfortunately, this new range may differ from the user's expectations, introducing bugs in spreadsheets. We describe a user study showing that standard reference semantics are error-prone, resulting in significant risk to users. We introduce Kale, a prototype system that eliminates the risk of inserting these kinds of bugs by restricting the kinds of references that can be expressed. We show that Kale can be used effectively by users to complete tasks that are error-prone in traditional spreadsheet systems. Finally, we describe a corpus study that evaluates the extent to which the reference restrictions in Kale might have implications on users.
Read the full abstract ▾
All authors Michael Coblenz, Jacob Yim, Ajinkya Bokade, Mounika Padala, Julia Epshtein, Priyanka Bhatia, Piyush Chauhan, Simran Gill, Aniket Gupta, Grishma Gurbani, Vaibhav Khetan, Arushi Munjal, Jeffery Tung, Joanna Yang
Human-Computer Interaction
Calibration-Free Cuffless Blood Pressure Estimation Using Multimodal ECG-PPG Fusion on a Google Pixel Watch
Jathushan Kaetheeswaran, Boyi Ma, Ali Abedi et al. (+2 more)
Inadequate blood pressure (BP) monitoring and management outside of clinical settings can worsen major cardiovascular risk factors such as hypertension. While cuff-based devices are commonly used for at-home monitoring, these devices can be inconvenient for daily use due to their sensitivity to body positions, upper-arm constrictions, and limited portability. A promising alternative is emerging in the form of consumer-grade smartwatches, where physiological signals related to cardiac activity can be used to estimate BP non-invasively and continuously across daily living conditions. In this work, we use data collected from a Google Pixel Watch in 40 participants to develop and compare several
algorithm approaches for BP estimation. We found that our proposed deep learning model achieved the strongest overall performance, and that fusing smartwatch signals with demographic information improved model generalizability to unseen individuals. However, we also identified that model accuracy was not consistent across participant subgroups, with obese individuals yielding higher estimation errors than others. This study highlights the feasibility of consumer-grade smartwatches as accessible platforms for deploying robust BP estimation
algorithms, though clinical reliability will require larger, more diverse populations and additional sensing modalities.
Read the full abstract ▾
All authors Jathushan Kaetheeswaran, Boyi Ma, Ali Abedi, Shehroz S. Khan, Milad Lankarany
Human-Computer Interaction · Cross-listed from cs.SE
"A Second Set of Eyes": The Process and Challenges of Software Documentation Review
Avinash Bhat, Ian Arawjo, Disha Shrivastava et al. (+1 more)
Organizations assign documentation work to technical writers, yet the knowledge required to produce it is distributed across developers, managers, and other practitioners. Prior work has established quality criteria for judging "good" documentation, but it has not examined how practitioners bring that expertise to improve documentation quality or the challenges they face in doing so. Through semi-structured interviews with experienced technical writers ($n=31$) from different organizations, our work reveals the individual and collaborative effort required to maintain documentation quality. We identify five distinct stages of the documentation review process: self review, technical review, editorial review, play testing, and post-publication feedback. Each stage draws on practitioners with distinct expertise to address quality across content, presentation, and user experience. Our findings surface organizational and technical challenges writers face in recruiting expert reviewers, navigating development timelines, and contending with tools not specifically designed for documentation workflows. Our work positions documentation review as a crucial yet understudied site of collaborative work and opens new research and design directions for process improvement and tool support.
Read the full abstract ▾
All authors Avinash Bhat, Ian Arawjo, Disha Shrivastava, Jin L.C. Guo
Human-Computer Interaction · Cross-listed from cs.CV
MyoMechanix: Biomechanically-Grounded Compositional Skilled Activity Understanding and Coaching
Hao Yin, Paritosh Parmar, Lijun Gu et al. (+6 more)
Existing action quality assessment (AQA)
datasets and methods rely primarily on visual inputs such as RGB and pose, overlooking physiological dynamics such as muscle mechanics and often modeling actions as monolithic patterns. These limitations hinder fine-grained, biomechanically grounded feedback. We introduce MyoMechanix, a
multimodal ecosystem for weight-loaded actions that aligns motion with muscle activity. Expert-annotated, it contains 7,500+ samples of 20 actions from 38 subjects, with synchronized multiview RGB
video, 3D pose, sEMG, and additional physiological signals, forming the largest
multimodal AQA
benchmark to date. We further construct the Fitness Knowledge Graph (FKG), which organizes expert annotations into structured relationships among actions, phases, key steps, errors, and corrective feedback, enabling compositional scoring and interpretable assessment. Building on these representations, we develop CUBIST (Compositional Ontological
Reasoning Engine), which performs decomposition-analysis-recomposition for fine-grained error attribution and feedback generation. We also establish MyoMechanix-AQA, MyoMechanix-VideoQA, and a novel MyoMechanix-
Video2EMG task. Experiments show that
multimodal sensing and structured representations improve performance, interpretability, and error attribution, with CUBIST achieving state-of-the-art results; VideoQA enhances
language-grounded action understanding; and
Video2EMG suggests
video-based alternatives to costly EMG sensing. MyoMechanix advances skilled activity understanding toward biomechanically grounded,
multimodal, and compositional
reasoning for Physical AI applications in fitness, rehabilitation, healthcare, and machine learning. Project page: https://haoyin116.github.io/MyoMechanix/
Read the full abstract ▾
All authors Hao Yin, Paritosh Parmar, Lijun Gu, Lin Xu, Tianxiao Guo, Xiujin Liu, Tianyou Zheng, Yang Zhang, Weiwei Fu
Human-Computer Interaction
From Producing to Validating: How AI Is Deskilling Freelancers
Nakul Rajpal
Generative AI is promoted as a way to enhance knowledge work, yet its benefits and drawbacks fall unevenly across the workforce. Freelance and gig workers, who commonly lack the upskilling pathways available to traditional employees, face heightened risks to both skill development and job security as AI adoption advances. We review empirical evidence on AI's impact on knowledge-worker workflows and upskilling, then predict the primary and downstream effects of AI adoption among clients and workers in the freelance economy. We anchor this in two cases of the same shift, machine-translation post-editing and software development. We argue that freelancers are the leading edge of a change that also reaches salaried HCI practitioners, and we close with questions for the platforms and clients that mediate this work, and for HCI researchers.
Read the full abstract ▾
Human-Computer Interaction
Gaming Together on Discord: Teen Gamer's Cross-Platform Practices
Elena Koung, Xinning Gui and Yubo Kou
Discord is one of the most popular communication platforms among gamers. While prior research has highlighted its role in community building, relatively little
attention has been paid to its original gaming context-how it shapes gameplay and social experiences. To address this gap, we conducted semi-structured interviews with 16 teenage Discord users. Through reflexive thematic analysis, we show how players leverage Discord to create more collaborative and socially enriched experiences that extend beyond the game itself. However, gaming together on Discord also resulted in social and security risks. We conceptualize gaming together on Discord as a cross-platform practice that extends gameplay beyond a game and supports players' social needs. Additionally, cross-platform practice also introduces the 'platform gap,' where fragmented governance between platforms exposed players to risks. To address this tension, we propose design implications aimed at bridging the platform gap, strengthening communication channels, and supporting safer gaming experiences.
Read the full abstract ▾
Human-Computer Interaction
Do Vision-Language Models Agree on the Affective Qualities of Shape? A Cross-Model Audit for Generative Design Interfaces
Luca Bux, Thiago Rios, Ingo Scholtes et al. (+2 more)
Generative design interfaces increasingly expose semantic controls that let users steer output with concepts such as "more elegant" or "more minimalist," typically encoded by a
vision-
language model (VLM). A practical question is whether state-of-the-art VLMs represent objects consistently in terms of the same concept. We audit 6 VLMs by ranking untextured 3D objects along Kansei adjective pairs, where Kansei describes affective impressions of product form, with each axis defined as the difference between the text representations of its two poles.
Geometric pairs serve as positive controls, and pairs of unrelated adjectives establish an empirical null. Across 10 categories of ShapeNet database, affective axes converge above the null (mean pairwise rank correlation 0.36 vs. 0.14) but below the
geometric ceiling (0.44). The agreement between models is partial and highly uneven: on the three axes shared by all categories, mean convergence ranges from 0.21 for bookshelves to 0.51 for jars. Convergence depends primarily on whether a category's representational variation aligns with the semantic direction being evaluated, rather than simply on how much the objects vary in shape overall. Cross-model convergence does not imply agreement with human judgments. Based on our findings, we implement a UI prototype that shows how the audit can inform which Kansei descriptors to expose as controls for a given object class and which to withhold.
Read the full abstract ▾
All authors Luca Bux, Thiago Rios, Ingo Scholtes, Stefan Menzel, Bernhard Sendhoff
Human-Computer Interaction · Cross-listed from cs.ET
Simultaneous Digital Communication and Deformation Sensing over a Single Stretchable Interconnect
Yuji Isano and Hiroki Ota
Stretchable hybrid electronics integrate rigid solid-state electronics with stretchable materials and structures to achieve both high deformability and stable electronic performance. However, most existing systems treat stretchability only as a mechanical attribute without exploiting device deformation to encode its own mechanical state. This problem arises from adapting conventional rigid
circuit architectures to stretchable substrates, affording a loss in compatibility with the sensors required for strain measurement. This study addresses this issue by proposing a communication-integrated deformation sensing architecture for stretchable hybrid devices. In the proposed approach, standard universal asynchronous receiver-transmitter digital signals transmitted between rigid nodes are amplitude-modulated by strain-induced resistance changes in stretchable
liquid metal interconnects. By reading both amplitude changes and digital patterns, the system enables simultaneous digital communication and self-deformation sensing without requiring additional stretchable sensing elements. The architecture is demonstrated in a multi-node system and applied to wearable sensing and self-deformation mapping devices. By extending the integration of rigid
circuits and soft elements from the hardware level to the system level, this study provides a novel design paradigm for stretchable electronic systems that inherently utilize their own deformation as functional information.
Read the full abstract ▾
Human-Computer Interaction
Large Language Model Few-Shot Prompting with Dilemma Training Outperforms Human Surrogates in Predicting Patient Preferences
Natasha Ureyang, Sebastian Porsdam Mann, Yuxin Liu et al. (+9 more)
In serious illness, human surrogates often struggle to accurately predict patient preferences (68% accuracy), causing decision conflict. Personalized Patient Preference Predictor (P4)
agents offer a potential solution, but prior prototypes treat values as static ratings, ignoring the contextual, situation-dependent nature of medical choices. Grounded in the 'logic of care', we present P4-DT (Dilemma Training), a P4
agent that constructs a patient decision policy by engaging users with varied medical dilemmas, eliciting individual preference
reasoning through bi-directional training. In a study with 12 patient-surrogate dyads, P4-DT predicted patient treatment choices with 81.7% accuracy, significantly exceeding chance (OR = 5.61 [2.03, 15.51], p < .001) and outperforming both unassisted surrogates (55.0%; OR = 3.67 [1.59, 8.47], p = .002) and surrogates assisted by P4-DT (61.7%). Comparative
prompt analyses showed that incorporating contextual scenario decisions and open-ended text improved accuracy by 15.0 percentage points over initial values ratings alone. We discuss implications for further testing and designing of context-aware AI
agents that embody richer human experience to partner in complex decision-making.
Read the full abstract ▾
All authors Natasha Ureyang, Sebastian Porsdam Mann, Yuxin Liu, Zuriel Hassirim, Melanie Almonte, Wenhao Chen, Joyce Ng, Thant Nay Lin, Aung Thiha, Gerald CH Koh, Brian David Earp, Pin Sym Foong
Human-Computer Interaction
AffectSim: A Controllable Interactive 3D Simulation Benchmark for Embodied Affective Perception
Ke Xing, Zhilong Wang, Zheng Lian et al. (+8 more)
Existing affective
benchmarks largely rely on fixed recordings, where observation conditions are predetermined before inference. Consequently, they mainly evaluate passive affect recognition while overlooking a key question for embodied
agents: how to actively acquire informative affective evidence. We introduce AffectSim, a controllable interactive 3D simulation
benchmark for embodied affective perception. AffectSim represents affective behaviors as replayable 3D episodes and separates the underlying behavior from how it is observed, enabling systematic control over distance, orientation, occlusion, scene geometry, and
agent viewpoint. It contains 27,647 episodes across five emotion categories and 57 scenes. Across 24 frozen perception-model configurations, we show that observation quality substantially affects recognition. To examine whether active sensing can recover this gap, we establish a training-free heuristic baseline, which improves recognition in 21 of 24 settings. Building on this, we propose Evidence-Aware Observation Gate (EAOG) to adaptively assess whether additional observation is still beneficial. These results show that affective perception depends not only on the observed behavior, but also on how evidence is acquired. By making observation acquisition experimentally controllable, AffectSim provides a foundation for studying embodied affective perception in interactive 3D environments. All
benchmark assets, code, and simulation environments will be released.
Read the full abstract ▾
All authors Ke Xing, Zhilong Wang, Zheng Lian, Sicheng Zhao, Haifeng Lu, Zhen Zhang, Zitong Yu, Xiaojiang Peng, Changxin Huang, Runhao Zeng, Xiping Hu
Human-Computer Interaction · Cross-listed from cs.AI
Using profiles of cognitive capability to assess AI suitability for workplace tasks
Jonathan Prunty, Marko Tešić, Patrick Quinn et al. (+2 more)
Organisations deploying AI face a scoping problem: which tasks can be automated, which should remain with humans, and which are best shared between the two. Aggregate
benchmark scores provide little insight into where systems will succeed or fail in practice, while human judgements of model capabilities quickly become outdated. We introduce a pipeline that profiles
agents and tasks using a shared set of core cognitive capabilities. Cognitive capability profiling infers an
agent's capabilities from performance on a
benchmark battery annotated for the cognitive demands of each item. Task requirements weighting elicits from domain experts the relative importance of these same capabilities for their work. As both use a common set of cognitive dimensions, they can be updated independently as models and roles change, and combined to estimate AI suitability at the level of a domain, organisation, role, or individual duty. We validate capability recovery on synthetic
agents, profile six AI systems, and elicit task requirements from 410 employees across six occupational domains. AI systems differed more across cognitive dimensions than across model families, while workplace activities converged on a shared cognitive core. The resulting scores provide a comparative scoping tool for identifying promising candidates for piloting and areas where current systems are unlikely to be well suited. We discuss extending the framework to profile human workers alongside AI systems, moving from AI suitability towards human-machine task allocation.
Read the full abstract ▾
All authors Jonathan Prunty, Marko Tešić, Patrick Quinn, José Hernández-Orallo, Lucy Cheke
Human-Computer Interaction
Are Concept Bottleneck Models Effective as Decision-Support Systems?
Alessandro Bogani, Nicola Debole, Emanuele Marconato et al. (+3 more)
Concept Bottleneck Models (CBMs) are interpretable-by-design neural networks that detect human-understandable concepts from the input and use them to generate predictions. By allowing users to inspect the concepts underlying a prediction and explore how predictions change under alternative concept configurations, CBMs have emerged as one of the most prominent approaches to supporting human-AI collaboration. However, user studies investigating their actual effectiveness as decision-support systems remain limited. We present two large-scale user studies (N participants = 705, N observations = 6,959) evaluating how concept-based explanations and user interventions on the model's concepts affect the performance of the human-AI team in two distinct binary classification tasks. Our results show that CBMs, and particularly their interactive component, can improve human-AI team accuracy relative to both unaided human performance and performance with non-interpretable AI support. However, these benefits emerge only under certain conditions: classification tasks perceived as difficult, easily identifiable concepts, and active interaction with the model. We also discuss how inaccurate concept detection may undermine users' trust in the model. Overall, this work provides practical guidance for the deployment of CBMs as effective decision-support tools.
Read the full abstract ▾
All authors Alessandro Bogani, Nicola Debole, Emanuele Marconato, Andrea Pugnana, Katya Tentori, Andrea Passerini
Human-Computer Interaction
Maru: Information Architecture as a Shared Language for Generating Aligned and Persistent User Interfaces
Eunhye Kim, DaEun Choi, Bryan Min et al. (+3 more)
Generative user interfaces (GenUIs) promise on-demand components tailored to users' needs. As users iterate on information tasks, they construct personal structures over information they encounter—how items are grouped, what gets prioritized, and what terms mean in their context. Yet, current systems leave these structural decisions to the model at each generation, ignoring the structural logic users have established. Without a persistent representational structure shared between user and system, GenUIs have no basis to remain aligned with what users have established. We draw on Information Architecture (IA), a design practice for organizing and structuring information, as a shared
language to bridge user-constructed structure and system generation. We present a framework identifying four IA elements—partition, hierarchy, order, and vocabulary—and characterize how each maps to concrete UI generation decisions. We instantiate this framework in Maru, a conversational system that captures user
prompts and interactions as IA preferences, persisting as rules both user and system draw on across generations. A user study revealed that IA persistence kept generated UIs aligned as sessions progressed, while
alignment without it degraded, with diverse patterns emerging across users and contexts, pointing to the value of IA persistence in aligning GenUI to individual needs.
Read the full abstract ▾
All authors Eunhye Kim, DaEun Choi, Bryan Min, Hyunjung Yi, Yue Jiang, Juho Kim
Human-Computer Interaction
The Well-Being Palette: An Action-Word Selection Tool Designed for Low-Burden Reflection on Workplace Well-Being
Nobuhiko Muramoto, Takayuki Nagaya, Tomoko Tanaka et al. (+2 more)
Background: Workplace well-being interventions need formats that can be used repeatedly with minimal disruption to daily work. We developed the Well-Being Palette, a web-based action-word selection tool designed for brief, low-burden reflection on workplace well-being. Methods: In a three-month exploratory field study at a private-sector corporate research institute in Japan, 88 analyzed participants selected up to three well-being-related action words after reflecting on positive actions or experiences from each workday. We examined application usage, PERMA Profiler scores, selected-word patterns across departments, selected-word diversity using Shannon
entropy, and exploratory associations with sharing workshops. Results: During the formal intervention period, the application captured 3,480 input records and 10,104 selected words, and all 72 available action words were selected. Overall PERMA scores increased from baseline to post-intervention, and no clear decline was observed at the one-month follow-up among available cases. Application logs revealed departmental differences in selected-word categories. Cumulative selected-word diversity increased over time, and sharing workshops showed exploratory associations with more sustained PERMA patterns and broader cumulative selected-word diversity. Conclusion: The Well-Being Palette was feasible for repeated use in a real workplace and provided complementary log-based information on how workers recognized and labeled well-being-related experiences. The findings should be interpreted as exploratory and hypothesis-generating, because the study did not include a randomized control condition and did not directly measure perceived burden or completion time.
Read the full abstract ▾
All authors Nobuhiko Muramoto, Takayuki Nagaya, Tomoko Tanaka, Koichiro Iwai, Katsunori Kohda
Human-Computer Interaction
ScentEcho: Exploring Adsorbent Materials for Accurate Odor Collection and Playback
Chih-Hung Lee, Yuchi Sun, Rui Zhang et al. (+2 more)
Delivering odors that feel realistic and recognizable remains a core challenge for olfactory interaction systems, particularly in applications that demand precise scent delivery. A key limitation lies in the difficulty of capturing, preserving, and playing back real-world scent sources in a reliable and scalable manner. This study explores the potential of adsorbent materials for supporting realistic scent playback. We present ScentEcho, a portable system that enables modular scent collection and release. Through user evaluations, we identify which adsorbent materials tend to perform better for specific odors, and observe that perceived intensity strongly influences similarity ratings. In addition, odor recognition follows a graded pattern, with users moving from broad category identification to more specific source recognition as similarity increases. These findings offer practical insights for designing olfactory interfaces that are both expressive and perceptually aligned with user expectations.
Read the full abstract ▾
All authors Chih-Hung Lee, Yuchi Sun, Rui Zhang, Suhang Wei, Qi Lu
Human-Computer Interaction
TailorCoPilot: Enabling Agentic Pattern Making with Version-Controlled State Tracking
Yuexin Sun, Zhaohui Wang, Ruiyang Liu et al. (+4 more)
Experience-driven manufacturing, such as garment pattern making, faces a severe generational skills gap because its core expertise relies on undocumented tacit knowledge forged through day-to-day practice. To address this challenge, we present TailorCoPilot, an agentic pattern-making system built upon a specially designed version-control backend TailorTrace. TailorTrace models sewing patterns as structured, discrete states and records their transformations during the pattern-making process as explicit operation sequences defined upon the geometry primitives in the sewing pattern (panels, edges, vertices and stitches). Integrated into a conventional pattern-making GUI, TailorTrace enables seamless documentation of senior experts' tacit pattern-making knowledge without breaking their daily workflow. The documented knowledge further offers interactive, pedagogical scaffolding for novices, while providing a robust foundation to power TailorCoPilot and train future generative AI models. In a user study with novices and advanced novices, TailorCoPilot improved task completion rates, reduced time and perceived workload, and yielded higher-quality artifacts compared to skill-appropriate baselines. Ultimately, TailorCoPilot demonstrates a viable pathway to capture practice-based expertise, operationalizing it to support both generative AI advancements and human apprenticeship.
Read the full abstract ▾
All authors Yuexin Sun, Zhaohui Wang, Ruiyang Liu, Demian Kong, Qian He, Gaofeng He, Huamin Wang
Human-Computer Interaction
Q&A or Document-Based? The Effects of Interface Type on How Screen Reader Users Access Interconnected Documents
Colleen F. Cipriano, Yichun Zhao, Miguel A. Nacenta et al. (+2 more)
Blind and low-
vision (BLV) users are increasingly engaging with large
language model (LLM) interfaces to access documents, but it is unclear how such systems support or hinder their ability to build interconnected knowledge. To examine this gap, we compared a Question-Answer Interface (QAI) that supports open-ended conversational inquiry, with a Document Interface (DI) based mostly on traditional structured text document navigation. We recruited 16 BLV screen reader users where they used both interfaces to explore two fictional worlds. Data from interaction logs, concept maps, decision-based tasks, and semi-structured interviews provide comparative insights into how interface design supports knowledge construction. Findings show that participants visited more distinct documents with the DI and formed larger and more correct mental models with the DI than with the QAI. They were also more able to apply knowledge they had gained. Simultaneously, many still preferred the QAI and often estimated that they had explored more, formed better mental models and applied their models better when acquiring the information with the QAI, despite this not being the case. Our analysis suggests possible interface design reasons for these differences and highlights some of the risks introduced by using question-answer interfaces to access information spaces.
Read the full abstract ▾
All authors Colleen F. Cipriano, Yichun Zhao, Miguel A. Nacenta, Kotaro Hara, Jaylee Soh
Human-Computer Interaction · Cross-listed from cs.CY
CompanionHarm: A Multi-Turn Benchmark for Detecting Harms in Real-World AI Companion Conversations
Renwen Zhang, Han Meng, Jian Chai et al. (+2 more)
As AI companions become increasingly embedded in everyday life, there is an urgent need to detect harms that emerge in social and emotional human-AI interactions. Yet research in this area is constrained by the lack of real-world, multi-turn conversational
datasets for operationalizing and evaluating harms that are relational and contextual. In this work, we introduce CompanionHarm, a publicly available
benchmark dataset comprising 2,111 real-world, multi-turn conversations (14,051 utterances) between users and the AI companion Replika. 7,016 AI utterances were annotated independently by three annotators across 13 harmful behavior categories grounded in a taxonomy of AI companion harms, and the
dataset includes both aggregated labels and annotator-level labels to support model evaluation and systematic disagreement analysis. Evaluations of seven large
language models (LLMs) show that harm detection using multi-turn conversational context outperforms detection based on isolated utterances, although current LLMs still struggle to consistently integrate contextual cues, calibrate harm severity, and interpret relational boundaries. We also find substantial annotator disagreement for context-dependent harmful behaviors, with disagreement varying according to annotators' political affiliation, conversation length, and the utterance's position. Together, CompanionHarm provides a foundation for detecting socio-emotional harms in multi-turn human-AI conversations and for rigorously examining how such harms are interpreted by both humans and LLMs. Our
dataset is available at https://github.com/HanMeng2004/CompanionHarm.
Read the full abstract ▾
All authors Renwen Zhang, Han Meng, Jian Chai, Yuntao Lin, Yi-Chieh Lee
Human-Computer Interaction
HRGuard: Gating Relationship Manipulation in Multi-Turn Agentic AI Conversations
Pei-Sze Tan, Tasuku Igarashi and Isao Echizen
Agentic AI assistants are increasingly used in everyday life. However, they may also be misused to support harmful manipulation in interpersonal relationships. This problem is role-sensitive. Requests from users who seek to manipulate others should be blocked. Users who seek protection from manipulation should instead receive supportive guidance. We study agentic relationship harm, which describes harm to human-human relationships that is mediated or assisted by AI
agents. In multi-turn settings, individually plausible actions may combine into a harmful workflow. We introduce a
benchmark of 1,000 five-turn conversations. It covers both attacker-side and victim-side scenarios. It also includes direct and adversarially paraphrased variants. We further propose HRGuard. It includes an online pre-generation gate and a turn-level post-generation gate. The post-generation gate maintains a decayed cumulative risk state and interrupts emerging manipulative workflows. Across eight generation models, HRGuard reduces harmful compliance while preserving victim-side protective guidance. It also outperforms a generic safety
prompt and three general-purpose guard models. Independent-judge evaluation supports the main findings. Under our evaluation protocol, the tested generic
prompt and general-purpose guards leave substantial residual risk, motivating turn-aware relationship-specific evaluation.
Read the full abstract ▾
Human-Computer Interaction
AVI-Personality: A Trait-Activated Multimodal Dataset for Personality and Competency Assessment in Asynchronous Video Interviews
Tianyi Zhang, Jinwenxi Shang, Antonis Koutsoumpis et al. (+3 more)
With the rapid development of AI-based personality and job-related competency assessment, Asynchronous
Video Interviews (AVIs) are increasingly used in recruitment. However, existing
multimodal personality
datasets are often based on short, task-free social media
videos and crowdsourced apparent personality labels, which limits their construct validity and relevance to structured interview assessment. To address these limitations, we introduce AVI-Personality, a trait-activated
multimodal dataset for personality and job-related competency assessment from AVIs. The
dataset contains 3,876 interview
videos from 646 participants who completed a simulated management traineeship application. Participants answered two generic questions and four personality-targeted questions designed according to Trait Activation Theory. Our
dataset provides both self and observer-reported HEXACO personality traits and job-related competency. We validate AVI-Personality through reliability, construct validity, internal nomological association,
fairness, and
benchmark analyses. Validation results show that the observer-rated personality traits have moderate to high reliability, especially when ratings are based on personality-targeted questions.
Benchmark results show that text-based AI
algorithms provide strong personality-relevant cues, while
multimodal methods achieve the best overall performance but only modestly outperform text-based baselines. In general, AVI-Personality provides a psychometrically grounded
dataset for developing and evaluating AI-based models for personality and competency assessment. The
dataset is available are released at https://github.com/APAL-SEU/AVI6
Read the full abstract ▾
All authors Tianyi Zhang, Jinwenxi Shang, Antonis Koutsoumpis, Yuan Zong, Reinout E. de Vries, Wenming Zheng