Peer Revenue Fully funded

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Computer Science Human-Computer Interaction

cs.HC Watch Human-Computer Interaction

Wednesday 5 August

Human-Computer Interaction

DeepConnect: A Visual Analytics System for Bridging Interdisciplinary Research Collaborations

Yingchaojie Feng, Zekai Shao, Yiqun Sun et al. (+2 more)

Interdisciplinary research collaboration is crucial for scientific innovation, but it remains difficult to initiate in practice. Existing collaborator discovery approaches are often constrained by disciplinary boundaries and static researcher profiles that do not reflect the specific context of a new collaboration goal. As a result, researchers struggle to translate open-ended collaboration goals into domain-specific tasks, evaluate candidate researchers' fit and complementarity, and establish common ground before initial contact. To address these challenges, we present DeepConnect, an LLM-augmented visual analytics system for interdisciplinary collaborator discovery. DeepConnect translates collaboration ideas into domain-specific tasks, retrieves relevant papers to ground cross-domain exploration, and provides coordinated visualizations for exploring and comparing candidate researchers. It further reveals terminology gaps and overlaps across domains and supports publication-grounded conversation rehearsal to help users prepare for outreach. We evaluate DeepConnect through two case studies, a user study, and a component-level evaluation, showing its value for complementary team formation, idea refinement, and pre-contact preparation. The DeepConnect website is available at https://deepconnect.sg.
Read the full abstract

All authors Yingchaojie Feng, Zekai Shao, Yiqun Sun, Yixuan Tang, Anthony K. H. Tung

Human-Computer Interaction · Cross-listed from cs.SE

Characterizing Visual Accessibility Issues in AI Developer Tools: An Empirical Study

Sabrina Haque and Christoph Csallner

AI-assisted developer tools increasingly mediate programming through chat panels, terminal agents, generated diffs, and streaming status output. These interaction surfaces may create visual accessibility barriers for blind, low-vision, and color-vision-deficient developers, yet little is known about how such barriers are reported in public tool ecosystems. We analyze issues and forum discussions from five AI developer tool ecosystems: GitHub Copilot in VS Code, Cursor, Claude Code, OpenAI Codex, and OpenCode. From 2,652 keyword-retrieved candidates, a three-model ensemble identified 600 unanimously positive visual accessibility reports. A stratified manual sanity check supported this conservative selection. Topic modeling and qualitative analysis identified three recurring categories: screen-reader and assistive-technology barriers; visual presentation, contrast, and differentiation problems; and readability, scaling, and control limitations in AI-specific interfaces. The relative prominence of these concerns varied across ecosystems and reflected differences in editor, terminal, chat, diff, and agent interaction surfaces. An exploratory metadata analysis further identified differences in reporter activity and, across the GitHub-based ecosystems, maintainer participation and closure processes. These findings show that the accessibility record of AI developer tools is shaped by both their interaction design and the reporting and maintenance practices of their surrounding ecosystems.
Read the full abstract

Human-Computer Interaction · Cross-listed from cs.CV

Robust and Efficient Motion Reasoning for Privacy-Aware Classroom Incident Recognition

Paritosh Parmar, Landy Lan, Hong Yang et al. (+2 more)

Can computer vision help make classrooms safer? In this pilot study, we investigate privacy-aware and computationally efficient classroom incident recognition from CCTV-style observations. This setting remains underexplored, with limited benchmarks and few methods designed for the privacy, efficiency, and generalization demands of real-world deployment. We introduce a novel hybrid benchmark combining generative CCTV-style videos with real-world classroom pose data, and propose a lightweight, but robust motion-reasoning framework motivated by the observation that many incidents differ more in motion direction, speed, acceleration, and intensity than in pose alone. To that end, our method first constructs hierarchical kinematic representations of human actions. Our method then distills hierarchical, multi-order kinematic reasoning from a large teacher into a much smaller single-order student, enabling efficient per-person inference while preserving expressive motion understanding. Experiments show that our model outperforms substantially larger baselines at less than one-tenth of their computational cost, while also demonstrating stronger out-of-domain motion reasoning and zero-shot synthetic-to-real generalization. We will publicly release the benchmark, codebase, and supporting tools to facilitate further research in privacy-aware classroom safety.
Read the full abstract

All authors Paritosh Parmar, Landy Lan, Hong Yang, Chen Yi, Chiat Pin Tay

Human-Computer Interaction

ArtAnno: Annotating Implicit Semantics in Artworks through LLM Agent-Driven Bidirectional Human-AI Augmentation

Xiaoyan Gu, Yifang Wang, Wenqing Zheng et al. (+6 more)

High-quality annotation of artworks is essential for computational art research, yet extracting implicit semantics remains challenging due to the reliance on culturally grounded meanings and deep contextual knowledge behind the images. Current AI-assisted annotation tools often lack assistance or rely on one-way workflows where experts have to perform extra manual calibrations to improve AI models, resulting in limited efficiency. To address this, we propose Bidirectional Human-AI Augmentation(BiHAA), a closed-loop framework in which skills and domain knowledge base evolve through real-time interaction and bidirectional HAI augmentation. Informed by a formative study with 20 artwork annotators from different backgrounds, we implement this framework in ArtAnno, an artwork annotation system driven by a multi-agent architecture. The system includes a Proactive Agentic Support Module, where AI augments humans through semantic mining and label suggestion, and an Interaction-Driven Evolution Module, where human expertise continuously enhances the AI through distilling annotation trajectories into reusable experience. Evaluation through a user study and two case studies demonstrates that our framework and system improve annotation efficiency, enable knowledge accumulation, and reduce the effort of information seeking and verification for annotators with limited domain expertise. We conclude by discussing broader implications and future directions.
Read the full abstract

All authors Xiaoyan Gu, Yifang Wang, Wenqing Zheng, Haozhong Liu, Yixia Zheng, Peiyi Jiang, Wenjie Ning, Wei Zhang, Wei Chen

Human-Computer Interaction

Exploring Cross-Reality Transitions between Projections and Head-Mounted Displays for Immersive Digital Art

Xiangpeng Fu and Mads Haahr

Immersive exhibitions increasingly combine large-scale projections with mixed reality (MR) head-mounted displays (HMDs), but how users perceive transitions across these heterogeneous visualization environments remains underexplored. We investigate cross-reality (CR) object- and scene-level transitions in a hybrid immersive art installation spanning projection, augmented reality (AR), and virtual reality (VR). In a within-subjects study (N=24), we compared a calibrated condition with a bundled inconsistency condition introducing noticeable differences in spatial alignment, visual appearance, and cross-device latency. This contrast was used not to establish perceptual thresholds or hardware targets, but as a diagnostic probe to make transition disruptions more perceptible and discussable. The inconsistency condition reduced presence and increased workload. Interviews showed that spatial misalignment disrupted action-outcome predictability, appearance mismatch reduced aesthetic coherence, and latency weakened perceived responsiveness. Effects also varied by asset type: rigid static assets exposed inconsistencies more clearly, animated skeletal assets shifted attention toward motion, and particle effects often masked small discrepancies. These findings clarify how users perceive projection-MR transitions and support asset-aware design strategies for coherent hybrid immersive art. We also release HUICRSync, an open-source Unreal Engine plugin for prototyping projection-MR CR experiences.
Read the full abstract

Human-Computer Interaction

Reply, Delete, or Ignore? Examining How Content Creators Perceive and Select Comment Moderation Strategies

Yunhee Shim and Shagun Jhaver

Content creators on social media sites occupy highly visible positions on their channels. As a result, creators, especially those with large followings, experience disproportionate levels of online harm. To address such harm, they enact a range of moderation strategies, which in turn shape the visibility of content that their audiences encounter. This paper examines how content creators perceive three moderation strategies to address hateful comments (deleting, replying to, or simply ignoring) and how they decide which strategy to deploy. While creator moderation is usually examined through the lens of safety, creators' regulation decisions may also be shaped by concerns about how their actions appear to audiences and how they are rewarded or penalized by platforms' recommendation algorithms. Conducting a survey of 584 content creators, we found that in their view, (1) deleting is the most beneficial for achieving safety, (2) both deleting and replying produce better impression management benefits than ignoring, and (3) replying is perceived to yield the highest algorithmic benefits. Crucially, while expectations of emotional safety and impression management benefits significantly predicted creators' willingness to adopt each comment moderation strategy, perceived algorithmic benefits did not. By unpacking how creators evaluate these trade-offs, this study contributes to HCI research on understanding creator-led, middle-level governance. We conclude with design implications for supporting creators as crucial governance actors without burdening them with sole responsibility for online safety.
Read the full abstract

Human-Computer Interaction

Transition Techniques for Externally-Guided Multi-Scale Viewpoint Changes

Matt Gottsacker, Mengyu Chen, David Saffo et al. (+3 more)

Extended reality (XR) is increasingly used to help users understand complex virtual environments through multiple viewpoints across different immersion levels, positions, and scales. While numerous techniques address viewpoint transitions for self-guided exploration, many scenarios require externally-guided transitions where a system or presenter controls the user's viewpoint, leaving the user with limited spatial knowledge and control over the transition process, which can increase susceptibility to disorientation and discomfort. We present three transition techniques for externally-guided multi-scale XR viewpoint changes and evaluate them against a fade-to-black baseline in a within-subjects study (N=20). Participants transitioned between world-in-miniature, street-level, and indoor destination views. We combined spatial recall measures, standardized questionnaires, and semi-structured interviews to assess orientation, workload, comfort, and continuity. Results showed that in our setup, techniques externalizing reference frames and the user's pose improved multi-scale spatial recall relative to a fade, while same-scale recall was insensitive to transition technique. We conclude with design implications for multi-scale XR viewpoint transitions.
Read the full abstract

All authors Matt Gottsacker, Mengyu Chen, David Saffo, Feiyu Lu, Benjamin Lee, Blair MacIntyre

Human-Computer Interaction

AutoCue: Multimodal LLM-Assisted Externalization of Implicit Inputs as Instructional Visual Cues in Screencast Tutorials

Shengyang Luo, Shengyao Luo, Xiaolei Guo et al. (+3 more)

Tutorial videos are widely used for learning feature-rich software, yet following screencast tutorials often breaks down in practice. Through a survey and contextual inquiry, we found that learners frequently rewind or get stuck because critical input information, especially mouse actions and keyboard-modified operations, is often implicit or missing in tutorials without input metadata. To address this problem, we present AutoCue, a multimodal LLM-assisted, human-in-the-loop tutorial augmentation pipeline for externalizing implicit inputs as instructional visual cues. AutoCue integrates frame-to-frame visual changes, narration signals, and operation guidance from official software manuals to infer likely mouse and key-modifier actions, then produces aligned cue layers and editable artifacts for human refinement. Grounded in multimedia learning and cognitive load theory, we further develop a visual cue grammar for representing mouse, keyboard, and combined inputs in software-learning tutorials. We instantiate and evaluate AutoCue in Autodesk Maya, focusing automatic inference on selected UI-mediated interactions with observable visual or textual feedback while supporting more ambiguous state changes through editable authoring artifacts. In a between-subjects study with 24 participants, the AutoCue-augmented tutorial reduced task completion time and interaction breakdowns and showed improved learner-reported experience.
Read the full abstract

All authors Shengyang Luo, Shengyao Luo, Xiaolei Guo, Fengze Zhang, James Liang, Yingjie Victor Chen

Human-Computer Interaction · Cross-listed from cs.CY

Exploring Fraction Comprehension and Interest in Elementary Education Through AI-Powered Personalized Learning

Kenneth Holman

Artificial intelligence systems that adapt instruction to individual learners are increasingly deployed in K-12 classrooms, yet empirical evidence on their effects in authentic elementary settings remains limited, particularly for students with mathematics learning difficulties. This dissertation examines AI-powered personalized learning during primary school fraction instruction, a domain that is foundational to later mathematics and STEM achievement. The first manuscript presents a systematic review of research on artificial intelligence in mathematics education published between 2020 and 2024. The second manuscript reports a quasi-experimental study evaluating Mathbot, a chatbot-based personalized learning platform, against business-as-usual classroom instruction. Repeated measures ANOVA was used to assess change in fraction comprehension and situational interest across time points. Results indicated modest improvements in fraction comprehension for students using Mathbot relative to traditional instruction, while changes in situational interest were not statistically significant. Findings suggest that automated personalization did not displace the instructional role of the teacher and that teacher decision-making remained central to student outcomes. The work contributes classroom-based evidence to ongoing discussion about the capabilities and limits of adaptive AI systems in elementary mathematics, and about accessibility and equity considerations when such systems are used with students with disabilities.
Read the full abstract

Human-Computer Interaction

Investigating Click Behaviors On Google Search Result Pages That Produce an AI Overview

Athena Chapekis, Anna Lieb, Sono Shah et al. (+1 more)

In 2024, Google introduced "AI Overviews," a feature that displays an AI-generated result summary at the top of many Google search pages. This study investigates the role of AI in Google search using one month of web browsing data from a representative panel of 900 U.S. adults. Our analysis of the panelists' Google searches sheds light on AI Overviews, when they appear in Google search results, and what user behaviors are associated with AI Overviews. We identify several attributes that make a search query more likely to generate an AI Overview, including the length of a query, whether the query begins with a question word, and whether the query contains both a noun and verb. When it comes to user behavior, we find that clicks to sources cited in AI Overviews are very rare, occurring in only about 1% of visits to AI Overviews. We also find that AI Overviews are associated with fewer clicks and higher rates of ending browsing sessions. Importantly, results from a mixed-effects logistic regression model indicate that these associations hold when controlling for random effects by panelist and query attributes that make AI Overviews more likely to appear.
Read the full abstract

All authors Athena Chapekis, Anna Lieb, Sono Shah, Aaron Smith

Human-Computer Interaction

Reference-Based Manipulation: A Framework and Pipeline for Multimodal Spatial Reasoning

Yangyang He, Zhuangze Hou, Yonglin Chen et al. (+1 more)

When manipulating objects in immersive platforms through speech and gesture, users naturally construct spatial references, referring to scene entities, their bodies, or the environment. Leveraging spatial cognition theories, this work systematically examines how users construct and communicate spatial intent. Using a custom toolkit, we conducted a Wizard-of-Oz study to observe unconstrained multimodal (speech + gesture) input patterns in Virtual Reality for scene construction. Based on these findings, we formalize a framework that decomposes spatial references into three core components: Source, Anchor, and Frame, while characterizing their compositional strategies and explicitness. We demonstrate the utility of this Reference-based Manipulation framework by implementing an LLM-based pipeline featuring a set of example interaction techniques with a preliminary technical evaluation. Finally, we discuss key lessons learned for supporting reference-based spatial interaction.
Read the full abstract

All authors Yangyang He, Zhuangze Hou, Yonglin Chen, Can Liu

Human-Computer Interaction

LiverPlan: A Stage-Adaptive Immersive Visual Analytics Framework for Anatomical Liver Surgical Planning

Qixuan Liu, Shi Qiu, Xiwen Wu et al. (+7 more)

Anatomical liver resection (ALR) surgery is the most important treatment for liver cancer, yet preoperative planning demands complex, multi-stage clinical reasoning under competing safety constraints. Current 2D desktop tools are not well equipped to support this process, exhibiting three fundamental limitations: reliance on monolithic interfaces that fail to adapt to the distinct cognitive demands of each planning stage; a perceptual bottleneck caused by limited anatomical spatial representation and missing plane-vessel intersection visualization; and an attention bottleneck stemming from fragmented critical safety criteria display across separate views. We present LiverPlan, a stage-adaptive immersive visual analytics framework for ALR planning, grounded in an 8-month collaboration with two expert hepatobiliary surgeons. Decomposing the surgical planning process into three sequential yet cognitively distinct stages, LiverPlan externalizes the cognitive demand of each stage via tailored techniques: (1) context-preserving focus and hue-preserving rendering for anatomical discovery; (2) direct 3D resection plane manipulation coupled with real-time, embedded visual feedback on critical safety criteria during plan refinement; and (3) explicit plane-vessel intersection visualization for anticipatory surgery preparation. A within-subjects study with eight hepatobiliary surgeons against a desktop baseline shows large-effect-size improvements in task completion time, perceived cognitive workload, and system usability on controlled planning tasks. Moreover, our study reveals broader insights: LiverPlan reduces cognitive burden and encourages a shift in surgeons from merely satisfying safety criteria to actively optimizing them, suggesting that explicit visualization of spatial relationships lowers the cognitive barrier to complex surgical planning.
Read the full abstract

All authors Qixuan Liu, Shi Qiu, Xiwen Wu, Yuqi Tong, Yinqiao Wang, Ruiyang Li, Jialun Pei, Shengdong Zhao, Chi-Wing Fu, Pheng-Ann Heng

Human-Computer Interaction · Cross-listed from cs.AI

AI Literacy for Legal Translation: Developing Digital Resilience

Łucja Biel

Generative AI is transforming legal translation by introducing opportunities alongside linguistic, technical, legal, ethical and cognitive risks. This chapter examines the implications of AI for professional legal translation and proposes an AI literacy framework tailored to the profession. It argues that AI does not change the fundamental objectives of legal translation but requires an extension of professional competence through AI literacy. The proposed framework comprises four mutually reinforcing dimensions, foundational, procedural, critical and strategic, and conceptualises AI literacy as a transversal component of legal translation competence that fosters digital resilience. It further discusses the pedagogical implications of this framework by proposing classroom activities designed to develop AI literacy in legal translator education, enabling future translators to integrate AI critically, responsibly and in accordance with professional standards.
Read the full abstract

Human-Computer Interaction

Emotion Dynamics in Social Deception Games: Analysis of Professional and Nonprofessional Players through Electrodermal Activity in Werewolf Games

Sho Mitarai, Chang Liu, Goshiro Yamamoto et al. (+1 more)

The development of AI systems capable of emotionally resonant communication remains a significant challenge. This study examines how humans influence emotions in social deception games by comparing professional and non-professional players. We measured electrodermal activity during gameplay to capture physiological emotional responses and analyzed communication patterns during periods of high emotional arousal. Our results revealed distinct communication strategies: professional players maintained persuasion-based approaches under high arousal, while nonprofessional players shifted toward information-focused communication. Statistical analysis confirmed significant differences in expression patterns between expertise levels. Professional players exhibited more stable emotional states during gameplay, indicating better emotional regulation. These findings inform the design of AI systems that can adapt their communication strategies based on recipient characteristics, advancing the development of emotionally intelligent artificial agents.
Read the full abstract

All authors Sho Mitarai, Chang Liu, Goshiro Yamamoto, Nagisa Munekata

Human-Computer Interaction

Super-Gaussian: Interactive Scene Editing for 3D Gaussian Splatting and NLI-Based Volume Visualization in Virtual Reality

Suemin Jeon, Kaiyuan Tang, Chaoli Wang et al. (+1 more)

Despite the promise of virtual reality (VR) for intuitive spatial interaction, volume visualization (VolVis) in VR remains constrained by high rendering costs and motion discomfort. Recent advances have shown that representing volumetric scenes with 3D Gaussian splatting enables high-performance rendering, making this representation well-suited for VR. However, existing Gaussian-based scene editing workflows remain limited by slow offline segmentation and fatigue-inducing manual selection. To address these challenges, we present Super-Gaussian, a novel VolVis framework that enhances scene editing and interaction in VR through intuitive 3D Gaussian selection and natural language interaction (NLI). Our approach groups Gaussian primitives into higher-level units via feature-aware clustering, enabling efficient selection of complex volumetric regions, such as tumors in medical images or filaments in cosmological data, without point-by-point interaction. Building on this, we introduce a hierarchical select-and-refine workflow that combines random-walk-based region propagation, cluster selection, and point refinement, allowing users to progressively specify regions of interest with reduced effort. We further support on-the-fly text labeling of selected regions using NLI, allowing users to semantically query, interpret, and manipulate content within a visualization-perception-action loop. By integrating multimodal interaction, including speech, visual feedback, and spatial manipulation in VR, our framework supports intuitive exploration, editing, and scientific analysis of volumetric data. We demonstrate the effectiveness of Super-Gaussian through four case studies, quantitative selection benchmarks against existing Gaussian-based techniques, and system-level evaluations. Implementation details and experiments can be found on the project page: https://smin0136.github.io/super-gaussian-project/
Read the full abstract

All authors Suemin Jeon, Kaiyuan Tang, Chaoli Wang, Won-Ki Jeong

Human-Computer Interaction

Revisiting Channel Effectiveness: A Multi-Dimensional Evaluation with Primitive Visual Stimuli

Soohyun Lee, Seokhyeon Park, Minsuk Chang et al. (+1 more)

Established channel effectiveness rankings primarily assess magnitude estimation accuracy in complete chart contexts, often neglecting other perceptual tasks such as discriminability, separability, and pop-out. To address this gap, we conducted crowdsourced experiments on seven core visual channels (position, length, tilt, area, curvature, luminance, and saturation) using primitive visual stimuli, a set of visual marks without chart-specific scaffolding to isolate channel-level variation. We evaluated these channels across four perceptual tasks (accuracy, discriminability, separability, and pop-out) and found that channel effectiveness is fundamentally multi-dimensional, with rankings shifting substantially across tasks. For instance, while spatial channels maintain an overall advantage, accuracy depends strongly on whether a fixed spatial anchor is available. Discriminability varies dramatically across channels and value ranges, a pattern we formalized with a novel Anchored Harmonic Weber model. Pairwise channel interactions are often strongly asymmetric. Finally, we identify a dissociation between estimation accuracy and preattentive detection: length shows only moderate detection effectiveness despite top-tier accuracy, while area achieves the highest detection rates despite poor quantitative accuracy, though the latter advantage may partly reflect stimulus-level cues. We synthesize these findings into a scenario-driven perspective for context-sensitive channel selection.
Read the full abstract

All authors Soohyun Lee, Seokhyeon Park, Minsuk Chang, Jinwook Seo

Human-Computer Interaction

Preference-Driven Online Adaptation for Personalized Interaction Initiation in Proactive AI Assistants

Yufeng Wang, Wei Zhang, Zhiquan Wen et al. (+5 more)

AI assistants are typically reactive, relying on users to initiate interactions. Proactive assistants go beyond this paradigm by autonomously initiating interactions based on users' activity contexts. However, appropriate interaction timing is user-specific and difficult to determine in advance, while online feedback offers valuable signals for personalization. Direct feedback-driven adaptation is therefore appealing, but remains challenging due to sparse interaction-worthy moments scattered across fine-grained user states. To address the issues, we propose Evidence-driven Online Preference Adaptation (EOPA), which grounds a user's interaction-timing preferences in measurable contextual evidence through two evidence carriers: temporal preference anchors and evidence-bearing activity prototypes. At each polling step, EOPA derives temporal and activity evidence from the carriers through user-prior-smoothed evidence estimation and uncertainty-guided evidence scaling, and adaptively fuses the evidence for interaction-or-silence decisions. When interaction is selected, an LLM uses high-quality historical responses as demonstrations to generate a context-aware response that better reflects user preferences. EOPA updates its evidence carriers and decision parameters from received online feedback without LLM-based reasoning or retraining. Extensive experiments on a ProPerSim-based benchmark show that EOPA improves the interaction-timing F1 score by 19.80 points over the strongest baseline in our experiments, substantially reduces inference latency for both silence and interaction steps, and lowers the average daily adaptation time from 11.41 to 0.39 seconds.
Read the full abstract

All authors Yufeng Wang, Wei Zhang, Zhiquan Wen, Jinwu Hu, Linhui Xiao, Tianlu Pan, Qingfang Zheng, Mingkui Tan

Human-Computer Interaction

IntentLint: Supporting Intent Scaffolding and Prompt-time Linting in Human-AI Collaborative Data Analysis

Felicia Li Feng, Jian Zhao and Anamaria Crisan

In human-AI collaborative data analysis, as analyses rapidly evolve, the artifacts meant to capture shared understanding often become incomplete or difficult to interpret, leading to undocumented assumptions, cross-user misaligned intent, context-poor prompts, and unwanted agent behaviors. To address these challenges, we introduce a rule-based coordination layer with two interaction mechanisms, intent scaffolding and prompt-time linting, that make analytic intent explicit and actionable during human-AI collaborative data analysis. We implement them in IntentLint, a proof-of-concept system that infers analytic intent from shared notebooks, represents it as structured, editable rules, and checks users' prompts against shared rules. IntentLint helps analysts externalize and refine their intent and proactively checks prompts for potential conflicts. A study with 16 data analysts shows that IntentLint improves awareness of collaborators' intent and encourages reflection on analytic strategies, and provides design implications for supporting more aligned and transparent human-AI collaborative data analysis.
Read the full abstract

Human-Computer Interaction

An immersive micro-manipulation system using real-time 3D imaging microscope and 3D operation interface for high-speed and accurate micro-manipulation

Kenta Yokoe, Tadayoshi Aoyama, Toshiki Fujishiro et al. (+2 more)

The use of intracytoplasmic sperm injection (ICSI), an assisted reproductive technique (ART), is increasing widely. ICSI is currently performed by specially skilled embryologists. However, with the increasing demand for ART, the shortage of skilled embryologists has become a problem. Therefore, we propose an immersive micromanipulation system that requires no special skills for efficient and accurate micromanipulation. Our proposed system is composed of a real-time three-dimensional (3D) imaging microscope and 3D operation interfaces. The 3D operation interfaces are stationary pen-type or wearable glove-type interfaces. In this system, an operator wearing a head-mounted display (HMD) and using 3D operation interfaces is immersed in a virtual micromanipulation space. The operator can move the pipettes by 3D operation interface and freely change the viewpoint. We verified that the proposed system improves the speed and accuracy of operating a pipette through two types of experiments with subjects.
Read the full abstract

All authors Kenta Yokoe, Tadayoshi Aoyama, Toshiki Fujishiro, Masaru Takeuchi, Yasuhisa Hasegawa

Tuesday 4 August

Human-Computer Interaction

EmAvatar: Multimodal Empathetic Response Generation via Conflict Resolution and Expressive Guidance

Xiaolin Chen, Xuemeng Song, Jinlan Fu et al. (+3 more)

Avatar-based multimodal empathetic response generation has emerged as a pivotal capability in human-centric systems, aiming to recognize user emotions and synthesize responses with synchronized text, audio, and talking-face video. Despite recent progress, existing methods still suffer from three critical limitations: (1) overlooking conflicting emotions across modalities, (2) lacking explicit multimodal synthesis guidance, and (3) neglecting inherent error propagation of multimodal response generation. To address these limitations, we propose EmAvatar, a novel framework for precise emotion perception and expressive response generation. It first performs deliberative multimodal emotion recognition by exposing inter-modal prediction conflicts and then initiates a multi-round QA process between a Conflict Inspector and an Evidence Collector to gather evidence for conflict resolution, leading to a robust, evidence-aware prediction. Regarding response generation, EmAvatar first synthesizes a composite script that couples the textual response with an expressive instruction. Moreover, to ensure high-quality synthesis, an iterative refinement mechanism evaluates and revises the script until it aligns with predefined criteria, serving as reliable guidance for subsequent audio and video synthesis. Extensive experiments across four tasks demonstrate that EmAvatar outperforms state-of-the-art methods. Our code will be publicly released.
Read the full abstract

All authors Xiaolin Chen, Xuemeng Song, Jinlan Fu, Weili Guan, Mong-Li Lee, Wynne Hsu

Human-Computer Interaction · Cross-listed from cs.CL

OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agents

Jingsheng Zheng, Xinyuan Fang, Jintian Zhang et al. (+3 more)

LLM agents are increasingly applied to open-ended everyday requests that span work, study, and life. These tasks are long-horizon, cross-environment, and multimodal, forcing the agent to preserve goals and constraints across many steps while navigating heterogeneous tools and attachments. While prior work has addressed individual failure modes such as goals drift, states loss, and context overflow, whether a single harness can manage them jointly and remain effective across backends has received less study. We present OneDayAgent, a long-horizon harness for autonomous agents. OneDayAgent turns an open-ended request into a managed execution process that decomposes tasks into bounded subtasks, maintains execution memory under context pressure, and verifies and repairs the final deliverable. We evaluate OneDayAgent on AgentIF-OneDay across 104 tasks. With the GLM-5.2 backend, OneDayAgent sets a new state of the art with an overall score of 0.821. The same harness runs across five backend LLMs from three model families, indicating the harness generalizes across backends without tuning, even as different models induce distinct execution styles under the same workflow.
Read the full abstract

All authors Jingsheng Zheng, Xinyuan Fang, Jintian Zhang, Zhengke Gui, Huajun Chen, Ningyu Zhang

Human-Computer Interaction

LEGOUI: Designing with UI-DSL Bricks to Balance Transparency and Controllability

Yinsi Zhou, Mingyue Yuan, Hongyue Xu et al. (+8 more)

Generative user interface design tools enable rapid prototyping but often operate as black boxes with limited transparency and controllability. When outputs diverge from the designer's intent, users are left tweaking prompts via trial-and-error with little insight into the model's reasoning. We present LegoUI, a staged generative framework that structures the interface design process into sequential, interpretable steps along key design dimensions, capturing each step's result in a UI domain-specific language (UI-DSL) enriched with provenance. This approach exposes the model's intermediate reasoning and enables user intervention and iterative refinement. In a technical evaluation on 40 real-world design prompts, LegoUI's requirement analysis stage captured explicit requirements with over 95% accuracy, near-complete coverage, and zero redundancy. In user studies, participants using LegoUI reported significantly greater transparency, controllability, and alignment with their intent compared to existing one-shot generative UI tools.
Read the full abstract

All authors Yinsi Zhou, Mingyue Yuan, Hongyue Xu, Jieshan Chen, Dong Wen, Shidong Pan, Xiwei Xu, Wenjie Zhang, Aaron Quigley, Zhenchang Xing, Gelareh Mohammadi

Human-Computer Interaction

Compass: Continuously Aligning Social Media Feeds via In-Situ Reflections

Aadit Barua, Leijie Wang and Amy X. Zhang

Social media recommendation feeds often optimize for users' immediate impulses rather than preferences they would hold after deeper reflection. Some systems address this misalignment by incorporating users' explicit preferences via a configuration page or in-feed controls instead of just behavioral signals. However, users typically have evolving preferences, and their stated preferences and behavior naturally diverge, necessitating continuous reflection and feed realignment. But existing strategies require the user to take initiative and are often effortful; as a result, in practice they are rarely invoked. We present Compass, a system that aligns a user's feed with their reflective preferences by helping users reflect on and articulate their preferences given their behavior. To enable continuous reflection during everyday browsing, Compass surfaces in-situ reflections via lightweight notifications, while feed alignment is achieved by periodically simulating behavioral signals and directly manipulating feed content. We embedded Compass within YouTube Shorts and compared it against a baseline without continuous support through a 10-day field study (N=15). We found that Compass promoted more reflective and purposeful feed consumption, iterative preference adjustment, and stronger feed alignment, without sacrificing the casual nature of feed browsing.
Read the full abstract

Human-Computer Interaction

Enacting Constructive Conflicts with AI Agents to Enhance Reconsideration among Novice Interaction Designers

Howard Ziyu Han and Nikolas Martelaro

Generative AI agents are increasingly used in interaction design to facilitate ideation and offer critique, often following their own internal reasoning. These interactions tend to add design ideas and expand the design space. Our work explores an antagonistic role for design agents, prompting designers to engage with stakeholder tension. We built an AI agent inspired by adversarial design theory that enacts constructive conflict. We examine the agent's influence in a between-subjects experiment with 45 design students across three conditions: Self Reflection (unsupported review of the design proposal), Stepwise Guidance (written prompts that walk designers through a constructive-conflict framework), and Interactive Engagement (an AI agent that enacts the constructive-conflict framework interactively by synthesizing stakeholder pushback). The latter two conditions share the framework but differ in whether it is self-enacted or agent-enacted. Results show that, compared with Self Reflection, both the Stepwise Guidance and Interactive Engagement groups reported significantly higher self-reconsideration and made more improvements to their design proposals. Compared with Stepwise Guidance, the antagonistic agent introduced more conflictual perspectives, and participants in the Interactive Engagement condition generated and discarded more ideas. These findings suggest that agent-enacted constructive conflict can turn reconsideration into concrete design actions and deepen engagement with divergent stakeholder perspectives.
Read the full abstract

Human-Computer Interaction · Cross-listed from cs.SE

AgentForge: An Immersive Role-Playing Platform for Learning Agentic Software Engineering

Zihan Fang, Yueke Zhang and Yu Huang

Agentic AI is increasingly used to coordinate planning, implementation, review, and testing in software development, yet it often offers limited transparency into its decisions and interactions. Many such systems also assume that users can effectively guide the AI's decisions and validate its outputs. This assumption poses a particular challenge for novices, who must simultaneously learn how agentic AI works, how to collaborate with it effectively, and how to evaluate its outputs critically. To address this challenge, we present AgentForge, an immersive learning system in which novices take on one of four software-engineering roles: Task Planner, Patch Author, Code Reviewer, or Test Runner, within a multi-agent code-repair workflow. In each practice session, the novices perform their chosen role while AI agents perform the remaining three. Through role-based scaffolding and metacognitive support, AgentForge clarifies role-specific responsibilities, makes agent coordination and intermediate artifacts visible, and encourages novices to monitor and evaluate their decisions. In a study with 37 novice developers, participants achieved high task-completion rates with AI-agent support. However, interaction demands differed significantly across practices: the Code Reviewer practice required more interaction turns, reroutes, and completion time ($p_{\mathrm{adj}} = .004$) and was perceived as the most challenging. Participants nevertheless reported significant gains in their understanding of software repair and agent collaboration ($p_{\mathrm{adj}} < .001$). These findings suggest that AgentForge can help novices develop practical software-engineering skills while learning to collaborate with agentic AI more critically and effectively.
Read the full abstract

Human-Computer Interaction

Echoes in the Sky: Computational Thematic Analysis of Online Public Discourse on Bluesky Across Trump's Reelection

Qile Wang, Ali Salloum, Carolina Coimbra Vieira et al. (+4 more)

As political disruption intensifies online discourse, Bluesky has become an important platform for political discussion and public reaction. In this study, we examine large-scale discourse on Bluesky related to U.S. policy developments associated with the Trump administration. Using the historical retrieval API, we collected all available posts matching Trump and related keywords from 2019 to 2026, yielding 38.5 million posts. We leverage a large language model (LLM)-assisted clustering pipeline, combined with human validation, to identify 14 interpretable thematic domains in English-language posts and 19 thematic categories across 258 executive orders (EOs) signed between January 20, 2025, and May 1, 2026. Our findings identify several dominant themes in Bluesky discourse, including executive governance, political identity, and national security, as well as recurring themes in EOs, including executive task forces, border enforcement, and foreign policy. We also find substantial variation in the persistence and volatility of issue attention, accompanied by an increasing proportion of negative sentiment over time. The dataset and resources are publicly available at https://github.com/Sensify-Lab/Echoes-in-the-Sky
Read the full abstract

All authors Qile Wang, Ali Salloum, Carolina Coimbra Vieira, Benjamin E. Bagozzi, Mikko Kivelä, Kenneth E. Barner, Matthew Louis Mauriello

Human-Computer Interaction · Cross-listed from cs.SD

Deep Learning for Real-Time Sound Order Recognition in Human-Robot Interaction

Rezaul Tutul, Usaid Khan, Andre Jakob et al. (+1 more)

Recognizing the temporal order of overlapping sounds is an underexplored challenge in human-robot interaction (HRI), with direct relevance to applications such as first responder detection systems. This paper presents a deep learning framework for real-time sound order recognition using recordable buzzers that emit distinct non-verbal sounds (cat meows, dog barks, helicopter noises). A multi-branch convolutional neural network (CNN) processes Mel spectrograms, Mel-frequency cepstral coefficients (MFCCs), and short-time Fourier transform (STFT) features, with an attention-based fusion mechanism to emphasize critical temporal cues. Experiments were conducted under same-amplitude, varied-amplitude, and unseen sound conditions. The proposed system achieved 99% accuracy in balanced overlaps, 91% under amplitude variation, and 74% on unseen test data with normalization. These results demonstrate that deep learning can reliably recognize sound order in overlapping conditions, supporting practical HRI scenarios. While experiments were conducted on carefully controlled synthetic overlaps, we additionally report latency benchmarks demonstrating real-time feasibility and provide an extended discussion on generalization, ecological validity, and deployment challenges in real-room environments.
Read the full abstract

All authors Rezaul Tutul, Usaid Khan, Andre Jakob, Ilona Buchem

Human-Computer Interaction

Calibrating Trustworthiness: Co-Designing Metrics and Visualizations for Evaluating LLMs in Education

Adam Coscia, Sujata Duwal, Langdon Holmes et al. (+2 more)

LLMs are reshaping educational technology, yet evaluating their responses for pedagogical alignment remains underexplored, relying heavily on the expertise of learning engineers building the technology. To bridge this gap, we explore trustworthiness as a structured lens for evaluation, leveraging existing measures of LLM trustworthiness to systematically identify potential pedagogical disruptions. Through a longitudinal co-design process with learning engineers developing an LLM-powered digital textbook, we: (1) co-constructed five trustworthiness metrics comprising 20 measures tailored to pedagogical use; (2) designed visualizations that map trustworthiness violations onto LLM responses; and (3) evaluated how these tools help learning engineers make A/B comparisons of LLM responses. Making trustworthiness explicit increased inter-rater reliability while helping learning engineers resolve conflicting objectives and produce more consistent judgments. We discuss the emergent benefits of trustworthiness as a lens for evaluating LLMs in education and propose new design guidelines for future evaluation tools that enable pedagogically-aligned, LLM-powered learning tools.
Read the full abstract

All authors Adam Coscia, Sujata Duwal, Langdon Holmes, Scott Crossley, Alex Endert

Human-Computer Interaction

Semantic Bundling: Interactive Node and Edge Bundling to Simplify Knowledge Graphs using Large Language Models

Adam Coscia, Zeyu Hua, Eric Krokos et al. (+2 more)

We present Semantic Bundling, a visual analytics technique for making sense of text documents represented as knowledge graphs (KGs). Representing a document corpus as a KG makes relationships between entities explicit, making KGs useful both to analyze directly and in computational workflows including ML pipelines and generative AI backends. However, as KGs grow they become difficult to interpret and visualize for specific tasks (e.g., the “hairball problem”), with the meaning of each relationship often buried in dense source text. Semantic Bundling uses large language models (LLMs) to support user-driven bundling of nodes and edges in a KG into higher-level graph structures: super nodes, which collapse and summarize a region of the graph, and super edges, which summarize the connection between two entities. Results are linked to underlying triples and source documents, grounding summaries in evidence. We implement Semantic Bundling in AgentK, an open-source system that builds a KG from text documents and maps graph interactions to bundling operations. Through use cases on movie reviews and an intelligence analysis scenario, we show how Semantic Bundling reveals new insights in document collections, and synthesize our findings into a discussion of emerging challenges and opportunities in knowledge graph sensemaking.
Read the full abstract

All authors Adam Coscia, Zeyu Hua, Eric Krokos, Timothy Lin, Alex Endert

Human-Computer Interaction

How Usable Are Geospatial Foundation Models? A Systematic Evaluation of 89 Models

Robin Young, Artyom Gabtraupov, Kenzy Soror et al. (+1 more)

Geospatial foundation models (GeoFMs) offer transformative potential for environmental monitoring, yet adoption among ecologists is uneven. Most evaluations are model-centric, focusing on architecture and benchmark accuracy, which overlooks whether the systems are usable by their intended audiences. To address this gap, we first conducted a pilot expert elicitation survey with ecology and conservation scientists that helped us identify misalignments between current GeoFM development priorities and their needs. Informed by these findings and based on HCI theory, we created a seven-dimension evaluation covering Access & Deployment, Interaction & Customization, Trust & Transparency, Community & Support, Scientific Permanence, Multilingual Support, and Offline Usability. Then, two raters applied this rubric to 89 GeoFMs. We found distinct accessibility gaps where nearly a third provide no support to practitioners beyond their source code. Dimensions along which ratings were highly consistent function as field-level diagnostics, revealing where there is room for improvement for current GeoFMs.
Read the full abstract

All authors Robin Young, Artyom Gabtraupov, Kenzy Soror, Srinivasan Keshav