Peer Revenue Fully funded

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Computer Science Human-Computer Interaction

cs.HC Watch Human-Computer Interaction

Monday 10 August

Human-Computer Interaction

VeriForge: Mitigating Latent Knowledge Gaps in Narrative Drafting via Mixed-Initiative Scaffolding

Ruqi Sun, Jiaping Li, Wenhui Tao et al. (+4 more)

Great fiction earns its verisimilitude through precise details, from how a longsword is gripped to pierce armor gaps to why a bleeding corpse cannot yet smell of decay, weaving domain expertise into the fabric of invented worlds. Current AI writing tools offer limited support for discovering and integrating unfamiliar domain knowledge into narrative. They require explicit queries that authors cannot formulate, generate finished prose that risks homogenizing voice, or assist only within the boundaries of what authors already know. We argue that AI should reveal latent knowledge gaps to writers while preserving their agency to transform discovered knowledge into authentic prose. Grounded in formative interviews with 9 fiction writers, we present VeriForge, a mixed-initiative writing system that divides cognitive labor so that the system assumes initiative over domain discovery while the author retains full initiative over narrative synthesis. VeriForge realizes this through three complementary mechanisms. Proactive inline highlighting flags potential knowledge gaps as authors draft. Dual-stream querying pairs conversational responses with source-anchored Knowledge Cards for direct fact extraction. A spatial Knowledge Canvas allows authors to organize and connect discovered knowledge across their writing. These mechanisms are powered by a graph-based retrieval-augmented generation pipeline grounded in domain-specific source materials. A within-subjects user study (N=12) provides preliminary evidence that this paradigm helps authors recognize previously overlooked knowledge gaps, supports creative exploration, and is perceived by expert raters to produce passages with stronger domain grounding in a controlled cold-start writing task.
Read the full abstract

All authors Ruqi Sun, Jiaping Li, Wenhui Tao, Ximing Zheng, Yuefeng Tan, Jiahao Wei, Yuxin Ma

Human-Computer Interaction · Cross-listed from cs.RO

Removing Infrastructure Barriers in Human-Robot Collaboration Through Wireless Reconfigurable Cells

Emma Takács, Mátyás Hajós, Ádám Juniki et al. (+7 more)

Human-Robot Collaboration (HRC) plays a vital role in dynamic, high mix, low volume industrial scenarios such as remanufacturing, which frequently face workcell rearrangements. Traditional setups are constrained by power and data cabling, restricting modularity and reconfigurations, while the selection of commercial wireless devices suitable for real-time perception and safe collaboration are limited in availability. This paper presents a highly flexible, wireless, 5G-based system that serves as a versatile experimental testbed for applications including remanufacturing, operator training, and user studies. To eliminate infrastructure barriers, the workcell integrates a novel battery-powered, multi-sensor platform prototype. Additionally, to support operator safety and system adaptability across environmental shifts, the system integrates a computer vision module for object detection and pose estimation, further augmented for robust hand recognition. Trained on synthetic and real data, the model reliably detects oriented grasping poses and human hands across varying lighting and background conditions (with an mAP@50-95 of 97.74 +- 0.10% and a mean inference time of 12.5 ms). Offloading these computationally intensive tasks to the edge via 5G, the proposed architecture contributes to resolving the bandwidth-latency trade-off. To demonstrate portability, the system was implemented in both Hungary and Norway, and was evaluated across a combination of public and private, Standalone and Non-Standalone 5G infrastructures. The performed network experiments produced results in round-trip response times down to 12 ms in case of compatible network-device pairings, suitable for safe, adaptive HRC. However, these measurements also revealed practical limitations related to interoperability in current 5G deployments that should be addressed in future works.
Read the full abstract

All authors Emma Takács, Mátyás Hajós, Ádám Juniki, Ádám Fischer, Zoltán Komáromi, Kristóf Abai, Dániel Horváth, Sándor Máthé, Konstantinos Kousias, Bence Tipary

Human-Computer Interaction · Cross-listed from cs.LG

LITEWAY: LIghtweight HAR via Temporal Efficient highWAY

Dominique Nshimyimana, Vitor Fortes Rey, Mengxi Liu et al. (+2 more)

Wearable human activity recognition (HAR) remains challenging due to the computational and energy constraints of deep learning models on resource-limited devices. Existing lightweight approaches often rely on recurrent architectures (e.g., GRU and LSTM), limiting parallelism and increasing inference latency. We propose LITEWAY, a modality-agnostic, fully convolutional framework for multichannel sensor time series that replaces recurrent temporal modeling with structured convolutional decomposition. LITEWAY combines lightweight convolutional blocks, strided temporal processing, and convolution-attention pooling to efficiently capture temporal dependencies while reducing computational complexity. We evaluate LITEWAY on 16 HAR datasets against TinyHAR, TinierHAR, and MLP-HAR. LITEWAY achieves competitive macro F1 while reducing model size by 4.06x-9.52x (Light) and 3.87x-9.07x (Full) compared with TinyHAR and TinierHAR. Deployment experiments further show energy reductions of 2.29x-3.14x (Light) and 1.46x-2.01x (Full) compared with TinierHAR and MLP-HAR, highlighting efficient fully convolutional temporal modeling for wearable HAR. The source code is publicly available at https://github.com/dominique-nshimyimana/liteway.
Read the full abstract

All authors Dominique Nshimyimana, Vitor Fortes Rey, Mengxi Liu, Bo Zhou, Paul Lukowicz

Human-Computer Interaction

Graphing the Everyday: A Neurosymbolic Approach to Eliciting Routines for Just-In-Time Adaptive Interventions

Shakyani Jayasiriwardene, Blake Mountford, Meican Ma et al. (+6 more)

Just-In-Time Adaptive Interventions (JITAIs) increasingly rely on conversational agents to elicit user routines, yet translating fluid human dialogue into rigid schedule data remains a significant challenge. We conducted a qualitative investigation of a neurosymbolic pipeline, combining Large Language Models (LLMs) with a Neo4j knowledge graph, to map unstructured verbal narratives into actionable interventions. Through human-centric evaluation using natural-language playbacks, we identified a critical "mental-model gap," where the linear extraction of LLMs clashes with hierarchical, non-linear human storytelling, causing severe entity fragmentation. Furthermore, we articulate an "ecological mismatch," demonstrating that algorithmic schedule availability frequently ignores the user's fluctuating psychological receptivity and physical energy levels. To resolve these tensions, we propose actionable design heuristics, including routine piggybacking, adaptive negotiation, and scalable transparency. Ultimately, these guidelines provide a foundational framework for evolving rigid schedule-trackers into empathetic, context-aware proactive agents capable of supporting long-term health behavior change.
Read the full abstract

All authors Shakyani Jayasiriwardene, Blake Mountford, Meican Ma, Niels van Berkel, Nicholas Koemel, Matthew Ahmadi, Jorge Goncalves, Emmanuel Stamatakis, Zhanna Sarsenbayeva

Human-Computer Interaction

Can Coding Agents Solve Repository-Level Issues with Rendered Code? An Exploratory Study of Visual Representations

Weijie Liang, Yuanfeng Song, Xing Chen et al. (+3 more)

Visual modality has recently been explored as a way to compress textual tokens, including rendering code as images for static code understanding. We study whether this representation can serve as operational context for agentic coding, where an agent must navigate repositories, edit source files, and verify executable patches. Using SWE-bench Verified, we evaluate rendered code in repository-level repair workflows and introduce controlled agent settings to separate unguided repository exploration from more structured repair stages. Our results show a mixed picture. Rendered code consistently reduces prompt-token cost, but the savings do not increase linearly with the nominal visual compression ratio. It largely preserves end-to-end repair accuracy, but does not overcome the performance limits of the underlying model or agent architecture, and can become unstable under aggressive compression. Further analysis suggests that visual code is most useful when raw source reading is a major bottleneck; once repository localization is structured, much of the remaining cost comes from patch–test trial-and-error, where visual compression has limited leverage. Overall, our study positions rendered code as a viable but conditional compression mechanism for realistic coding agents.
Read the full abstract

All authors Weijie Liang, Yuanfeng Song, Xing Chen, Caleb Chen Cao, Sirui Han, Yike Guo

Human-Computer Interaction

Intuitive Directional Sense Presentation to the Torso Using McKibben-Based Surface Haptic Sensation in Immersive Space

Kenta Yokoe, Tadayoshi Aoyama, Yuki Funabora et al. (+2 more)

In recent years, systems that utilize immersive space have been developed in various fields. Immersive spaces often contain considerable amounts of visual information; therefore, users often fail to obtain their desired information. Therefore, various methods have been developed to guide users toward haptic sensations. However, many of these methods have limitations in terms of the intuitive perception of haptic sensation and require practice for familiarization with haptic sensation. Fabric actuators are wearable haptic devices that combine fabric and McKibben artificial muscles to provide wearers with surface haptic sensation. These sensations can be provided to a wide area of the body with intuitive perception, instead of only to a part of the body. This paper presents a novel air pressure adjustment method for whole-body motion guidance using surface haptic sensations provided by a wearable fabric actuator. The proposed system can provide users with a directional sense without visual information in an immersive space. The effectiveness of the proposed system was evaluated through subject experiments and statistical data analysis. Finally, a directional sense presentation was conducted for users performing micromanipulations in a mixed-reality space to demonstrate the applicability of the proposed system for teleoperation.
Read the full abstract

All authors Kenta Yokoe, Tadayoshi Aoyama, Yuki Funabora, Masaru Takeuchi, Yasuhisa Hasegawa

Human-Computer Interaction

Intuitive Hand Positional Guidance Using McKibben-Based Surface Tactile Sensations to Shoulder and Elbow

Kenta Yokoe, Yuki Funabora and Tadayoshi Aoyama

Hand positional guidance with intuitive perception is crucial for enhancing user interaction and task performance in immersive environments. However, conventional hand positional guidance methods, relying on tactile sensations, lack intuitiveness. Consequently, users require instruction on the relationship between the tactile sensation and target position of the guidance before using these methods. Additionally, the user needs training to become familiar with tactile sensations. This study presents a hand positional guidance system with intuitive perception that leverages McKibben-based surface tactile sensations directed to the shoulder and elbow. We developed a wearable fabric actuator that provides McKibben-based surface tactile sensations to induce six specific movements: elbow flexion, extension, shoulder abduction, adduction, horizontal abduction, and horizontal adduction. The effectiveness of the actuator was experimentally validated, demonstrating its high accuracy in intuitively inducing six movements. An algorithm based on the equilibrium point hypothesis and Weber-Fechner law was implemented to regulate the intensity of the tactile sensations for hand positional guidance. Furthermore, the accuracy and speed of the proposed system were compared with that of conventional guidance methods utilizing synthesized speech and vibrotactile guidance.
Read the full abstract

Human-Computer Interaction

Diagnosing Performance in Invasion-Based Esports: The Esports Performance Screening (EPS) Framework

Michael Trotter, Frauke Kubischta, Matthew Watson et al. (+2 more)

Coaching in esports continues to become more common, yet conceptual tools for coaches to analyse performance breakdowns in esports remain limited. Existing approaches lack a way to distinguish between mental errors (i.e., mistakes relate to strategy and tactics), and physical slips (i.e., motor execution). This paper introduces the Esports Performance Screening (EPS) framework, that integrates several frameworks and models from sport and computing science which can be used to analyse mental and motor performance in esports. The EPS framework organises performance into five interconnected levels: strategy, tactics, tasks, actions and operations. Across these levels, teams pursue forms of superiority that influence transitions between stability and instability during invasion-based esports competition. The framework supports two complementary modes of screening use: diagnostic during reflective review and real-time screening during live play. Through illustrative esports scenarios, we demonstrate how instability can originate at different hierarchical layers and how misattribution of failure can obscure underlying causes. The purpose of the EPS framework is to help coaches systematically diagnose performance breakdowns, and act as heuristics to aid gameplay analysis during a match. The paper contributes a theoretically grounded, title-agnostic esports framework to support systematic performance analysis and pedagogical development in invasion-based esports.
Read the full abstract

All authors Michael Trotter, Frauke Kubischta, Matthew Watson, Bastian Hougaard, Hendrik Knoche

Human-Computer Interaction

CatPAL: Task-Aware Learning for Categorical Palette Recommendation

Chin Tseng, Arran Zeyu Wang, Yunqi Li et al. (+1 more)

Designing effective categorical palettes requires balancing a range of factors, including perceptual distinctiveness, category count, and task effectiveness. The effectiveness of categorical encodings can vary substantially depending on the target analytical tasks; however, existing recommendation tools largely ignore task context when evaluating palette quality, resulting in inconsistent performance across tasks. We synthesize findings from a series of multi-stage user studies into a unified model of task-based effectiveness for color encodings, shape encodings, and their redundant combination across category counts and seven common scatterplot tasks. Our results show that task and palette choice jointly influence perceptual accuracy: different color and shape palettes exhibit varying levels of robustness across tasks, indicating that palette effectiveness is task-dependent. We estimate task-specific perceptual strengths for 39 colors and 39 shapes using Bradley-Terry models, refined through adaptive sampling to target uncertain and task-sensitive comparisons. We further quantify cross-channel interactions using a redundant gain Delta G metric to model performance across color and shape pairings. We then train a predictive model that scores candidate palettes based on task, category count, and perceptual features. This model drives effective palette recommendations in CatPAL, a task-aware palette recommendation system grounded in empirical data responsive to user constraints. Our findings highlight the importance of selecting categorical palettes aligned with specific analytical tasks and demonstrate how task-aware modeling enables more reliable palette design. CatPAL translates empirical results into a practical tool that supports user-specified colors or shapes and returns ranked palette recommendations adaptable to a range of tasks.
Read the full abstract

All authors Chin Tseng, Arran Zeyu Wang, Yunqi Li, Danielle Albers Szafir

Human-Computer Interaction

ITO: Real-time Browser Tab Orchestration Through Intent Detection

Elvin Hu, Jasmine Sumpter, Takatoshi Yoshida et al. (+2 more)

People routinely interleave activities while browsing the web, often simultaneously and with overlapping boundaries. Yet organizational primitives in modern browsers treat every tab uniformly, offering no structural awareness of which items serve which purpose. While task-based organization approaches exist, they typically require users to manually organize or invoke reorganization features, and quickly fall out of sync as user intents evolve. To address this, we present Ito, a mixed-initiative approach that infers user intent in real time and organizes browsing activities into dynamic collections called Flows. Ito continuously reads unfolding user context, determines moment-to-moment focus, and creates, restructures, hibernates, and awakens Flows in real time while preserving user control through a mixed-initiative loop of proposals and corrections. In a controlled lab study (N=12) and a two-week field study (N=11), results suggest that Ito's intent-driven Flows aligned more closely with participants' mental models, facilitated task switching and navigation more effectively, and reduced management burden without loss of perceived control.
Read the full abstract

All authors Elvin Hu, Jasmine Sumpter, Takatoshi Yoshida, Mauricio Sousa, Kouta Minamizawa

Human-Computer Interaction · Cross-listed from cs.CL

When Confidence Fails: Overconfidence in LLMs under Uncertainty and Missing Clinical Information

Maryam Tahermazandarani, Adnan Mahmood, Fahmida Islam et al. (+1 more)

Large Language Models (LLMs) have achieved strong performance in medical question answering and clinical reasoning tasks. However, their reliability under uncertainty remains poorly understood which raises critical concerns for deployment in high-stakes clinical settings. In such environments, incorrect predictions are inherently risky, but confident incorrect predictions can be particularly harmful as they may mislead clinical decision-making. In this paper, we conduct a systematic behavioral analysis of LLMs under clinical information uncertainty. We propose an evaluation framework based on the MedMCQA dataset consisting of two complementary uncertainty settings. First, we introduce linguistic uncertainty cues through prompt modifications to simulate ambiguous clinical contexts. Second, we construct an answer removal setting, wherein the correct option is deliberately excluded mandating the model to recognize insufficient information and abstain. We analyze both model accuracy and confidence behavior using multiple calibration metrics including calibration gap, Expected Calibration Error (ECE), and Unsafe Confident Error Rate (UCER) across 500 medical questions. Our results reveal a consistent failure mode, i.e., although accuracy degrades under increasing uncertainty, model confidence remains misaligned with accuracy. This leads to a substantial increase in unsafe confident errors, indicating that model confidence remains largely insensitive to clinically meaningful information loss. Furthermore, we observe significant variation across models in their ability to abstain when the correct answer is unavailable, with some models persistently producing high confidence hallucinated answers. These findings expose critical limitations in the epistemic reliability of current LLMs and highlight the need for uncertainty aware evaluation methods prior to their deployment in clinical workflows.
Read the full abstract

All authors Maryam Tahermazandarani, Adnan Mahmood, Fahmida Islam, Quan Z. Sheng

Human-Computer Interaction

How People Evaluate AI-, Expert-, and Peer-Style Financial Advice

Aryan Ramchandra Kapadia, Eshwar Chandrasekharan and Koustuv Saha

As generative AI increasingly becomes a common source of daily decision-making, including financial choices, it is critical to understand how people evaluate AI-generated financial advice. We conducted a preregistered vignette experiment (N = 285) in which substantive financial content—including facts, numerical values, recommendation direction, and core reasoning—was held constant while communication style varied across AI Financial Assistant (AI), Certified Financial Planner (Expert), and Online Community Forum (OC) advice. Displayed source attribution was independently manipulated through correctly labeled, unlabeled, and mislabeled conditions, allowing us to separate attribution effects from source-specific communication cues. Expert advice was rated more favorably than AI advice on 9 of 10 outcomes (|d|=0.20–0.47), and this advantage remained visible without source labels, where Expert advice outperformed AI advice on 8 of 10 outcomes (up to d=0.60). Correct labels added limited differentiation, whereas mislabeling increased ratings of AI advice for situational fit and overall quality (d=0.42 for each) and attenuated the Expert advantage in situational fit (d=-0.36). Descriptive analyses further showed that AI advice was most responsive to displayed attribution and, conversely, that advice-style differences were most visible under an AI label. These findings show that financial-advice evaluations are shaped jointly by displayed attribution and message-level communication cues. We position disclosure not as a neutral transparency mechanism, but as an interpretive frame whose accuracy and interaction with message cues can shape trust and reliance.
Read the full abstract

Human-Computer Interaction

AI-Guided Learning: Research on Knowledge and Skill Acquisition Support Methods Using Deep Learning Audio-Video Processing Techniques

Kazuki Kawamura

Audio and video have become major learning media, but learners face two persistent challenges: the time cost of consuming long-form content sequentially and the lack of scalable feedback for imitation-based skill acquisition. This dissertation proposes an AI-guided learning framework that supports three interconnected stages: Consume, Understand, and Imitate. It develops and evaluates three systems. AIxSpeed dynamically adjusts audio playback speed at the phoneme level using speech-recognition-model confidence as a proxy for listening difficulty. FastPerson generates multimodal video summaries that preserve visual and auditory information and lets learners switch between summarized and full versions by chapter. Profy learns proficiency from largely unannotated speech data and visualizes classifier-relevant regions and model-derived acoustic distances to support pronunciation practice. Technical and user evaluations show that AIxSpeed achieved average playback factors of 1.30x on LibriSpeech and 1.29x on UME-ERJ and received higher mean opinion scores than matched constant-speed playback; FastPerson reduced viewing time by 53% with no statistically significant difference in quiz scores compared with normal playback; and Profy showed an observed improvement in pronunciation intelligibility, with non-overlapping pre- and post-practice confidence intervals. Together, these systems demonstrate how deep learning can support efficient content consumption, multimodal understanding, and repeated skill practice while retaining learner access to the original material.
Read the full abstract

Human-Computer Interaction

Generative Proxy: Synthesizing Proxy-Based Interfaces for Real-World Interaction Across AR Glasses

Xianhao Carton Liu, William Chastek, Eric J Gonzalez et al. (+2 more)

Interacting with real-world objects in AR is difficult, especially when targets are distant, cluttered, or occluded. These challenges are amplified on emerging lightweight AR glasses, which often lack binocular or large field of view on display, but also continuous inputs, such as hand or eye tracking. Proxy-based interfaces offer an alternative by allowing users to interact with virtual abstractions of physical objects that can be repositioned, reorganized, and adapted to the task and device. However, designing such interfaces is currently manual and highly device-specific. We present Generative Proxy, a method for automatically generating proxy-based interfaces from three specifications: scene, intent, and device capabilities. We formulate generation as a constrained synthesis problem that first produces valid interfaces for the target device and task, then ranks candidates using semantic and articulatory distance inspired by direct manipulation theory. We demonstrate Generative Proxy across diverse scenes, device profiles, and user intents. Expert evaluation shows initial evidence that generated proxy UIs are useful and usable, highlighting proxy-based abstraction as a promising interaction paradigm for future AR glasses.
Read the full abstract

All authors Xianhao Carton Liu, William Chastek, Eric J Gonzalez, Mar Gonzalez-Franco, Chen Zhu-Tian

Sunday 9 August

Human-Computer Interaction

EvoMaestro: Toward Interpretable and Steerable LLM-Driven Program Evolution

Feng Liang, Sizhe Cheng, Yikai Li et al. (+3 more)

Recent program evolution systems use large language models (LLMs) to generate and iteratively improve populations of programs, producing striking advances in mathematics, algorithm design, and scientific computing. Yet these systems largely operate as fully automated black boxes. As populations grow, domain experts must make sense of the scores, code changes, reasoning, and algorithmic ideas across many programs, while current interfaces provide limited support for understanding or redirecting the evolution. We characterize this need to understand and steer populations of evolving algorithmic ideas as semantic oversight. A formative study with 8 domain experts yields six design requirements for this emerging human-computer interaction problem. We then propose a steerable program evolution framework that lets expert judgments shape subsequent evolution. Built on this framework, EvoMaestro is an interactive visual analytics system that organizes evolution information from population overview to source code. It helps experts locate and compare noteworthy programs, guide evolution with natural language, combine promising ideas, and prune unproductive directions. A seven-day system demonstration illustrates how an expert applied these capabilities throughout a long-running evolution process. A within-subjects study with 12 participants shows that EvoMaestro improves users' understanding of evolution processes, reduces cognitive workload, and supports expert steering. These findings suggest that preserving expert agency in open-ended LLM-driven search requires support for both informed judgment and the ability to shape subsequent automation.
Read the full abstract

All authors Feng Liang, Sizhe Cheng, Yikai Li, Ruijie He, Xiaolin Wen, Yong Wang

Human-Computer Interaction · Cross-listed from cs.CY

Make it SewSimple: Navigating UK Curriculum and Classroom Practice in Secondary Computing Education with E-textiles

Yifan Feng, Hanlin Zhang, Yishan Du et al. (+3 more)

This paper explores the potential of integrating e-textiles as part of the approach to delivering computing in UK secondary schools. As one of the few UK-based exploratory studies of teachers experiences, it investigates how e-textile platforms such as the SewSimple maker kit and the BBC micro:bit can be incorporated into Key Stage 3 computing education (ages 11-14), taking into account both English national curriculum requirements and the realities of classroom practice. Our research question is: How do teachers perceive the potential of including e-textiles as part of computing education in English secondary schools? In summary, our research contributes to secondary computing education in three ways. First, we examine teachers direct, cross-disciplinarity experiences in two participatory design workshops using a newly designed e-textile platform and extend the limited discussion on supporting the BBC micro:bit in e-textile education. Second, we specifically identify opportunities, barriers, and challenges across three dimensions: school planning, national curriculum guidance, and practical e-textile implementation. Third, we offer insights into best practices for supporting maker technology adoption and pedagogical practices within existing institutional structures to maximize students' benefits for secondary school computing education.
Read the full abstract

All authors Yifan Feng, Hanlin Zhang, Yishan Du, Weihong Tang, Jennifer A. Rode, Bea Wohl

Human-Computer Interaction · Cross-listed from cs.CV

Can Webcam Gaze Constrain Mesa-Objectives in Driving Models? An Instrument Precision Analysis

Lennox Anderson, Ahmed Boutar, Jonah Mulcrone et al. (+1 more)

Current hazard detection systems in autonomous driving may develop mesa objectives, learned internal goals that achieve high training performance through spurious correlations rather than genuine hazard recognition. We investigate whether human gaze patterns, captured via webcam-based eye tracking (WebGazer.js), can serve as privileged information to constrain mesa-objective formation. We collected 137,663 frame-level gaze samples synchronized with hazard annotations across 388 real dashcam clips, then test this hypothesis across two calibration protocols (9-point/45-click and 11-point/440-click), two model architectures (Random Forest and causal Transformer), and five random seeds per experiment with paired t-tests. No experiment yields a statistically significant improvement from gaze (p = 0.919, 0.578, and 0.667 respectively). A geometric analysis reveals the root cause: WebGazer's reported error (~130-257 px depending on configuration) exceeds 93% of detected hazard object sizes (median 36 px), rendering object-level gaze attribution physically impossible at this instrument precision.
Read the full abstract

All authors Lennox Anderson, Ahmed Boutar, Jonah Mulcrone, Tal Erez

Human-Computer Interaction

Imagining the Musical Metaverse: a study on the needs and creative practices of composers and musicians

Alberto Boem, Matteo Tomasetti, Alessio Gabriele et al. (+3 more)

This study explores user needs for the Musical Metaverse (MM) through a series of workshops with electroacoustic composers, classical musicians, and music producers. Using a design approach based on the prompt "as if by magic," participants were invited to reflect on how the MM could impact their practice in composition, performance, and education domains. While groups maintain distinct priorities based on their roles, they share interests in educational applications and creative learning environments. Key themes include preference for mixed reality over purely virtual environments, virtual space as a creative paradigm, and tensions between democratizing tools and maintaining authentic musical experiences. Education emerged as the most promising initial use case, particularly for understanding complex musical concepts. However, participants expressed skepticism about fully virtual performances, emphasizing physical connections to instruments and spaces. Results suggest MM development should enhance rather than replace traditional practices while supporting embodied creativity.
Read the full abstract

All authors Alberto Boem, Matteo Tomasetti, Alessio Gabriele, Agostino Di Scipio, Alessandra Micalizzi, Luca Turchet

Human-Computer Interaction · Cross-listed from cs.RO

SHRIMP: Iterative Refinement of Robot Task Plans

Mya Schroder, Yuna Hwang, Callie Y. Kim et al. (+5 more)

As collaborative robots have entered domains such as manufacturing, agriculture, and healthcare, programming or adapting robot behavior typically requires robotic expertise that most end users lack. Natural language lowers this barrier. Recent advancements in large language models (LLMs) have made it feasible to translate natural language into robot task plans. However, language-based task specification suffers from semantic ambiguity, and generative models lack transparency for how language instructions become robot actions, making it difficult for users to validate the plan before execution. To address these issues, we introduce SHRIMP, a system that allows users to automatically generate a hierarchical robot primitive plan using natural language and iteratively revise their plan through re-prompting and explicit correction. At each revision, SHRIMP allows users to validate their plan in simulation, and once satisfied, execute it on the physical robot. Through a user study involving participants planning tabletop kitchen tasks (n=35), we validate that SHRIMP improves perceived control and enhances robot transparency. System videos and source code are available at https://wisc-hci.github.io/SHRIMP.
Read the full abstract

All authors Mya Schroder, Yuna Hwang, Callie Y. Kim, Leqian Cheng, Jeffrey Li-cheng Liu, Chenchen Zheng, Xinning He, Bilge Mutlu

Human-Computer Interaction

Epistemic Transfer in AI-Assisted Verification: A Framework and Evaluation Protocol

Christoph Trattner

AI tools can improve claim judgments while leaving open what users can do later without them. This paper develops an evaluation framework for epistemic transfer: the effect of prior AI-assisted verification on delayed judgments of novel claims under a specified access regime. The contribution is a verification-specific synthesis of learning, transfer, and human–AI evaluation, organized around two complementary estimands. The Epistemic Transfer Effect (ETE) compares delayed performance after alternative practice conditions. Tool-Removal Cost (TRC) compares immediate performance with and without assistance after practice; despite its name, it measures a current availability effect, not skill loss or psychological dependence. The proposed randomized protocol includes answer-first and evidence-first interfaces, active practice, a no-additional-practice comparator, and held-out claims. It specifies how to account for learning opportunities introduced by assessment, elicit confidence probabilities, average model predictions over a target population, and handle attrition and uncertainty. Reading ETE and TRC together distinguishes relative capability gains, equivalence, transfer penalties, and unresolved outcomes. A “verification-on-loan” profile is explicitly comparator-relative and cannot be inferred from a nonsignificant delayed contrast. A brief illustration from a two-wave verification study shows why these distinctions matter: an uncertain delayed interface contrast and an ordered assisted–unassisted probe cannot establish a clean transfer profile. The framework makes a practical demand: when independent judgment matters, evaluate both what assistance contributes now and what prior use changes later.
Read the full abstract

Human-Computer Interaction

Inductive Graph Layout with Implicit Neural Fields

Berfin Inal and Daniel Probst

A graph layout is normally a table of $N$ free coordinates. We optimise a function with a fixed number of parameters instead. This gives a drawing a sample complexity and an extensible domain. Force-directed algorithms remain the standard tools for graph drawing. The most accurate among them minimise stress in the Kamada-Kawai formulation by directly optimising the node coordinates, at a full objective cost of $O(N^2)$ in time and space. Here, we propose Fling (Field Layout via Implicit Neural Geometry), a small neural network mapping the distances of each node to a set of landmarks, positioning it in the plane by training on the layout energy. The full spring system then becomes tractable without its distance matrix, as rest lengths follow from a landmark bound in constant time per pair while a second network learns the majorisation sums from exact anchor rows, at $O(|\mathcal{A}|N)$ per step for $|\mathcal{A}|\ll N$ anchors. Unlike neural drawers that read the graph by message passing, we represent the drawing as a function of node features. An unseen node costs one forward pass, where sparse and low-rank majorisation remain transductive. As the unknowns are weights rather than coordinates, the energy only requires a small fraction of the nodes, and a field fitted that way outperforms PivotMDS, landmark MDS, and a kernel ridge trained on the same energy and features, when the task is fitting the energy of a graph from a sample of its nodes. In addition, the same parameterisation enables a stochastic pivot stress variant, an aesthetics-optimised variant carrying a neighbour-embedding energy with node-edge clearance and crossing terms on the same field, and conditioning on the weight between two energies gives a whole layout family from one run.
Read the full abstract

Human-Computer Interaction · Cross-listed from eess.SY

Preview-Based Relative-Motion Control of an Insertion Tool for Neural-Thread Placement in Pulsating Tissue

Yongyan Cao and Xiaobo Li

Flexible neural electrode threads must be placed at a prescribed depth while the cortical surface moves with cardiac and respiratory pulsation. A controller tracking a fixed point in the laboratory frame cannot distinguish commanded insertion from tissue motion; the error appears as both a depth offset and relative tip–tissue velocity during contact. This paper formulates thread insertion in tissue-relative coordinates: a harmonic observer predicts delayed cortical-surface motion over the control horizon, a constrained MPC regulates the tip relative to that prediction while limiting actuator effort and lateral relative velocity, and an augmented disturbance state removes the steady offset from persistent contact force and model mismatch. In a 1-DOF MuJoCo benchmark, the controller reaches RMS relative-placement errors of 12.0\um\ free-space and 1.9\um\ in contact, versus 18.3/176.8\um\ for delayed-feedback impedance and 286.1/275.5\um\ for laboratory-frame PD – the lower contact offset costs more peak contact force (3.43 vs.\ 2.00~mN), since it drives to commanded depth rather than yielding to tissue. A 3-DOF extension reduces lateral shear velocity from 1.34 to 0.50~mm/s at 2.1\um\ lateral placement error, and a feasibility-restoring soft-slack formulation keeps the shear constraint solvable under degraded sensing where a matched hard-constraint controller fails. A two-vertex Lyapunov certificate for the finite-horizon gain holds over $-40\%/{+}50\%$ reflected-mass mismatch, and the 1-DOF QP solves in under 0.4~ms at the 95th percentile. These results are a simulation-based control benchmark, not a clinical safety claim: the modeled tip is a rigid contact point, and flexible-thread mechanics, a validated force constraint, biological damage thresholds, and hardware-realistic sensing and timing remain necessary before deployment.
Read the full abstract

Human-Computer Interaction

Wearing Trust: How Older Adults Calibrate Reliance on Health Wearables Through Bodily Experience and Everyday Use

Yibo Meng, Bingyi Liu, ZhiMing Liu et al. (+1 more)

Older adults increasingly use health wearables, yet often cannot inspect the properties that matter for reliance. Through 31 semi-structured interviews in China, we examined how participants judged whether wearable outputs were reliable enough for everyday use. Participants relied on brand and price, visible interface activity, lived interaction experience, and comparison with bodily sensation. These cues supported conditional trust, but did not reveal sensor validity, data continuity, or failure conditions. We describe this mismatch as an observability gap and outline design directions for showing signal quality, reliability by context, human-system fit, and alert provenance.
Read the full abstract

All authors Yibo Meng, Bingyi Liu, ZhiMing Liu, Ruiqi Chen

Human-Computer Interaction · Cross-listed from cs.AI

Findings of the First Teaching Monster Challenge: A Benchmark of Pedagogical Content Knowledge in AI Agents

Yi-Cheng Lin, Yu-Kai Guo, Szu-Chi Chen et al. (+15 more)

AI agents can now solve problems, answer like subject experts, and generate long-form multimodal content. However, whether they can adapt a lesson to fit a specified learner, which education calls Pedagogical Content Knowledge (PCK), has not been benchmarked. To measure it, we introduce the Teaching Monster Challenge, the first instructional video generation benchmark to treat the learner persona as an explicit evaluation criterion. Each system is given a topic and a learner persona and must generate a complete instructional video. Every video is screened by an LLM-judge, ranked by crowd pairwise voting, and finalized by an expert panel. The first edition shows that today's systems handle the content well but are far weaker at presenting it and adapting it to the learner. The same process exposes a limit of automatic judging. The LLM-judge separates a clear low-performing tail but ranks the strongest systems poorly. The strongest systems receive nearly identical scores from the judge, so its ranking of them does not match human preference. Progress therefore requires not only better teaching systems but also better automatic judges, and we release the benchmark, rubric, and human judgments as a testbed for both.
Read the full abstract

All authors Yi-Cheng Lin, Yu-Kai Guo, Szu-Chi Chen, Bo-Han Feng, Yun-Man Hsu, Hsiang Hsieh, Yu-Jung Lin, Yue-Ling Wu, Jia-Kai Dong, An-Yu Cheng, Yu-Han Huang, Lok-Lam Ieong, Kuan-Yu Chen, Ming-Douo Tchouang, Shao-Hua Sun, Che Lin, Jian-Jiun Ding, Hung-yi Lee

Human-Computer Interaction

From Scaffolding to Internalization: Enhancing CPR Training with In-Situ Visualization and Kinesthetic Feedback

Jiahe Dong, Shuhao Zhang, Yutao Ming et al. (+3 more)

CPR training requires learners to not only understand explicit procedural targets, such as compression depth and rate, but also to internalize these targets as stable psychomotor skills. However, existing CPR training systems often rely on feedback presented outside the action space, which divides learners' attention between performing compressions and monitoring external guidance. This separation weakens the coupling between action and bodily sensation and may lead to an over-reliance on external feedback, compromising skill retention once support is removed. To address this challenge, we conducted a formative study with novice trainees and certified BLS instructors, from which we derived three design goals: embedding feedback within the task space, providing active kinesthetic guidance, and gradually fading assistance based on learning phases. Informed by these insights, we designed Kinesthetic-CPR, a stage-adaptive multimodal mixed reality CPR training system, and evaluated it in a controlled user study across two sub-studies (N = 60). This work offers design implications for CPR training systems that aim to better support skill retention.
Read the full abstract

All authors Jiahe Dong, Shuhao Zhang, Yutao Ming, Jinkai Zhang, Yurui Zhang, Quan Li

Human-Computer Interaction

FOX: Visual Exploration of Data Fact Outliers

Yikai Li and Yong Wang

Exploratory Data Analysis (EDA) systems extract and present data facts to summarize meaningful patterns such as trends and correlations for efficient dataset exploration. However, existing approaches rarely consider outlier detection at the level of data facts,and heterogeneous facts from different analytical scopes are often aggregated in a single view, making it difficult to define meaningful metrics and effectively analyze data fact outliers. To fill this gap, we present FOX, a novel visual analytics system for interactive data Fact Outlier eXploration. FOX organizes data facts into groups with consistent analytical scopes and computes a unified outlier score that combines distribution-based and pattern-based components. Its interface comprises an Upload Panel for data preparation and two coordinated exploration panels: the Overview Panel employs a matrix-based visualization to enable an intuitive overview of all data facts, and the Main Panel provides four linked views for cluster-level and fact-level analysis. We evaluated the usability and effectiveness of the system through two usage scenarios on public datasets and in-depth interviews with 12 participants. The results show that FOX enables meaningful detection, analysis, and explanation of data fact outliers.
Read the full abstract

Human-Computer Interaction

A Dynamic-Semantics Framework for Grounding Human Referring Expressions in Visual Perceptual Data

Joseph Bingham

Humans converge on shared names for novel, hard-to-describe objects through repeated interaction, a process psycholinguists call lexical entrainment. Leading vision-language models fail at this: recent empirical work documents that they do not shorten references, reuse successful expressions, or maintain stable pact state across turns. We present a framework that addresses the gap by externalizing pact state into three explicit, inspectable sets of referent-object bindings ($\Gamma, \Xi, \Omega$), updated by a dynamic-semantics context-change rule. The symbolic layer sits on top of a lightweight perceptual-alignment pipeline that grounds noisy human referring expressions in crowd-sourced imagery via SIFT homographies and the Universal Quality Index. Evaluated on the Stanford Repeated Reference Game corpus (over 15{,}000 director-matcher utterances on abstract tangram stimuli), the framework places the correct target in its top-5 hypothesis set 83.56% of the time from a single director utterance. Human matcher top-1 accuracy on the same corpus is approximately 77-80%. We also report results on a held-out condition in which obvious tangram-adjacent images are excluded from the retrieved set, which provides a more conservative measurement of the grounding signal. Ablations isolate the contribution of each component: SIFT alignment, UQI, query preprocessing, and image augmentation. The central contribution is the combination: a transparent, auditable symbolic layer that recovers the structure of lexical entrainment turn by turn, paired with a perceptual channel whose behavior can be examined ablation by ablation. We also discuss in detail what the framework does not do. It is not interactive, it does not close the loop with the director, and its retrieval-driven perceptual channel is vulnerable to a class of leakage effects that we quantify and bound rather than wave away.
Read the full abstract

Human-Computer Interaction

Charting Public Health: A Taxonomic Study of Visualization Practices in the Public Health Field

Mia Hines and Alvitta Ottley

Public health organizations regularly produce and publish data visualizations to raise awareness of critical issues, influence decision-making processes, and promote overall well-being. However, the design practices shaping these visualizations in real-world settings remain largely unexamined, limiting the research community's ability to evaluate their effectiveness, accessibility, and alignment with communication goals. To address this gap, we construct and analyze a large-scale corpus of over 4,000 real-world data visualizations drawn from more than two dozen websites associated with U.S. and international public health organizations. We evaluate salient design characteristics like chart type, visualization accessibility, use of embellishments like iconography, and design flaws. This work contributes to understanding real-world decisions in designing data visualizations and supports public health officials in improving data visualization-related communications. Visualizations in our finalized corpus and the labeled dataset can be found at https://github.com/washuvis/ChartingPublicHealth.
Read the full abstract

Human-Computer Interaction · Cross-listed from cs.AI

Walking through Discussions: A Mobile Visual Analytics System for In-Situ Group Discussion Analysis

Yiping Sun, Ziyao Kang, Wei Zeng et al. (+2 more)

Group discussion-based teaching is widely used to foster collaborative learning, yet teachers in physical classrooms often struggle to simultaneously monitor multiple groups and quickly diagnose a target group before intervening. Existing visual analytics tools primarily support post-hoc analysis on desktop, providing limited support for in-situ walk-around teaching. To address this gap, we present MobileGroupVis, a mobile visual analytics system for in-situ analysis of classroom group discussions. MobileGroupVis integrates multi-group monitoring, single-group diagnosis, and instructional intervention into a concise analytical workflow tailored for small-screen touch interaction. The system is powered by a lightweight streaming analysis pipeline that converts group audio into structured discussion data and further extracts interaction patterns, topic progression, and topic deviation through a dialogue analysis module. To enable both glanceable overview and traceable diagnosis, we design six coordinated views, including a compact glyph that visually encodes word count, interaction intensity, and topic deviation for efficient cross-group comparison and anomaly localization, along with detailed views for opinion evolution, interaction dynamics, topic coverage, and dialogue records. We evaluate MobileGroupVis through two case studies and expert interviews. The results provide preliminary evidence that MobileGroupVis supports teachers in understanding discussion processes, identifying groups in need of attention, and facilitating in-class intervention.
Read the full abstract

All authors Yiping Sun, Ziyao Kang, Wei Zeng, Minli Wu, Jiazhi Xia

Human-Computer Interaction

TeachUp: Facilitating Early-Stage Teachers to Learn Instructional Strategies from Classroom Videos with Reflective Support

Haoxiang Fan, Ding Lei, Zaihong Zheng et al. (+4 more)

Recorded videos of offline open classes provide good examples for early-stage teachers to learn instructional strategies, e.g., how to organize cooperative learning. However, learning by watching these videos is challenging, as these strategies are implicitly performed, and it lacks in-situ reflective support. In this paper, via a formative study (N=9), we design TeachUp to support the learning of instructional strategies from classroom teaching videos. TeachUp adopts an LLM-powered pipeline to detect nine instructional strategies in videos (precision = 63.4%), provides reflective questions and hints while watching, and generates customized practices with reflective feedback. A within-subjects study (N=16) shows that compared to a traditional video-playing and self-practicing baseline, early-stage teachers with TeachUp are more engaged in learning and perform better in applying learned strategies to new tasks. Interviews with four in-service teachers further generalize our findings and TeachUp's use cases. We discuss practical implications for fostering video-based learning of instructional strategies.
Read the full abstract

All authors Haoxiang Fan, Ding Lei, Zaihong Zheng, Jiale Li, Jionghao Lin, Jian Yin, Zhenhui Peng