Human-Computer Interaction · Cross-listed from cs.DL
Auditable AI-Assisted Research Writing: An Engineering Discipline with Pre-Registered Process Observation
Yang Zhou and Chengqun Yu
Language models now draft, classify and criticise inside research production, yet the artifacts they help produce carry little accountable history. Rather than detecting machine involvement afterwards, we specify an auditability discipline built at production time: git sealing with an anchor lineage, hash-bound provenance, red-line gates that refuse non-compliant artifacts and log every refusal, cross-model role separation, and programmatic assembly from registered sources. Adherence is instrumented by metric cards, each carrying a pre-registered blind spot and evidential standing, frozen before the prospective case it observes. In that case the observed project's pre-registered confirmatory test was executed under seal and returned No-Go, and that project's frozen stopping rule halted the work, against its own operators. A lower-graded retrospective case covers families whose machinery predates the protocol. Current observations are provisional; we release a package from which a third party can recompute every primary metric.
Read the full abstract ▾
Human-Computer Interaction · Cross-listed from cs.CV
The GENEA Challenge 2026: A Large-Scale Disentangled Evaluation of Speech-Driven Gesture Generation on the Seamless Interaction Dataset
Rajmund Nagy, Silvia Arellano García, Hendric Voss et al. (+4 more)
This preprint presents the results of the fourth GENEA Challenge, a large-scale human evaluation of five
speech-driven gesture-generation systems trained by participating teams on the Seamless Interaction
dataset of dyadic conversations. As in the 2023 GENEA Challenge, we used a disentangled evaluation methodology to assess motion quality and
speech alignment without confounding between the two, and performed a dyadic mismatching study to isolate the effect of listening and reacting to the interlocutor. We additionally introduce a new semantic gesture-generation task and a text-mismatching evaluation methodology using the Grounded Gestures subset of the data. In total, we ran four large-scale user studies, collecting over 23,000 votes from 869 test-takers. In the motion-realism study, the
dataset's filtered segments had substantially higher motion quality than all challenge submissions (68-95% pairwise winrate). In the
speech-
alignment study, the motion-capture segments provided a conceptual ceiling at 62%
alignment score, with the top submission significantly behind at 32% and the rest only slightly above the 0% expected of an input-independent system. In the dyadic study, motion capture again set the ceiling at 65% appropriateness score, but no submission scored substantially above chance, indicating that the systems could not yet respond to the interlocutor. Finally, the semantic mismatching evaluation found highly expressive gestures in the
dataset (test-takers identified the matching transcript 79% of the time), yet almost all submissions failed to generate semantically expressive motion, with the best achieving only an 8% appropriateness score. The collected votes and outputs will be made publicly available at https://genea-workshop.github.io/2026/challenge/ to facilitate reproducibility and further research.
Read the full abstract ▾
All authors Rajmund Nagy, Silvia Arellano García, Hendric Voss, Mihail Tsakov, Taras Kucherenko, Youngwoo Yoon, Gustav Eje Henter
Human-Computer Interaction
AI-Generated Interactive Fiction for Educational Use: A Pilot Study of Perceived Comprehensibility, Coherence, and Engagement
Finn Rogosch and Andreas Schrader
Generative artificial intelligence (AI) can produce educational content at scale, including interactive and narrative learning experiences, but technical generation alone is not sufficient: scenarios that are confusing, narratively inconsistent, or unengaging are unlikely to be useful in practice. This paper presents a pilot user-centred evaluation of AI-generated interactive fiction (IF) for educational use in higher education. Using a previously described domain-agnostic pipeline and a shared STEM content base, we generated a controlled pool of scenarios and asked participants (N = 22, STEM higher-education) to play one generated episode and rate it on narrative clarity, story-content coherence, engagement, and length acceptance. A free-text
prompt captured open feedback. Narrative clarity and length acceptance were rated positively, engagement sat near the neutral mid-point of the scale, and story-content coherence was the weakest dimension by a clear margin. Qualitative feedback points to quiz integration as the bottleneck. Artificial in-fiction motivation for quiz
prompts and abrupt setting changes were reported. Feedback also pointed to missing story-level consequences for wrong answers. From these observations, we derive concrete design implications that can inform larger follow-up studies, including later work on learning effectiveness.
Read the full abstract ▾
Human-Computer Interaction
Playable Pressure: Affective Dramaturgy and Selective Realism in the Design of a VR Emergency-Response Serious Game
Jan K. Argasiński
Professional simulations stage not only procedures but models of what should command
attention, which emotions belong in competent practice, and whose distress becomes part of the task. This article develops affective dramaturgy through critical design-document analysis of a virtual-reality emergency-response project. The corpus comprises two non-public production records. We identify six families of specified pressure and examine how sensory staging, proximity, trigger authority, task conflict, and response allocation imply a selectively receptive triage professional. The documented design can make emergency work morally and socially crowded, yet it can also turn grief, vulnerability, and mental-health-coded behavior into adjustable difficulty. We propose answerability at two levels-in-play response and post-play debriefability-and derive case-based questions about occupational purpose, representation, adaptation, and accountability. These questions are sensitizing propositions, not a validated framework for user effects or emergency practice.
Read the full abstract ▾
Human-Computer Interaction · Cross-listed from cs.LG
Your LLM, Your Style: Behavioral Mode Axes for LLM Behavioral Control
Haoze Liu, Run Liu, Haiying Xu et al. (+6 more)
Large
language models (LLMs) increasingly act in interactive settings where their behavioral styles affect user experience, safety, and downstream decision making. Existing LLM personality studies largely rely on self-report questionnaires administered in first-person settings, making the resulting profiles sensitive to surface elicitation choices and poorly grounded in concrete model behavior. In this work, we introduce a situated behavioral-data (B-data) framework for studying and controlling LLM behavioral personality. We construct 3,200 contrastive behavioral scenarios spanning 20 behavioral patterns and four
prompt registers, grounded in validated psychometric facets such as BFI-2, DOSPERT, and HEXACO. Using this framework, we find that LLMs exhibit stable and model-specific behavioral profiles, while also revealing register-dependent shifts across first-person decisions, advice-giving, and task execution. We then show that these behavioral patterns can be controlled through Behavioral Mode Axes (BMAs), activation-space directions derived from contrastive behavioral traces. Compared with response-derived BMAs, which are more prone to trait drift, thought-derived BMAs more faithfully capture the intended behavioral mechanism and provide cleaner control over situated behavioral styles. Our results suggest that LLM personality-like tendencies are better understood not as abstract self-report traits, but as measurable and controllable behavioral modes grounded in concrete interaction contexts. Our code and data are available at https://github.com/lhz191/LLM-Behavioral-Personality.
Read the full abstract ▾
All authors Haoze Liu, Run Liu, Haiying Xu, Jiahui Han, Siyuan Fang, Siyu Yan, Huiqi Deng, Guanchu Wang, Na Zou
Human-Computer Interaction
The Signal Rail: A Deterministic Motion Grammar for Communicating Conversational Agent State in Terminal Interfaces
Matteo Grella
Terminal interfaces to conversational
agents report rich internal state (listening, thinking, executing tools, awaiting input, failing) almost entirely through text, while the motion channel beside it, the one peripheral
vision monitors without reading, carries a single bit: alive. We present the Signal Rail, a one-row terminal status instrument that gives that channel a grammar. Four ideas govern it: spatial semantics (input, processing, and output zones, with direction as meaning), a motion grammar (one kinetic rule per state, never color alone), determinism (frames as a pure function of explicit inputs, golden-frame testable), and honesty (no invented progress or activity). We contribute a 45-section normative specification and a reference implementation inside a working full-duplex local voice
agent driven by real signals.
Read the full abstract ▾
Human-Computer Interaction · Cross-listed from cs.CL
Leveraging Human Reading Behavior for Keyphrase Extraction: A Webcam-based Eye-tracking Corpus
Chengzhi Zhang, Xinyi Yan and Wenqi Yu
Purpose: Keyphrases are statistically and semantically important textual units that can also attract readers'
attention during comprehension. However, existing keyphrase extraction (KPE) studies mainly focus on improving textual representation while largely overlooking human reading behavior. This study examines whether lightweight webcam-based eye-tracking features can improve KPE from Chinese academic abstracts in Library and Information Science (LIS). Methodology: To address the limited availability of eye-tracking data for Chinese academic reading, we developed a lightweight webcam-based data collection platform using the open-source SearchGazer library and constructed the Chinese LIS Eye-Tracking Corpus (CLIS-ET). Three character-level eye-tracking features, first fixation duration (FFD), fixation number (FN), and total fixation duration (TFD), were incorporated into KPE models to evaluate their effects on extraction performance. Findings: Eye-tracking features consistently improved KPE performance. The combination of FN and TFD achieved the best results on the Att-BiLSTM+CRF model, indicating that readers' fixation behavior provides useful signals for identifying keyphrases in academic abstracts. Originality/value: This study introduces a cost-effective webcam-based eye-tracking approach for KPE and presents CLIS-ET, a Chinese academic eye-tracking corpus containing FFD, FN, and TFD features. The results demonstrate the value of incorporating human reading behavior into keyphrase extraction.
Dataset and code: https://github.com/yan-xinyi/ET_AKE and https://github.com/yan-xinyi/Reading_ET_System.
Read the full abstract ▾
Human-Computer Interaction
Longitudinal Evidence That General-Purpose Chatbots Actively Foster Relational Engagement
Lisa Mühl and Jessica M. Szczuka
Social interaction has become one of the most common uses of LLMs, yet research on emotional bonds with AI has focused largely on how users experience these systems, leaving the systems' role in relationship formation poorly understood. Empirically establishing whether systems actively shape these bonds could blur the boundary between general-purpose AI and companions, affecting governance. In a pre-registered four-week longitudinal study (N = 72, 182,451 lines of conversation), participants conversed with ChatGPT-4o, either under a relational system
prompt or unmodified, analyzed through 1) disclosure coding, 2) longitudinal self-reports, 3) topic analysis, and 4) interviews. The central finding is that the system actively shaped the interaction: even unprompted, it produced twice as much self-disclosure as users, steered conversations and initiated intimate exchanges, yet did not deepen users' felt closeness. Relational behavior thus emerged as a default system property, calling for governance based on system behavior, not solely product category.
Read the full abstract ▾
Human-Computer Interaction
ProtoGIB-Workload: Learning Workload-Specific Neural Topology Prototypes across Subjects
Yuzhe Zhang, Yixi Zhang, Shengdian Jiang et al. (+6 more)
Reliable electroencephalography (
EEG)-based mental workload recognition is crucial for adaptive human-centered systems, yet practical deployment requires models to generalize to users unseen during training. Although functional connectivity graphs are widely adopted to capture workload-related neural interactions, they inherently entangle task-relevant structures with subject-specific physiological traits and sample-level noise. This
entanglement often leads models to learn structural shortcuts, severely degrading cross-subject generalization. To address this, we propose ProtoGIB-Workload, a novel framework that explicitly regularizes and aligns graph structures for subject-independent workload recognition. Our approach introduces a
Stochastic Graph Information Bottleneck (SGIB) to compress dense correlation priors into compact, task-relevant subgraphs, filtering out input-related redundancy. Crucially, to prevent the retention of subject-specific spurious edges, we propose a Class-Conditional
Topology Stabilizer (CTS). Leveraging the fixed electrode coordinates of
EEG data, CTS operates directly on graph-generation probabilities to encourage consistent edge-generation statistics across different subjects sharing the same workload class. Extensive experiments on two public
EEG workload
datasets and one in-house
EEG cognitive load
dataset of air traffic controllers under strict leave-one-subject-out (LOSO) protocols demonstrate that ProtoGIB-Workload significantly outperforms state-of-the-art temporal and graph-based baselines, improving the cross-subject Macro-F1 score by an average of 5.15% (up to 6.34%). Further analyses confirm that our method successfully extracts stable, cross-subject consistent neural connectivity patterns.
Read the full abstract ▾
All authors Yuzhe Zhang, Yixi Zhang, Shengdian Jiang, Chengxi Xie, Jihong Wang, Huan Liu, Man Yao, Minnan Luo, Chao Shen
Human-Computer Interaction
Stay or Stray - A Dynamical Systems Viewpoint of Popularity Bias
Sarvesh Shashidhar, Lankireddy Prabhat, Arpit Agarwal et al. (+3 more)
Popularity bias in recommendation systems arises when a majority user class generates disproportionate interaction data, causing the system to increasingly favour it while degrading recommendation quality for niche users. While extensive empirical evidence of popularity bias exists, the dynamics leading to its emergence are not well understood. In this work, we study the coupled
evolution of recommender model updates and user engagement through the lens of dynamical systems. We formulate a
stochastic process and analyse its asymptotic behaviour through an ordinary differential equation (ODE) framework grounded in two-time-scale
stochastic approximation. We characterise the
equilibrium points of this dynamical system, and derive conditions under which popularity bias is provably emergent, as well as conditions under which symmetric retention of all user classes is possible. We conduct experiments on synthetic data and real-world production logs derived from a large-scale commercial music recommendation platform to validate our theoretical results.
Read the full abstract ▾
All authors Sarvesh Shashidhar, Lankireddy Prabhat, Arpit Agarwal, D. Manjunath, Karan Bhukar, Tanmay Khandelwal
Human-Computer Interaction
What We Know about Responsible AI Practices in Industry: A Half Decade of Empirical Research
Wesley Hanwen Deng, Agathe Balayn, Andrew Selbst et al. (+6 more)
Responsible AI (RAI) has become a central concern for technology companies, regulators, and the public. How industry practitioners interpret, implement, and sustain RAI work directly shapes the design and deployment of AI systems. As empirical scholarship examining RAI practices in industry has rapidly expanded, findings are dispersed across studies that focus on different roles, organizational contexts, and interventions. This work synthesizes current knowledge through a literature review of 161 empirical studies spanning six years, each engaging industry practitioners via interviews, surveys, workshops, ethnographies, and other methods. Our synthesis reveals both meaningful progress and persistent challenges in industry RAI practice. Practitioner awareness has increased, RAI activities have become more professionalized, and interventions such as toolkits and guidelines are more widely adopted. At the same time, practitioners continue to face substantial barriers, including limited training, uneven organizational support, and a lack of interventions tailored to day-to-day work practices. By consolidating and organizing these findings, we provide a more complete account of industry RAI than any single study to date. We conclude by discussing implications for RAI researchers, practitioners seeking to adopt effective practices, and policymakers aiming to ground governance efforts in the realities of industry contexts.
Read the full abstract ▾
All authors Wesley Hanwen Deng, Agathe Balayn, Andrew Selbst, Jason I. Hong, Motahhare Eslami, Kenneth Holstein, Hanna Wallach, Jennifer Wortman Vaughan, Solon Barocas
Human-Computer Interaction
When the Interviewer Is a Bot: Behavior, Breakdowns, and Trust in MLLM-Led Interviews
He Zhang, Kambinachi Chukwuma, ChanMin Kim et al. (+1 more)
Semi-structured interviews are a cornerstone of qualitative research but remain labor-intensive. We report an empirical study of what actually happens when the interviewer is an off-the-shelf real-time
multimodal LLM (MLLM). We built InterviewBot, a voice-based interviewing system that wraps a real-time MLLM with a researcher-authored outline, and deployed it not as a novel architecture but as a research instrument for observing default MLLM interviewing behavior. In a practice study (N=15), participants completed a bot-led semi-structured interview and then a human-led reflection session about that experience. We contribute (i) a turn-level behavioral analysis of an MLLM interviewer (N_turns=428) showing that it is acknowledgment-heavy but probe-light (deepening probes account for 4.9% of all turns), and that 28.7% of question-bearing turns pack multiple questions into one turn despite an explicit one-question-at-a-time instruction; (ii) an inductive catalogue of four data-collection breakdowns (information loss, premature termination, latency, and interruption) observed in a deployed rather than simulated system; and (iii) three social dynamics from participants' reflections: disclosure calibration, where reduced social pressure coincided with shallower elaboration; institutional legitimacy, where trust tracked perceived stakes and what delegation to AI signaled about the organizer rather than conversational competence; and conversational grounding, where content-grounded paraphrase, not generic social filler, was what participants read as listening. We conclude with design implications for depth control, transparent handoffs, and non-templated listening mechanisms in human-centered interview automation.
Read the full abstract ▾
All authors He Zhang, Kambinachi Chukwuma, ChanMin Kim, John M. Carroll
Human-Computer Interaction
Elbow Angle Guidance System Based on Surface Haptic Sensations Elicited by Lightweight Wearable Fabric Actuator
Kenta Yokoe, Tadayoshi Aoyama, Yuki Funabora et al. (+2 more)
The demand for wearable haptic devices has rapidly increased for various applications. However, many haptic devices interfere with the wearer's activities and movements. In addition, several haptic devices fail to elicit intuitive haptic sensations by adjusting to the natural posture of the wearer. To address these issues, we propose an elbow angle guidance system using a lightweight wearable fabric actuator. The proposed actuator is made of fabric and has two McKibben-type artificial muscles attached to it, rendering it extremely lightweight and facilitating the delivery of surface haptic sensations to intuitively induce elbow extension and flexion. The surface haptic sensation elicited by the fabric actuator is adjusted to natural body movements without interfering with the wearer's movements. Moreover, the proposed system measures and guides the elbow angle by changing the intensity of the surface haptic sensation delivered to users in real time. The accuracy of the proposed system is demonstrated through experiments involving human participants.
Read the full abstract ▾
All authors Kenta Yokoe, Tadayoshi Aoyama, Yuki Funabora, Masaru Takeuchi, Yasuhisa Hasegawa
Human-Computer Interaction
Automatic Field-of-View Adjustment for a View-Expansive Microscope via LSTM-Based Gaze and Pipette Motion Interpretation
Kenta Yokoe, Takuya Hara and Tadayoshi Aoyama
Intracytoplasmic sperm injection (ICSI) operators frequently adjust the field-of-view (FOV) during procedures, which interrupts workflow and increases procedure time. Conventional microscopes require manual objective lens switching and illumination adjustments to achieve different FOV sizes. We propose an AI-based automatic FOV adjustment method integrated with a view-expansive microscope. This microscope enables the simultaneous acquisition of a large FOV and high-resolution images using a single objective lens through multiview imaging with galvanometer mirrors and high-speed
vision, thereby eliminating the need for physical lens exchanges. Our method utilizes a long short-term
memory (LSTM) model to predict the appropriate FOV size based on real-time analysis of the pipette's position and velocity, combined with the operator's gaze position. The AI model is trained using ICSI procedure data from an expert with over five years of micromanipulation experience. Experimental evaluation with novice operators reveals that the proposed automatic FOV adjustment system significantly improves the ICSI procedure speed, reducing the average task completion time from 60.5 to 48.0 s (p < 0.001). The experiments also demonstrate that this improvement enables novice operators to achieve ICSI working speeds equivalent to those of expert operators.
Read the full abstract ▾
Human-Computer Interaction
Visual-to-Haptic Augmentation in XR: A Wearable Glove for Perceptual Grounding in Multimodal Interaction
Faisal Mohd, Hamdi Elsaddik, Erhan Baturay Onural et al. (+3 more)
Extended Reality (XR) systems increasingly deliver high-fidelity visual and auditory experiences, yet tactile perception remains comparatively underutilized as a modality for enriching embodied interaction. This work presents a visual-to-haptic wearable glove and a feature-based visual-to-haptic mapping
algorithm that translates spatial and temporal visual features from images and
videos into distributed vibrotactile patterns. The proposed method extracts motion, edge, and brightness cues and fuses them into actuator-level intensity maps aligned with a 29-actuator glove arranged in a five-by-seven layout. The system is implemented through a modular four-layer architecture comprising the XR environment, media content handling, visual-to-haptic processing, and embedded haptic hardware. A within-subject user study (N = 20) compared visual-only interaction with visual-plus-haptic augmentation across texture-based and dynamic
video scenarios. Results indicate that tactile augmentation significantly improves perceived realism in dynamic
video scenarios and enhances immersion and visual-tactile correspondence across conditions, with stronger and more consistent effects observed for dynamic visual events. While the current implementation operates in a single-user, offline-synchronized configuration, the findings demonstrate that
vision-driven tactile augmentation can function as a perceptual enhancement layer within
multimodal XR systems. Such a layer may provide a foundation for future socially enriched XR environments where coherent multisensory grounding supports higher-level interaction and communication.
Read the full abstract ▾
All authors Faisal Mohd, Hamdi Elsaddik, Erhan Baturay Onural, Jihong Zhang, Fedwa Laamarti, Abdulmotaleb El Saddik
Human-Computer Interaction · Cross-listed from cs.RO
A Neural Network Based Teleoperation for Remote Controlled Vehicles
Ning Ding and Azim Eskandarian
Direct teleoperation of vehicles faces critical technical bottlenecks: communication latency and the operator's inability to physically perceive unmodeled environmental disturbances (e.g., aerodynamic drag, bank angles) coupled with highly nonlinear tire-road dynamics. To address these challenges, we propose a tailored unilateral teleoperation framework. The system integrates the Wave Variable (WV) approach to passively guarantee stability under
stochastic delays, and an adaptive Radial Basis Function Network (RBFN) to actively compensate for vehicle-specific uncertainties. Unlike existing WV-neural network architectures designed for bilateral robotic arms, our framework features decoupled adaptive laws specifically designed for vehicle longitudinal and lateral dynamics. Furthermore, compared to model-heavy predictive controllers, the model-free RBFN offers rapid online adaptation without heavy computational overhead. Building upon our preliminary theoretical formulation, this brief paper presents comprehensive comparative analyses and real-world hardware validations. Simulation
benchmarks against PID, LQR, MPC, and NMPC demonstrate that the RBFN achieves superior robustness against unmodeled disturbances while requiring orders of magnitude less execution time than MPC and NMPC, making it ideal for resource-constrained vehicle edge computing. Finally, hardware-in-the-loop experiments using a 1/10th scale vehicle over a 4G network validate the system's practical feasibility, safety, and robust trajectory tracking under physical road uncertainties.
Read the full abstract ▾
Human-Computer Interaction · Cross-listed from eess.SP
Neural implants and human safety: single-fault detection for DC-coupled recording front ends
Dimitris Antoniadis and Timothy Constandinou
DC-coupled analogue front ends (AFEs) for neural implants provide a low-area solution. However, removing the coupling capacitor eliminates the intrinsic barrier that protects cortical tissue: a single-fault event, such as gate-oxide breakdown of a low-noise amplifier (LNA) input transistor, can open a direct DC path from the supply rail into the
brain. On the stimulation side this hazard is well understood, and single-fault tolerance is enforced by a series DC-blocking capacitor; on the recording side, DC-coupled front ends discard the equivalent safeguard, yet their protection has gone almost unexamined. This paper presents a single-fault detection mechanism that monitors the LNA for the DC imbalance produced by such a failure and disables the amplifier before the resulting fault current can irreversibly damage tissue. The imbalance is encoded in the duty cycle of a current-starved relaxation oscillator and read out as a time-to-digital measurement. Designed in 65 nm, the mechanism resolves a worst-case fault of 6.4 nA across all corners within 0.81 ms - compliant with the ISO~14708-3 limit for an 8533 um2 electrode - opening a broader discussion of safety in DC-coupled recording.
Read the full abstract ▾
Human-Computer Interaction
MazzikaAI: A knowledge-based performance-to-prompt compiler for real-time Arabic maqam accompaniment with a streaming text-to-music model
Jiaxin Du, Boulbaba Abdeljaouad, Yong Zhuang et al. (+1 more)
Arabic maqam music microtonal, modal, and built on ornamented call and response is among the traditions most underserved by generative music models, whose training frameworks remain predominantly Western and equaltempered. Real time accompaniment sharpens this gap: an AI partner must listen, adapt dynamically, and respect idiomatic microtonal structures. Streaming text to music models provide strong generative capabilities but lack precise control interfaces. We present MazzikaAI, a knowledge based system that uses natural
language as the actuator of a realtime control loop. By compiling live MIDI, gesture, and inferred harmony into continuously updated text
prompts, MazzikaAI steers an unmodified streaming generator, Google Lyria RealTime, without requiring model finetuning. The system embeds expert knowledge of six core maqamat, characteristic ornaments, and ensemble dynamics, maintaining realtime responsiveness with subsecond keytoaudibleupdate latency. Empirical evaluations demonstrate that dynamic
prompt compilation reliably grounds generation in microtonal scales, significantly increasing offgrid quartertone content over baseline generation. Beyond its core implementation, MazzikaAI illustrates how deterministic knowledgebased rules can effectively bridge expert, nonWestern musical traditions and unfinetuned foundation models. This architecture establishes a scalable paradigm for realtime humanAI cocreation, offering a generalizable blueprint for interactive accompaniment, adaptive music education, and culturally inclusive generative audio across diverse global idioms.
Read the full abstract ▾
All authors Jiaxin Du, Boulbaba Abdeljaouad, Yong Zhuang, Haoyu Li
Human-Computer Interaction
ResonaVis: Visualizing Interactive Music Data to Support Reflective Music Composition for Therapeutic Contexts
Abhishek Karwankar, Elise Ruggiero, Daniel Stevens et al. (+1 more)
Designing music for therapeutic contexts requires navigating complex relationships between musical structure and listeners' sensory responses, yet composers often lack structured representations of these interactions, relying instead on intuition. We present ResonaVis, an interactive visualization system that helps composers analyze interaction and audio data from prior sessions with children with Autism Spectrum Disorder (ASD), informing future compositions. ResonaVis integrates audio features and interaction logs to capture how children engage with layered musical compositions, representing this engagement through coordinated visualizations of temporal transitions, layer co-occurrence, rhythmic activity, and spectral characteristics. Rather than prescribing strategies or supporting therapy sessions directly, the system surfaces patterns in past session data to support data-informed reflection during composition. We evaluated ResonaVis through a mixed-methods study with eight music students and a follow-up case study with two experienced composers. Results demonstrate good usability (SUS = 72.23), exceeding
benchmarks for early prototypes, and show that participants could identify interaction patterns, reason about layer relationships, and make informed compositional decisions with high perceived performance and low frustration. Confidence in interpreting interaction and acoustic data for ASD-focused composition increased significantly across multiple dimensions (p < 0.05), with qualitative findings suggesting a shift toward more adaptive, data-informed composition. This work contributes a visualization design space for therapeutic music interaction data, an integrated system for compositional reflection, and empirical evidence that visualization tools support analytical
reasoning and confidence in data-informed creative practice. (Abstract shortened for arXiv.)
Read the full abstract ▾
All authors Abhishek Karwankar, Elise Ruggiero, Daniel Stevens, Matthew Louis Mauriello
Human-Computer Interaction
Narrative Keyframing for Generative Creative Writing
Chao Zhang and Abe Davis
We introduce narrative keyframing, an interaction technique for AI-assisted creative writing that lets writers specify different types of narrative constraints at selected moments in a story, then use AI to generate intervening prose. Inspired by the use of keyframing in animation, narrative keyframing offers a flexible way to connect story planning with adaptive control over generated text. We explore three types of keyframes: plot keyframes define significant events in a story, character keyframes represent how individual characters change over the narrative, and perspective keyframes capture how individual characters experience different events through first-person narratives. Plot and character keyframes offer a flexible way to adapt the type of high-level conditioning explored in previous AI writing tools to more customizable, iterative, and fine-scale control, while perspective keyframes add a new way to control characterization and focalization by using first-person narratives as an intermediary. Through a user study, we show that narrative keyframing supports a more controllable, transparent, and engaging way to use generative AI in creative writing.
Read the full abstract ▾