Human-Computer Interaction · Cross-listed from cs.AI
Agency as a New Organizing Abstraction for Computer Science
Ian Foster
Advances in large-scale machine learning have enabled agentic AI systems: persistent, autonomous computational entities that pursue goals, initiate actions, and interact with open environments. These systems are often framed as applications of foundation models. We argue that this framing is incomplete: agency is also a candidate new organizing abstraction for computer science. What is new is not the concept of an
agent, nor runtime policy generation per se, but its general-purpose, open-ended form, in which systems can synthesize courses of action from goals, observations, and
memory, over tools and environments for which no complete task-specific model was specified in advance. Thus agency becomes a systems-level abstraction as well as an AI modeling concept. Where the process abstraction organizes computation around executing programs, agency organizes it around pursuing goals under bounded resources: a process executes a program; an
agent pursues a goal. From this perspective, the recurring failures of deployed agentic systems (runaway resource consumption, behavior redirected by
adversarial content, opaque continuous operation) are symptoms of structural abstraction gaps, not isolated engineering defects. We examine these gaps across six dimensions (execution, interfaces, correctness, oversight, coordination, and persistence) and argue that closing them requires coordinated advances in systems,
Read the full abstract ▾
Human-Computer Interaction · Cross-listed from cs.AI
Get In, Decide, Prove: A Merchant-Side Readiness Benchmark for Agentic Commerce
Adrian de Valois-Franklin, Alex Bogdan, Anas Ibrahim et al. (+2 more)
AI
agents now browse and buy on people's behalf, and
benchmarks measure how well the
agents perform. Far less is known about the other side of the transaction: whether a merchant's website lets an
agent in, publishes terms precise enough to decide a purchase, and leaves evidence of what was agreed. This paper presents a merchant-side
benchmark built on those three questions and reports a pilot of it. The central measurement, decision sufficiency, rates each criterion of a concrete buyer mandate (total cost, cancellation or return terms, service date, variant identity) as determinate, indeterminate, or unknown from the merchant's own published evidence. Readiness scores carry their evidence coverage and missingness bounds; access is reported separately for a declared bot, a declared headless browser, and a user-present browser; and each observed term is mapped to the RAILS admissibility class of the evidence that could support it in a dispute. Four large merchants received the full assessment: three refused the declared profiles at their offer pages, and no mandate could be confirmed as met, because each merchant left a total or a refund term undetermined. An access census of 555 sites of merchants that two payment processors publicly name as clients found the home page unreadable to both declared profiles at 105 sites (18.9%). A declared assessment of 330 merchants, scored by deterministic rules without a rater, found that evidence coverage rather than the score decides who can be ranked: 115 merchants (35%) reached the ranking threshold, rankable profiles scored a median of 89 to 90, and material terms was the one pillar well below the others, because the all-in total was decidable at 18 merchants. The pilot shows that the measurements can be taken; it does not validate them or rank merchants, and the weights and the projected clearing band await validation against outcomes.
Read the full abstract ▾
All authors Adrian de Valois-Franklin, Alex Bogdan, Anas Ibrahim, Ibrahim Shaer, Tamaz Andguladze
Human-Computer Interaction · Cross-listed from cs.CV
Brain2Face: A Study on the Relation of EEG Signals and Spatio-temporal Head Avatars
Camilo Andrés Martínez Martínez, Olaf Dünkel, Kartik Teotia et al. (+3 more)
Driving digital head avatars is classically done by hand-crafted artist design or by image-based facial performance capture. In this work, we ask whether spatio-temporal tracking of the humans's head geometry and appearance can be performed solely from
brain signals, i.e., Electroencephalography (
EEG). Since prior work mostly focused on activity and emotion recognition, this task is relatively unexplored without even having publicly available high-quality
datasets and
benchmarks. Thus, we contribute a
dataset featuring paired
EEG and human head geometry as well as appearance annotations. We synchronize 64-channel
EEG with per-frame expression latent codes of a personalized Gaussian head avatar. Second, we introduce a baseline to the task of expression recovery of a head avatar from
EEG signals. To this end, we propose a dedicated
EEG-to-expression model that takes a temporal sequence of
EEG signals as input and regresses the latent expression parameters of the 3D Gaussian head avatar. Lastly, we validate our capture and algorithmic design choices through a
benchmark protocol that we introduce to this task. We believe this work paves the way towards better understanding of the relation between
EEG signals and facial surface as well as appearance changes.
Read the full abstract ▾
All authors Camilo Andrés Martínez Martínez, Olaf Dünkel, Kartik Teotia, Maurice Rekrut, Christian Theobalt, Marc Habermann
Human-Computer Interaction
Human Situational Awareness and Workload during Heterogeneous Autonomous Vehicle Hazard Monitoring: A Cross-Platform Study
Ziyue Wang, Yang Xing, Argyrios Zolotas et al. (+3 more)
As autonomous ground and aerial vehicles become increasingly capable, human operators are transitioning from direct control to supervisory monitoring roles. Maintaining Situational Awareness (SA) during rare but safety critical hazard events remains a fundamental challenge, particularly when operators must monitor heterogeneous autonomous platforms with different spatial and cognitive demands. This study develops a unified experimental framework for investigating human SA and workload during supervisory monitoring of autonomous ground and aerial vehicles under hazard conditions. Eighteen participants completed six within-subject conditions combining three interface modalities, namely, visual-only, haptic-only, and visual-haptic, with two platform types, ground vehicle and aerial drone. SA was measured using Situation Awareness Rating Technique (SART), while workload was assessed using the NASA Task Load Index (NASA-TLX). The key findings are: 1) Aerial monitoring produced significantly lower overall SA, higher situational demand, and greater workload than ground monitoring, driven by elevated Demand and reduced Understanding rather than diminished Supply. 2) Workload was consistently coupled with situational demand across all conditions, suggesting a stable SA-workload mechanism. 3) Both haptic-only and visual-haptic significantly improved overall SA relative to visual-only without increasing workload, demonstrating that interface design can enhance awareness within the existing workload envelope. These findings establish a cross-platform behavioural baseline for designing
multimodal interfaces that enhance operator SA without elevating workload in heterogeneous autonomous vehicle systems.
Read the full abstract ▾
All authors Ziyue Wang, Yang Xing, Argyrios Zolotas, Adolfo Perrusquia, Weisi Guo, Antonios Tsourdos
Human-Computer Interaction
Calibrating LLM Explanations to Human Decisions with Reinforcement Learning
Dong Shu, Ziyang Guo, Negar Kamali et al. (+2 more)
Explanations of LLM outputs are considered an important way to make model behavior legible, helping human users make good decisions on downstream tasks. However, default training approaches for improving explanations target model task performance or human preference rather than human decision quality. To bridge this gap, we introduce HD-RL, a reinforcement learning framework for optimizing LLM explanations for human decision making. Our approach calibrates a human decision simulator on observed human decisions, applying a proxy reward model to optimize the LLM to generate explanations that help people to make good decisions. We apply this framework to AI-generated text detection. In our initial data collection phase, we observed that while LLMs possess strong autonomous detection capabilities, the explanations they generate for users yield near-random human accuracy (51.3%). By applying the HD-RL framework to the open-source Qwen 3.6 model, we generate optimized explanations that improve human performance, achieving a GLMM-estimated human accuracy of 63.7% in a randomized between-subjects experiment with repeated measures and outperforming RLVR, RLAIF, and other baselines by over 8%.
Read the full abstract ▾
All authors Dong Shu, Ziyang Guo, Negar Kamali, Matthew Groh, Jessica Hullman
Human-Computer Interaction
TouchTwin: Human-AI Haptic Authoring for 3D-Scanned Tabletop Scenes through Language and Touch
Wanli Qian, Xinlei Yu and Heather Culbertson
Creating haptic experiences from real-world environments remains difficult: visual scans capture appearance but not material feel, while manual haptic authoring requires specialized expertise. We present TouchTwin, a human-AI approach that turns phone scans of tabletop scenes into editable haptic scenes. TouchTwin uses AI to propose material regions that users can experience and revise through
language and spatial edits. A model trained on measured tool-surface interactions generates the haptic responses, allowing users to work with material descriptions rather than low-level signal parameters. In a study with N=12 participants and physical reference objects present, automatic drafts scored 4.0/7 for realism and 5.4/7 as useful starting points. After a median of 6.6 minutes of editing, realism increased to 5.6/7, while scoring the saved material layers against ground truth showed material-family accuracy improving from 52% to 92%. Participants also used the same workflow to author material feels without a physical reference.
Read the full abstract ▾
Human-Computer Interaction
Particle Selection in Virtual Reality
Boaz Keren-Gil, James Gain, Lucia Marchetti et al. (+2 more)
Point-cloud representations of 3D data are used in many fields, from astronomical surveys of
galaxies to medical imaging and
laser scans of buildings. A task common to all of these is interactive selection: picking out a meaningful subset of points for closer analysis. The data is inherently 3D, yet the traditional workflow uses a 2D mouse on a 2D projection and inherits the limitations of a flat screen. This paper compares Virtual Reality (VR) and desktop environments for 3D particle selection, asking whether VR's six-degree-of-freedom hand interaction yields measurable gains in accuracy, speed, usability, workload and flow. In a counterbalanced within-subjects experiment (N = 34), participants selected 90 target
galaxies from a catalogue of 16,816
galaxies, using mouse-driven selection with keyboard camera control on the desktop and a hand-held spherical brush that adds or removes particles in VR. We measured task accuracy, completion time, and three standard questionnaires: the System Usability Scale (SUS), the NASA Task Load Index (NASA-TLX) and the Flow State Scale (FSS). Accuracy was comparable (91.6% VR vs. 93.1% desktop; p = 0.858), but every other measure favoured VR. Tasks were 23% faster (p = 0.001, d = 0.93), usability was higher (SUS 84.6 vs. 54.0; d = -1.78), workload was lower (NASA-TLX 20.4 vs. 34.7; d = 0.90), and flow was higher (FSS 83.2 vs. 70.2; d = -1.23). Because the underlying problem, making a 3D selection through a 2D interface, is not specific to astronomy, we expect the pattern to carry over to point-cloud selection in other fields.
Read the full abstract ▾
All authors Boaz Keren-Gil, James Gain, Lucia Marchetti, Patrick Marais, Alexander Sivitilli
Human-Computer Interaction
Better together? A performance theory of human-machine skill policies under generalization difficulty
Riccardo Zanardelli
Organizations pairing a human and a machine on a task decide, often by default, which executes each instance and what happens when both do. We account for what this attains under the squared rule, with performance distributed over instances to capture generalization difficulty. A gross stake lies between the skills' mean and best per instance. It splits at the best single skill into a systematic part, known from separate assessments and secured by commitment, and a residual, the expected value of perfect information on which skill each instance favors, known only from paired observation and split further by classes. Skills set the stake; workflows decide the take, via two channels. Routing directs work, weighting or selecting responses; with binary truth it is capped by the best skill per instance. Reading responses as joint evidence is not routing; what they jointly reveal caps it. Synergy lets skills interact before responding; bounded by no skill either way, its lift over independent responses must clear a break-even and can fall below one. As
algorithms, arrangements form a ladder stating what each reads, holds and records, and a procedure whose checks keep earlier gains when class means are known and lose little on finite records given a margin. The theory extends to many skills, each worth what it adds where it beats all others held, and to competing skills, where at parity a single failure's allowed cost decides. In a public chest-radiograph study with 227 radiologists, they are near parity with the model and the stake is mostly residual; assisted reading, no better on average originally, returned less than averaging both responses; recalibrating the model alone beats any deployable arrangement of raw responses. Organizations can learn from their records what is at stake, which arrangement collects it and what evidence each step needs; pricing costs and errors is future work.
Read the full abstract ▾
Human-Computer Interaction
Integrating Feature Understandability into Feature Selection for Human-Centred Tabular Explanations
Nicola Rossberg, Bennett Kleinberg, Barry O'Sullivan et al. (+2 more)
Existing human-centred XAI research predominantly evaluates explanations after they have been generated. However, this fails to provide actionable insights into improving accessibility. This study incorporates human-based feature understandability into feature selection to improve the generation of human-centred explanations. Natural
language explanations for machine learning models trained on tabular data often cite influential features in the input data to explain a given decision. As a consequence, users' ability to understand the produced explanation hinges on their understanding of the cited features. To quantify this natural-
language feature understandability, Rossberg et al. proposed the Feature Understandability Scale, which assigns numerical `understandability scores' to features based on user feedback. To evaluate the degree to which these scores can improve machine learning explanation quality, this study collects understandability scores across two
datasets, proposes an understandability-aware feature selection
algorithm to select features that are both understandable and informative, and trains a random forest based on these feature subsets. The SHAP-based explanations generated by this model are evaluated using user feedback, and changes in explanation quality are assessed. The individual evaluation of the explanations showed that while the understandability-based explanations scored higher on average, the difference was not statistically significant. In contrast, the comparative evaluation found that participants preferred these explanations significantly more frequently than expected at the chance level. It is concluded that the integration of understandability can yield more understandable explanations at minimal cost to classification performance, although changes to quality require elucidation in larger cohorts.
Read the full abstract ▾
All authors Nicola Rossberg, Bennett Kleinberg, Barry O'Sullivan, Luca Longo, Andrea Visentin
Human-Computer Interaction
Foresee: Making Procedural Futures Tangible through Generative Previews
Yusi Sun, Ying Jiang, Jiayin Lu et al. (+4 more)
Procedural guidance can show users how to act without explaining what an action enables later. We present Foresee, an interface combining next-step guidance with consequence previews, method comparisons, and cross-step access. A hybrid pipeline combines structured rendering, image editing, and
video generation. A literature-informed framework and six-participant formative study shaped its design. A 24-participant study compared AR guidance, AR with generated
video, and Foresee across digital painting, gameplay, and cocktail making. Exploratory painting comparisons showed higher correct completion without a recorded error than AR and higher post-task quiz accuracy than generated-
video guidance under task-wise correction. Interviews revealed that users could overlook valued explanations, match visible outcomes without understanding later possibilities, or wait for concrete demonstrations when symbolic cues were unclear. These findings inform generative previews that explain what actions enable or constrain while supporting ongoing execution.
Read the full abstract ▾
All authors Yusi Sun, Ying Jiang, Jiayin Lu, Jing Qian, Yin Yang, Yong-Hong Kuo, Chenfanfu Jiang
Human-Computer Interaction
What UI Design Can Do For "Good" Oversight Work: A Three-Month Action Research Study
Cedric Faas, Lisa Dargasz, Julius Gabelmann et al. (+4 more)
With AI automating consequential tasks, human oversight is required for ethical, legal, and practical reasons. Yet these oversight tasks can be demanding and often make for poor work. Empirical evidence on overseers' lived work experiences remains underexplored, yet critical for designing “good” oversight. We provide longitudinal, in-the-wild insights through action research accompanying (N=8) tutors who oversaw an AI-supported learning assistant over three months. Our thematic analysis combines insights from twelve weekly meetings and two workshop iterations and was enriched by four repeated questionnaires. We identified stimulation, mastery, and relatedness as key challenges, stemming from the task's repetitiveness, unclear impact of interventions, and limited connections with students and peers. Informed by our initial findings, we deployed targeted UI modifications with mixed success. This paper documents the long-term
evolution of the work characteristics of a real-world oversight task and discusses which oversight work characteristics can and cannot be supported by interface design.
Read the full abstract ▾
All authors Cedric Faas, Lisa Dargasz, Julius Gabelmann, Felix Jahn, Markus Langer, Kevin Baum, Anna Maria Feit
Human-Computer Interaction
STEPwise: Supporting Teachers in Diagnosing Students' Mathematical Problem-Solving Skills through LLM-Generated Questions and Rubric
Yoonsu Kim, Hayeon Doh, Eunhye Kim et al. (+2 more)
Problem-solving is central to mathematics education. However, diagnosing where students'
reasoning breaks down requires individualized probing and consistent criteria that teachers cannot sustain at classroom scale. We introduce STEP (STage-based Evaluation of Problem-solving skills), co-designed with six mathematics education experts, which operationalizes problem-solving as four stages and eight diagnostic sub-skills. By specifying what evidence to elicit and assess at each stage, STEP enables an LLM pipeline to generate stage-specific questions and rubrics for any math word problem, reducing the manual design of diagnostic materials. An expert review (N=8) assessed the pedagogical validity of the generated materials and informed STEPwise, a teacher-in-the-loop system for diagnostic material generation and process-level assessment. In a classroom study with 5 teachers and 64 students, STEPwise helped teachers and students locate
reasoning breakdowns and supported students in reflecting on their
reasoning. Together, STEP and STEPwise offer a structured approach to fine-grained diagnosis at classroom scale.
Read the full abstract ▾
All authors Yoonsu Kim, Hayeon Doh, Eunhye Kim, Jinho Son, Juho Kim
Human-Computer Interaction · Cross-listed from cs.LG
BehaviorFormer: Discovering Behavioral Patterns via Clusterable Next-Token Prediction
Tobias Schneider, Danial Fahrner, Soren Antebi et al. (+4 more)
Discovering behavioral patterns in open-ended games is challenging due to the lack of clear objectives or ground-truth annotations. In this work, we extend a next-
token prediction
Transformer with the deep clustering
algorithm DCSS to learn meaningful representations with well-separated clusters. In a controlled Minecraft session with 26 players, the method discovers 23 distinct behavioral patterns that generalize across the studied players, further refining structures present in self-supervised representations into clearly separable clusters. We group the clusters into seven interpretable categories covering stationary, roaming, travel, mining, base work, logging, and farming behavior.
Read the full abstract ▾
All authors Tobias Schneider, Danial Fahrner, Soren Antebi, Daniel Uedelhoven, Lennard Bodden, Sandra Halscheidt, Rafet Sifa
Human-Computer Interaction · Cross-listed from cs.CV
FlowAvatar: Real-Time Full-Body Avatars from Sparse Egocentric Inputs on Consumer XR Devices
Chenfeng Gao, Taeyoung Yeon, Sungheon Park et al. (+4 more)
Prior avatar embodiment research has studied tracking completeness (how faithfully the body, hands, and face are sensed and reproduced) and personalized appearance in isolation, each requiring specialized lab equipment. How these factors interact on consumer hardware remains an open question, because no existing system unifies body pose, face, hands, and personalized appearance on a standalone headset. We present FlowAvatar, a system enabling real-time personalized look-alike avatars with full-body pose, hand articulation, facial expression, and eye gaze tracking from sparse egocentric inputs on consumer XR devices (Meta Quest Pro). Our pose model achieves competitive spatial accuracy and state-of-the-art temporal consistency on the AMASS
benchmark against prior works with comprehensive spatial and temporal losses validated through architecture and loss ablations, while running in real-time. Two user studies (N=12, N=10) provide converging evidence that tracking completeness and personalized appearance serve complementary roles: tracking completeness broadly improves perceived motion quality, while only their combination achieves positive total embodiment.
Read the full abstract ▾
All authors Chenfeng Gao, Taeyoung Yeon, Sungheon Park, Vasco Xu, Anish Prabhu, Mar Gonzalez-Franco, Karan Ahuja
Human-Computer Interaction
{Access|Use|Sustain|_}-ability: Exploration and Quantification of a Linguistic Structure and Its Use in HCI
Alarith Uhde and Paul Haimes
Language plays a crucial role in research and technology design, throughout problem formulation, method descriptions, analysis of findings, and interpretation. In this paper, we focus on a particular, trending linguistic pattern in the Human-Computer Interaction literature, which we call -abilities. Examples include usability, accessibility, and explainability. These -abilities have become increasingly popular, with potential impacts on the discipline. They take a verb, turn it into an adjective, and finally nominalize it. We approach the -abilities phenomenon with a survey of the HCI literature between 1988 and 2025. All in all, we identified 349 different -abilities. Moreover, the use of -abilities notably increased since the late 2000s, both in the absolute number of different -abilities and the frequency of use of the most popular ones. The majority of -abilities we found had a positive notion, and most -abilities with a negative notion are prefixed counterparts of their positive opposite (e.g., un+sustainability). We critically discuss implications of their increased use. First, -abilities change the literal meaning of design intentions. For example, acceptability reduces the concrete verb accepting into only its potential (acceptable), and finally nominalizes it, claiming the existence of acceptability as a somewhat stable and real, abstract characteristic of a technology. Second, the prefix-based opposite construction can weaken the design goal. Un+sustainability avoids clarification of the -ability's meaning (here e.g., ecological or financial), and does not spell out the expected impact of designs that are not sustainable (e.g., the destruction of nature). Future work should compare the use of -abilities in English with other
languages to better understand the ways in which they transform meaning.
Read the full abstract ▾
Human-Computer Interaction
Exploring Mixed Reality for Engagement in Robotics: A Comparison of Robot Control Methods
Samantha Edvell, Dinh Tung Le and Gavin Paul
This paper presents a Mixed Reality (MR) robotics experience that provides an accessible introduction to interacting with a physical
robot. The application combines a TurtleBot with a virtual animal-tending activity and three control methods: Joystick, Waypoint, and Path-Drawing. A mixed-methods user study with 10 participants investigated changes in robotics interest and confidence following the experience and how the control methods influenced engagement, workload, and preference. Confidence increased following the experience, with the exploratory analysis indicating a significant pre–post difference, while the increase in interest was not statistically significant. The control methods produced clear differences in perceived workload and ease of use, with Path-Drawing generally more demanding and difficult to use. Joystick was preferred by eight of ten participants and was associated with better perceived performance and lower frustration. However, no statistically significant differences were detected in the other engagement dimensions, suggesting that the most preferred method was not necessarily more engaging across all aspects of the experience. These findings highlight the importance of considering both usability and engagement when designing how users interact with
robots in introductory MR experiences.
Read the full abstract ▾
Human-Computer Interaction · Cross-listed from cs.AI
GUIDE: Computationally Augmented Consensus for Trustworthy Clinical Practice Guidelines
Yuanfan Lin, Jie Di, Xujin He et al. (+21 more)
Maintaining up-to-date clinical practice guidelines (CPGs) is essential for evidence-based medicine, but rapid evidence growth and complex expert consensus formation limit timely updates. We developed GUIDE, a multi-
agent copilot system for human-AI collaborative guideline generation and updating. GUIDE delegates labor-intensive tasks to specialized AI
agents while preserving expert judgment and final arbitration. In 12 NICE ophthalmology
datasets, GUIDE achieved 88.85% literature recall, 86.06%
alignment in certainty-of-evidence ratings, and 91.67% recommendation agreement, requiring 162.17 minutes per
dataset on average (28.68% human intervention). Performance remained stable across 20 additional
datasets from cardiology, endocrinology, gastroenterology, and respiratory medicine. Ablation analysis showed that removing expert intervention reduced performance in judgment-intensive steps such as Risk-of-Bias assessment. In six post-cutoff
datasets, GUIDE maintained robust end-to-end performance on unseen evidence. Prospective real-world validation showed that GUIDE reduced expert workload and produced more clinically acceptable recommendations than the original decade-old guideline (mean score, 3.73 vs. 1.53; agreement, 66.7% vs. 13.3%), with outcomes comparable to human-only updating (P = 0.588) but substantially less time and effort. GUIDE offers a practical framework for timely, trustworthy, expert-centric guideline development and validates computationally augmented consensus.
Read the full abstract ▾
All authors Yuanfan Lin, Jie Di, Xujin He, Zhenzhe Lin, Jialing Chen, Ruiyang Li, Changyu Zheng, Yuanrui Shang, Weiqin Peng, Jingxi Qin, Xiong Wang, Dongxin Ye, Yining Huang, Guoqiang Li, Andi Xu, Sijia Yu, Ruixin Wang, Yahan Yang, Xiaohang Wu, Duoru Lin, Jingjing Chen, Zhenzhen Liu, Wenben Chen, Haotian Lin