Human-Computer Interaction · Cross-listed from cs.SE
Look Before You Prompt, and After: Scaffolding Human-AI Collaboration in Software Tutorial Creation
Avinash Bhat, Vy Bui and Jin L.C. Guo
With LLMs, creating software tutorials now involves steering the model's output and shaping it into a coherent, accurate learning resource, yet existing LLM tools offer writers little support for this work. By analyzing interviews with technical writers ($N=17$), we identify three requirements for how they assemble and structure multiple LLM responses, curate the context the model uses, and verify the generated content. We designed a tool called dBlocks with the following features: blocks to scope content, a context manager to edit context, and inline execution to verify code. Following a human-centered design method, we iteratively refined the design through a user study ($N=5$). In a within-subjects lab study ($N=16$) comparing dBlocks with participants' preferred workflows for LLM-assisted authoring, participants reported significantly higher confidence in the tutorials they produced with dBlocks. In addition, the tool reduced friction in verification, with writers verifying code as they drafted rather than deferring or skipping it, and helped them avoid searching long chat histories by scoping their work into blocks that kept each tutorial section and its LLM conversation together. More broadly, our work offers implications for tools that scaffold human-AI collaboration in SE workflows and shows how human-centered design can guide the development of LLM-integrated tools.
Read the full abstract ▾
Human-Computer Interaction
An Agent Model Abstraction for Human-AI Teaming Cognitive Coupling
Kolitha Kottagaha W.M, Jos A.C. Bokhorst, Ben Gaffinet et al. (+1 more)
Industrial environments increasingly rely on collaboration between humans and AI-enabled
agents. Effective teamwork requires aligning how
agents perceive situations, plan actions to pursue goals, and adapt to changing conditions, yet existing systems lack mechanisms for cross-
agent cognitive processes coupling. This paper presents a conceptual cognitive
agent model that formalises cognitive coupling through eight components: Input, Process, Output, State, Value,
Memory, World Model, and Goal. The model abstracts how
agents coordinate and co-regulate their cognitive cycles, providing a basis for analysing distributed cognition and designing cognitively interoperable human-AI systems.
Read the full abstract ▾
All authors Kolitha Kottagaha W.M, Jos A.C. Bokhorst, Ben Gaffinet, Christos Emmanouilidis
Human-Computer Interaction
Matched Starts, Divergent Objects: Research-Object Formation in Human-AI Scholarly Work
Mehmed Zahid Çögenli
Human-artificial intelligence interaction can shape what scholarly inquiry becomes about. This study examines research-object formation in two isolated scholarly trajectories developed by the same expert researcher with Claude and ChatGPT under matched starting conditions. Both used the same five-session, 843-turn project corpus, frozen research problem, initial
prompt, publication objective, and conduct rules. The subsequent scholarly-development conversations were reconstructed ex post through consequential-event analysis, a Socioduality relational-process overlay, and tracing into completed manuscripts. Both trajectories shifted the continuity problem from recall toward usability, then developed different research objects: continuity labour and distributed, evolving project state. Failed analytical units, rejected explanations, counterexamples, and changes in analytical scale shaped these objects and propagated into research questions, methods, findings, evidence logic, and theoretical contributions. Consequential scholarly change, reciprocal relational continuity, and local substantive re-formation had distinct boundaries; continued interaction did not itself establish epistemic endorsement. The findings demonstrate empirically constrained research-object formation under matched scholarly starts. For systems supporting sustained scholarly knowledge work, they motivate recoverable analytical provenance, inspectable proposal status, and source-faithful contribution records.
Read the full abstract ▾
Human-Computer Interaction
EyeMakeYou: Identity-, Task-, and Subjective-State-Conditioned Diffusion for High-Frequency Gaze Synthesis
Kamrul Hasan, Mehedi Hasan Raju and Oleg V. Komogortsev
Eye movement biometrics (EMB) is an emerging behavioral modality for user authentication, particularly in virtual- and augmented-reality systems, where gaze dynamics contain distinctive subject-specific features. However, robust EMB systems require diverse, high-quality gaze recordings that are expensive to collect and often unavailable at the scale needed for model development. Generative models can mitigate data scarcity, but existing methods either synthesize generic gaze behavior or personalize signals primarily by identity, without jointly representing the user's task and subjective state. Consequently, generated signals may appear visually realistic while failing to retain the behavioral properties required for biometric applications. To address this limitation, we propose EyeMakeYou, a multi-conditional denoising
diffusion framework for subject-specific, high-frequency gaze synthesis. EyeMakeYou generates 5-s, 1000-Hz bivariate gaze-velocity sequences from an identity-removed reference trajectory and conditions the denoising process on an identity
embedding, a task
embedding, and self-reported ratings of overall difficulty, mental tiredness, and eye tiredness. Its objective combines
diffusion noise prediction and identity preservation with multi-resolution spectral, drift-consistency, and event-weighted local-smoothness losses. Experiments on GazeBase show that EyeMakeYou achieves higher median spatial accuracy and greater real–synthetic similarity in the
embedding feature space than the existing generative approaches, while retaining selected task-dependent associations between subjective reports and oculomotor features. These findings support conditional
diffusion as a practical approach for augmenting gaze
datasets for biometric and interactive applications.
Read the full abstract ▾
Human-Computer Interaction · Cross-listed from cs.CL
You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments
Shiwei Hong, Junjie Ma, Emma Jiren Wang et al. (+5 more)
Chinese online comments often convey social meaning through indirect and playful
language that is hard to interpret without context. Existing evaluations largely organize items around predefined phenomena or controlled pragmatic categories, leaving open whether models can distinguish plausible readings of what a naturally occurring comment is doing in a particular exchange. We introduce a
benchmark for evaluating whether LLMs can recover such situated pragmatic meanings. From more than 200,000 public Chinese social media interaction records, we construct 4,735 human-validated diagnostic items, each pairing a target comment with reconstructed preceding context and plausible misreadings. We evaluate eight LLMs as both question writers and solvers in a cross-writer setting. The task is challenging: the strongest model achieves 81.42% leave-writer-out accuracy. Across all eight models, the mean leave-writer-out accuracy is 68.70% while human accuracy was 90.8%. Case analysis shows that models often recognize broad irony or playfulness while misidentifying the mechanism or interactional move.
Read the full abstract ▾
All authors Shiwei Hong, Junjie Ma, Emma Jiren Wang, Ethan Z. Rong, Siying Hu, Haichang Li, Ziying Wang, Zhicong Lu
Human-Computer Interaction · Cross-listed from cs.RO
VLA-Precision: Asymmetric Co-Bootstrapping for Efficient Real-World Online RL of Vision-Language-Action Models
Chenyu Su, Zhaolong Shen, Yuan Qian et al. (+8 more)
Pretrained
vision-
language-action (VLA) models enable broad manipulation but remain unreliable in tasks demanding precision and repeatability. Applying real-world online reinforcement learning (RL) to VLA post-training enables autonomous trial-and-error improvement beyond demonstrations alone, but exposes two bottlenecks: 1) unreliable value signals can induce policy drift; 2) large-VLA overhead constrains throughput and sample efficiency. To address these challenges, we present VLA-Precision, an efficient real-world online RL framework featuring the Asymmetric Co-Bootstrapping (ACoB)
algorithm and the ACoB-Stream architecture. Specifically, ACoB establishes asymmetric co-bootstrapping across timescales: early intervention-guided behavioral learning rapidly improves policy performance while enhancing online experience quality. As autonomous experience accumulates, global return propagation and local preference ranking progressively calibrate value estimates, yielding relative action advantages for reference-regularized policy improvement while suppressing drift. To enable ACoB on large VLAs, we develop ACoB-Stream, a closed-loop experience–policy architecture that establishes invariant-state decoupling and on-demand streaming as design principles, delivering up to 10.9$\times$ improvements in throughput and computational efficiency. Extensive evaluations on nine high-precision chemistry tasks across four categories and four
robot embodiments show that VLA-Precision achieves 98.3% mean success rate in 45.8 min/task, with 27.6 s episodes running at 1.2$\times$ and 1.8$\times$ the speeds of VLA and RL baselines. Resources are available at https://vla-precision.github.io.
Read the full abstract ▾
All authors Chenyu Su, Zhaolong Shen, Yuan Qian, Chen Qian, Rui Zhang, Feng Yan, Weixing Chen, Fei Zhang, Jiamin Wang, Shuang Cong, Weiwei Shang
Human-Computer Interaction
Toward Frontier-Quality Declarative UI Generation at Small-Model Cost
Yingxiang Yang, Weihang Xiao, Ben Bullough et al. (+2 more)
Declarative UI protocols such as A2UI let applications generate interactive UIs by selecting pre-built components from a catalog and binding their props to application data, rather than emitting frontend code from scratch. This contract is attractive for production systems because of safety and consistency. An open question is: can low-latency and low-cost small models achieve the required quality for A2UI-based UI generation? To answer this, we systematically study three controllable design choices for catalog-conditioned A2UI generation: supervised fine-tuning (SFT) data construction method, model size, and component-catalog size. Across two React/TypeScript domains and four base checkpoints spanning two model families (Qwen 3.5 0.8B/2B/4B; SmolLM 3B), we find: (i) a 4B fine-tuned student recovers ~98% of teacher semantic quality and ~97% of teacher visual quality at more than an order of magnitude lower cost than frontier API calls; (ii) both augmented strategies (Perturbed-catalog and Constrained-GT) Pareto-dominate the unaugmented Full-catalog baseline, while specializing on different axes; (iii) even small models can handle and benefit from relatively large component catalog size. We distill these results into practitioner-facing trade-offs and deployment recommendations across the three design choices.
Read the full abstract ▾
All authors Yingxiang Yang, Weihang Xiao, Ben Bullough, Tushar Deshpande, Niresh Agarwal
Human-Computer Interaction · Cross-listed from cs.CY
Decreasing Digital Distraction in College Students: Associated Online Learning Strategies Identified by Unsupervised Data Mining Approaches
Hui Shi, Ran Bi, Xi Lin et al. (+1 more)
The proliferation of digital tools in education offers numerous benefits but also introduces significant challenges, notably digital distractions that hinder academic performance, especially in online learning contexts. This study employed unsupervised data mining techniques, specifically association rule mining and clustering analysis, to identify effective learning strategies associated with lower levels of digital distractions among college students. Data from 530 participants revealed that self-regulated learning strategies (i.e., goal setting, environment structuring, and time management) co-occurred most consistently with lower digital distractions. Additionally, learner-instructor and learner-content engagement strategies, as well as technical competencies, also tended to appear in the same profiles as lower distraction. Interestingly, reliance on peer help-seeking and learner-learner engagement strategies appeared less often in those lower distraction profiles. These findings offer actionable implications for educators to design targeted interventions that foster focused and productive online learning environments.
Read the full abstract ▾
All authors Hui Shi, Ran Bi, Xi Lin, Yan Dai
Human-Computer Interaction · Cross-listed from cs.CY
Making Gender-Inclusive Practices Actionable: Evaluating a Research-Informed Computing Education Toolkit
Alina Berry, Susan McKeever, Brenda Murphy et al. (+1 more)
The persistent gender imbalance in computing remains a global concern, and universities offer a key part of the pipeline to address it. Although research has identified practices that support under-represented student groups, translating this evidence into actionable guidance remains challenging. This paper first presents a novel web- based toolkit (TechMate) designed to address this gap by helping computing educators implement gender- inclusive initiatives through practical research-informed guidance. The toolkit defines over 25 actions, ranging from operational to strategic, and provides case studies and implementation resources. Second, this work reports on the evaluation of TechMate, capturing educators first impressions of its usefulness and usability through authentic tasks, and eliciting unanticipated insights about structural barriers to gender-inclusive practice in computing higher education. Eighteen computing educators of varying seniority from eight universities in Ireland assessed the toolkit using nine research-driven attributes: four usefulness measures (novelty, relevance, trustworthiness, actionability) and five usability measures (effectiveness, aesthetics, navigation, terminology, user satisfaction). Data was analysed using mixed methods: deductive coding, quantitative sentiment scoring, and inductive thematic analysis. Educators rated TechMate highly for user satisfaction and praised its novelty and trustworthiness, with over 60% of participants reporting positively on all attributes, suggesting that research -informed tools can be designed for real-world use. Thematic analysis highlighted other challenges, including women students isolation, institutional resistance to change, and the scarcity of women role models, while also surfacing proposed solutions such as leveraging student ambassadors and fostering supportive learning environments.
Read the full abstract ▾
All authors Alina Berry, Susan McKeever, Brenda Murphy, Sarah Jane Delany
Human-Computer Interaction
Grounding GUI Design in Computational Psychology
Xianni Wang, Javier Romero Davila, Saku Sourulahti et al. (+2 more)
Creating visually appealing user interfaces often requires extensive manual iteration. We propose an approach that applies answer set programming (ASP) to automatically generate and optimize UI layouts while satisfying design objectives such as grid
alignment, grouping, color harmony, and whitespace, along with designer-specified preferences. Our method encodes constraints on element properties and relative positioning, producing layouts that balance functional and aesthetic goals. We evaluate this approach in three studies. Across two user studies, participants rated layouts generated with the full ASP model higher than both random designs and those based on simple heuristics. Designers reported that ASP-generated layouts supported early-stage sketching and exploration.
Read the full abstract ▾
All authors Xianni Wang, Javier Romero Davila, Saku Sourulahti, Torsten Schaub, Jussi P. P. Jokinen
Human-Computer Interaction
From Misconceptions to Evidence: What Science Teachers Make Visible When Co-Designing Agentic Learning Apps
Nizam Kadir, Wei Ting Liow, Sumbul Khan et al. (+1 more)
Science educators increasingly encounter AI tools that generate content, yet disciplinary teaching depends on eliciting learners' models, diagnosing misconceptions, interpreting evidence, and preserving professional judgment. This study asks how science teachers translate such epistemic work into specifications for agentic learning applications. It contributes to the conference theme, "Innovating Pedagogies, Inspiring Minds: Transforming Science Learning," and the Teachers' Professional Learning strand by examining app co-design as a form of pedagogical
reasoning. We conducted a bounded qualitative cross-case analysis of four de-identified artifacts produced in a teacher professional-learning workshop: an experimental-design diagnostic, a Kinetic Particle Theory dialogue guide, a chemistry prior-knowledge checker, and a physics application/scaffolding tool. Each artifact was coded for the disciplinary problem, learner interaction, evidence made visible, teacher authority, and safeguard. All four connected a science-learning problem to an interaction and pedagogically interpretable evidence: misconceptions and gaps, explanations-in-progress, class-level readiness patterns, or investigation performance. However, only two made teacher control or evaluation explicit, and only two named a safeguard. The proposals therefore positioned AI less as an answer generator than as an elicitor, scaffold, and evidence-return mechanism, while leaving decision rights and protections unevenly specified. We argue that teacher professional learning should treat AI app ideation as epistemic specification work. A five-question design protocol–problem, learner interaction, evidence, teacher authority, and safeguard–can help teachers transform science-learning needs into accountable human-AI arrangements before building or adopting a tool.
Read the full abstract ▾
All authors Nizam Kadir, Wei Ting Liow, Sumbul Khan, Lay Kee Ang
Human-Computer Interaction
GazeFS: Target-Centered Gaze-Trajectory Forecasting and Stabilization from Gaze-Head History
Yaozheng Xia, Zaiping Zhu, Bo Pang et al. (+4 more)
Target-centered gaze interaction requires more than suppressing frame-to-frame fluctuations: target acquisition produces task-aligned changes in gaze-head dynamics, while a gaze trace may retain a persistent target-relative residual direction. We formulate gaze correction as online target-centered gaze-trajectory forecasting and stabilization and introduce GazeFS, which maps a variable-length gaze-head history to the next target-center direction and a short-horizon Search/Focus estimate without target information at inference. Across 7,960 acquisition episodes from 30 participants, Search-Focus differences remain stable under quality control, onset exclusion, and duration matching. History windows improve phase decoding over the current endpoint, but explicit task progress remains a strong control. Under the 30-participant, five-fold grouped out-of-fold protocol across three seeds, the reductions relative to raw hold in Focus episode bias, within-episode dispersion, and P90 target error are 0.182 degrees, 0.257 degrees, and 0.400 degrees, with participant-bootstrap 95% confidence intervals excluding zero. Endpoint-free replay from empty history preserves the Focus advantage and yields raw-network phase balanced accuracy/AUPRC of 0.925/0.993; coordinate controls further show that recent history contributes beyond explicit progress metadata. GazeFS therefore improves Focus target centering and empirical residual contraction while leaving temporal smoothness as a separate objective.
Read the full abstract ▾
All authors Yaozheng Xia, Zaiping Zhu, Bo Pang, Minghao Xie, Hui Li, Shaorong Wang, Sheng Li
Human-Computer Interaction · Cross-listed from cs.AI
Govern the Model, Not Only the Data: Storage, Circulation, and Learning in Creative AI
Phoenix Perry, George Simms, Elizabeth Wilson et al. (+9 more)
Federated learning is increasingly presented as a
privacy-preserving advance: personal data remain on the device, and only model updates are shared. It borrows the vocabulary of the
federated social web, yet inverts its logic, distributing computation while the resulting model stays with whoever convened the training. We argue that federation is not in itself a remedy for extractive AI, because outcomes depend on who governs the data and the model and who has agency over the practices that shape them. We describe three layers at which a creative community can hold its work: storage, circulation, and learning. Examining artist-governed trusts, cooperatives, and consent infrastructures, we show that creator governance is established at storage and circulation but stops at learning: contributors can consent to training, yet have little say over the resulting model or its federation. We map the research space this opens, pairing technical open problems with the human questions from which they unfold. We propose four design principles for a creative data commons that governs models and their federation, not only
datasets: govern the model, not only the corpus; make the terms legible at the moment of contribution; design for refusal as a first-class state; and decide stewardship in the open and account for it.
Read the full abstract ▾
All authors Phoenix Perry, George Simms, Elizabeth Wilson, Yasmine Boudiaf, Nick Bryan-Kinns, Tega Brain, R. Luke DuBois, Alix Rule, Rachel Meade Smith, Kelani Nichole, Atharva Pravin Pawar, Rebecca Fiebrink
Human-Computer Interaction · Cross-listed from cs.AI
Transfiver: Human-AI Co-Inference through a Shared Editable State
Minji Park, Seunghyun Yoon and Hyuk Lim
Long-term human-AI interaction is difficult because the information that guides inference is updated implicitly by the model and is not directly inspectable or controllable by the user. We introduce the TRANSparent Framework for Interactive, Verifiable, Editable Representation (Transfiver), an architecture for human-AI co-inference through a shared editable state. Its central idea is that interaction-specific information is maintained in a single persistent state $(S_t)$ that both the model and the human update. Transfiver distinguishes two modes of state
evolution. In an implicit stream update, the model interprets ongoing interaction and decides whether new information revises an existing state item or creates a new one. In an explicit directed edit, a human inspects and modifies an addressed item. Both act on the same underlying state, so a human correction changes the state that subsequent computation reads, rather than adding another instruction or separate record. The architecture separates shared parameters $(\theta)$, learned before ordinary use, from the persistent state $(S_t)$, which evolves during deployment without parameter retraining. Extending Transfiver to rich natural-
language, relational, and large-scale shared states remains open.
Read the full abstract ▾
Human-Computer Interaction
Auditing Contextual Bias in Human Ball-Strike Calls Using KBO's Automated Umpiring Transition
Kichang Lee and JeongGil Ko
This paper uses the Korean Baseball Organization's adoption of the Automated Ball-Strike (ABS) system to audit long-standing claims about contextual bias in human ball-strike calls. Using pitch-level KBO data from 2021 through the available portion of the 2026 season, we model called-strike probability for taken pitches near the strike-zone boundary, with 2022-2023 as the primary human-umpire baseline and ABS seasons (2024 and onward) as a diagnostic
benchmark. The strongest evidence concerns count pressure. Relative to 0–0 counts, human umpires called substantially fewer strikes in two-strike counts and more strikes in hitter-ahead three-ball counts. Specifically, in the main 0.25-ft boundary band, 0–2 was associated with a -17.17 percentage-point effect and 3–0 with a +6.61 percentage-point effect. Under ABS, the corresponding effects were close to zero and did not survive false-discovery-rate correction. Game progression shows a smaller but coherent pattern as human calls were less strike-prone in early innings and more strike-prone in innings 7–9+, especially in late-close situations, while complete ABS seasons were essentially flat. Other suspected biases are weaker or more localized. Salary-based reputation proxies provide suggestive but proxy-sensitive evidence, and catcher identity shows human-period residual heterogeneity that disappears under ABS. Home-context evidence is mostly null at the umpire level, with one FDR-significant human-period exception and an exploratory umpire-team gap best treated as an audit lead. Overall, the results do not show that human umpires were biased everywhere. Instead, they map where the human strike zone was most context-sensitive, where evidence was weaker, and where common suspicions received little support.
Read the full abstract ▾
Human-Computer Interaction
Exploratory Unstructured Data Analysis: A Formative Study and Implications for Human-AI Collaboration
Johannes Eschner, Dominik Eitler, Max Irendorfer et al. (+3 more)
We propose a conceptual framework for exploratory data analysis of (large) unstructured data (EluDA), combining classical elements (querying, visualization) with active knowledge construction in the "search for structure". In a formative study, users conceptualized a structure for an image
dataset during exploration. We found that users conceptualize by building faceted classifications bottom-up and rarely create meaningful spatial categorization during this process. We also evaluated CLIP for zero-shot assignment and semantic categorization, finding that it remains unreliable for assigning user-defined concepts to images but does support semantic grouping. Based on these findings, we identify and discuss four key opportunities for human-AI collaboration in EluDA: intelligent sampling and visualization to maximize data visibility; incremental and few-shot learning to minimize effort for reliable assignment; automatic category, concept, and facet suggestions to reduce effort during the search for structure; and the necessity for effective trust calibration methods.
Read the full abstract ▾
All authors Johannes Eschner, Dominik Eitler, Max Irendorfer, Patrick Kramml, Matthias Zeppelzauer, Manuela Waldner
Human-Computer Interaction
PlanePivoting: Exploration and Optimization of Gaze-Mouse Cursor Alignment for Spatial Object Translation
Jinwook Kim, Sangmin Park, Jihyeon Lee et al. (+2 more)
As XR matures into a ubiquitous computing platform, the disconnect between 2D and 3D input modalities remains a critical barrier to seamless workflow. Frequent transitions between the mouse for 2D precision and hand gestures for 3D manipulation induce significant physical fatigue and cognitive load. To address this, we introduce PlanePivoting, a
multimodal interaction technique that extends standard mouse input into 3D space by leveraging gaze-mouse
alignment. This technique dynamically modulates the translation plane based on the spatial overlap between the gaze and mouse cursor, eliminating the need for physical input modality switching. To systematically explore the foundational design space of gaze-mouse coordination and optimize key variables, we conducted a user study comparing PlanePivoting with a standard 3D Gizmo interface across two translation mapping profiles and two gaze cursor apertures. Results demonstrate that PlanePivoting outperforms the Gizmo on efficiency metrics while maintaining comparable precision and yielding higher subjective satisfaction. This study demonstrates the potential of gaze-mouse
alignment for efficient spatial manipulation between 2D and 3D environments.
Read the full abstract ▾
All authors Jinwook Kim, Sangmin Park, Jihyeon Lee, Sang Ho Yoon, Jeongmi Lee
Human-Computer Interaction
Point&Spawn: Mid-Air Reference-Free Object Instantiation Using Gaze and Hand Gestures in Extended Reality
Jihyeon Lee, Ken Pfeuffer, Jinwook Kim et al. (+1 more)
Mid-air object instantiation in XR requires users to specify a 3D position without spatial references, such as surfaces or existing objects. We present Point&Spawn, a staged pipeline for pre-instantiation position specification through Direction Setting, Depth Setting, and Position Refinement within a continuous gesture flow. We evaluated six controller-free techniques combining Gaze or Non-Dominant Hand (NDH) direction setting with Ray Intersection, Relative Gain, or Drag&Hold depth setting in a user study (N=24) across Near and Far spawn depths. Relative Gain and Drag&Hold yielded faster and more accurate spawning, lower workload, higher usability, and greater preference than Ray Intersection. The shoulder-referenced NDH ray improved speed and coarse accuracy, whereas the viewpoint-based Gaze ray reduced hand movement with comparable final accuracy. Farther spawn depth imposed greater temporal costs as well as Gaze and accuracy costs with Ray Intersection. These findings offer empirical guidance for designing direction and depth control in spawning in XR.
Read the full abstract ▾
All authors Jihyeon Lee, Ken Pfeuffer, Jinwook Kim, Jeongmi Lee
Human-Computer Interaction · Cross-listed from cs.GR
ReRoom: Blending Virtual and Physical Contexts for In Situ Room Planning in Mixed Reality
Hongliang Yang, Yanjing Xu, Anhang Zhang et al. (+2 more)
Planning a real domestic space is an in situ authoring process: users evaluate candidate layouts at true scale, refine their intent, and carry accepted decisions into later iterations. Existing approaches either separate layout editing from the physical room or provide limited support for evaluating and refining whole-room proposals in situ. We present ReRoom, a mixed-reality system for in situ room-layout authoring. ReRoom presents a shared layout state through a virtual room proxy spatially registered to the target room, allowing interaction and layout generation to remain grounded in the physical context. Users refine the current proposal through direct manipulation or
language and preserve accepted placements, allowing each generated update to continue the same evolving design. To balance layout quality with generation efficiency, ReRoom uses a skill-guided layout
agent whose room-layout design skill operationalizes three principles that we formulate by synthesizing established interior-design guidance for real-room layout generation. The skill grounds these principles in a normalized representation of the scanned room and reusable
geometric checks. Evaluations show that ReRoom produces high-quality layouts for non-rectangular rooms, while its in situ workflow improves the room-planning experience over an otherwise equivalent off-site VR workflow. Code will be released upon acceptance of the paper.
Read the full abstract ▾
All authors Hongliang Yang, Yanjing Xu, Anhang Zhang, Hui Ye, Pengfei Xu
Human-Computer Interaction · Cross-listed from cs.CV
Occlusion-Robust Multimodal Emotion Recognition in VR via Fusion of Facial Images and EMG
Birgit Nierula, Karam Tomotaki-Dawoud, Mert Akguel et al. (+5 more)
Head-mounted displays (HMDs) fundamentally limit emotion recognition in virtual reality (VR): by occluding the upper face, they render conventional image-based facial expression analysis incomplete, particularly for applications requiring real-time affective assessment. We address this challenge by fusing lower-face
video with facial electromyography (EMG) from the occluded upper face to classify seven emotional categories (six basic emotions plus neutral). We introduce a synchronized
multimodal dataset from 20 participants, pairing lower-face
video with seven-channel upper-face EMG elicited by validated emotion stimuli. Under subject-independent test, our proposed late-fusion architecture merging convolutional visual
embeddings with RBF-kernel EMG representations achieves 51% macro-F1, outperforming both image-only (41%) and EMG-only (43%) baselines. These results demonstrate that upper-face EMG provides robust complementary information under HMD-induced visual occlusion and establish a foundation for
multimodal emotion recognition in naturalistic VR environments. This approach facilitates affect-adaptive applications, including communication training and therapeutic interventions. The
dataset will be shared upon request under an ethical-use agreement.
Read the full abstract ▾
All authors Birgit Nierula, Karam Tomotaki-Dawoud, Mert Akguel, Mustafa Tevfik Lafci, David Przewozny, Anna Hilsmann, Peter Eisert, Sebastian Bosse
Human-Computer Interaction
Code Black: Desktop-Mediated Co-Design of AR-HMD Microinteractions for Emergency Department Teamwork
Jonathan Segal, Jalynn Nicoly, Francisco Ortega et al. (+1 more)
Emergency Department (ED) teams coordinate shifting roles, medication decisions, and time-critical interventions under uncertainty. Augmented reality head-mounted displays (AR-HMDs) have shown potential to spatially anchor information during care, creating opportunities to examine how spatial interfaces might support teamwork. We conducted a speculative co-design study with 12 healthcare workers (HCWs) using an editable, desktop-mediated Unity-based 3D design probe to visualize and refine work-as-imagined AR-HMD interfaces for role-based notifications, task-specific timers, and dosage verification. Guided by microinteraction rules, participants identified future spatial user interfaces (SUI) requirements such as how they appear, update, or are dismissed in relation to clinical practice, safety concerns, and existing tools. Five returning participants and 26 additional HCWs subsequently provided follow-up feedback on derived visual interface alternatives. Findings show that desktop-mediated spatial co-design elicited formative specifications for role visibility, task-linked timing, and verification-oriented dosage assistance, while revealing tensions involving clutter, shared awareness, communication,
privacy, and reliability. Rather than evaluating a functional AR-HMD system or team-based clinical performance, this study contributes the Speculative Co-Design Framework for AR-HMD Teamwork (SCF-HMD) and a visual design catalog for translating expert critique of work-as-imagined (WAI) concepts into situated goals for future AR-HMD systems.
Read the full abstract ▾
All authors Jonathan Segal, Jalynn Nicoly, Francisco Ortega, Angelique Taylor