Human-Computer Interaction
"I don't know anything about laptops!" - User Perception of Digital Product Advisors Adapting to Their Knowledge Levels
Kevin Schott, Andrea Papenmeier, Daniel Hienert et al. (+1 more)
Conversational commerce uses digital assistants to support the search process and decision-making in e-commerce. Effective communication in these interactions can be facilitated by assistants adapting their communication style to users and supporting shared understanding. An open challenge in this context is adapting the presentation of complex product information to users with varying levels of domain knowledge. To investigate strategies for such knowledge-level adaptation, we set up a chatbot-assisted laptop search scenario. In a between-subjects experiment (n = 251), we examined novice and expert perceptions of product attribute recommendations presented as technical information only (T), or augmented with performance categories (TC), attribute explanations (TE), or both (TCE). For novices, approaches with explanations (TE, TCE) were perceived as more helpful and led to higher perceived learning than those without. Novices also rated the combined approach (TCE) more appropriate than the baseline (T) and TC in terms of information quantity, indicating that explanations are crucial to understand and benefit from performance categories. Critically, experts showed no significant differences across conditions, suggesting that providing supplementary information beneficial to novices did not detract from their experience. We distill these findings into four concrete design guidelines for inclusive text-based product advisors in technical domains: use TCE by default; keep a single inclusive interface; avoid standalone categories; and support user agency and personalize to the stated use case.
Read the full abstract ▾
All authors Kevin Schott, Andrea Papenmeier, Daniel Hienert, Dagmar Kern
Human-Computer Interaction
Cleo: A Transparent and Controllable Chatbot for Conversational Commerce
Kevin Schott, Jan Lattenkamp, Daniel Hienert et al. (+1 more)
We demonstrate Cleo, a transparent and controllable conversational product advisor that addresses the challenges of opacity, unpredictability of LLMs, and the complexity of comparisons in conversational commerce. With our chatbot system, we make four contributions: First, we introduce transparency by prompting the LLM to reflect on interpreted user needs, while an auditable ranking mechanism reveals loss values per attribute, explaining ranking decisions. Second, we propose controllability through a hybrid architecture separating deterministic ranking from
language generation. A ranker applies categorical filters and numeric loss functions over 3,638 product specifications. Meanwhile, a constrained LLM generates grounded descriptions constrained to catalog evidence, thus mitigating the risk of hallucinated or persuasive content. Third, we provide decision support in the form of natural-
language comparisons and a highlights feature. These aim to reduce mental workload by contextualizing specifications relative to user needs. Fourth, we contribute an extensible experimental system for IR and HCI researchers, as well as practitioners of conversational search and recommendation. Unlike traditional faceted search or opaque LLM-only recommenders, our approach allows for fluid conversation while maintaining algorithmic transparency. In a live demonstration, attendees will experience information needs elicitation and reflection, conversational refinement with real-time re-ranking, inspection of per-attribute loss explanations, and AI-generated multi-item comparisons. The system aims to advance the design of transparent and controllable conversational systems that provide support for decision-making during online product search.
Read the full abstract ▾
All authors Kevin Schott, Jan Lattenkamp, Daniel Hienert, Dagmar Kern
Human-Computer Interaction · Cross-listed from cs.CL
FormBharo: Designing and Evaluating a Voice Agent for Conversational Form Filling in Rural India
Aman Dalmia, Sanskriti Midha and Jigar Doshi
In India, almost every social benefit starts with a form, yet the people who need these benefits most are often unable to read or write. Reaching them requires a spoken conversation. Today that work falls to frontline health workers who enroll beneficiaries one at a time, a poor use of stretched capacity. We built FormBharo ("fill the form" in Hindi), a voice
agent that fills a structured form over a phone call under tight latency and cost budgets by pairing Large
Language Models (LLMs) with deterministic, rule-based validation and flow control. It is being piloted with ARMMAN, an NGO running large-scale maternal and child mobile-health programs in India, to enroll low-income, Hindi-speaking mothers in antenatal and postnatal care. To our knowledge, it is the first voice
agent piloted to fill an enrollment form for this population. We openly release FormVoiceAgentBench, a
benchmark pairing human-recorded Hindi audio with 3,760 multi-turn conversation tests across 960 simulated calls, to evaluate our
agent's components (transcription, extraction, reply generation) and end-to-end form completion under real acoustic variations. Form completion drops by up to ~41 points when LLMs receive error-prone real-
speech transcripts instead of reference ones. The rule-based controls recover many turn-level extraction errors, helping smaller, cheaper models match or surpass frontier models on form completion. Component performance does not predict end-to-end performance: GPT-5.5 leads turn-level extraction accuracy on reference transcripts (99.8%) but ranks lower on form completion. Since errors both propagate and cancel across the pipeline, the optimal model choice of models emerges only through end-to-end evaluation. Finally, no single model is best across accuracy, cost, and latency at once, so we use a Pareto-based weighted-sum scalarization to select a deployable configuration balancing the three.
Read the full abstract ▾
Human-Computer Interaction
OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction
Jiahao Huang, Zheng Lian, Jingyi Zhang et al. (+3 more)
Multimodal Large
Language Models (MLLMs) have demonstrated remarkable capabilities in emotional intelligence. However, prevailing research predominantly focuses on task-specific specialization, often neglecting inter-task synergy and leaving latent
reasoning potential underexplored. To bridge this gap, we introduce OneEmo, a unified affective generalist capable of mastering emotion perception, comprehension, and interaction. For this purpose, we first construct EmoWorld-130K, a comprehensive
dataset that distills specialized affective knowledge into explicit
reasoning trajectories via a human-in-the-loop workflow. Supervised fine-tuning on this corpus reveals significant mutual benefits derived from multi-task learning. Second, to fully unlock the latent
reasoning potential, we propose Emo-Chord, a novel reinforcement learning strategy that stabilizes optimization through unified multi-task reward allocation. Extensive experiments demonstrate that OneEmo achieves state-of-the-art performance against similarly sized baselines across most
benchmarks. Notably, despite having significantly fewer parameters than commercial models, OneEmo delivers highly competitive results. This paper paves the way for more reliable and interpretable affective computing. The code is available at https://github.com/waHAHJIAHAO/OneEmo.
Read the full abstract ▾
All authors Jiahao Huang, Zheng Lian, Jingyi Zhang, Zhide Chen, Xiaojiang Peng, Shaonan Wang
Human-Computer Interaction
PoseForge: Editable Pose Analytics for AI-Assisted Sports Coaching
Shuvam Swapnil Dash and Arpit Narechania
Athletic coaching increasingly relies on
video analysis, yet raw footage lacks tools to quantify motion or simulate valid technique corrections. Drawing on formative interviews with eleven cricket experts (coaches, performance analysts, captains, and players), we introduce PoseForge, a visual analytics system that extracts 3D skeletal poses from single-camera sports
videos for interactive movement analysis. In a cricket batting case study, PoseForge computes interpretable kinematic metrics such as feet gap and elbow angle, compares them against scientifically derived norms, and uses an AI coach to suggest targeted adjustments, presented visually and through natural-
language feedback (e.g., "increase feet gap by 10 cm"). Users can directly modify poses via mouse interaction or natural-
language instructions, with inverse kinematics maintaining anatomical plausibility and real-time updates of metrics and comparisons. An evaluation with the same eleven cricket experts found PoseForge effective for diagnosing movement issues and exploring corrective alternatives, highlighting its applicability in low-resource, academy, and grassroots coaching settings, while identifying opportunities for enhanced sport-specific metrics and longitudinal tracking. PoseForge is available as open-source software at https://github.com/DataVisards/PoseForge.
Read the full abstract ▾
Human-Computer Interaction
A Modular Workflow for Multimodal Reading Experiments
Thomas Krämer, Thomas Kosch, Dagmar Kern et al. (+1 more)
We introduce a web-based modular workflow for real-time
multimodal experiments in naturalistic online reading. The workflow integrates eye tracking,
EEG, and interaction data from mouse and keyboard, synchronizes them via Lab Streaming Layer, and links gaze to browser-based text at the word, sentence, and AOI levels. It is designed as a reusable experimental procedure that can be adapted to different sensors, tasks, and analysis goals. As a use case, we apply the workflow to a study of selective exposure in online news search and reading. During the experiment, gaze-derived measures are computed online, while
EEG and other synchronized streams are processed immediately after task sessions based on fixation-triggered segmentation. The resulting behavioral, neural, and linguistic metrics support selecting text passages for targeted post-task rating or labelling within the same lab session. The workflow thus provides a general basis for
multimodal research on reading and related cognitive processes, and supports the empirical validation, in ecological contexts, of constructs that are typically operationalised through self-report measures.
Read the full abstract ▾
All authors Thomas Krämer, Thomas Kosch, Dagmar Kern, Daniel Hienert
Human-Computer Interaction
Topic Matters: How Linguistic Properties can Shape Reading Behaviour in Selective Exposure Studies
Thomas Krämer, Dagmar Kern, Thomas Kosch et al. (+1 more)
Research on selective exposure frequently relies on eye tracking to study reading behaviour, often assuming that texts across different controversial topics are comparable once basic controls are applied. This assumption is problematic if topic-dependent linguistic properties systematically shape how users read and allocate
attention. We therefore examine whether such properties relate to differences in reading behaviour in selective exposure contexts. We analyse linguistic features and eye-tracking data from a laboratory study in which 68 participants searched for and read news articles on
climate change and migration policy. Our results reveal systematic differences in both textual characteristics and reading behaviour across topics. These findings identify an important methodological confound in selective exposure research and highlight the need to account for topic-specific linguistic properties when interpreting eye-tracking measures and designing systems intended to mitigate biased information consumption.
Read the full abstract ▾
All authors Thomas Krämer, Dagmar Kern, Thomas Kosch, Daniel Hienert
Human-Computer Interaction · Cross-listed from cs.SE
Mapping the Emerging Curriculum for AI-Assisted Software Engineering via Syllabus Analysis
Francis Geng, Anshul Shah, Mia Chen et al. (+5 more)
As Generative AI coding tools reshape professional software development, universities have begun designing courses to prepare students for AI-assisted development workflows. By analyzing the syllabi of these courses, we can gather empirical evidence about these courses, reveal how this emerging curricular area is being defined, and gain guidance for future curriculum design. We analyzed 23 publicly available syllabi and course materials of upper-division, credit-bearing courses that meet specific criteria, including explicitly addressing Generative AI in software engineering. Through iterative qualitative coding, we characterized courses' learning objectives, assessments, topics, and documented AI tools. Our analysis reveals commonalities and differences among these courses that allow researchers and educators to study and develop future courses.
Read the full abstract ▾
All authors Francis Geng, Anshul Shah, Mia Chen, Paul Denny, Juho Leinonen, Bill Griswold, Gerald Soosai Raj, Leo Porter
Human-Computer Interaction
Temporal Tracking of Reeb-Space Sheets
Mohit Sharma, Petar Hristov, Talha Bin Masood et al. (+1 more)
Time-varying bivariate fields arise in many scientific applications, where the relationship between two scalar quantities evolves over time. While topological methods such as merge trees provide an effective framework for identifying and tracking features in univariate data, analogous approaches for bivariate fields remain comparatively underexplored. Reeb spaces extend topological analysis to multivariate data by representing fiber connectivity through a collection of interconnected sheets, making these sheets natural candidates for describing bivariate structures. However, establishing temporal correspondences between sheets is challenging due to the structural complexity of Reeb spaces, sensitivity to noise, and the difficulty of defining meaningful similarity measures across timesteps. We present a framework for tracking Reeb space sheets in time-varying bivariate fields. The method establishes correspondences between sheets in consecutive timesteps using complementary similarity measures defined in the spatial domain and the range space. We evaluate the method on a synthetic
torus dataset and two time-varying molecular electronic structure
datasets. The results show that Reeb space sheet tracking reveals persistent structures and highlights interesting intervals of temporal change. Overall, the results demonstrate that Reeb space sheets can serve as trackable topological structures and provide a foundation for the visual analysis of time-varying bivariate data.
Read the full abstract ▾
All authors Mohit Sharma, Petar Hristov, Talha Bin Masood, Ingrid Hotz
Human-Computer Interaction
SpaceVLA: Spatially Grounded VLA for Robotic Manipulation with User-Authored Grasp and Place Anchors
Daniia Zinniatullina, Iaroslav Kolomiets, Mikhail Konenkov et al. (+2 more)
Vision-
language-action (VLA) models follow
language commands but often lack explicit spatial intent for manipulation. We present Visual Intent Anchors, an XR pipeline that lets users specify grasp and placement regions and renders them as image-space overlays for VLA control. We collect 200 Unity pick-and-place demonstrations and fine-tune OpenVLA-7B with LoRA on temporally subsampled annotated observations. The policy predicts tokenized 7-DoF incremental actions from marked RGB observations and
language. We evaluate the policy in closed-loop Unity trials, achieving a grasp success rate of 91.25% and mean grasp and placement errors of 0.5 cm and 0.7 cm, respectively.
Read the full abstract ▾
All authors Daniia Zinniatullina, Iaroslav Kolomiets, Mikhail Konenkov, Miguel Altamirano Cabrera, Dzmitry Tsetserukou
Human-Computer Interaction · Cross-listed from cs.AI
Unified Agent: Managing Interactions across Devices
Xinshuang Liu, Runfa Blark Li, Shaoxiu Wei et al. (+2 more)
As capabilities rapidly increase, AI
agents can move from running inside one app to acting across a user's devices over time. Yet existing
agent systems still fall short in this scenario. This is because observations are scattered across devices and moments, but mainstream systems are not designed around this fact: a single
agent that treats devices as tools lacks effective state management for all devices across time, and multi-
agent systems coordinate across
agents but do not maintain the compact carried state a cross-device, cross-time request needs. We argue that the
agent should maintain an effectively designed state that organizes engagement evidence, stated facts, and the standing request in a compact, action-ready form for deciding its action given the current observation. To compare state designs, we construct a
benchmark of user-
agent interaction across devices and time. We instantiate this principle in Unified
Agent, a stateful
agent that carries interaction evidence across devices and moments and uses it with the current observation to act. In the default setting, it significantly outperforms our adaptations of four published designs. Across changes in
multimodal large
language model (MLLM) family, capability, and
reasoning effort, it remains ahead of all compared systems, demonstrating that the state-design advantage is robust across MLLM settings. Our code and data will be publicly available on GitHub.
Read the full abstract ▾
All authors Xinshuang Liu, Runfa Blark Li, Shaoxiu Wei, Xin Lin, Truong Nguyen
Human-Computer Interaction
ASIDE: From Conflict Participants to Co-Observers Through Dyadic Spectator Reflection
Xinyi Zhang, Jingting He, Zicheng Zhu et al. (+1 more)
When two people argue over text, each knows what they meant and can only guess what the other was thinking. Existing AI reflection tools work from one person's account, and dyadic tools support co-expression without making the gap between accounts inspectable. We propose Dyadic Spectator Reflection (DSR), an interaction structure in which both partners externalize their models of each other independently and then encounter them together, and present ASIDE, a system that operationalizes it. ASIDE replays a past text conflict as a pixel-art theatrical scene where each character's unspoken state appears as an AI-inferred thought bubble either partner can contest and rewrite. Each edits alone, and the two versions meet only when both are done, in a scene they watch together. In an exploratory study, 10 couples revisited real conflicts and described the scene as a shared position from which to observe their own argument, stepping out of their roles without disengaging from it, and Divergence Cards as a way to locate specific interpretation gaps afterward. We contribute DSR as a reusable interaction structure, ASIDE as its system realization, and exploratory empirical findings on how couples used it.
Read the full abstract ▾
All authors Xinyi Zhang, Jingting He, Zicheng Zhu, Yuxin Su
Human-Computer Interaction · Cross-listed from cs.CY
Studying People to Study AI: Expert Perspectives on the Epistemic Fit and Barriers of Human Research in AI Safety & Ethics
Jessica Y. Bo, Paula Akemi Aoyagui, Shalaleh Rismani et al. (+3 more)
Safety risks of AI are becoming increasingly evident in human interactions with AI technologies. The prominent approaches to evaluating these risks favor technical methods, such as model
benchmarks and LLM simulations, often sidelining empirical research with human subjects. To examine this apparent gap in the acceptance of human research, we conduct an expert survey (n=93) and expert interviews (n=17) with AI Safety & Ethics (AISE) researchers from Technical, Sociotechnical, Governance, and Normative backgrounds. Our findings suggest that although there is a consensus that human research is valuable for generating evidence for AISE, its adoption and acceptance are constrained by perceived validity issues, tangible resource barriers, epistemic and personal preferences in methods, and infrastructural constraints from the broader research community. In particular, Technical researchers tend to value human research less and collaborate across disciplines less, suggesting an epistemic tension towards human methods. We propose recommendations for establishing the epistemic fit of human research within AISE and bridging the prohibitive limitations that researchers face, while avoiding performative 'human-washing'.
Read the full abstract ▾
All authors Jessica Y. Bo, Paula Akemi Aoyagui, Shalaleh Rismani, Dipto Das, Syed Ishtiaque Ahmed, Ashton Anderson
Human-Computer Interaction
CaRing: Preventing Carpal Tunnel Syndrome based on Daily Activities from Always-Available Input Device
Shuowei Li, Houdong Liang and Xingjian Dong
We present CaRing, a ring worn on the base knuckle of the index finger, a wearable system for detecting the start and end of mouse use to help prevent Carpal Tunnel Syndrome, in which the damage to the median nerve is permanent. CaRing senses finger movement, which neither a software timer nor a wrist-worn device detects. The displacement reported by an optical flow sensor is accumulated into a running value, then a zero point is measured while the hand rests on the desk at the start of each session. With this formulation, the start and end thresholds are expressed relative to the session's zero point. CaRing does not introduce any per-user parameter. We empirically demonstrate that approximately $90\%$ of start and end events are detected within two seconds of the researcher's label, using 35 recordings and a lab study with ten users.
Read the full abstract ▾
Human-Computer Interaction
Toward Resilient Human-AI Collaboration: A Lifecycle Taxonomy of Sociotechnical Risks and Cascading Failures
Md Foysal Ahmed, Isaac Kobby Anni and Md Main Uddin Rony
As AI systems become increasingly integrated into consequential domains such as healthcare, journalism, education, scientific research, organizational decision-making, and defense, effective human-AI collaboration has emerged as a critical challenge. However, the sociotechnical risks that undermine collaboration are often studied in isolation, obscuring the recurring failure mechanisms that cut across domains. This paper presents a lifecycle-oriented synthesis of human-AI collaboration risks spanning four stages: task allocation, interaction, feedback, and adoption. Drawing on evidence from diverse application domains, we identify six recurring cross-domain risk clusters: Trust Miscalibration, Cognitive Burden, Accountability Gap, Capability Erosion, Goal Misalignment, and AI Anxiety and Technostress. We further propose a conceptual interaction model that illustrates how these risks emerge from sociotechnical drivers, interact through cascading pathways, and ultimately affect team performance and human well-being. Our analysis shows that many collaboration failures stem not from isolated technical deficiencies but from interconnected sociotechnical dynamics, helping explain why piecemeal interventions frequently create unintended consequences. By synthesizing fragmented literature into a unified framework, this work provides a foundation for future empirical research, lifecycle-oriented governance, and the design of more resilient, trustworthy, and human-centered human-AI collaboration systems.
Read the full abstract ▾
Human-Computer Interaction · Cross-listed from cs.AI
Epistemic Trustworthiness in Generative AI: A Normative Framework for Warranted Reliance in High-Stakes Workflows
Nimisha Karnatak, Max Van Kleek and Nigel Shadbolt
Generative AI systems are increasingly deployed in high-stakes professional contexts, where their outputs shape what users believe, how they reason, and what they treat as settled. This raises a central question for responsible AI: under what conditions is reliance on generative AI outputs epistemically warranted rather than behaviourally induced? Existing frameworks largely ask whether AI outputs are accurate, fair, explainable, safe, or trusted by users. These questions remain necessary, and each can contribute to warranted reliance. However, they do not directly specify warranted reliance as a distinct evaluative target: the conditions under which users are justified in treating AI outputs as inputs into their own
reasoning. We argue that this requires an account of epistemic trustworthiness: what makes a system epistemically worthy of reliance. Drawing on philosophical accounts of trustworthiness as competence and audience-orientation, we develop a constitutive normative framework comprising three jointly necessary and non-fungible conditions. First, epistemic humility requires systems to represent and communicate the limits of their competence. Second, epistemic access requires systems to enable users to inspect, question, and contest outputs in context. Third, resistance to epistemic injustice requires systems to recognise users as legitimate epistemic
agents and avoid marginalising their knowledge and experience. Through real-world case analyses in legal
reasoning, medical
reasoning, and hiring, we show how failures of epistemic humility, epistemic access, and resistance to epistemic injustice can produce consequential harms that standard measures of accuracy,
fairness, and usability do not address on their own. We conclude by outlining design and evaluation implications for GenAI systems organised around epistemically warranted reliance rather than output correctness alone.
Read the full abstract ▾
Human-Computer Interaction
A Multi-Layer System for Ultra-High-Resolution Static 360-Degree Telepresence
Jiapeng Chi, Gerd Bruder, Carsten Neumann et al. (+2 more)
360-degree
video telepresence offers strong immersive potential but remains constrained by the limited resolution of current capture and display hardware. Many telepresence installations feature fixed viewpoints and largely static scenes, yet optimization strategies tailored to such setups have received limited
attention. We present a multi-layer, ultra-high-resolution system for static 360-degree telepresence that combines an 8K panoramic camera with a rotatable 4K pan-tilt-zoom (PTZ) camera. Our approach builds a three-layer representation: (1) a tile-based ultra-high-resolution panoramic background, generated by offline stitching high-detail 4K PTZ scans onto the base 8K panorama to achieve effective resolution beyond native capture, and represented as a set of spatial tiles; (2) a dynamic update layer that composites foreground motions from the 8K stream via real-time high-resolution background matting; and (3) a region-of-interest 4K layer that streams a real-time PTZ view of the selected region and additionally updates the corresponding background tiles over time. We evaluate the proposed system through comparisons with representative
video super-resolution approaches and a user study assessing perceived detail and immersive experience. Our results indicate that tile-based background refinement, together with user-guided updates, provides a practical way to balance panoramic fidelity and interactivity in static 360-degree telepresence.
Read the full abstract ▾
All authors Jiapeng Chi, Gerd Bruder, Carsten Neumann, Carolina Cruz-Neira, Dirk Reiners
Human-Computer Interaction
Turing's First Imitation Game: Design Concepts and a Human-Approximates-Machine Reading
Sharon Temtsin and Christoph Bartneck
This paper examines Turing's 1948 report, "Intelligent Machinery", as an important conceptual source for the later imitation games. Its first contribution is to identify and integrate the design concepts underlying the 1948 chess-based imitation game: the possibility that intelligent machines may make mistakes, the exclusion of irrelevant physical features, the role of the human judge, and Turing's claim that intellectual activity consists mainly of search. The paper's second contribution is to argue that restricting the human contestant to a rather poor chess player increases the role of intellectual search and makes human behaviour more comparable to machine behaviour. This interpretation presents the 1948 game as a human-approximates-machine game and suggests that the imitation game framework can be used not only to ask whether machines imitate humans, but also to examine when human intelligence becomes machine-like under specific task constraints.
Read the full abstract ▾
Human-Computer Interaction · Cross-listed from cs.CY
Vibe Compiler: A Research-Logic Synthesis Tool That Runs without Prompt Engineering -Toward Enhancing Metacognition for Sustaining Agency in the Age of Generative AI-
Riichiro Mizoguchi, Tomoki Aburatani, Kento Koike et al. (+1 more)
Used as a capable servant, generative AI has greatly accelerated intellectual work, yet it also risks eroding human epistemic agency by encouraging uncritical acceptance of AI-generated
reasoning. Preserving that agency calls for mechanisms that augment human metacognition during AI-assisted work. We therefore propose the Synthesis-Analysis Reciprocity Model, which views intellectual construction as a reciprocal interaction between two cognitive functions. Synthesis selects and combines the components of the artifact; Analysis evaluates them critically against objective indicators and constrains the Synthesis that follows. Grounded in this model, we present the Vibe Compiler, a research-logic compiler that helps researchers turn vague intuitions (Vibes) into coherent research logic. The system attempts to compile those intuitions against a paper ontology of 16 academic parameters. It treats compilation failures as signs that logical components are missing. Rather than fill those gaps autonomously, it returns reflective questions that
prompt researchers to develop the missing
reasoning themselves. We further characterize the origins of structural gaps along two orthogonal dimensions: cognitive function (Synthesis versus Analysis) and executing
agent (human versus AI). The four resulting types of origin give a principled way to identify where breakdowns in intellectual construction arise. Crucially, our design implements the type in which the AI probes its own synthesized output, itself driven by the user's Vibes, and thereby stimulates human metacognition. This choice raises researchers from passive "Makers" of the output into "Managers" who critically direct and validate what the AI produces. In a prototype on NotebookLM and Gemini, AI behavior depended less on prompting than on the structure of the knowledge supplied. The framework spans a learner layer and a researcher layer.
Read the full abstract ▾
All authors Riichiro Mizoguchi, Tomoki Aburatani, Kento Koike, Machi Shimmei
Human-Computer Interaction · Cross-listed from cs.CR
PromptShield Home: Ambient Multimodal Prompt Injection Defense for Smart-Home Agents
He Zhang, Feilong Li, Dingning Long et al. (+5 more)
Smart-home assistants increasingly use
multimodal large
language models (MLLMs) that perceive
video and audio directly. This raises a safety question specific to the home: can the
agent tell a genuine user command from ambient or externally-sourced content, television
speech, on-screen text, or an overheard conversation, that merely looks like a command? We introduce PromptShield-Home, a pilot
benchmark of realistic smart-home scenarios spanning addressee ambiguity, screen/audio injection, health-monitor false triggers, mixed occupancy, and a legitimate-command floor, and use it to compare three abstraction layers: traditional detectors (L0), a single MLLM
agent (L1;
vision,
vision+ASR, and audio-visual), and multi-
agent mediation (L2; voting, role specialists, cross-model arbitration). Because the label distribution is skewed toward inaction, aggregate accuracy is misleading, a constant always-block predictor scores 82%, so we report unsafe-execution and safe-completion rates separately. The two paradigms fail in opposite ways: detectors act on everything, while every MLLM configuration over-refuses, completing almost no genuine command and missing a true fall in every case. Crucially, their correct sets are disjoint: an oracle that always picks the right layer reaches 94.1%, against 76.5% for the best single layer. We report this as an upper bound, not a system - no router is implemented - and argue that home-
agent safety is best served by learned routing and sensor fusion, not by replacing detectors with an MLLM.
Read the full abstract ▾
All authors He Zhang, Feilong Li, Dingning Long, Yilin Cui, Peijun Zhang, Yuewen Zhang, Qianyao Xu, Xinyi Fu
Human-Computer Interaction
Mixed Uncertainty in One View: Co-Visualizing Statistical Variability and Qualitative Confidence
Racquel Fygenson, Lace Padilla and Laura E. Matzen
Forecasting involves multiple forms of uncertainty, including both uncertainties that can be quantified directly (quantitative uncertainty) and those that must be expressed through experts' subjective judgments about the forecast and its context (qualitative confidence). Past work has established that conveying both quantitative uncertainty and qualitative confidence in forecasts can alter readers' decision making, but little research investigates the impact of how these forms of uncertainty are presented. In this work, we present three preregistered human-subjects studies (total n = 923) on how different methods of visualizing qualitative uncertainty alongside line charts' confidence intervals affects non-experts' decision making. In particular, we investigate representing qualitative uncertainty separately via text and icons, and integrated into quantitative confidence intervals via color, transparency, and a blurred stroke design. In Experiment 1, we confirm that showing qualitative confidence alongside statistical variability can change patterns of decision making, replicating findings from previous work in the new context of time-series line charts. In Experiments 2 and 3, we find several non-textual encoding techniques that produce similar effects in participants' incorporation of qualitative confidence into their judgments. Our findings suggest actionable guidelines for visualization designers who seek to represent multiple forms of uncertainty for a single line chart forecast. A free copy of this paper and all supplemental materials are available at https://osf.io/7ya2c/overview.
Read the full abstract ▾