Computers and Society
From Efficiency to Sufficiency: The Cost-Aware Measurement Design Framework for Automated Essay Scoring
Yi Gui
Compute cost in automated essay scoring (AES) is usually treated as a technical efficiency issue rather than a measurement design concern. Yet deployment decisions require more than reporting latency, model size, or inference expense. In high-stakes settings, total burden includes validation, monitoring, adjudication, compliance, and governance; in instructional settings, pricing, licensing, and integration may dominate cost. This paper reframes AES as a cost-constrained measurement system and introduces the Cost-Aware Measurement Design (CAMD) Framework, a four-layer structure consisting of metering, allocation, unit economics, and decision, for translating computational and workflow resource records into decision-relevant cost constructs. Direct cost is specified as monetary outlay plus attributable labor and time, and two criteria, design attribution and auditability, determine what the cost boundary excludes. Burdens that satisfy neither criterion, such as institutional capability and governance posture, are represented as constraints on which designs are admissible rather than priced. The paper defines cost per defensible score (CPDS) and argues that AES systems should be compared by quality-cost sufficiency: minimizing cost among defensible scoring designs that satisfy measurement requirements. Scoring volume conditions all four layers, so cost per defensible score is treated as a function of scale rather than as a single transferable number. CAMD provides a portable basis for representing, comparing, and evaluating cost across AES studies and deployment contexts.
Read the full abstract ▾
Computers and Society · Cross-listed from cs.CL
Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources
Mir Tafseer Nayeem and Davood Rafiei
Benchmarks shape how model capabilities are measured and interpreted, yet their scores can obscure whose conventions, contexts, and judgments define successful performance. We introduce a user-centric audit framework that distinguishes users who contribute
benchmark content from those whose needs evaluation results are intended to represent. The framework operationalizes seven dimensions across
benchmark content, metadata, documentation, and scoring procedures. Across 16 resources, we find that American-centered conventions account for 62.4-94.4% of matched signals in 11 text
benchmarks, while geographic visibility in five image-text resources is concentrated and strongly associated with national GDP share (mean Pearson r = 0.780; Spearman p = 0.778). Documentation reveals a systematic source-target gap: all 16 resources operationalize source-user context, but none operationalize target-user context. Paired scoring experiments further show that meaning-preserving answer variants reduce DROP exact-match accuracy from 100.0% to 33.3%, while TruthfulQA exhibits metric-dependent reductions in reference similarity. Our framework and documentation checklist make user assumptions auditable, clarifying what
benchmark scores represent and where evaluation claims are supported.
Read the full abstract ▾
Computers and Society · Cross-listed from cs.AI
MemeTrace: A Multimodal LLM-Based Agent for Safety-Aware Meme Provenance Analysis
Bo Shao, Yihan Ma, Yiting Qu et al. (+2 more)
Online memes are often carriers of unsafe content such as disinformation and propaganda. Their rapid
evolution in response to news events and strong dependence on contextual information make it difficult for platforms and moderators to detect and moderate them effectively. Moreover, existing moderation studies typically classify memes as safe or unsafe based on individual images, lacking the ability to perform broader provenance analysis, i.e., where a meme originates, how it spreads, what cultural symbolism it represents, whether it poses potential risks to society, etc. To bridge this gap, we introduce MemeTrace, a
multimodal LLM-based
agent that automatically traces the provenance and potential risks of emerging memes. We evaluate MemeTrace on Meme100, a
dataset of 100 real-world memes, using both human annotation and automatic metrics. The results demonstrate that MemeTrace substantially outperforms the VLM baseline. More importantly, MemeTrace achieves higher accuracy in detecting unsafe memes, especially those involving disinformation and sensitive content, and remains robust under visual perturbations.
Read the full abstract ▾
All authors Bo Shao, Yihan Ma, Yiting Qu, Michael Backes, Yang Zhang
Computers and Society · Cross-listed from cs.CV
Trust Before Autonomy: Rethinking Agentic Remote Sensing for Crisis Response
Georgios Angelidis, Aggelos Psiris, Spyridon Evangelatos et al. (+4 more)
Agentic remote sensing is evaluated on how many questions it can answer, whereas crisis response depends on whether its answers can be relied on and how much of the decision can safely be handed to the
agent. This paper argues that autonomy should be earned, not assumed: An Earth-Observation (EO)
agent should be granted autonomy in proportion to demonstrated reliability, epistemic honesty and verifiability. The position is supported by an evidential meta-analysis of twelve agentic RS/EO/GIS
benchmark studies from peer-reviewed venues. Only one repeats runs, three contain unanswerable tasks and none measures calibration. Their own tables show that completed workflows are often not correct (task success exceeds tool-call correctness by up to 54pp on identical queries), that the gap to an expert trajectory widens overall with trajectory length (from 7 to 56pp), and that capability does not predict abstention (Spearman r from -0.25 to 0.50, all n.s.). The paper then contributes three EO-specific trust properties, an autonomy ladder for crisis EO tasks, and a readiness protocol that replays archived emergency activations before any live use. In this context, current evidence justifies assistive roles only.
Read the full abstract ▾
All authors Georgios Angelidis, Aggelos Psiris, Spyridon Evangelatos, Thomas Lagkas, Panagiotis Sarigiannidis, Iraklis Varlamis, Georgios Th. Papadopoulos
Computers and Society · Cross-listed from cs.CL
Looking Complete Is Not Being Grounded: Evaluating LLM Sitreps Against Human-Authored Situation Reports
Sachin Kumar
Evaluation for automatic crisis reporting should measure what practitioners need, not what is convenient to compare. Through LLM-assisted, author-approved content analysis, we derive a 15-slot information schema from professional humanitarian situation reports using qualitative content analysis, then evaluate 121 human and 261 LLM-generated reports across five disasters and three hazard types. Under google/gemini-3.7-flash on Cyclone Idai, the best machine arm is nominally above the empirical human ceiling (0.827 vs. 0.821), but the difference is statistically indistinguishable from zero (95% bootstrap CI $[-.049,.059]$; permutation $p=.852$), and the ordering does not replicate under openai/gpt-5.6-luna. On a blind 100-item Idai sample, the author agrees with the Gemini judge at 0.910 accuracy and unweighted Cohen's $\kappa=.799$. All four judges clear the plan's raw-agreement criterion of .7; only Gemini reaches .7 on chance-corrected $\kappa$. Across the three primary events, machine reports reproduce 5.4-11.0% of same-day official figures. Across those same events, raw exact-number matching links 14.3-19.6% of official figures and 85.4-90.1% of machine figures to the social stream; manual audit confirms 45/50 declared absences, 4/25 official matches, and 24/25 machine matches semantically. The issuing-provenance gap is positive in all 12 observed event-judge
cells: report date, organisation, series, and authority were not supplied at professional-report coverage in any
cell. On the three primary events, ROUGE-L is uninformative, BERTScore is negatively associated with completeness after multiplicity correction, and a generic LLM judge changes sign.
Read the full abstract ▾
Computers and Society · Cross-listed from cs.AI
AgentStripes: An Evidence-Graded Maturity Model for AI Agents
Nutan Kumar Naik, Vijay Prasad Poudel, Aditya Kumar Saroj et al. (+2 more)
Benchmarks report what an Artificial Intelligence (AI)
agent can do on a task suite, but not whether one released version has been engineered with the controls that make it reliable, safe and observable, or how strongly that is shown. This paper proposes AgentStripes, an evidence-graded maturity model for AI
agents. It defines levels (0 to 7 stripes), process areas (12 areas holding 40 capabilities), three gates (basic, safety, proof) that hold an
agent at a level however high its score, and evidence grades E0 to E5. Each applicable capability receives a rating from 0 to 3 and a separate evidence level; these are combined into a credit, aggregated into a severity-weighted score and converted into stripes that the gates limit. Instruments gather evidence,
language-model judges interpret it under explicit criteria, and the arithmetic is deterministic. The model is compared with standards at topic level without a claim of conformance, and an unexecuted validation protocol is specified. An empirical evaluation (exploratory, non-confirmatory) of 30 public Python
agent repositories, with static evidence only, is a demonstration and not a validation: static evidence caps every final stripe at 2, 26 repositories end at one stripe and 4 at two, and this distinction is unreliable, since re-judging changed the final stripe of 4 of 22 repositories (engine, dossier and panel changed), whereas none of 8 panel-only repeats, all at stripe 1, changed. The parameters are expert conventions without outcome based calibration, and the stripes have not been shown to predict reliability or safety.
Read the full abstract ▾
All authors Nutan Kumar Naik, Vijay Prasad Poudel, Aditya Kumar Saroj, Saurav Samantray, Abhishek Patel
Computers and Society · Cross-listed from cs.LG
Fairness under Prediction Error in Predict-then-Optimize Pipelines
Hilda Adwubi Osei
Fairness constraints in predict-then-optimize pipelines are necessarily enforced on predicted outcomes $\hat{y}$, since true outcomes $y$ are unobserved at decision time. We study whether a demographic parity constraint enforced this way continues to hold against $y$, and how the resulting violation scales with prediction error. Using a budget-constrained allocation problem solved as a linear program, we show empirically that the true-outcome
fairness gap grows with prediction error $\sigma$ and saturates, well fit by $c(1 - e^{-\sigma/\tau})$ with $c = 0.1545 \pm 0.0103$ and $\tau = 3.9045 \pm 0.3745$; its mean is invariant to sample size while its variance decays at the standard $O(1/\sqrt{n})$ rate. We derive a two-part theoretical bound: a closed-form local sensitivity result showing the gap's first-order response to prediction error depends on only two individuals per group, validated against the solver to within three significant figures; and a universal, error-independent ceiling following from the boundedness of the decision variable, confirmed consistent with but looser than the empirical saturation level. On US Census income data, enforcing demographic parity on a trained predictor's output reduces an unconstrained disparity in true income to roughly one-third of its original size, leaving 34.4% $\pm$ 3.3% (California) and 35.7% $\pm$ 1.8% (Texas) of the disparity intact and undetectable from the predictions alone – consistent not only in mechanism but in approximate magnitude across two states with different demographics and income distributions, and stable across ten independent train/test splits in each. A
fairness constraint that appears fully satisfied at decision time can mask a substantial, systematic violation of the outcome it was meant to protect.
Read the full abstract ▾
Computers and Society · Cross-listed from cs.CL
Decomposing and Predicting Annotation Disagreement in Harmful Content Across Languages
Irem Ecem Yelkanat, Faeze Ghorbanpour and Alexander Fraser
Annotator disagreement is increasingly recognized as a meaningful property of subjective tasks, rather than merely noise to be removed. Predicting which items are likely to be contested could help allocate annotation effort and route difficult cases to human review. We propose predicting disagreement from interpretable semantic properties of the text. An LLM answers a set of binary questions covering harmful-content characteristics, disagreement-related cues such as borderline severity and missing context, and
language-sensitive cues such as local slurs. A classifier predicts disagreement from these answers, and we analyze which questions contribute most to the predictions. Across four
datasets in English, German, and Turkish and three open-weight LLMs, our structured question features outperform direct prompting by 10.5 ROC-AUC points and the strongest text-based baselines by 5.0 points. A compact subset of eight questions performs within 1.5 ROC-AUC points of the full twenty-question set. In leave-one-
dataset-out evaluation, models retain roughly 90-95% of their in-domain ROC-AUC. Across models and
datasets, questions about derogation, missing context, and decision boundaries serve as the strongest predictors, suggesting that what makes harmful content contested is partly observable in the text.
Read the full abstract ▾
Computers and Society
A Global View of How LLMs Value Human Time
Yifan He, Manon Reusens, Sofie Goethals et al. (+1 more)
Large
language models (LLMs) increasingly act as personal assistants; however, little is known about how they capture national-level preferences on a global scale. Given that "time" is a universal and representative concept, we examine LLMs' time preferences by varying only users' country and measuring the compensation LLMs deem appropriate for a 60-minute appointment delay. We then compare LLMs' responses with GDP per capita on Purchasing Power Parity (GDP_PPP), Mean Income, and a literature-based index (Patience Score). Our results across 193 countries and 5 LLMs demonstrate that people's time across the world is evaluated in noticeably different ways among LLMs, as well as varying levels of robustness under
prompt variations. LLMs consistently recommend the highest compensation for Europe, but mostly the lowest for Africa. Based on GDP_PPP, LLMs underestimate the value of time for users living in Asia and South America. These findings indicate that geographical patterns are embedded in LLMs and may translate in different treatment in agentic settings.
Read the full abstract ▾
All authors Yifan He, Manon Reusens, Sofie Goethals, David Martens
Computers and Society
How Safe Is ChatGPT Health Advice? Evidence from Real User Conversations in Four Global South Countries
Shreyasi Roy Chowdhury and Kiran Garimella
People increasingly bring their health questions to ChatGPT, yet safety evaluations of LLM health advice rest almost entirely on clinician-authored vignettes, synthetic
prompts, or simulated user profiles. The gap is widest for the Global South, where concerns about demographic bias and unsafe recommendations are well documented but have never been tested on real usage. We present, to our knowledge, the first demographically grounded safety evaluation of real-world health conversations with ChatGPT. From a consent-based data-donation corpus of 202,590 conversations from 1,252 users in India, Nigeria, Brazil, and Pakistan (December 2022 to February 2026), each linked to self-reported age and gender, we identify 15,555 health conversations, classify their health intent using an LLM classifier adapted from an established taxonomy, and score response safety under both a context-blind and a user-context-aware rubric. Roughly one in five health conversations receives a high-risk rating. Risk concentrates in symptom and condition queries and recurs through identifiable failure modes: unsafe medication or dosage guidance, harmful self-diagnosis, missing safety warnings, encouraging over-reliance on the model, and failure to refer users to professionals. Supplying the user's own accumulated context raises the high-risk share by 13-17 percentage points, and about one in four conversations rated safe in isolation turns out to be unsafe for the specific person asking. Safety evaluations built on synthetic cases are likely to miss these harms.
Read the full abstract ▾
Computers and Society · Cross-listed from cs.AI
The Singularity Has a Distribution: Exact Timing Laws and Detection Limits for a Stochastic Intelligence Explosion
Xiaoyu Li, Zheng Gao, Chiwun Yang et al. (+1 more)
Automating AI research creates a positive feedback loop between software capability and research effort. We study a
stochastic realisation of this loop in which independent, positive, multiplicative improvements arrive at a rate proportional to a power of current capability. Within this model, randomness leaves the deterministic explosion threshold unchanged but replaces the blow-up date by an explicit distribution. For equal-sized improvements the explosion time is a geometrically weighted sum of independent exponentials; its $q$-series representation gives its transforms, moments and tail behaviour, including a two-term small-time asymptotic. For Lomax-distributed improvement sizes the explosion time is exactly Gamma distributed. Self-similarity implies that, with parameters known, the relative uncertainty about the remaining time does not vanish as explosion approaches. We formulate a fixed-sample test between constant and geometrically increasing idea-arrival rates. If the baseline rate and the alternative log-rate increment $\ell$ are known, the minimum sample size at error level $\alpha$ is asymptotic to $(12z_\alpha^2)^{1/3}\ell^{-2/3}$, where $z_\alpha$ is the upper-$\alpha$ standard normal
quantile. Uncertainty about the growth exponent near its critical value produces a $1/t$ tail with an explicit constant, and hence an infinite conditional mean waiting time, although explosion is certain given a positive exponent. Separate extensions show that a fixed deployment delay prevents explosion in the specified delayed-feedback models, that power-law diversion of effort shifts the growth threshold, and that in a deterministic oversight race a fixed allocation share does not change the asymptotic regime, except at a critical equality of exponents. Numerical illustrations, which are not empirical forecasts, show how timing and detectability depend on the granularity of improvements.
Read the full abstract ▾
All authors Xiaoyu Li, Zheng Gao, Chiwun Yang, Zhizhou Sha
Computers and Society
Institutionalizing Trust in Data Exchange: Trust-Anchor Model of the Data Credit Guarantee Organization and Its Simulation-Based Comparison
Hiroshi Mano, Chie Sugiyama and Teruaki Hayashi
Data trading
markets that span industries and regions are expanding, but the implicit trust that supports data use within an organization does not carry over to cross-domain exchange. This paper examines the Data Credit Guarantee Organization (DCGO), a proposal to institutionalize trust through third-party verification and certification, and asks what it means to trust data, what such certification can support, and under what conditions DCGO outperforms existing trading models. We decompose trust in data into four layers: authenticity of entity, authenticity and integrity of data, rights and conditions of use, and history and auditability. A third party can certify authenticity but not the correctness of content, and DCGO is positioned as a trust anchor spanning existing marketplaces and data spaces. We then extend the Law Merchant model with three data-specific properties, namely imperfect ex-post detection, cheap pseudonyms, and non-rivalry of harm, and compare bilateral trading with DCGO by
agent-based simulation. DCGO keeps the buyer's payoff near its theoretical upper bound regardless of the
market environment, whereas bilateral trading loses even when fraud is always detected. Identity verification proves to be a prerequisite for record sharing, and pre-verification is the only function that limits the spread of harm. On the seller side, honest sellers join voluntarily whenever the price premium exceeds the fee. Lock-in and a critical mass for
diffusion, however, call for fee governance and support for entry.
Read the full abstract ▾
Computers and Society · Cross-listed from cs.PF
How Reliable Are Large Language Models for Soil Organic Carbon Assessment? An Australian Benchmark
Yingxuan Cheng, Tong Li, Yayong Li et al. (+5 more)
Soil organic carbon (SOC) is an indicator of agricultural soil condition and carbon storage, but numerical outputs from large
language models (LLMs) require validation before use in monitoring or management. We evaluated seven general-purpose LLM configurations on three direct-response tasks: estimating 0–10 cm SOC concentrations for 308 Australian paddocks under two information conditions, calculating SOC stock in 120 matched scenarios, and comparing 300 paddock pairs. References were measured SOC records or deterministic calculations; response validity, repeated-call dispersion and estimated API cost were assessed separately. Additional measured soil attributes reduced concentration mean absolute error by 13–62% across configurations, although high-SOC underprediction and paddock-specific errors persisted. Six configurations were highly accurate in stock calculations using equation-native units without coarse fragments, but accuracy declined with physically equivalent alternative units. Pairwise accuracy was 90.5–94.6% overall and 59.1–70.9% for 52 closely separated pairs; some configurations were sensitive to presentation order. Nearly all outputs were parseable, without necessarily being correct. The
benchmark identifies three distinct requirements for AI-mediated SOC assessment: agreement with measured concentrations, unit-consistent stock calculations and reliable discrimination between sites. These findings support task-specific validation and auditable calculations, rather than unverified use of LLM outputs as ecological evidence.
Read the full abstract ▾
All authors Yingxuan Cheng, Tong Li, Yayong Li, Yuchuan Fan, You-Gan Wang, Jinran Wu, Scott Chapman, Geoffrey J. McLachlan
Computers and Society
Technology-Supported Moderation in Online Deliberation
Lodewijk L. Gelauff, Ashish Goel and Sukolsak Sakshuwong
Mini-publics are an increasingly popular family of democratic innovation to broaden citizen engagement. However, convening a representative sample in person is resource-intensive, and requires significant organizational efforts to coordinate travel, venue logistics, and facilitator training. This chapter outlines the design and development of the Stanford Online Deliberation Platform, initially tailored for Deliberative Polling, building upon insights from in-person moderation in that setting, but supporting a broader range of methods and approaches in the deliberative space. The platform supports structured conversations for small groups (ideally 8-15 people) with automated moderation functionality, enabling the groups to assume ownership of the deliberative conversation rather than relying on a moderator persona. We describe the platform's architecture and key design decisions, including the deliberation flow, room assignment, speaker management, speaking nudges, agenda management, outcome development, and real-time monitoring. Between its inception in 2018 and 2025, the platform has facilitated over 30,000 unique participants engaging in deliberations; we catalog 54 deliberations with at least 100 participants up to December 2025, spanning a diverse range of
languages, countries, and topics. Finally, we establish a standardized vocabulary for future analyses by defining the terminology employed in the platform's data and documenting significant software revisions over time.
Read the full abstract ▾
Computers and Society · Cross-listed from cs.SI
A null model to detect coordinated online behavior integrating evidence from multiple indicators
Filipi Nascimento Silva, Manita Pote, Lake Yin et al. (+3 more)
Coordinated online accounts can amplify messages and manipulate
attention, but detecting such behavior is challenging due to confounding factors like account activity and item popularity. We propose a statistical null-model framework that assesses behavioral similarity between accounts by comparing observed coordination indicators against activity- and popularity-conditioned reference distributions. By combining evidence across multiple indicators and partners, our method can identify suspicious coordination links more accurately. Evaluations on nine labeled information-operation
datasets involving over 152 thousand accounts show that aggregating p-values multiplicatively across indicators and partners yields improved detection performance. While results vary across information operations, this approach offers a generalizable and statistically principled tool for uncovering coordinated online behavior.
Read the full abstract ▾
All authors Filipi Nascimento Silva, Manita Pote, Lake Yin, Luca Luceri, Filippo Menczer, Alessandro Flammini
Computers and Society · Cross-listed from cs.LG
Towards Green AI: Evaluating Energy Efficiency for Quantized LLM Inference
Petar Basta
While it is known that quantization of large
language models (LLMs) reduces
memory usage via lower-bit weights, studies quantifying the resulting impact on energy usage and carbon intensity are scarce. We present the first unified evaluation of weight-only quantization strategies for LLM inference across varying input/output lengths and GPUs, highlighting differences in energy efficiency in addition to accuracy degradation and runtime. We study quantized inference on Llama 2 7B, Phi 3.5 Mini Instruct, OLMo 1B, Qwen 2.5 7B, and Qwen 2.5 14B across GLUE MNLI, MMLU, HumanEval, and GSM8K
datasets, evaluating 10 post-training quantization (PTQ) strategies on NVIDIA H100, A6000, A100, and L4 GPUs. We identify a novel trend showing that quantization techniques tend to exhibit peak energy efficiency relative to full-precision baselines when inputs are sufficiently long and outputs are short. Furthermore, we show that quantization strategies become marginally more energy-efficient relative to full-precision models as batch sizes increase, though gains are modest. Notably, fused-kernel implementations such as EETQ int8 and Bitsandbytes int8 offer the highest energy savings, up to 4x compared to FP32 on short text generation, with negligible accuracy loss. Finally, we observe that energy usage closely tracks runtime on our evaluated
benchmarks, indicating that, in practice, latency optimization can serve as a reliable proxy for reducing the environmental footprint of LLM services. We conclude with suggested directions for strategically selecting low-carbon quantization strategies based on specific inference requirements.
Read the full abstract ▾