Machine Learning · Cross-listed from stat.ME
Null-Corrected Conditional Coverage in Materials Property Prediction: A Five-Dataset Replication
Irene Boruah
Uncertainty quantification for materials property prediction increasingly reports conditional coverage, coverage within strata of the predicted value, alongside marginal coverage. The worst-bin statistics used for this are downward-biased in finite samples, a known defect that applied materials work does not generally correct for. We replicate a single-dataset conditional coverage study across five regression datasets (n = 437 to 21,263), seven interval methods and 50 random splits each, correcting the worst-bin statistic against its expected value under exact conditional validity. The null is simulated from realized per-bin test counts and referenced to each method's own marginal coverage, separating marginal miscalibration from uneven coverage. On the smallest dataset the correction runs from 0.061 to 0.096, between a third and two thirds of the shortfalls actually reported, against 0.009 to 0.011 on the two largest. Split conformal prediction's failure survives the correction on four of five datasets; two conclusions of the antecedent study do not. Conformalized quantile regression, characterized there as a negative case, fails on one dataset only. Cross-fitted normalized conformal prediction, grouped there with the successful methods, fails on three of five. Group-conditional (Mondrian) conformal prediction stays below our significance criterion on every dataset, yet exceeds the null on three of five under an octile refinement of the variable it calibrates on. A level-matched ablation locates that uniformity in group-conditional calibration rather than in the elevated quantile level, and removes Mondrian's apparent width cost entirely. Conditional coverage evaluations should report the null distribution of the worst-bin statistic used, the method's realized marginal coverage, and the resolution of the evaluation partition relative to any partition the method calibrates on.
Read the full abstract ▾
Human-Computer Interaction
VIVID: Bringing Tool-Augmented Agents into Enterprises
Yichu Fang, Ziqiang Zhang, Qiyang Shi et al. (+11 more)
Tool-augmented large
language model
agents connect natural-
language requests to enterprise services, yet their operational viability depends on reliable execution, organizational control, and system responsiveness. We present VIVID, a deployed tourism-management
agent system designed to study these requirements in a real enterprise environment. On an audited
dataset of 283 queries, its Qwen3-8B configuration achieves 59.06% strict end-to-end success across the evaluated decision stages, indicating useful but bounded autonomous capability. Stage-wise KV-cache eviction reduces mean latency by 34.4%, while targeted optimization reduces the mean latency of the five slowest measured business tools by 81.1–92.3%. In a three-month deployment serving more than 1,500 personnel, VIVID reduced average per-order processing time from approximately ten minutes to under two minutes, with a final-task error rate below 1% after human-in-the-loop control. These results show that enterprise
agent viability requires system-level consideration beyond isolated tool-calling accuracy.
Read the full abstract ▾
All authors Yichu Fang, Ziqiang Zhang, Qiyang Shi, Zilong Wang, Lin Shen, Zhikang Ou, Jing Ma, Jiayuan Chen, Lei Sun, Dai Cheng, Yi Qiao, Yu He, Wei Zhang, Xiaoyu Shen
Human-Computer Interaction
Cellery: Gradual, Composable Structure for Spreadsheets
Jacob Yim, Sathwika Peechara, Brian Hempel et al. (+1 more)
Structure in spreadsheets is typically restricted to ranges of
cells and flat tables. However, replacing the open-ended two-dimensional grid with a more structured alternative risks limiting users' freedom and impeding exploratory workflows. How can we enable expressive structure in spreadsheets while preserving their flexibility? We propose a model that allows users to gradually add structure on top of the traditional grid. To identify common structural patterns, we conducted a content analysis of 30 real-world spreadsheets. Based on our findings, we developed Cellery, a novel spreadsheet system that makes these structures explicit. In Cellery, users gradually define the structure of their spreadsheets using named groupings of
cells, composing together four structural primitives: Atoms, Lists, Tables, and Blocks. Once defined, users can directly manipulate structures, dynamically reference them in formulas, and incorporate them in reusable abstractions. We evaluate Cellery with an expressiveness study reproducing spreadsheets from our sample, demonstrating that it is capable of representing types of structure that traditional spreadsheets cannot. In a preliminary user study of five participants, we find that they successfully use Cellery to design spreadsheets with explicit structure, and we uncover opportunities for further work on the formula
language.
Read the full abstract ▾
All authors Jacob Yim, Sathwika Peechara, Brian Hempel, Michael Coblenz
Machine Learning · Cross-listed from physics.comp-ph
pcNN-SN: a physics-enhanced predictor-corrector neural network algorithm coupled with the discrete ordinates method for neutron transport equations
Cong Liu, Junxia Wei, Shuai Wang et al. (+1 more)
Super-resolution machine learning methods hold promise for addressing the high computational costs associated with high-dimensional features in traditional transport simulations. This paper proposes a physics-enhanced neural network coupling
algorithm for discrete ordinates (SN) transport calculations. Based on high- and low-accuracy data, we develop a predictor-corrector neural network (pcNN) to predict correction factors for coarse-grid solutions. Based on transport equations, local physical features containing subgrid-scale information are selected as network inputs. Dimensionless transformation of physical quantities reduces the input space dimensionality and narrows the range of parameter distributions, thereby facilitating network training. Physical constraints are incorporated into the network model, and the corrected solution retains its original conservation properties. pcNN and SN transport sweeps are internally coupled, and numerical solutions are corrected online. Numerical results demonstrate that this
algorithm can effectively improve coarse-grid solutions to achieve a level of accuracy comparable to that of fine-grid results. The physics-based network design enables the model to demonstrate a certain degree of reusability under different computational conditions. Compared to mesh refinement, the pcNN-SN
algorithm reduces the computation time for the three-dimensional test problems by more than 60%.
Read the full abstract ▾
All authors Cong Liu, Junxia Wei, Shuai Wang, Heng Yong
Databases
BuildGraph: A Synthetic Multi-Archetype Building Knowledge Graph Dataset
Wooyoung Jung
Semantic querying of building knowledge graphs (KGs) underpins the integration of artificial intelligence into building operations, from natural-
language access to cross-building analytics, but such KGs are rarely public owing to proprietary, security, and cost barriers. BuildGraph is a synthetic building KG
dataset of 120 buildings in the Brick Schema ontology, grounded in U.S. Department of Energy prototype models and sensor-placement patterns from real buildings. It spans eight commercial building types across three ASHRAE energy-code vintages, with five realizations per archetype varying URI naming, sensor-attachment predicates, and
topology. A 75-query SPARQL
benchmark confirms structural completeness: BuildGraph reaches 91.5% Query Answerability Rate versus 35.8% for 59 real-world Brick files. Independently, it reproduces real buildings' sensor-type proportions on their shared vocabulary (cosine 0.86-0.94), evidence of realistic instrumentation where measurable. A downstream text-to-SPARQL experiment with Gemma 4 (26B) reaches 27.3% Row-Matching F1 (+17.5 pp over zero-shot) on 12 held-out buildings. BuildGraph gives facility managers and digital-twin developers a testbed for portable analytics and natural-
language interfaces, and its
dataset, generator, and
benchmark are openly available at https://github.com/humanbuildingsynergy/BuildGraph.
Read the full abstract ▾
Databases
Guarded Commits: Transactional Human Approvals for LLM Workflows
Laurent Bindschaedler, Ferdi Kossmann, Chunwei Liu et al. (+1 more)
LLM workflows often require human approval before an irreversible external action. Most systems keep that approval outside the workflow, as an interface click or an audit entry. The workflow therefore lacks a commit-time check that every risky path reached an approval gate. Its logs may not preserve the reviewed evidence or the conditions for reusing an earlier decision. We present a guarded-commit design that makes human approval part of workflow state. Before the external action runs, a decision source appends a resolution record and its referenced evidence to a ledger. A credential-confined commit adapter then checks that record against the artifact, policy version, and executed path. Our evidence is limited to trace reconstruction. A validator test on synthetic acyclic workflow plans accepts unfaulted plans and rejects plans with each injected fault: a missing gate, the wrong gate type, or incomplete path coverage. Across three public workloads totaling 271,035 traces, replay reproduces recorded artifact hashes when present. Avoided reviews and disagreement with recorded decisions vary by workload. The resulting record supports audit and replay under the policy in force when the resolution was recorded.
Read the full abstract ▾
All authors Laurent Bindschaedler, Ferdi Kossmann, Chunwei Liu, Jason Mohoney
Machine Learning
Integrating Fairness and Explainability in a Multiple Instance Reinforcement Learning System
Bente Hinkenhuis, Seyed Sahand Mohammadi Ziabari and Ali Mohammed Mansoor Alsahag
Predicting student performance from educational interaction data requires models that are both accurate and sufficiently transparent to support meaningful intervention, while demographic information introduces an additional risk of unfair predictions. This study investigates a multi-objective framework that combines reinforcement learning-based multiple instance learning (RL-MIL),
adversarial debiasing, and preference-conditioned hypernetworks for student-at-risk prediction. MIL represents each student as a bag of weakly labeled interactions, while an RL
agent selects informative instances for downstream classification. Two hypernetwork variants are evaluated to determine whether a user-defined preference scalar can continuously control the trade-off between predictive performance and Equalized Odds. The underlying RL-MIL baseline achieves strong classification performance, but both hypernetwork extensions exhibit mode collapse: changing the preference weight produces little systematic movement along the intended
fairness-performance frontier. The failure is associated with objective dominance, weak gradient propagation through the conditioning mechanism, and interactions between dynamically generated parameters. The results show that
fairness objectives can be incorporated into an interpretable RL-MIL pipeline, but preference conditioning alone does not guarantee controllable multi-objective behavior. Robust fair RL-MIL therefore requires explicit mechanisms for gradient balancing, objective separation, and stability analysis.
Read the full abstract ▾
Systems and Control
Development of an EMT model of the Balearic power system
Yousef Pipelzadeh, Dharshana Muthumuni, Farid Mosallat et al. (+5 more)
The Balearic Islands are striving to achieve 100% renewable energy, which poses new challenges in operating the power system securely and reliably. With a higher integration of inverter-based resources (IBRs) and a reduced presence of conventional synchronous generators, the system strength, particularly short
circuit levels, becomes weaker and more susceptible to disturbances; and the total inertia of the system becomes lower. Technology enablers are planned to achieve energy transition in the Balearic power system: a new 2x200 MW VSC-HVDC link (bipole with metallic return), Synchronous Compensators (SC) and Battery Energy Storage Systems (BESS), as fully-integrated network components. Detailed ElectroMagnetic Transient (EMT) simulation studies may be needed to analyse stability of the system, due to the high amounts of power-electronics devices in the system. This paper describes the implementation of an EMT model of the Balearic power system.
Read the full abstract ▾
All authors Yousef Pipelzadeh, Dharshana Muthumuni, Farid Mosallat, Javier Renedo, Silvia Sanz Verdugo, Antonio Cordón, Edgar Nuño, Macarena Martín
Computer Vision
Seeing the City or Recognizing the Place? What Street-View Imagery Adds Beyond Existing Urban Data in VLM-Based Urban Sensing
Kaizhen Tan
Street-view imagery is increasingly analysed with
vision-
language models (VLMs) to infer urban attributes, but predictive accuracy alone does not show how much a photograph contributes beyond data already available for the same place. Using three VLMs, we compare image-based predictions with existing urban data for seven attributes drawn from five public resources. Each urban unit is evaluated with imagery, with location or text context, and against non-image predictions from nearby observations or public records. We also replace images and add conflicting records to test which source the predictions follow. Nearby observations or public records matched or exceeded image-only predictions for road damage, curb ramps, population, and house price. Images were more informative for building type, building function, and the floor count of low-rise buildings. For floor count, the advantage of images over nearby OpenStreetMap labels increased by 5.7 percentage points per doubling of distance to the nearest labelled building and declined for tall buildings whose rooflines often fell outside the frame. When images and records disagreed, predictions usually moved towards the supplied record. Released OpenFACADES floor annotations, generated with OpenStreetMap floor values as input, showed the same dependence: their agreement with the reference increased with building height, whereas that of image-only reruns decreased. The value of street-view imagery therefore depends on whether an attribute is visible and how well the place is already covered by existing data. Because machine-derived labels are often reused as references, these comparisons also bear on how urban
datasets are documented and evaluated.
Read the full abstract ▾
Computer Vision
Domain generalization and synthetic data in object detection: the enabler, the probe, and the gap
Elfi I.S. Hofmeijer, Ella P. Fokkinga, Friso G. Heslinga et al. (+2 more)
Object detection models often experience performance degradation when deployed under distribution shifts, caused by for example changes in weather type, operational environment, or object appearance. Domain Generalization (DG) aims to develop models that remain robust under such shifts and generalize well to unseen domains. DG research specifically focused on object detection models is scarce, although these models face additional challenges around localization and multi-scale representations. Synthetic data is a promising tool to support in DG, by enabling large-scale generation of diverse new samples. In this paper, we present an object detection-centric review of DG and examine the role of synthetic data from three complementary perspectives. First, synthetic data acts as an enabler of DG through diversification and
alignment strategies that aim to improve robustness to distribution shifts. Second, it serves as a probe that enables controlled experimentation to identify and understand failure modes. Third, we discuss the synthetic-to-real gap, a particularly challenging form of domain shift that arises when models trained on synthetic imagery are deployed on real-world data. Through reviewing these perspectives, we identify limitations of current DG approaches for object detection and argue that future research requires representation-aware methods that explicitly address both localization and classification under domain shift.
Read the full abstract ▾
All authors Elfi I.S. Hofmeijer, Ella P. Fokkinga, Friso G. Heslinga, Klamer Schutte, Jörgen M. Karlholm
Software Engineering
Software Project Management with LLM-Based Automation: Coordination, Validation, and Governance in Practice
Ronnie de Souza Santos, Cleyton Magalhaes and Italo Santos
As software engineering has evolved, development environments have increasingly integrated automated support for tasks such as testing, analysis, and code generation, requiring project management to coordinate both human work and automated workflows. In this paper, we investigate how software project managers perceive changes in their work and learning demands associated with LLM-based automation. Motivated by a gap in software engineering research, which has largely focused on task-level and developer-centered uses of LLMs, we adopted an exploratory case study approach to capture managerial perspectives on automation in practice. Based on the experience of software project managers working in a large, multi-project software organization, our analysis indicates that LLM-based automation influences planning, estimation, coordination, monitoring, and governance activities rather than introducing new formal management practices. Participants described LLMs as becoming embedded in everyday project work, producing uneven effects on productivity, increasing the need for review and validation, and reducing visibility into task execution. These effects contribute to greater reliance on managerial judgment and coordination. Learning demands were perceived as experiential and incremental, centered on understanding LLM capabilities and limitations, critically assessing generated artifacts, and guiding responsible use within teams. The findings provide empirical evidence on how software project management work is adapted in contexts where LLM-based automation is integrated into ongoing software development practice.
Read the full abstract ▾
Sound
High-Value Synthetic Supervision for Parameter-Efficient Adaptation of a Compact Japanese Speech Model
Sidi Chang and Peiying Zhu
Private domain
speech is difficult to collect and redistribute, while compact models need task-specific supervision. We study an auditable synthetic pipeline that maps Japanese care handoffs directly to six-field structured notes. Using 182 synthetic training and development clips, we adapt a 1.47B audio model by full fine-tuning and rank-16 LoRA. On a 39-clip scenario-seed-disjoint synthetic test, an unadapted model obtains a model-judged factuality-recall score of 0.0500, full tuning 0.8664, and LoRA 0.8461. LoRA reaches 97.7% of the full-tuning aggregate as a descriptive ratio while project telemetry reports 12.4M trainable parameters, about 0.85% of the backbone. Both adaptations show large paired gains over the same base; the full-versus-LoRA interval crosses zero, and differing optimization settings preclude an equivalence claim. This is a parameter-efficient capability-acquisition result, not a device-performance result: latency,
memory, energy, and real-time factor were not measured. All evaluation
speech and targets are synthetic, references are model-proposed, and the judge is uncalibrated. The evidence shows that a compact model can acquire a narrow audio-to-structure transformation from a few hundred provenance-linked synthetic examples; it does not establish clinical validity, real-
speech transfer, or superiority to a clean cloud system.
Read the full abstract ▾
Machine Learning
PALM: Point-in-Time Adaptation for Financial Language Models
Seunghan Lee, Jun Seo, Jaehoon Lee et al. (+9 more)
Language models used in financial backtests suffer from look-ahead bias, as a model trained on text published after the study period has already observed the outcomes it is asked to predict. To handle this issue, point-in-time (PIT)
language models are pretrained on chronologically filtered corpora and released as one checkpoint per calendar year, each with a documented cutoff. However, each additional year costs another pretraining run, and whether that run is necessary has never been tested. In this paper, we show that a newer checkpoint is not detectably better on the decision it informs, as its information coefficient on a fixed evaluation window shows no decay with age that we can detect. Motivated by this observation, we propose PALM (Point-in-time Adaptation for financial
Language Models), a simple yet effective alternative to annual pretraining that fits a low-rank adapter on financial text the checkpoint is allowed to read, without modifying any pretrained weight. We further find that the gain comes from the financial domain rather than from recency, which leaves the point-in-time constraint without a detectable cost. We validate PALM on a decade of financial news and on three families of PIT models, whose cutoffs span two decades. The pooled gain keeps its sign across experimental settings and comes at a small fraction of the cost of pretraining. Code is available at: https://github.com/seunghan96/palm.
Read the full abstract ▾
All authors Seunghan Lee, Jun Seo, Jaehoon Lee, Junhyeok Kang, Sangjun Han, Sungdong Yoo, Minjae Kim, Tae Yoon Lim, Dongwan Kang, Hwanil Choi, Soonyoung Lee, Wonbin Ahn
Artificial Intelligence
Cross-Modal Emotion Understanding: A Transformer-GAT Approach for Dialogue Emotion Recognition
Jiaqi Qiao, Yifan Lyu and Xiujuan Xu
Multimodal emotion recognition is a key research area in affective computing, with applications in sentiment analysis, intelligent customer service, and human-computer interaction. However, existing methods often rely on single-modal features or simple
multimodal fusion, failing to capture the synergy between global and local contexts, which limits model performance and
emotion understanding. To address this challenge, we propose
Transformer-GAT, a hybrid framework that combines
Transformer and the Graph
Attention Network to enable cross-modal
emotion understanding. The
Transformer is used to capture global semantic information, while the Graph
Attention Network is employed to model fine-grained relationships between modalities, thereby enhancing the representation of emotional features. Experiments on the IEMOCAP and MELD
datasets show that our model achieves weighted F1 scores of 72.45% and 77.37%, outperforming state-of-the-art methods. These results demonstrate that
Transformer-GAT effectively integrates
multimodal features, balances global and local contexts, and provides deeper emotional insights, offering new directions for
multimodal emotion computing.
Read the full abstract ▾
Artificial Intelligence
ERRAND: Budgeted Maintenance of Agent Memory
Beining Wu, Zihao Ding and Jun Huang
Deployed
agents run on handed-over knowledge: a frozen policy consults a briefing of consolidated items written before the stream begins. The world then moves while the store stands still: paths close, flags change, price bands move; every item was true at handover, and the failure is staleness, not ignorance. We introduce ERRAND, which treats revalidation as a priced errand: a recheck competes with the task it protects for the same scarce actions, funded only when the value per action of resolving a doubt clears a running wage. The errand index is single-peaked, vanishing at both ends of belief, so certainty in either direction costs nothing; free en-route receipts maintain on-path knowledge, and repair writes a version, never a deletion. Under equal action budgets in two drifting tool-use worlds, ERRAND clears every non-oracle policy on the preregistered calibers, primary in every setting and conditional at every binding budget, leading eager revalidation by 10.0pp at the base cap. Restraint wins: given no cap, ERRAND stops on its own, spending 11.0% of steps, while uncapped eager revalidation spends 70.7% and still finishes 4.5pp behind capped ERRAND. The margin sits where the briefing's coverage is thinnest, the shadow price of long-tail knowledge: a small budget, well priced, beats a bigger store that never rechecks.
Read the full abstract ▾
Machine Learning
The Impossible Trinity of Time-Series Validation: A Conservation Law among Training Sufficiency, Test Coverage, and Temporal Causality
Jiayu Li
Validating a model on a time series asks for three things at once: each training run should use most of the sample (sufficiency), the test sets should together cover most of the sample (coverage), and training data should come before test data (causality). We prove that the three cannot be had together and price each one. Let $\alpha$ be the smallest training fraction over folds, $\beta$ the fraction of the sample covered by tests, $\Lambda$ the fraction of the sample used as training data from the future of a test point, and $\delta$ the distance from a test point to the nearest training point in its future. Every scheme on a sample of length $T$ satisfies $\alpha+\beta \le 1+\Lambda$ and $\alpha+\min\{\beta,\delta/T\} \le 1$, and under $\beta$-mixing the leakage bias at a test point is at most $2M\beta_{\mathrm{mix}}(\delta)$. In words: going beyond the causal frontier $\alpha+\beta=1$ requires training on the future; that future data must sit within $(1-\alpha)T$ of a test point; and its harm depends on its distance, not its amount. Hence expanding walk-forward is exactly the Pareto frontier of causal validation, $k$-fold cross-validation buys the most future data, and purged $k$-fold with an embargo pays in distance instead, which is cheap when the process forgets quickly but cannot repair the part of causality demanded by non-stationarity. On pure noise, shuffled 5-fold reports an information coefficient of $+0.32$, while contiguous 5-fold, using the same amount of future data, reports $+0.004$.
Read the full abstract ▾
Robotics
Silent Sabotage: Internal State Triggered Backdoor Attacks on LLM-Powered Robotic Systems
Doniyorkhon Obidov, Shivayogi Akki, Tan Chen et al. (+1 more)
The integration of Large
Language Models (LLMs) into robotic control systems is enabling a new generation of autonomous
agents capable of complex
reasoning and planning. While this paradigm shift accelerates progress, it also introduces novel security risks that remain largely unexplored. Current research into LLM
backdoors has focused on attacks triggered by external stimuli, such as specific words, visual objects, or environmental states. These attacks, while potent, overlook a more insidious class of vulnerability where the trigger is internal to the
agent's own operational logic. This paper presents the first comprehensive study of history-based
backdoor attacks on LLM-powered robotic systems. We demonstrate that an attacker can embed a stealthy
backdoor into an LLM-based
robot controller by manipulating its instructions. This
backdoor is triggered not by an external cue, but by a specific, rare sequence of the
robot's own past actions. It remains dormant during normal operation, preserving the
robot's utility, but can be activated to induce a malicious behavior, such as a complete stop or a collision. Our experiments, conducted in a simulated environment with a variety of
robots and LLMs, show that this history-based attack is highly effective, achieving a near-perfect attack success rate while remaining exceptionally difficult to detect. These findings reveal a critical and previously unaddressed vulnerability in autonomous systems and underscore the urgent need for security measures that account for an
agent's internal state.
Read the full abstract ▾
All authors Doniyorkhon Obidov, Shivayogi Akki, Tan Chen, Kaichen Yang
Computation and Language
Peerify: Benchmarking Peer-Review Claim Verification
Alireza Daghighfarsoodeh, Sajad Ebrahimi, Ali Ghorbanpour et al. (+4 more)
Peer review plays a central role in scholarly publishing, yet verifying whether reviewer claims are supported by manuscript evidence remains a largely manual and time-consuming process. We present Peerify, a pipeline for manuscript-grounded verification of peer-review claims. Given a manuscript and a review comment, the Peerify pipeline decomposes reviews into atomic claims, retrieves relevant manuscript evidence, and determines whether each claim is supported by the paper. To support the development and evaluation of the pipeline, we construct a
benchmark of 800 claims derived from authentic peer-review interactions collected from NeurIPS 2024 and ICLR 2024, including a 300-claim hand-labeled subset used to audit the automated supervision. We evaluate state-of-the-art
language models and
retrieval strategies within the Peerify pipeline, together with entailment baselines. Our results demonstrate the importance of
retrieval-centered verification and claim decomposition, while highlighting the challenges posed by ambiguous and interpretive reviewer claims. Automated labels agree with human consensus on 90.3% of audited claims ($\kappa = 0.87$), while off-the-shelf entailment models stay below 0.24 macro-F1.
Read the full abstract ▾
All authors Alireza Daghighfarsoodeh, Sajad Ebrahimi, Ali Ghorbanpour, Soroush Sadeghian, Radin Cheraghi, Negar Arabzadeh, Ebrahim Bagheri
Computation and Language
Do LLMs Choose Like Humans? Using Cognitive Theory to Evaluate LLM Decision-Making
Johnathan Sun, Andrei Shleifer and Yonatan Belinkov
Large
language models (LLMs) exhibit a range of human-like decision-making behaviors, but whether these reflect similar underlying mechanisms or surface-level mimicry remains unclear. We evaluate whether LLM context sensitivity aligns with a cognitive economic theory that explains human behavior through problem categorization and
attention allocation. Across 12 open-source and commercial LLMs on a novel 140,000-trial product choice
benchmark, context induces human-like shifts in choice and problem categorization, but does not reliably reweight
attention between features like price and quality. Neither scale nor chain-of-thought
reasoning reliably attenuates context sensitivity or generates human-like behavior. These results suggest that LLM decision mechanisms are distinct from human ones.
Read the full abstract ▾
Computation and Language
From Trait Vectors to Circuits: Tracing Refusal and Sycophancy Through Language Models
Oscar Miró López-Feliu and Maya Ozbayoglu
A direction in activation space that changes safety-relevant behavior when steered is not necessarily one the model uses to produce that behavior on its own. We therefore ask whether steering acts through the computation of the unmodified model or through a different set of components, studying two traits whose directions have been extracted and validated in prior work: refusal and sycophancy in Qwen2.5-7B-Instruct. For each, we use the trait vector to split the computation into a reconstruction
circuit before the vector and a transmission
circuit after it. We then test whether restoring the coordinate returns behavior removed by ablation. For refusal, both
circuits are compact and faithful, restoring the coordinate alone recovers almost all of the refusal signal lost to ablation, and a
circuit built around the vector matches the faithfulness of a direct input-to-output
circuit at roughly half the edges. For sycophancy, transmission is compact, but reconstruction is broader and only partially faithful.
Circuits that transmit a steering intervention are therefore not automatically the
circuits that produce the behavior, and we report the counterfactual, target, and behavioral test for every
circuit accordingly.
Read the full abstract ▾
Computation and Language
EAVer: Long-Form Factuality Verification as an End-to-End Agentic Policy
Kening Zheng, Aoying Zheng, Zhigang Chang et al. (+21 more)
Long-form factuality verification is commonly implemented as a static decompose-search-verify pipeline, with separately prompted modules processing claims and invoking external search. Treating claims independently makes LLM and search calls scale with claim count and causes repeated searches for overlapping evidence about related claims. We introduce EAVer, an End-to-end Agentic Verifier that learns to control the complete response-level verification workflow as a unified policy. EAVer groups semantically related claims, routes each group to direct verification or targeted search based on confidence, and keeps evidence returned by search in compact in-context memos for cross-claim reuse. To train this policy, we develop a privileged-teacher synthesis pipeline that converts gold claim annotations into executable multi-turn tool-interaction trajectories with live search rather than post-hoc rationales. Structural, label-
alignment, tool-use, search-budget, and leakage checks yield 1,447 quality-controlled trajectories. We further construct 794 bidirectional same-trajectory preference pairs that keep claim grouping, search, and evidence fixed, enabling decision-focused Direct Preference Optimization (DPO) over factuality-decision
tokens. The results with Qwen3-8B show that EAVer outperforms the strongest search-based baseline on each
benchmark by 2.88 Macro-F1 points on VeriFastScore and 4.73 points on the out-of-distribution FaStFact-Bench, while using about 80% fewer searches than the most search-efficient baseline. Moreover, EAVer consistently improves performance across models ranging from 4B to 32B parameters, demonstrating its strong generalizability.
Read the full abstract ▾
All authors Kening Zheng, Aoying Zheng, Zhigang Chang, Yazhi Guo, Miaotian Guo, Qingwei Zong, Xianhai Xie, Weiqiang Jin, Chengze Li, Hanrong Zhang, Jie Yang, Wei-Chieh Huang, Lingzhe Zhang, Liancheng Fang, Xin Zou, Hanqian Li, Jiahao Huo, Yibo Yan, Zizhuang Deng, Lei Miao, Wei Guo, Haihong Tang, Bo Zheng, Philip S. Yu
Machine Learning
Can Coding Agents Reproduce Official Statistics? Metadata, Retry Budget and the Limits of Execution Feedback in a Controlled Eurostat Benchmark
Sabina-Cristiana Necula
Large
language models can generate executable data-analysis code, but successful execution is not equivalent to a valid official-statistics result. This study asks whether authoritative metadata and execution feedback improve the reproducibility of Eurostat answers produced by a coding
agent, and isolates what execution feedback actually contributes. A
benchmark of 30 natural-
language tasks covering seven domains, seven Eurostat
datasets and four difficulty tiers was run under four conditions: task only (A), task plus a frozen
dataset metadata card (B), metadata plus a repair loop driven by sanitized execution feedback (C), and metadata plus the same attempt budget with no diagnostics of any kind (D). Claude Sonnet 5 generated Python through the Anthropic Messages API in three independent replicates, yielding 360 task-runs. Exact correctness required successful execution, the correct
dataset, filters, output shape, values and unit. A companion experiment run under an under-specified output contract, in which the required ranking key and unit representation were never stated to the model, understated condition C by 23.4 points, showing that evaluator and contract design can dominate measured
agent error. Reliable statistical coding
agents need semantic validation against frozen specifications, a fully specified output contract, and a retry budget - not execution diagnostics.
Read the full abstract ▾
Computation and Language
Team DArgk at the 2026 ELOQUENT lab for evaluating generative language model quality: Residuals of Humanity: AI Detection Evasion via GRPO Fine-Tuning
Antonela Tommasel and Juan Manuel Rodriguez
Large
language models (LLMs) can generate fluent and coherent text that is increasingly difficult to distinguish from human writing, motivating the development of automatic AI-generated text detectors. However, the robustness of such detectors under
adversarial generation remains uncertain. This paper presents SHADE (
Stochastic Human-like generation via
Adversarial Detector Evasion), a reinforcement learning framework that formulates detector evasion as a policy optimization problem. Instead of applying post-hoc perturbations or prompting-based rewriting, SHADE fine-tunes an instruction-tuned LLaMA model with Group Relative Policy Optimization (GRPO), using feedback from a surrogate detector based on the PAN 2025 mdok system. Our experiments show that full fine-tuning with a small KL regularization penalty achieves $98.5\%$ surrogate evasion, compared to $1.5\%$ for the base model, while LoRA-based adaptation is substantially less effective under regularization. Linguistic analysis reveals that successful evasion is associated with shorter, simpler, and less lexically diverse outputs, suggesting that high detector evasion does not necessarily correspond to more human-like writing. In the official Voight-Kampff competition setting, our submissions ranked sixth and seventh, indicating that optimization against a single surrogate detector only partially transfers to unseen evaluation classifiers. These results highlight both the potential and limitations of reinforcement learning for
adversarial AI-text generation and motivate more robust, multi-detector evaluation protocols for AI-generated text detection.
Read the full abstract ▾
Machine Learning
Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles
Mingzhe Du, Anh Tuan Luu, Dong Huang et al. (+1 more)
Benchmarks for LLM-generated GPU kernels decide correctness with a few random inputs and a loose floating-point tolerance, and their verdicts now feed leaderboards and reinforcement-learning rewards. Recent work agrees these checkers are weak and patches them by hand—extra input distributions, fuzzing recipes, tighter tolerances—with no way to measure whether any patch suffices. We introduce mutation analysis as an adequacy metric for kernel-
benchmark oracles: deterministic rules inject 10{,}303 compilable faults into verified CUDA implementations of 188 KernelBench problems, 7{,}384 of them with an independent kill witness; any test protocol is scored by the fraction it detects. The official check misses one in six witnessed faults (16.9%), deterministically, and the misses are skewed by family: 8.7% of arithmetic faults escape, but 78.6% of precision faults do. The metric explains why (a tolerance blind band growing with reduction size; a measured ceiling on input aggressiveness set by legitimate floating-point variance), audits the strongest existing patch (KernelBench-Verified's gain splits into $+4.0$ points from hidden inputs and $+4.5$ from tighter tolerance, a split its authors could not compute), and exposes a published fuzzing recipe that rejects correct kernels 107 times. Optimizing suites over the kill
matrix reaches 98.0% detection with two inputs per problem (94.8% held-out), and the measurement's fault taxonomy teaches a test generator more than the raw faults themselves. Across 48 whole architectures, the blindness grows with scale, concentrating in deep homogeneous pipelines, and two problems prove unrefereeable: their official references violate the
benchmark's own tolerance against fp64. We release everything as \href{https://huggingface.co/
datasets/Elfsong/KernelBench-M}{KernelBench-M}.
Read the full abstract ▾
All authors Mingzhe Du, Anh Tuan Luu, Dong Huang, See-Kiong Ng