Computation and Language
From Generation to Detection: Exploration of Discourse Driven Scenario based LLM Generated Fake News
Zeynep Özdemir, Murat Osmanoğlu, Sevgi Yiğit-Sert et al. (+2 more)
In this study, we examine how modern LLMs generate and detect fake news under controlled settings across four manipulation scenarios. These are open-ended generation, rewriting, manipulation
prompts and attribute based
prompts grounded in the journalistic discourse framework. Firstly, using seven widely adapted models, we created a synthetic fake news corpus with 14000 generated articles across these four scenarios. Then we analyzed its linguistic properties to assess how closely model-generated news resembles real news structurally and semantically. Finally, to evaluate detection performance, we conducted experiments where each model judges generated fake news, starting with a basic detection
prompt and improved
prompts developed through an iterative refinement process that extracts misleading patterns from real-fake pairs. Our results revealed substantial variation across models in both generating and detecting misinformation, demonstrated that the generation strategy strongly influences detectability, and show that the refined
prompt does not improve and often harms detection performance. Therefore, the study provides a systematic assessment of LLMs detection capability of LLMs generated fake news across typical generation scenarios.
Read the full abstract ▾
All authors Zeynep Özdemir, Murat Osmanoğlu, Sevgi Yiğit-Sert, Ömer Özgür Tanr{\i}över, Y{\i}lmaz Ar
Computation and Language
PhysioBench: A Unified Benchmark for Physiological Signal Question Answering
Mengxuan Li, Junfa Chen, Jinze Xia et al. (+5 more)
Physiological signals support diverse clinical and monitoring tasks, yet existing physiological signal foundation models typically require task-specific adaptation for each task. Natural
language provides a common interface for specifying different prediction objectives, but the ability of current models to follow such instructions across physiological signal modalities remains insufficiently evaluated. To address this gap, we introduce PhysioBench, a unified
benchmark for physiological signal question answering. PhysioBench harmonizes annotations from 22 public
datasets into 61.4 million questions across 30 tasks. Each question-answer pair is grounded in a signal segment and traceable to its source annotation. We evaluate 21 representative models, including large
language models,
vision-
language models, time-series
language models, and physiological signal foundation models under three complementary settings. The results show that none of the evaluated models achieves consistently strong performance across physiological signal modalities and tasks. The incorporation of natural
language supports unified prediction across tasks, although performance remains sensitive to question formulation. Beyond these findings, PhysioBench offers an extensible platform for fine-grained analysis and future research on physiological signal understanding. Our codes are available at https://github.com/Leanna97/PhysioBench.
Read the full abstract ▾
All authors Mengxuan Li, Junfa Chen, Jinze Xia, Yundan Chen, Lixin Fan, Ke Liu, Keyue Shi, Haishuai Wang
Computation and Language
VisKG-LM: Compiling Knowledge Graphs into Visual Memory for Multiple-Choice Question Answering
Yixin Peng, Er Jin, Shiwei Luo et al. (+2 more)
Knowledge graphs are usually integrated into question answering by encoding a retrieved subgraph with a graph neural network and fusing it with the
language model in the online inference path. The same subgraph is therefore re-encoded from scratch every time a pair is scored, across training epochs, seeds, and evaluation runs, even though the knowledge graph never changes. We ask whether the retrieved knowledge graphs can instead be compiled once, offline, and then accessed as read-only
memory. VisKG-LM shows that it can, by decoupling graph encoding from
language reasoning. It serializes each retrieved candidate-specific subgraph as Relation-Labeled Paths and renders the result as an image whose two-dimensional layout preserves the branching structure of the paths. Each image is encoded once, offline, and cached for reuse. At inference, the
language model contextualizes the question and candidate from text alone, and only its final layer consults the cached visual
memory, reading both its global layout and its local relational detail. The graph information thus enters only after the text has been understood. On the test sets of CommonsenseQA, OpenBookQA, and MedQA-USMLE, VisKG-LMimproves over GreaseLM by $1.2$, $0.8$, and $4.3$ points, respectively, while matching or surpassing GraphVis, a $7$B
vision-
language model, with only about $400$M online parameters. Against a matched text-only control that receives the identical Relation-Labeled Paths, it gains $4.2$, $6.5$, and $5.1$ points across the three
benchmarks. These gains show that the complete visual-
memory interface adds value beyond path textualization alone and support compiled visual
memory as an alternative to online graph propagation.
Read the full abstract ▾
All authors Yixin Peng, Er Jin, Shiwei Luo, Diego Collarana, Stefan Decker
Computation and Language
From GenAI Virtual Patient Dialogue Logs to Teacher-Interpretable Process Evidence: A Learning Analytics Study in Higher Education
Xinyu Li, Zijian Li, Mengyu Xia et al. (+6 more)
Medical history taking is a dialogue-based clinical
reasoning task in which learners must gather, organise, and integrate patient information while the consultation unfolds. Generative AI-powered virtual patients (GenAI VPs) make repeated history taking practice scalable and preserve full turn by turn dialogue. However, these logs are educationally difficult to use directly. Complete transcripts are too detailed for routine teacher review, whereas final scores obscure whether learners followed up patient cues, checked uncertainty, or used summaries to guide later questioning. This study examined whether coded GenAI VP dialogues can provide teacher-interpretable process evidence of clinical
reasoning. We analysed 1{,}030 GenAI VP dialogues from 210 second-year medical learners across five weeks chest-pain cases. Each consultation was teacher-scored using a rubric assessing the full history taking dialogue, and consultations were classified within each week as high- or low-rated using the weekly median score. To explain how rated performance was reflected in the dialogue process, we applied three analytic layers to the same coded dialogue data: behavioural prevalence, local co-occurrence using Epistemic Network Analysis, and sequential transition using Transition Network Analysis. High-rated consultations involved more history taking activity, but differences were not simply about volume. High rated consultations more often connected information gathering and symptom exploration with communication, checking, organisation, and synthesis. Summarising and organising moves more often led to verification or mechanism-oriented follow-up. These findings show how layered analysis of GenAI VP dialogue logs can reveal process patterns associated with high rated history taking and support process-focused feedback in medical education.
Read the full abstract ▾
All authors Xinyu Li, Zijian Li, Mengyu Xia, Luzhen Tang, Naping Chen, Changmin Lin, Danijela Gasevic, Dragan Gasevic, Yizhou Fan
Computation and Language
LingShu: A Large-Scale Symptom-Centric Contextualized Knowledge Graph Bridging Traditional Chinese Medicine and Modern Biomedicine
Rui Hua, Zixin Shu, Kai Chang et al. (+18 more)
Biomedical knowledge graphs (KGs) are pivotal for knowledge organization, yet traditional binary relations often struggle to represent the conditional nature of biomedical knowledge. Symptoms provide a shared phenotypic layer for linking Traditional Chinese Medicine (TCM), which relies on symptom patterns for syndrome differentiation and treatment selection, with modern biomedicine, which connects clinical manifestations to diseases and molecular mechanisms. We present LingShu, a large-scale symptom-centric contextualized knowledge graph designed to bridge TCM and modern biomedicine. The exported version of LingShu analyzed in this study comprises 17.33 million atom-level entity records and 39.47 million relation records, including 17.19 million semantic triples and 22.29 million contextualized quadruples. LingShu integrates multi-source data, including clinical electronic medical records, authoritative TCM texts, biomedical ontologies, and curated knowledge bases, through a pipeline combining natural
language processing, terminology normalization, and human-in-the-loop verification. A key innovation of LingShu is its hybrid data model: it maintains 64 typed triple relation patterns to ensure broad connectivity, while incorporating 35 contextual quadruple relation patterns to capture conditional medical associations. This dual-structure approach explicitly encodes conditional knowledge, providing a granular representation of the contexts associated with medical relations. These contextualized relations cover syndrome-dependent herb efficacy, disease-contextualized drug effects, population-specific clinical associations, and mechanism-related therapeutic responses. Furthermore, we developed a web platform (http://www.tcmkg.com/) that integrates graph visualization, graph-based
reasoning, and an evidence-grounded knowledge question-answering
agent.
Read the full abstract ▾
All authors Rui Hua, Zixin Shu, Kai Chang, Dengying Yan, Jianan Xia, Hui Zhu, Shujie Song, Shurui Yang, Tongxin Wang, Yue Yin, Yu Wei, Lijuan Pei, Yunhui Hu, Hao Xu, Mingzhong Xiao, Xiaodong Li, Haibin Yu, Runshun Zhang, Wenjia Wang, Baoyan Liu, Xuezhong Zhou
Computation and Language · Cross-listed from cs.LG
DUET: Dual-Teacher On-Policy Distillation via Same-Weight Disagreement for Prohibition Compliance
Zihan Li, Feifei Li and Wenhui Que
Real-world LLM deployments increasingly rely on runtime-injected prohibitions–enterprise policies, PII redlines, tool boundaries–that vary per request and per tenant. Conventional post-training is structurally ill-suited: SFT hides the violation signal in compliant labels, and DPO's sequence-level preferences mismatch
token-localized violations. We propose DUET, a
token-selective on-policy
distillation method for prohibition compliance. DUET pairs a teacher that sees the prohibition (positive) with an identical-weight teacher that does not (negative). Because the two teachers differ only in prohibition visibility, their per-
token disagreement isolates the prohibition's causal effect–yielding a clean supervision signal uncontaminated by model capacity or mismatch. This disagreement drives two complementary mechanisms: signal cleaning, which discards agreement
tokens as redundant or prefix-corrupted, and preference-directed learning, which pushes the student away from the negative teacher and toward the positive one at
token granularity,
embedding DPO-style optimization directly into OPD without offline preference data. We construct an industrial Prohibition-Compliance
benchmark spanning five task families covering explicit-refusal, paraphrase robustness, and over-refusal. Across 1.5B-8B Qwen variants, DUET achieves 72.3-85.2% violation compliance while preserving 88-93% normal utility, dramatically outperforming teacher model and other
distillation baselines. External evaluation on SysBench confirms improved safety
alignment with minimal degradation on GSM8K and MATH-500.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.NE
An evolutionary model of animats with VLM-based subjective evaluation
Shota Miyazaki, Takaya Arita and Reiji Suzuki
In this study, we propose a framework that incorporates subjective evaluations provided by a
Vision-
Language Model (VLM) into the fitness evaluation and selection processes of a genetic
algorithm. As the target of
evolution, we employ virtual soft
robots with flexible morphologies and locomotion and present the VLM with sequence images representing the locomotion of two individuals. Selection is performed via pairwise comparisons based on subjective evaluation terms such as adorably and weirdly. The outcomes of these comparisons are used as selection pressure within the genetic
algorithm, enabling the simultaneous
evolution of morphology and locomotion. Experimental results demonstrate that subjective selection by the VLM accelerates population convergence compared to random selection, while also giving rise to distinctive morphologies and motions corresponding to each evaluation term. An auxiliary experiment with human participants further showed that, although individual pairwise choices only partly agreed with the VLM selections, the resulting morphological and locomotion tendencies were qualitatively similar and repeated human evaluations imposed noticeable fatigue. Moreover, the observation that similar evolutionary outcomes emerged across different evaluation terms suggests that the VLM does not apply these terms in a purely literal manner but instead decomposes them into multiple internal evaluation criteria when making judgments. This work visualizes the evolutionary process through which subjective linguistic expressions are mapped onto
embodied phenotypes and provides a foundational framework for analyzing the structure of subjective judgment in VLMs. The proposed approach is expected to contribute to new developments in evolutionary computation and artificial life research based on subjective evaluation.
Read the full abstract ▾
Computation and Language
TELLER: Dual-Path Iterative Preference Optimization for Table Entity Linking
Yixin Peng, Kehao Li and Stefan Decker
Entity linking in tables matches short and ambiguous
cell mentions to their corresponding knowledge-base entities. Existing approaches typically rely on data preprocessing pipelines that retain either compact or extensive table content as contextual evidence, and then formulate entity linking as a
language generation task for instruction-tuned models; recent systems further incorporate explicit
reasoning to disambiguate challenging mentions. However, their training supervision is usually static: fixed preference data cannot adapt to the residual errors of an evolving model, while variations in
reasoning length can bias sequence-level preference learning. To address these limitations, we present TELLER: Table Entity Linking through Learning from Errors and
Reasoning. We first retrieve and rank Wikidata candidates and retain reduced table evidence in the
prompt. The direct-answer path applies iterative direct preference optimization and refreshes its preference data with residual errors from the updated model. The
reasoning path uses filtered and compressed chain-of-thought rationales for supervised fine-tuning, followed by our iterative length-normalized regularized preference optimization. On the TableInstruct entity-linking subset, the direct-answer path improves accuracy from 94.35% to 94.50%; on the MammoTab V2 evaluation set, it improves accuracy from 87.59% to 88.20%. The
reasoning path improves accuracy from 92.90% to 92.95% on TableInstruct and from 79.09% to 81.85% on MammoTab V2, while maintaining high rates of complete
reasoning generation. These results show that iterative preference learning benefits both concise entity prediction and explicit
reasoning.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
SciDataSailor: Deep Scientific Data Exploring
Jiyong Rao, Yicheng Qiu, Chi Zhang et al. (+2 more)
Scientific
datasets are commonly organized as hierarchical repositories containing heterogeneous and interdependent files, making their inspection, integration, and analysis labor-intensive and reliant on domain expertise. Although large
language model (LLM)
agents have advanced substantially in planning,
reasoning, and tool use, existing research has largely overlooked their ability to interact with real scientific data assets through executable environments. We introduce Deep Scientific Data Exploration, an agentic task paradigm in which
agents navigate repositories, interpret heterogeneous files and schemas, execute analyses, integrate cross-file evidence, and produce conclusions grounded in executed observations. To operationalize this paradigm, we present SciDataSailor, a framework for synthesizing tool-interactive trajectories by balancing broad exploration with targeted exploitation. SciDataSailor instantiates trajectory synthesis as Monte Carlo Tree Search (MCTS) with four task-specific mechanisms: difficulty-stratified exploration seeds, dual-feedback first-play urgency, hierarchical strategy-to-tool action generation, and
entropy-guided branching. Using this framework, we construct SciDataSailor-SFT-2K for supervised fine-tuning and SciDataSailor-Bench for evaluation, with the latter comprising 627 meta-information summarization tasks and 586 scientific question-answering tasks across 27
datasets spanning the life, earth, and physical sciences.
Read the full abstract ▾
All authors Jiyong Rao, Yicheng Qiu, Chi Zhang, Chunfeng Song, Runkai Zhao
Computation and Language · Cross-listed from cs.CR
ThreatForest: Multi-Agent Attack Tree Generation with Pluggable TTP Framework Mapping
Cristian Leo, Anton Dykyi, Danny Cortegaca et al. (+2 more)
Threat modeling is essential for secure software development, yet manual analysis of cloud-native architectures is slow and demands scarce security expertise. We present ThreatForest, a multi-
agent system that generates structured attack trees from source code repositories, maps attack steps to adversary tactics, techniques, and procedures (TTPs) from a pluggable set of frameworks (MITRE ATT&CK, CAPEC, and cloud-specific threat
matrices), and synthesizes actionable mitigations. ThreatForest decomposes threat modeling into a multi-stage
agent pipeline – repository analysis, context refinement, threat generation, parallel attack-tree construction with TTP mapping and mitigation synthesis, and report generation – orchestrated as a directed graph with deterministic verification gates, bounded retries, and three human-in-the-loop validation points. A domain-specific sentence-
transformer maps each attack step to candidate techniques by cosine similarity; we show empirically that this
embedding stage, not the surrounding pipeline, is the dominant accuracy bottleneck. We evaluate ThreatForest across seven application domains on a sixteen-dimension rubric, scored by a panel of independent LLM raters with an
adversarial verification pass and expert review. Panel-measured quality reaches 0.63-0.68 (on a 0-1 scale) for threat statements, attack trees, and mitigations, but only 0.29 for
embedding-only TTP mapping – a gap stable across all seven domains that isolates the binding constraint. A controlled single-call baseline on the same model more than doubles mapping defensibility, pinning the limitation on the
embedding encoder rather than the multi-
agent design. To our knowledge, ThreatForest is the first end-to-end system that turns a code repository into TTP-mapped attack trees with evidence-based mitigations across adversary frameworks, with a reusable framework for benchmarking such systems.
Read the full abstract ▾
All authors Cristian Leo, Anton Dykyi, Danny Cortegaca, Daniel Begimher, Prakash Jha
Computation and Language
Belief Coevolution in a Social Network of Generalist and Specialist Large Language Models: A Controlled Study
Germans Savcisens, Samantha Dies, Courtney Maynard et al. (+1 more)
Large
language models (LLMs) are increasingly deployed in multi-
agent environments. However, the processes by which beliefs form and propagate among interacting LLMs remain poorly understood. We introduce CoevolveSim, a framework for studying belief
diffusion within networked LLM populations. CoevolveSim allows us to isolate and study three factors: domain specialization, social-role assignment, and social network structure. Within this framework, generalist and specialist LLM
agents exchange and revise beliefs. In each round, an LLM
agent observes a summary of its neighbors' beliefs before updating its own. We find that persona-style role assignment reshapes individual belief revision but has little effect on population-level consensus. In contrast, introducing finetuned specialist LLMs produces a large change in consensus and is accompanied by persistent asymmetries in outgoing influence. We further show that an opinion-dynamics surrogate combining individual persistence with population-level belief composition reproduces consensus with high fidelity in every scenario, whereas predicting individual final-round beliefs in heterogeneous LLM populations additionally benefits from
agent identity.
Read the full abstract ▾
All authors Germans Savcisens, Samantha Dies, Courtney Maynard, Tina Eliassi-Rad
Computation and Language
Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Models
Shreyas Subramanian, Mecit Gungor and Vikram Elango
Language and
embedding models used in RAG systems are conventionally assumed to require large-scale pretraining and explicit grounding supervision. We present B1ade, an efficient RAG architecture comprising two purpose-built components: a compact
embedding model and a purpose-built SLM. B1ade-embed, a 335M parameter
retrieval model constructed via parameter-free fusion of five pretrained encoders achieves top MTEB scores among sub-500M models with zero additional training, and B1ade-1B, an SLM trained on low-cost GPUs using Group Relative Policy Optimization (GRPO) on 723M
tokens (2.2M examples) of curated context-question pairs with rewards that optimize only answer similarity. Our central finding is emergent attribution: despite receiving no explicit supervision for source citation, B1ade-1B cites retrieved passages in 42.4% of responses, exceeding the attribution rate of its training distribution by 5.5 percentage points. This demonstrates that grounding behavior can emerge as an accuracy-maximizing strategy under RL training, without explicit reward engineering. On standard QA
benchmarks, B1ade-1B achieves 81.82% on PopQA, 65.8% on PubMedQA, and 51.09% on FEVER. In end-to-end RAG evaluation, B1ade-1B achieves an average score of 0.654 across correctness, completeness, coherence, and faithfulness, a 10.8% improvement over the SFT, while closing the gap with models 1.5x its size. These results show that strategic model composition and reward design suffice for resource-efficient RAG, without large-scale pretraining.
Read the full abstract ▾
Computation and Language
SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge
Lucio M. Dery, Benedict Aaron Tjandra, Siavash Samiei et al. (+4 more)
Agentic systems driven by large
language models (LLMs) regularly feature two key mechanisms to autonomously solve complex problems: synthesizing text-based knowledge and procedures from past experiences and building parametric (weight-space) skill libraries for recurring sub-goals. To date, research has largely treated these as orthogonal pursuits: either organizing textual knowledge through composition and reflection, or consolidating parametric skills via weight-space merging. Consequently, the seamless integration of text and model weights for targeted performance improvements remains largely unexplored. This work bridges this modality gap by treating model weights as an additional modality that an LLM can natively reason over. We instantiate parametric learning via prefix-tuning and augment an LLM to ingest both prefix weights and rich textual data which capture relationships to a target capability. Our augmented LLM, which we call SkillSmith, synthesizes these inputs to perform instruction-steered parametric synthesis, directly outputting new prefix weights that manifest the target skill. We demonstrate that our approach significantly outperforms both text-only and weight-space-only baselines, unlocking performance gains that are out of reach for uni-modal (text-only or weight-only) adaptations.
Read the full abstract ▾
All authors Lucio M. Dery, Benedict Aaron Tjandra, Siavash Samiei, Adhiguna Kuncoro, Zohar Yahav, Jiajun Shen, Arthur Szlam
Computation and Language · Cross-listed from cs.LG
Latent States in Neural Networks: Recovering the Temporal Structure of Drifting Data from Model Weights
Kevin Guan
A temporally drifting data stream may pass through discrete regimes rather than changing continuously. We ask whether such regimes are recoverable from the weights of models trained on the stream, using a hidden Markov model (HMM) fit to the chronologically ordered trajectory of those weights. We study this question in two domains known to drift over time:
multimodal misinformation detection, using the Fakeddit
dataset; and sentiment analysis, using the Yelp
dataset. We train classifiers on consecutive temporal windows and fit an HMM to the trajectory of their aligned weights, recovering latent states that partition each timeline into coherent phases. On both
datasets, classifiers generalize better to data from windows sharing the state of their training window than to windows across state boundaries. This within-state transfer advantage survives a control for temporal proximity and modestly exceeds the advantage recovered by a naive partition into contiguous states of equal size. Although the states are estimated solely from model weights, they correlate more strongly with shifts in the data's class distribution than with the weight-space geometry used to estimate them. After class divergence and lag are residualized out, the within-state advantage exceeds its permutation null on both tasks, indicating that the states recover structure relevant to transfer beyond the data distribution. Every effect replicates on both tasks but is attenuated on Yelp, whose label distribution is more temporally stable.
Read the full abstract ▾
Computation and Language
Selecting Open-Weight Language Models for Zero-Shot Intent Classification: A Systematic Evaluation of 41 Models
Parishruthi Ganesh, Gerry Dozier and Cheryl Seals
Intent classification is a core component of task-oriented dialogue systems, yet practitioners have limited systematic guidance for selecting deployable open-weight
language models under compute, latency, and robustness constraints. We present a systematic zero-shot evaluation of 41 open-weight
language models spanning 15 families and the 135M–9B parameter range across eight English single-label intent-classification
datasets. A ninth
dataset, ATIS, uses five labeled demonstrations and is reported as an auxiliary five-shot result. The evaluation includes standard
benchmarks, a large-scale voice-assistant corpus, and production-derived e-commerce
datasets. Beyond exact-match accuracy, we analyze confidence calibration, robustness to realistic input perturbations, statistical reliability of model rankings, deployment efficiency, and
benchmark saturation. Our results show that instruction-tuned 3B models can outperform several evaluated 7B base models, that differences among leading models on MASSIVE are statistically indistinguishable under pairwise McNemar tests, and that widely used
benchmarks such as SNIPS have become saturated and no longer meaningfully discriminate among current open-weight models. Instruction tuning's effect on confidence calibration is inconsistent rather than uniformly harmful. These findings provide practical guidance for selecting and evaluating open-weight
language models for intent classification.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Dimensionality and Measurement Precision in HLE's Multiple-Choice Subset
Mayank Sharma, Savira Nadela and Tyler Matteson
Humanity's Last Exam (HLE) is widely used to evaluate frontier
language models. HLE organizes its questions into eight subject-domain categories, whose subscores are often interpreted as evidence of distinct capabilities. However, no study has assessed whether these labels correspond to empirically separable latent constructs, nor whether the
benchmark effectively differentiates between models of similar ability. We evaluate 29 LLMs on the text-only multiple-choice subset of HLE and apply psychometric methods to assess both the dimensionality of the
benchmark and the distribution of its measurement precision. Fitting a two-parameter logistic IRT model, we find that HLE is dominated by a single general
reasoning factor: domain labels explain only 3.5% of variance in the leading principal components of item responses, and within- and between-domain residual correlations are nearly identical (Cohen's $d = 0.016$). A simulation study with the same number of models ($N = 29$) shows that our analyses would detect clearly distinct domain abilities, so their absence is informative; however, small domain-specific differences cannot be ruled out, and model rankings vary across domains somewhat more than a single factor predicts. A separate analysis of the test information function reveals that measurement precision concentrates at moderate ability levels and drops sharply above $\theta = 0$, where the strongest models sit. These findings suggest that the subset's domain subscores do not warrant distinct capability interpretations and that its ability to discriminate among the strongest models might be limited.
Read the full abstract ▾
Computation and Language
Benchmarking LLM Competence on Logical Inference over Probability Operators
Nayera Hasan, Jack Greff and Alvin Grissom II
Both expressions of uncertainty and inferences are ubiquitous in natural
language, and valid inferences over natural-
language expressions of uncertainty are necessary for not only everyday conversations but also for high-stakes domains such as medicine and law. While large
language models are increasingly evaluated on logical
reasoning tasks, disentangling principled, symbolic
reasoning from clever surface-level pattern matching is fraught with difficulty. We introduce a
benchmark for
reasoning over probability operators–inference over sentences with gradable epistemic modals (e.g., probably, might, must) containing 14,320 procedurally-generated English
prompts across fifteen inference templates, systematically varying question form, negation strategy, and surface content. Evaluating 29 models, we find that most show answer biases independent of the logical form, a systematic preference for Yes or No. We summarize this with a competence floor: the worse of a model's accuracy on Yes-correct and No-correct items. Only 9 of 29 models exceed random chance. We also test variations in question form, verb phrases/activity, and both the gender and origin of names used in the
prompts, finding biases across every axis.
Read the full abstract ▾
Computation and Language
AHA-Memes: A Fine-Grained Multimodal Benchmark for Understanding Hate in Arabic Memes
Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, Abul Hasnat et al. (+3 more)
Hateful memes are a growing form of
multimodal online harm, where hostile intent is often conveyed through the joint interpretation of images, text, cultural references, and implicit targets. While hateful meme detection has advanced in high-resource
languages, Arabic remains underexplored, with existing meme resources focusing mainly on propaganda or coarse harmful-content labels. We introduce AHA-Memes (Arabic HAteful Memes), which is, to our knowledge, the first large-scale Arabic hateful meme
benchmark with fine-grained, multi-label annotations. The
dataset includes 5K manually annotated memes using a taxonomy that captures hate types, i.e., attack strategies. We further provide ~66K silver-labeled memes to support future studies. We
benchmark text-only, image-only, and late-fusion
multimodal models, as well as few-shot in-context learning (ICL) and open- and closed-weight
Vision-
Language Models (VLMs) under zero-shot and fine-tuning settings. Our results establish strong baselines and highlight key challenges in culturally grounded Arabic hateful meme detection. We release the
dataset, annotation guidelines, and evaluation scripts to support future research. WARNING: This paper contains examples that may be disturbing to readers.
Read the full abstract ▾
All authors Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, Abul Hasnat, Md. Rafiul Biswas, Wajdi Zaghouani, Firoj Alam
Computation and Language
Same Illness, Different Voice: Restating the Patient's Account Narrows Narrative Anchoring in Clinical Language Models
Prabhjot Singh, Pritam Deka and Vijay Chennareddy
Clinical
language models are evaluated on vignettes written in a clinician's voice, but patients describe the same illness in their own. We pair 1,000 MedQA cases with three retellings that carry the same audited original facts, in a clinical, a socioeconomic and a cultural voice, and evaluate 12 open models (3B to 24B, including two medical fine-tunes). Every model is less accurate when the patient speaks in a marked voice: 26% more errors, a pooled penalty of $-9.6$ accuracy points (95% CI $[-11.0, -8.2]$), with no detectable reduction at larger scale. The penalty is not explained by any retelling property we measured (length, omitted age, MedQA split, added clinical content) and persists when the marked narrative contains the original vignette verbatim; because generation settings differ between voices, we read it as sensitivity to these marked retellings, not proof that register alone is the cause. Chain-of-thought and explicit debiasing instructions together remove only about a fifth of the gap. Having the same model first rewrite the patient's account as a clinical vignette, then answer from the rewrite, halves the gap while raising marked-voice accuracy by 3.9 points (means over the 11 models that can run the pipeline). A matched pipeline using a structured schema instead of prose closes the same gap but is 7.1 points less accurate in all 11 models, and adding
reasoning agents or tools costs more still. Rewriting is not free: it recovers more facts than extraction but also invents plausible details, which we quantify. In our experiments, the representation a model reasons over matters more than how much it reasons.
Read the full abstract ▾
Computation and Language
HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs
Ru Peng, Tianyu Zhao, Xijun Gu et al. (+9 more)
High-quality, diverse data are vital for large
language models (LLMs) but remain scarce and costly. Data synthesis is a viable alternative and succeeds on closed tasks, yet the humanities and social sciences (HSS) are overlooked, and their open-ended nature makes synthesis challenging. Moving beyond prior capability-centric, fragmented attempts, we adopt a subject-centric paradigm, define the first HSS domain system covering 14 mainstream fields, and introduce HSS-Synth, the first data synthesis pipeline for HSS. HSS-Synth comprises: (1) constructing seed documents from web corpora via multi-step filtering and text refinement evaluated by a judge; (2) specifying "requirements + persona" to backtranslate seed documents into diverse yet faithful instructions with a strict Q&A
alignment check; and (3) breaking LLM response limits via teacher-forced Answering that feeds seed documents during response generation to anchor semantics, reduce
hallucinations, and preserve tone and integrity. HSS-Synth yields 237k high-quality, diverse instruction-tuning samples that outperform 14 leading baselines on 16
benchmarks. The fine-tuned Qwen3-8B-Base sets a new SOTA and approaches the official Qwen3-8B, improving both human preference and knowledge capabilities without performance seesaws. Extensive experiments demonstrate HSS-Synth's robustness and transferability. Our code is publicly available at https://github.com/pengr/HSS-Synth.
Read the full abstract ▾
All authors Ru Peng, Tianyu Zhao, Xijun Gu, Zhiting Fan, Haokai Xu, Jinyang Zhang, Yawen Zeng, Yihong Zhuang, Kexin Yang, Junyang Lin, Dayiheng Liu, Junbo Zhao
Computation and Language · Cross-listed from cs.LG
Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation
Alexi Gladstone, Heng Ji and Yilun Du
The deep learning revolution, kicked off by AlexNet, taught us that end-to-end training beats decomposing a problem into hand-designed stages. Generative modeling, however, has remained the exception-despite generative models being remarkably capable, they are still not trained end-to-end. This is because, at its core, generative modeling is about handling distributions with many modes, and existing scalable approaches handle this the same way, by factoring the generation procedure, which prevents end-to-end generation. In this work, we introduce Explorative Modeling, a new paradigm that instead factors the training loop, exploring K candidate matches between model generations and data, and training on the best, so predictions commit to modes rather than blurring them. We find Explorative Models (XMs) useful in two settings. First, increasing exploration adds a third pretraining axis beyond parameters and data for existing generative models-where scaling exploration monotonically improves performance across both continuous and discrete domains (images,
video, and
language). Notably, gains from exploration increase with scale, climbing from 7% to 36% as data scales and from 13% to 23% as models grow, with efficiency gains more than doubling at 3x the compute. Concretely, exploration improves FLOP efficiency by 4.1x, sample efficiency by 6.2x, parameter efficiency by 47%, lifts the strongest of image-generation recipes to a near-state-of-the-art 1.43 FID on ImageNet without guidance, enables scaling how end-to-end existing models are, and unlocks scaling generalization. Second, XMs enable end-to-end reconstructive generative modeling, matching
diffusion on control tasks with 16-256x fewer inference steps. Together, these results establish XMs as both a new pretraining axis for existing generative models and a standalone end-to-end generative modeling paradigm.
Read the full abstract ▾
Computation and Language
BridgeAlign: Bridging Preference Alignment for Humanities and Social Sciences
Ru Peng, Haokai Xu, Xijun Gu et al. (+11 more)
While data synthesis for large
language models (LLMs) is prevalent, it primarily targets domains with verifiable answers, overlooking open-ended humanities and social sciences (HSS), where nuanced quality judgments matter more than objective correctness. This makes preference
alignment a natural paradigm for broad HSS tasks. Yet existing methods are either costly or not tailored to broad HSS disciplines. We thus propose BridgeAlign, among the first preference-
alignment pipelines for broad HSS disciplines, with three phases: i) Seed Curation: curating HSS seed documents from web corpora via heuristic/LLM-based filtering and text refinement; ii) Preference Data Synthesis: generating preference triplets via persona-based instruction inversion with Q&A consistency checks; iii) Preference Optimization: moving beyond naive human-vs-model heuristics by first grounding preferences in HSS quality rubric, then generating transitional responses via controlled quality degradation to form near-boundary preference pairs for finer-grained quality discrimination. Aligning over 210k synthetic preference samples, BridgeAlign enables Qwen3-8B to achieve the best average across 17
benchmarks against 11 strong baselines; importantly, leading on both human-preference and knowledge-based capabilities at once, with no trade-off between them, as supported by extensive experiments and contextualized by existing theories.
Read the full abstract ▾
All authors Ru Peng, Haokai Xu, Xijun Gu, Tianyu Zhao, Zhiting Fan, Yawen Zeng, Yihong Zhuang, Jinyang Zhang, Kexin Yang, Jian Wu, Hao Chen, Junyang Lin, Dayiheng Liu, Junbo Zhao
Computation and Language
LayerRAG-Bench: A Cross-Layer Reliability Benchmark for Agentic Retrieval-Augmented Generation
Musa Shams
Agentic
retrieval-augmented generation systems can produce answers that appear grounded while failing at the evidence, tool-contract, authorization, or session-state layer. We introduce LayerRAG-Bench, a controlled cross-layer reliability
benchmark with 8 enterprise domains, 240 tasks, 9 fault scenarios, 2 contract modes, and 38,880 live task-level records across nine models from OpenAI, Anthropic, and Gemini. Schema normalization raises schema-drift success from 0.000 to 0.913, but stale evidence, missing tool output, denied permissions, and wrong-session context are not recovered by schema normalization. Groundedness-only evaluation also produces substantial false positives under stale and wrong-session evidence. These results support a layer-specific evaluation principle: a reliability intervention should be credited for repairing its target layer without being mistaken for a universal fix.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
THGFM: Dual-Branch Temporal Heterogeneous Graph Fusion Model
Yixin Peng, Diego Collarana, Er Jin et al. (+1 more)
Temporal heterogeneous graphs offer a natural abstraction for dynamic relational systems in which diverse node and relation types co-exist and evolve over time. Learning on such graphs requires jointly modeling cross-type structural heterogeneity and the temporal dynamics of interactions, yet existing methods still struggle to reconcile parameter-efficient cross-type transfer with relation-aware specialization, and typically inject time only as additive features outside the
attention kernel. We propose THGFM, a web-scale temporal heterogeneous graph fusion model that addresses both limitations within a unified dual-path architecture. THGFM couples a Shared-Space Temporal
Attention branch for parameter-efficient cross-type transfer with a Relational Type-Partitioned Temporal
Attention branch for relation-aware specialization, and integrates them through Dual-Path Relational–Shared Fusion, instantiated with Type-Conditioned Non-Competitive Gated Sum Fusion: a adaptive mechanism that assigns independent, type-conditioned feature-wise gates to the shared and specialized branches, allowing both to be amplified or suppressed without zero-sum competition. To directly incorporate relative time into the
attention score, THGFM further introduces Rotary Temporal
Attention, which rotates queries and keys by half-phases of relative time before matching. THGFM consistently outperforms baseline graph
transformer models on academic graphs
benchmarks, delivering a $+3.25\%$ six-task mean gain, with peak relative gains of $+12.37\%$ on OAG-CS PV, $+4.87\%$ on PF-$L_2$, and $+1.18\%$ on PF-$L_1$, and $+4.24\%$, $+3.73\%$, and $+4.61\%$ on OGBN-MAG, HTAG-ArXiv, and HTAG-DBLP, respectively.
Read the full abstract ▾
All authors Yixin Peng, Diego Collarana, Er Jin, Stefan Decker
Computation and Language
Mental World Modeling
Hao Fei and Yiran Zhao
World models enable a predictive substrate for planning and action, yet existing formulations merely answer a physical question: what/where it is, and how will it evolve. Human behavior, however, is driven by hidden mental state (what a person believes, wants, intends, feels, and considers socially permissible), so a model that tracks the physical scene but not what each
agent knows and believes about it predicts the wrong action for the right-looking scene. We formulate Mental World Modeling (MWM), a generic theoretical framework that makes mental variables core components of a world model rather than posthoc rationales: MWM aintains a coupled physical-mental world state, renders a target-specific partial observation, and simulates how candidate actions jointly update both components. We instantiate the framework in MENTIS, a training-free and fully inspectable baseline that decomposes the process into state parsing, target-observation generation, action decomposition, coupled physical and mental transition, and branch-level value evaluation. On a manually constructed, quality-controlled
dataset of situated decision scenarios spanning text, image, and sounding-
video stories, experiments with 8 modern LLM-based world models demonstrate that explicitly modeling the mental state is essential for predicting human decisions. Deeper analyses further expose the bottlenecks of current mental world modeling. We expect MWM as a next stage of world modeling, from simulating physical scenes to simulating the minds that act in them.
Read the full abstract ▾
Computation and Language
APEX-Accounting
Julien Benchek, Austin Bennett, Jasmin Kern et al. (+8 more)
We introduce APEX-Accounting, a
benchmark built by Mercor in partnership with Ramp, to assess whether frontier models can do the real work of accountants. Tasks include reconciling accounts, accruing expenses, posting transactions, and producing reports. The private eval set comprises 160 tasks, split across 10 worlds. Each world contains an accounting system, as well as spreadsheets, PDFs, and other files. Every task was authored and solved by experts in accounting and bookkeeping, who also wrote grading rubrics. Across nine frontier models, Claude-Fable-5 (Max) leads with 56.4% Mean Criteria@3, ahead of Muse-Spark-1.1 (xHigh) at 52.6%. No model scores more than 2.6% Pass^8 (GPT-5.6-Sol (Max+Pro)) and the highest Pass@8 is 21.5% (Muse-Spark-1.1 (xHigh)). We experiment with increasing the
token budget from $1 to $50 and observe an instance of Simpson's paradox: scores increase as the
token budget increases but within a given budget-constrained harness, scores are lower on tasks where the model spends more
tokens. As APEX-Accounting is a closed
benchmark, leaderboard evals can be run for any frontier model on request.
Read the full abstract ▾
All authors Julien Benchek, Austin Bennett, Jasmin Kern, Ryan Stevens, Rene Sultan, Charis Ching, Hayley Popiel, Vaibhav Mittal, Felix Mercier, Brendan Foody, Bertie Vidgen
Computation and Language
Pangram 4 Technical Report
Ben Glickenhaus, Katherine Thai, Jenna Russell et al. (+4 more)
We present Pangram 4, the latest deep-learning-based AI-text classification model from Pangram Labs. We achieve an AUROC of 0.9916 with a false positive rate of 0.0041% and a false negative rate of 0.3396%. In addition to its increased overall accuracy compared with Pangram 3, Pangram 4 exhibits superior out-of-distribution generalization and robustness to
adversarial attacks. Another novel contribution of Pangram 4 is its improved ability to distinguish fine-grained edits and mixed AI-human co-authored text. We demonstrate improvements to both boundary detection tasks and the detection of interleaved AI assistance. Finally, we report metrics on standard AI detection
benchmarks showing that Pangram 4 achieves state-of-the-art performance on the AI text detection task across a wide variety of settings and domains.
Read the full abstract ▾
All authors Ben Glickenhaus, Katherine Thai, Jenna Russell, Elyas Masrour, Yue Han, Max Spero, Bradley Emi
Computation and Language
DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search
Raphaël Sourty, Antoine Chaffin, Paulo Roberto Moura Junior et al. (+1 more)
State-of-the-art
retrieval models increasingly rely on closed training data, creating a reproducibility gap. We present an open end-to-end recipe for training
retrieval models and study how English supervision transfers to
multilingual retrieval through translate-train. We first reconstruct and curate 665M English contrastive pre-training pairs from 1.4B pairs across 34 public sources and build 1.88M supervised fine-tuning pairs with mined hard negatives. Training yields two 149M-parameter models: DenseOn, a single-vector dense model, and LateOn, a ColBERT-style late-interaction model. They achieve 56.20 and 57.22 average nDCG@10 on BEIR, respectively, setting new state-of-the-art results for this size class. We then translate the validated English data into eight
languages, yielding 2.8B pairs with cross-lingual samples, and train mDenseOn and mLateOn, two 307M-parameter models built on mmBERT-base. Despite sharing their backbone, data, and objectives, their representations behave differently: the dense model is strong on English and translated
languages but degrades outside translate-train support, whereas the late-interaction model generalizes better to unseen
languages and scripts. This suggests that
token-level matching turns translate-train from a target-
language expansion strategy into a
multilingual generalization recipe. We publicly release the models,
datasets, and training code.
Read the full abstract ▾
All authors Raphaël Sourty, Antoine Chaffin, Paulo Roberto Moura Junior, Amélie Chatelain
Computation and Language · Cross-listed from cs.SE
SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch
Yihao Chen, Shi Chang, Feng Lin et al. (+4 more)
LLM-based
agents excel at software engineering tasks where an existing codebase provides context, but constructing a program from scratch remains fundamentally harder. Recent
benchmarks such as ProgramBench quantify this gap: given only natural-
language documentation and an execute-only binary as a behavioral oracle, even frontier models solve fewer than 1% of instances. Existing frameworks conflate documentation reading, behavioral exploration, and code synthesis into a single pass, causing
agents to probe insufficiently, lose behavioral intent as context drifts, and propagate early misinterpretations into the final implementation. Inspired by classical requirements engineering, we argue that behavioral specification elicitation should be a first-class phase that precedes implementation. We present SpecFirst, a two-stage framework that forces the specification elicitation before code synthesis. A dedicated spec
agent first probes the binary and combines observations with documentation into a structured specification. Next, a code synthesis
agent then uses this specification to drive implementation. This decomposition resolves documentation ambiguities before coding begins and provides a stable behavioral reference throughout synthesis. We evaluate SpecFirst on all 200 ProgramBench instances across four models spanning two families and an order of magnitude of capability. SpecFirst consistently outperforms the single-loop baseline, improving test pass rates by 6.9%-21.3% and binary exploration coverage by 9.4%-18.5%, all statistically significant. Behavioral analysis on code synthesis further shows that a prior specification enables earlier and more sustained code construction. Our results demonstrate that an explicit requirements-engineering phase is an effective paradigm for from-scratch program construction.
Read the full abstract ▾
All authors Yihao Chen, Shi Chang, Feng Lin, Khaled Chawa, Boyuan Chen, Shaowei Wang, Ahmed E. Hassan
Computation and Language · Cross-listed from cs.AI
OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding
Jingbo Zhou, Yusai Zhao, Qi Bao et al. (+12 more)
Large
language model (LLM)
agents are increasingly expected to assist users in completing tasks. However, existing
benchmarks provide limited support for evaluating whether
agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a
benchmark for evaluating LLM
agents on long-horizon office-suite tasks with task-level economic grounding. The
benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a
privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the
benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and
dataset are fully open-sourced, and more information is available on our project website: https://omegause-officeval.github.io.
Read the full abstract ▾
All authors Jingbo Zhou, Yusai Zhao, Qi Bao, Jingjia Cao, Zhenghai Chen, Chang Gao, Kaiqi Guo, Muxin Guo, Mingxuan Li, Xinjiang Lu, Yanru Ma, Yixiong Xiao, Zenghui Zhang, Le Zhang, Hua Wu