Computation and Language
Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models
Christopher Ormerod
Traditional methods for determining assessment item parameters, such as difficulty and discrimination, rely heavily on expensive field testing to collect student performance data for Item Response Theory (IRT) calibration. This study introduces a novel approach that implicitly models these psychometric properties by fine-tuning Large
Language Models (LLMs) to simulate student responses across a spectrum of latent abilities. Leveraging the Qwen-3 dense model series and Low-Rank Adaptation (LoRA), we train models to generate responses to multiple choice questions conditioned on discrete ability descriptors. We reconstruct the probability of a correct response as a function of student ability, effectively generating synthetic Item Characteristic Curves (ICCs) to estimate IRT parameters. Evaluation on a
dataset of Grade 6 English
Language Arts (ELA) items and the BEA 2024 Shared Task
dataset demonstrates that this method competes with or outperforms baseline approaches. This simulation-based technique seems particularly effective at modeling item discrimination.
Read the full abstract ▾
Computation and Language
DataParasite Enables Scalable and Repurposable Online Data Curation
Mengyi Sun
Many questions in computational social science rely on
datasets assembled from heterogeneous online sources, a process that is often labor-intensive, costly, and difficult to reproduce. Recent advances in large
language models enable agentic search and structured extraction from the web, but existing systems are frequently opaque, inflexible, or poorly suited to scientific data curation. Here we introduce DataParasite, an open-source, modular pipeline for scalable online data collection. DataParasite decomposes tabular curation tasks into independent, entity-level searches defined through lightweight configuration files and executed through a shared, task-agnostic python script. Crucially, the same pipeline can be repurposed to new tasks, including those without predefined entity lists, using only natural-
language instructions. We evaluate the pipeline on multiple canonical tasks in computational social science, including faculty hiring histories, elite death events, and political career trajectories. Across tasks, DataParasite achieves high accuracy while reducing data-collection costs by an order of magnitude relative to manual curation. By lowering the technical and labor barriers to online data assembly, DataParasite provides a practical foundation for scalable, transparent, and reusable data curation in computational social science and beyond.
Read the full abstract ▾
Computation and Language
Fact-Checking with Large Language Models via Probabilistic Certainty and Consistency
Haoran Wang, Maryam Khalid, Qiong Wu et al. (+2 more)
Large
language models (LLMs) are increasingly used in applications requiring factual accuracy, yet their outputs often contain hallucinated responses. While fact-checking can mitigate these errors, existing methods typically retrieve external evidence indiscriminately, overlooking the model's internal knowledge and potentially introducing irrelevant noise. Moreover, current systems lack targeted mechanisms to resolve specific uncertainties in the model's
reasoning. Inspired by how humans fact-check, we argue that LLMs should adaptively decide whether to rely on internal knowledge or initiate
retrieval based on their confidence in a given claim. We introduce Probabilistic Certainty and Consistency (PCC), a framework that estimates factual confidence by jointly modeling an LLM's probabilistic certainty and
reasoning consistency. These confidence signals enable an adaptive verification strategy: the model answers directly when confident, triggers targeted
retrieval when uncertain or inconsistent, and escalates to deep search when ambiguity is high. Our confidence-guided routing mechanism ensures that
retrieval is invoked only when necessary, improving both efficiency and reliability. Extensive experiments across three challenging
benchmarks show that PCC achieves better uncertainty quantification than verbalized confidence and consistently outperforms strong LLM-based fact-checking baselines. Furthermore, we demonstrate that PCC generalizes well across various LLMs.
Read the full abstract ▾
All authors Haoran Wang, Maryam Khalid, Qiong Wu, Jian Gao, Cheng Cao
Computation and Language
LoRA-Drop: Temporal LoRA Decoding for Efficient LLM Inference
Hossein Rajabzadeh, Maryam Dialameh, Chul B. Park et al. (+2 more)
Autoregressive large
language models (LLMs) are bottlenecked by sequential decoding, where each new
token typically requires executing all
transformer layers. Existing dynamic-depth and layer-skipping methods reduce this cost, but often rely on auxiliary routing mechanisms or incur accuracy degradation when bypassed layers are left uncompensated. We present LoRA-Drop, a plug-and-play inference framework that accelerates decoding by applying a temporal compute schedule to a fixed subset of intermediate layers: on most decoding steps, selected layers reuse the previous-
token hidden state and apply a low-rank LoRA correction, while periodic refresh steps execute the full model to prevent drift. LoRA-Drop requires no routing network, is compatible with standard KV caching, and can reduce KV-cache footprint by skipping KV updates in droppable layers during LoRA steps and refreshing periodically. Across LLaMA2-7B, LLaMA3-8B, Qwen2.5-7B, and Qwen2.5-14B, LoRA-Drop achieves up to \textbf{2.6$\times$ faster decoding} and 45–55% KV-cache reduction while staying within 0.5 percentage points (pp) of baseline accuracy. Evaluations on
reasoning (GSM8K, MATH, BBH), code generation (HumanEval, MBPP), and long-context/
multilingual benchmarks (LongBench, XNLI, XCOPA) identify a consistent safe zone of scheduling configurations that preserves quality while delivering substantial efficiency gains, providing a simple path toward adaptive-capacity inference in LLMs. Codes are available at https://github.com/hosseinbv/LoRA-Drop.git.
Read the full abstract ▾
All authors Hossein Rajabzadeh, Maryam Dialameh, Chul B. Park, Il-Min Kim, Hyock Ju Kwon
Computation and Language · Cross-listed from cs.SE
Compressed code: the hidden effects of quantization and distillation on programming tokens
Viacheslav Siniaev, Iaroslav Chelombitko and Aleksey Komissarov
Large
Language Models (LLMs) have demonstrated exceptional code generation capabilities, yet their
token-level mechanisms remain underexplored, particularly in compressed models. Through systematic analysis of programming
language token representations, we characterize how programming
languages are encoded in LLM tokenizers by analyzing their vocabulary distribution and keyword coverage patterns. We introduce a novel cold-start probability analysis method that provides insights into model behavior without requiring explicit
prompts. Additionally, we present a comprehensive evaluation of how different model optimization techniques - including quantization,
distillation, model scaling, and task-specific fine-tuning - affect
token-level representations and code generation quality. Our experiments, supported by comprehensive probability distribution analysis and evaluation metrics, reveal critical insights into
token-level behavior and provide empirically-validated guidelines for maintaining code generation quality under various optimization constraints. These findings advance both theoretical understanding of LLM code generation and practical implementation of optimized models in production environments.
Read the full abstract ▾
Computation and Language
ModeX: Evaluator-Free Best-of-N Selection for Open-Ended Generation
Hyeong Kyu Choi and Sharon Li
Selecting a single high-quality output from multiple
stochastic generations remains a fundamental challenge for large
language models (LLMs), particularly in open-ended tasks where no canonical answer exists. While Best-of-N and self-consistency methods show that aggregating multiple generations can improve performance, existing approaches typically rely on external evaluators, reward models, or exact string-match voting, limiting their applicability and efficiency. We propose Mode Extraction (ModeX), an evaluator-free Best-of-N selection framework that generalizes majority voting to open-ended text generation by identifying the modal output representing the dominant semantic consensus among generated texts. ModeX constructs a similarity graph over candidate generations and recursively applies spectral clustering to select a representative centroid, without requiring additional inference or auxiliary models. We further instantiate this selection principle as ModeX-Lite, an improved version of ModeX with early
pruning for efficiency. Across open-ended tasks – including text summarization, code generation, and mathematical
reasoning – our approaches consistently outperform standard single- and multi-path baselines, providing a computationally efficient solution for robust open-ended text generation. Code is released in https://github.com/deeplearning-wisc/ModeX.
Read the full abstract ▾
Computation and Language
Losses that Cook: Topological Optimal Transport for Structured Recipe Generation
Mattia Ottoborgo, Daniele Rege Cambrin and Paolo Garza
Cooking recipes are complex procedures that require not only a fluent and factual text, but also accurate timing, temperature, and procedural coherence, as well as the correct composition of ingredients. Standard training procedures are primarily based on cross-
entropy and focus solely on fluency. Building on RECIPE-NLG, we investigate the use of several composite objectives and present a new topological loss that represents ingredient lists as point clouds in
embedding space, minimizing the divergence between predicted and gold ingredients. Using both standard NLG metrics and recipe-specific metrics, we find that our loss significantly improves ingredient- and action-level metrics. Meanwhile, the Dice loss excels in time/temperature precision, and the mixed loss yields competitive trade-offs with synergistic gains in quantity and time. A human preference analysis supports our finding, showing our model is preferred in 62% of the cases.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
Diagnostic-Driven Layer-Wise Compensation for Post-Training Quantization of Encoder-Decoder ASR Models
Xinyu Wang, Ziyu Zhao, Yajie Luo et al. (+6 more)
Layer-wise post-training quantization reconstructs each layer from inputs already altered by the quantized prefix. QEP compensates for this drift with one model-wide coefficient, conflating the model-level operating point with residual variation across layers. We present FADE, which constructs layer-specific coefficients from normalized round-to-nearest distortion and a heuristic calibrated-solver response. It requires no training or per-model coefficient search and adds no inference-time operation. We evaluate seven Whisper, Moonshine, and Qwen3-ASR models at 3 and 4 bits on four English ASR
benchmarks. Across 38 settings, FADE lowers mean word error rate relative to fixed QEP-0.5 in 31, although a development-tuned global coefficient recovers much of this gap. The largest absolute reductions occur in 3-bit settings whose final error remains too high for practical use; these results measure collapse mitigation rather than deployable accuracy. In two lower-WER 3-bit cases, FADE reduces the tuned-global result from 3.67 to 3.10 and from 13.03 to 11.63. Most 4-bit differences from the tuned control are within a descriptive tolerance, and one reverses. Paired reruns and within-seed permutations on an outcome-informed subset support assignment sensitivity in selected cases, but do not estimate a
matrix-wide success rate. FADE is therefore a layer-wise alternative when per-model coefficient search is unavailable, not a universal replacement for tuned global compensation.
Read the full abstract ▾
All authors Xinyu Wang, Ziyu Zhao, Yajie Luo, Yihong Wu, Liheng Ma, Jingrui Tian, Lei Ding, Xiao-Wen Chang, Peng Lu
Computation and Language
Robust Persona-Aware Toxicity Detection with Prompt Optimization and Learned Ensembling
Berk Atil, Rebecca J. Passonneau and Ninareh Mehrabi
Toxicity detection is inherently subjective, shaped by the diverse perspectives and social priors of different demographic groups. While “pluralistic” modeling as used in economics and the social sciences aims to capture perspective differences across contexts, current Large
Language Model (LLM) prompting techniques have different results across different personas and base models. In this work, we conduct a systematic evaluation of persona-aware toxicity detection, showing that no single prompting method, including our proposed automated
prompt optimization strategy, uniformly dominates across all model-persona pairs. To exploit complementary errors, we explore ensembling four prompting variants and propose a lightweight meta-ensemble: an SVM over the 4-bit vector of
prompt predictions. Our results demonstrate that the proposed SVM ensemble consistently outperforms individual prompting methods and traditional majority-voting techniques, achieving the strongest overall performance across diverse personas. This work provides one of the first systematic comparisons of persona-conditioned prompting for toxicity detection and offers a robust method for pluralistic evaluation in subjective NLP tasks.
Read the full abstract ▾
Computation and Language
Estimating Text Temperature with Language Models
Nikolay Mikhaylovskiy
Autoregressive
language models typically use temperature parameter at inference to shape the probability distribution and control the randomness of the text generated. After the text was generated, this parameter can be estimated using maximum likelihood approach. Following it, we propose a procedure to estimate the temperature of any text, including ones written by humans, with respect to a given
language model. We evaluate the temperature estimation capability of a wide selection of small-to-medium Large
Language Models (LLMs). We then use the best-performing Qwen3 14B to estimate temperatures of popular corpora, finding that while most measured temperatures are close to 1, notable exceptions include Jokes, GSM8K, and AG News (1.1), and Python code (0.9).
Read the full abstract ▾
Computation and Language
Classifying several dialectal Nawatl varieties
Juan-José Guzmán-Landa, Juan-Manuel Torres-Moreno, Miguel Figueroa-Saavedra et al. (+3 more)
Mexico is a country with a large number of indigenous
languages, among which the most widely spoken is Nawatl, with more than two million people currently speaking it (mainly in North and Central America). Despite its rich cultural heritage, which dates back to the 15th century, Nawatl is a
language with few computer resources. The problem is compounded when it comes to its dialectal varieties, with approximately 30 varieties recognised, not counting the different spellings in the written forms of the
language. In this research work, we addressed the problem of classifying Nawatl varieties using Machine Learning and Neural Networks.
Read the full abstract ▾
All authors Juan-José Guzmán-Landa, Juan-Manuel Torres-Moreno, Miguel Figueroa-Saavedra, Carlos-Emiliano González-Gallardo, Graham Ranger, Martha Lorena-Avendaño-Garrido
Computation and Language
Power-of-Two Quantization-Aware-Training (PoT-QAT) in Large Language Models (LLMs)
Mahmoud Elgenedy
In Large
Language Models (LLMs), the number of parameters has grown exponentially in the past few years, e.g., from 1.5 billion parameters in GPT-2 to 175 billion in GPT-3 to possibly more than trillion in higher versions. This raises a significant challenge for implementation, especially for Edge devices. Unlike cloud computing,
memory and processing power for Edge devices are very limited, which necessitates developing novel ideas to make such applications feasible. In this work, we investigate compressing weights with a special quantization that limits numbers to only power-of-two (PoT). This helps save a huge amount of
memory as only exponents need to be stored, more importantly, it significantly reduces processing power by replacing costly multiplication with low cost bit shifting. To overcome performance loss due to this strict quantization, we investigate Quantization Aware Training (QAT) to enhance performance through additional training. Results on GPT-2 124M show a major enhancement for quantized PoT model after additional training, with a perplexity enhancement of 66% and BERT-Score loss to baseline GPT-2 of 1%. The
memory saving is estimated to be 87.5% while the inference speed is expected to be 3-10x faster with PoT quantization versus full-precision.
Read the full abstract ▾
Computation and Language
pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFs
Tobias Schimanski, Imene Kolli, Yu Fan et al. (+4 more)
PDFs are the second-most used document type on the internet (after HTML). Yet, existing QA
datasets commonly start from text sources or only address specific domains. In this paper, we present pdfQA, a multi-domain 2K human-annotated (real-pdfQA) and 2K synthetic
dataset (syn-pdfQA) differentiating QA pairs in ten complexity dimensions (e.g., file type, source modality, source position, answer type). We apply and evaluate quality and difficulty filters on both
datasets, obtaining valid and challenging QA pairs. We answer the questions with open-source LLMs, revealing existing challenges that correlate with our complexity dimensions. pdfQA presents a basis for end-to-end QA pipeline evaluation, testing diverse skill sets and local optimizations (e.g., in information
retrieval or parsing).
Read the full abstract ▾
All authors Tobias Schimanski, Imene Kolli, Yu Fan, Ario Saeid Vaghefi, Jingwei Ni, Elliott Ash, Markus Leippold
Computation and Language
Yihao Liang, Ze Wang, Hao Chen et al. (+7 more)
Autoregressive large
language models achieve strong results on many
benchmarks, but decoding remains fundamentally latency-limited by sequential dependence on previously generated
tokens.
Diffusion language models (DLMs) promise parallel generation but suffer from a fundamental static-to-dynamic misalignment: Training optimizes local transitions under fixed schedules, whereas efficient inference requires adaptive "long-jump" refinements through unseen states. Our goal is to enable highly parallel decoding for DLMs with low number of function evaluations while preserving generation quality. To achieve this, we propose CD4LM, a framework that decouples training from inference via Discrete-Space Consistency
Distillation (DSCD) and Confidence-Adaptive Decoding (CAD). Unlike standard objectives, DSCD trains a student to be trajectory-invariant, mapping diverse noisy states directly to the clean distribution. This intrinsic robustness enables CAD to dynamically allocate compute resources based on
token confidence, aggressively skipping steps without the quality collapse typical of heuristic acceleration. On GSM8K, CD4LM matches the LLaDA baseline with a 5.18x wall-clock speedup; across code and math
benchmarks, it strictly dominates the accuracy-efficiency Pareto frontier, achieving a 3.62x mean speedup while improving average accuracy. Code is available at https://github.com/yihao-liang/CDLM
Read the full abstract ▾
All authors Yihao Liang, Ze Wang, Hao Chen, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Emad Barsoum, Zicheng Liu, Niraj K. Jha
Computation and Language
From XAI to Stories: A Factorial Study of LLM-Generated Explanation Quality
Fabian Lukassen, Jan Herrmann, Christoph Weisser et al. (+2 more)
Explainable AI (XAI) methods like SHAP and LIME produce numerical feature attributions that remain inaccessible to non expert users. Prior work has shown that Large
Language Models (LLMs) can transform these outputs into natural
language explanations (NLEs), but it remains unclear which factors contribute to high-quality explanations. We present a systematic factorial study investigating how
Forecasting model choice, XAI method, LLM selection, and prompting strategy affect NLE quality. Our design spans four models (XGBoost (XGB), Random Forest (RF), Multilayer Perceptron (MLP), and SARIMAX - comparing black-box Machine-Learning (ML) against classical time-series approaches), three XAI conditions (SHAP, LIME, and a no-XAI baseline), three LLMs (GPT-4o, Llama-3-8B, DeepSeek-R1), and eight prompting strategies. Using G-Eval, an LLM-as-a-judge evaluation method, with dual LLM judges and four evaluation criteria, we evaluate 660 explanations for time-series
forecasting. Our results suggest that: (1) XAI provides only small improvements over no-XAI baselines, and only for expert audiences; (2) LLM choice dominates all other factors, with DeepSeek-R1 outperforming GPT-4o and Llama-3; (3) we observe an interpretability paradox: in our setting, SARIMAX yielded lower NLE quality than ML models despite higher prediction accuracy; (4) zero-shot prompting is competitive with self-consistency at 7-times lower cost; and (5) chain-of-thought hurts rather than helps.
Read the full abstract ▾
All authors Fabian Lukassen, Jan Herrmann, Christoph Weisser, Benjamin Saefken, Thomas Kneib
Computation and Language
ARCADE: A City-Scale Corpus for Fine-Grained Arabic Dialect Tagging
Omer Nacar, Serry Sibaee, Adel Ammar et al. (+12 more)
The Arabic
language is characterized by a rich tapestry of regional dialects that differ substantially in phonetics and lexicon, reflecting the geographic and cultural diversity of its speakers. Despite the availability of many multi-dialect
datasets, mapping
speech to fine-grained dialect sources, such as cities, remains underexplored. We present ARCADE (Arabic
Radio Corpus for Audio Dialect Evaluation), the first Arabic
speech dataset designed explicitly with city-level dialect granularity. The corpus comprises Arabic
radio speech collected from streaming services across the Arab world. Our data pipeline captures 30-second segments from verified
radio streams, encompassing both Modern Standard Arabic (MSA) and diverse dialectal
speech. To ensure reliability, each clip was annotated by one to three native Arabic reviewers who assigned rich metadata, including
emotion,
speech type, dialect category, and a validity flag for dialect identification tasks. The resulting corpus comprises 6,907 annotations and 3,790 unique audio segments spanning 58 cities across 19 countries. These fine-grained annotations enable robust multi-task learning, serving as a
benchmark for city-level dialect tagging. We detail the data collection methodology, assess audio quality, and provide a comprehensive analysis of label distributions. The
dataset is available on: https://huggingface.co/
datasets/riotu-lab/ARCADE-full
Read the full abstract ▾
All authors Omer Nacar, Serry Sibaee, Adel Ammar, Yasser Alhabashi, Nadia Samer Sibai, Yara Farouk Ahmed, Ahmed Saud Alqusaiyer, Sulieman Mahmoud AlMahmoud, Abdulrhman Mamdoh Mukhaniq, Lubaba Raed, Sulaiman Mohammed Alatwah, Waad Nasser Alqahtani, Yousif Abdulmajeed Alnasser, Mohamed Aziz Khadraoui, Wadii Boulila
Computation and Language
Toward Global Large Language Models in Medicine
Rui Yang, Huitao Li, Weihao Xuan et al. (+47 more)
Despite continuous advances in medical technology, the global distribution of health care resources remains uneven. The development of large
language models (LLMs) has transformed the landscape of medicine and holds promise for improving health care quality and expanding access to medical information globally. However, existing LLMs are primarily trained on high-resource
languages, limiting their applicability in global medical scenarios. To address this gap, we constructed GlobMed, a large
multilingual medical
dataset containing more than 800,000 entries spanning 20
languages, including seven low-resource
languages. Building on this, we established GlobMed-Bench, which systematically assesses 56 state-of-the-art proprietary and open-weight LLMs across 12 of the 20
languages represented in GlobMed, comprising over 20,000 independent experiments and more than 32 million model responses. This large-scale evaluation revealed significant performance disparities across
languages, particularly for low-resource
languages. Additionally, we introduced GlobMed-LLMs, a suite of
multilingual medical LLMs trained across the 12
benchmark languages, with parameters ranging from 1.7B to 8B. GlobMed-LLMs achieved an average performance improvement of nearly 40% relative to baseline LLMs, with a maximum performance improvement of more than threefold in low-resource
languages. These resources provide an important foundation for advancing the equitable development and application of LLMs globally, enabling broader
language communities to benefit from technological advances.
Read the full abstract ▾
All authors Rui Yang, Huitao Li, Weihao Xuan, Heli Qi, Xin Li, Kunyu Yu, Yingjian Chen, Rongrong Wang, Jacques Behmoaras, Tianxi Cai, Bibhas Chakraborty, Qingyu Chen, Lionel Tim-Ee Cheng, Marie-Louise Damwanza, Chido Dzinotyiwei, Aosong Feng, Chuan Hong, Yusuke Iwasawa, Yuhe Ke, Linah Kitala, Taehoon Ko, Jisan Lee, Irene Li, Jonathan Chong Kai Liew, Hongfang Liu, Lian Leng Low, Edison Marrese-Taylor, Yutaka Matsuo, Isheanesu Misi, Yilin Ning, Jasmine Chiat Ling Ong, Marcus Eng Hock Ong, Enrico Petretto, Hossein Rouhizadeh, Abiram Sandralegar, Oren Schreier, Iain Beehuat Tan, Patrick Tan, Daniel Shu Wei Ting, Junjue Wang, Chunhua Weng, Matthew Yu Heng Wong, Fang Wu, Yunze Xiao, Xuhai Xu, Qingcheng Zeng, Zhuo Zheng, Yifan Peng, Douglas Teodoro, Nan Liu
Computation and Language
Confidence Estimation for LLMs in Multi-turn Interactions
Caiqi Zhang, Ruihan Yang, Xiaochen Zhu et al. (+5 more)
While confidence estimation is a promising direction for mitigating
hallucinations in Large
Language Models (LLMs), current research overwhelmingly focuses on single-turn settings. The dynamics of model confidence in multi-turn conversations, where context accumulates and ambiguity is progressively resolved, remain largely unexplored. This work presents the first systematic study of confidence estimation in multi-turn interactions, establishing a formal evaluation framework grounded in two key desiderata: per-turn calibration and monotonicity of confidence as more information becomes available. To facilitate this, we introduce novel metrics, including a length-normalized Expected Calibration Error (InfoECE), and a new "Hinter-Guesser" paradigm for generating controlled evaluation
datasets. Our experiments reveal that widely-used confidence techniques struggle with calibration and monotonicity in multi-turn dialogues. In contrast, a novel logit-based probe we introduce, P(Sufficient), proves comparatively more effective, robustly tracking evidence accumulation and distinguishing it from conversational filler. Our work provides a foundational methodology for developing more reliable and trustworthy conversational
agents.
Read the full abstract ▾
All authors Caiqi Zhang, Ruihan Yang, Xiaochen Zhu, Chengzu Li, Tiancheng Hu, Yijiang River Dong, Deqing Yang, Nigel Collier
Computation and Language · Cross-listed from cs.AI
EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoning
Chuanrui Hu, Xingze Gao, Zuyi Zhou et al. (+8 more)
Large
Language Models (LLMs) are increasingly deployed as long-term interactive
agents, yet their limited context windows make it difficult to sustain coherent behavior over extended interactions. Existing
memory systems often store isolated records and retrieve fragments, limiting their ability to consolidate evolving user states and resolve conflicts. We introduce EverMemOS, a self-organizing
memory operating system that implements an engram-inspired lifecycle for computational
memory. Episodic Trace Formation converts dialogue streams into MemCells that capture episodic traces, atomic facts, and time-bounded Foresight signals. Semantic Consolidation organizes MemCells into thematic MemScenes, distilling stable semantic structures and updating user profiles. Reconstructive Recollection performs MemScene-guided agentic
retrieval to compose the necessary and sufficient context for downstream
reasoning. Experiments on LoCoMo and LongMemEval show that EverMemOS achieves state-of-the-art performance on
memory-augmented
reasoning tasks. We further report a profile study on PersonaMem v2 and qualitative case studies illustrating chat-oriented capabilities such as user profiling and Foresight. Code is available at https://github.com/EverMind-AI/EverMemOS.
Read the full abstract ▾
All authors Chuanrui Hu, Xingze Gao, Zuyi Zhou, Dannong Xu, Yi Bai, Xintong Li, Hui Zhang, Tong Li, Chong Zhang, Lidong Bing, Yafeng Deng
Computation and Language
FormationEval, an open multiple-choice benchmark for petroleum geoscience
Almaz Ermilov
This paper presents FormationEval, an open multiple-choice question
benchmark for evaluating
language models on petroleum geoscience and subsurface disciplines. The
dataset contains 505 questions across seven domains including petrophysics, petroleum geology and reservoir engineering, derived from three authoritative sources using a
reasoning model with detailed instructions and a concept-based approach that avoids verbatim copying of copyrighted text. Each question includes source metadata to support traceability and audit. The evaluation covers 72 models from major providers including OpenAI, Anthropic, Google, Meta and open-weight alternatives. The top performers achieve over 97% accuracy, with Gemini 3 Pro Preview reaching 99.8%, while tier and domain gaps persist. Among open-weight models, GLM-4.7 leads at 98.6%, with several DeepSeek, Llama, Qwen and Mistral models also exceeding 93%. The performance gap between open-weight and closed models is narrower than expected, with several lower-cost open-weight models exceeding 90% accuracy. Petrophysics emerges as the most challenging domain across all models, while smaller models show wider performance variance. Residual length bias in the
dataset (correct answers tend to be longer) is documented along with bias mitigation strategies applied during construction. The
benchmark, evaluation code and results are publicly available.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting
Muxi Diao, Lele Yang, Wuxuan Gong et al. (+6 more)
Supervised Fine-Tuning (SFT) is the standard paradigm for domain adaptation, yet it frequently incurs the cost of catastrophic forgetting. In sharp contrast, on-policy Reinforcement Learning (RL) effectively preserves general capabilities. We investigate this discrepancy and identify a fundamental distributional gap: while RL aligns with the model's internal belief, SFT forces the model to fit external supervision. This mismatch often manifests as "Confident Conflicts"
tokens characterized by low probability but low
entropy. In these instances, the model is highly confident in its own prediction but is forced to learn a divergent ground truth, triggering destructive gradient updates. To address this, we propose
Entropy-Adaptive Fine-Tuning (EAFT). Unlike methods relying solely on prediction probability, EAFT utilizes
token-level
entropy as a gating mechanism to distinguish between epistemic uncertainty and knowledge conflict. This allows the model to learn from uncertain samples while suppressing gradients on conflicting data. Extensive experiments on Qwen and GLM series (ranging from 4B to 32B parameters) across mathematical, medical, and agentic domains confirm our hypothesis. EAFT consistently matches the downstream performance of standard SFT while significantly mitigating the degradation of general capabilities.
Read the full abstract ▾
All authors Muxi Diao, Lele Yang, Wuxuan Gong, Yutong Zhang, Zhonghao Yan, Yufei Han, Kongming Liang, Weiran Xu, Zhanyu Ma
Computation and Language
Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts
Boxuan Lyu, Soichiro Murakami, Hidetaka Kamigaito et al. (+1 more)
Mixture-of-Experts (MoE) architectures scale large
language models efficiently by employing a parametric “router” to dispatch
tokens to a sparse subset of experts. Typically, this router is trained once and then frozen, rendering routing decisions brittle under distribution shifts. We address this limitation by introducing kNN-MoE, a
retrieval-augmented routing framework that reuses locally optimal expert assignments from a
memory of similar past cases. This
memory is constructed offline by directly optimizing
token-wise routing logits to maximize the likelihood on a reference set. Crucially, we use the average similarity of retrieved neighbors as a confidence-driven mixing coefficient, thus allowing the method to fall back to the frozen router when no relevant cases are found. Experiments show that kNN-MoE outperforms the zero-shot baseline and is competitive with computationally intensive supervised fine-tuning.
Read the full abstract ▾
All authors Boxuan Lyu, Soichiro Murakami, Hidetaka Kamigaito, Peinan Zhang
Computation and Language
Towards Multi-Level Transcript Segmentation: LoRA Fine-Tuning for Table-of-Contents Generation
Steffen Freisinger, Philipp Seeberger, Thomas Ranzenberger et al. (+2 more)
Segmenting
speech transcripts into thematic sections benefits both downstream processing and users who depend on written text for accessibility. We introduce a novel approach to hierarchical topic
segmentation in transcripts, generating multi-level tables of contents that capture both topic and subtopic boundaries. We compare zero-shot prompting and LoRA fine-tuning on large
language models, while also exploring the integration of high-level
speech pause features. Evaluations on English meeting recordings and
multilingual lecture transcripts (Portuguese, German) show significant improvements over established topic
segmentation baselines. Additionally, we adapt a common evaluation measure for multi-level
segmentation, taking into account all hierarchical levels within one metric.
Read the full abstract ▾
All authors Steffen Freisinger, Philipp Seeberger, Thomas Ranzenberger, Tobias Bocklet, Korbinian Riedhammer
Computation and Language
DeCode: Decoupling Content and Delivery for Medical QA
Po-Jen Ko, Chen-Han Tsai and Yu-Shao Peng
Large
language models (LLMs) exhibit strong medical knowledge and can generate factually accurate responses. However, existing models often fail to account for individual patient contexts, producing answers that are clinically correct yet poorly aligned with patients' needs. In this work, we introduce DeCode (Decoupling Content and Delivery), a training-free, model-agnostic framework that adapts existing LLMs to produce contextualized answers in clinical settings. We evaluate DeCode on OpenAI HealthBench, a comprehensive and challenging
benchmark designed to assess clinical relevance and validity of LLM responses. DeCode boosts zero-shot performance from 28.4% to 49.8% and achieves new state-of-the-art compared to existing methods. Experimental results suggest the effectiveness of DeCode in improving clinical question answering of LLMs.
Read the full abstract ▾
Computation and Language
Deferred Commitment Decoding for Diffusion Language Models
Yingte Shu, Yuchuan Tian, Chao Xu et al. (+2 more)
Diffusion language models (DLMs) have recently emerged as a strong alternative to autoregressive models by enabling parallel text generation. To improve inference efficiency and KV-cache compatibility, prior work commonly adopts block-based
diffusion, decoding
tokens block by block. However, this paradigm suffers from a structural limitation that we term Boundary-Induced Context Truncation (BICT): undecoded
tokens near block boundaries are forced to commit without access to nearby future context, even when such context could substantially reduce uncertainty. This limitation degrades decoding certainty and generation quality, especially for tasks requiring precise
reasoning, such as mathematical problem solving and code generation. We propose Deferred Commitment Decoding (DCD), a novel, training-free decoding strategy that mitigates this issue. DCD maintains a certainty-aware sliding window over masked
tokens, resolving low-uncertainty
tokens early while deferring high-uncertainty
tokens until sufficient contextual evidence becomes available. Extensive experiments across multiple
diffusion language models,
benchmarks, and caching configurations show that DCD improves generation accuracy by 1.73% with comparable time on average compared to fixed block-based
diffusion methods, with the most significant improvement reaching 16.5%. These results demonstrate that deferring
token commitment based on uncertainty is a simple yet effective principle for improving both the quality and efficiency of
diffusion language model decoding.
Read the full abstract ▾
All authors Yingte Shu, Yuchuan Tian, Chao Xu, Yunhe Wang, Hanting Chen
Computation and Language
Cost-Efficient Cross-Lingual Retrieval-Augmented Generation for Low-Resource Languages: A Case Study in Bengali Agricultural Advisory
Md. Asif Hossain, Nabil Subhan, Mantasha Rahman Mahi et al. (+1 more)
Access to reliable agricultural advisory remains limited in many developing regions due to a persistent
language barrier: authoritative agricultural manuals are predominantly written in English, while farmers primarily communicate in low-resource local
languages such as Bengali. Although recent advances in Large
Language Models (LLMs) enable natural
language interaction, direct generation in low-resource
languages often exhibits poor fluency and factual inconsistency, while cloud-based solutions remain cost-prohibitive. This paper presents a cost-efficient, cross-lingual
Retrieval-Augmented Generation (RAG) framework for Bengali agricultural advisory that emphasizes factual grounding and practical deployability. The proposed system adopts a translation-centric architecture in which Bengali user queries are translated into English, enriched through domain-specific keyword injection to align colloquial farmer terminology with scientific nomenclature, and answered via dense vector
retrieval over a curated corpus of English agricultural manuals (FAO, IRRI). The generated English response is subsequently translated back into Bengali to ensure accessibility. The system is implemented entirely using open-source models and operates on consumer-grade hardware without reliance on paid APIs. Experimental evaluation demonstrates reliable source-grounded responses, robust rejection of out-of-domain queries, and an average end-to-end latency below 20 seconds. The results indicate that cross-lingual
retrieval combined with controlled translation offers a practical and scalable solution for agricultural knowledge access in low-resource
language settings
Read the full abstract ▾
All authors Md. Asif Hossain, Nabil Subhan, Mantasha Rahman Mahi, Jannatul Ferdous Nabila
Computation and Language · Cross-listed from cs.AI
Hendrik Kempt and Alon Lavie
Reasoning has long been understood as a pathway between stages of understanding. Proper
reasoning leads to understanding of a given subject. This
reasoning was conceptualized as a process of understanding in a particular way, i.e., "symbolic
reasoning". Foundational Models (FM) demonstrate that this is not a necessary condition for many
reasoning tasks: they can "reason" by way of imitating the process of "thinking out loud", testing the produced pathways, and iterating on these pathways on their own. This leads to some form of
reasoning that can solve problems on its own or with few-shot learning, but appears fundamentally different from human
reasoning due to its lack of grounding and common sense, leading to brittleness of the
reasoning process. These insights promise to substantially alter our assessment of
reasoning and its necessary conditions, but also inform the approaches to safety and robust defences against this brittleness of FMs. This paper offers and discusses several philosophical interpretations of this phenomenon, argues that the previously apt metaphor of the "
stochastic parrot" has lost its relevance and thus should be abandoned, and reflects on different normative elements in the safety- and appropriateness-considerations emerging from these
reasoning models and their growing capacity.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Output Embedding Centering for Stable LLM Pretraining
Felix Stollenwerk, Anna Lokrantz and Niclas Hertzberg
Pretraining of large
language models is not only expensive but also prone to certain training instabilities. A specific instability that often occurs at the end of training is output logit divergence. The most widely used mitigation strategies, z-loss and logit soft-capping, merely address the symptoms rather than the underlying cause of the problem. In this paper, we analyze the instability from the perspective of the output
embeddings' geometry and identify anisotropic
embeddings as its source. Based on this, we propose output
embedding centering (OEC) as a new mitigation strategy, and demonstrate that it suppresses output logit divergence. OEC can be implemented in two different ways: as a deterministic operation called $\mu$-centering, or a regularization method called $\mu$-loss. Our experiments show that both variants outperform z-loss in terms of training stability, while being on par with logit soft-capping. This holds true both in the presence and the absence of weight tying. As a secondary result, we find that $\mu$-loss is significantly less sensitive to regularization hyperparameter tuning than z-loss.
Read the full abstract ▾
Computation and Language
Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs
Amirali Ebrahimzadeh and Seyyed M. Salili
As Large
Language Models (LLMs) increasingly utilize massive context windows as working
memory for autonomous tasks, their reliability fluctuates significantly depending on how information is distributed in real-world corpora. We investigate how fact placement, corpus-level distributions, and anti-
hallucination ("Don't Make It Up")
prompts influence model behavior by introducing a model-agnostic extended needle-in-a-haystack
benchmark designed for scalability, which we apply to evaluate Gemini-2.5-flash, ChatGPT-5-mini, Claude-4.5-haiku, and Deepseek-v3.2-chat. Unlike prior work, we separately evaluate literal extraction, logical inference, and
hallucination risk. We identify two critical failure modes: Distributional Collapse, where performance degrades significantly when evidence is dispersed; and a Safety Tax, where anti-
hallucination prompts cause over-conservative refusal of present facts and evidence, sharply reducing accuracy. Our results suggest that many failures stem from ineffective context utilization, as models struggle to prioritize relevant information even when it is present. These findings highlight the need for model-specific robustness and effective context management to ensure reliable deployment in long-horizon agentic workflows.
Read the full abstract ▾
Computation and Language
Surprisal and Metaphor Novelty Judgments: Moderate Correlations and Divergent Scaling Effects Revealed by Corpus-Based and Synthetic Datasets
Omar Momen, Emilie Sitter, Berenike Herrmann et al. (+1 more)
Novel metaphor comprehension involves complex semantic processes and linguistic creativity, making it an interesting task for studying
language models (LMs). This study investigates whether surprisal, a probabilistic measure of predictability in LMs, correlates with annotations of metaphor novelty in different
datasets. We analyse the surprisal of metaphoric words in corpus-based and synthetic metaphor
datasets using 16 causal LM variants. We propose a cloze-style surprisal method that conditions on full-sentence context. Results show that LM surprisal yields significant moderate correlations with scores/labels of metaphor novelty. We further identify divergent scaling patterns: on corpus-based data, correlation strength decreases with model size (inverse scaling effect), whereas on synthetic data it increases (quality-power hypothesis). We conclude that while surprisal can partially account for annotations of metaphor novelty, it remains limited as a metric of linguistic creativity. Code and data are publicly available: https://github.com/OmarMomen14/surprisal-metaphor-novelty
Read the full abstract ▾
All authors Omar Momen, Emilie Sitter, Berenike Herrmann, Sina Zarrieß