Computation and Language
Where Does Robustness Live? Neuron-Guided Adaptation for Retrieval-Augmented Language Models
Jae O Lee, Jaemin Kim, Sumyeong Ahn et al. (+1 more)
Retrieval-Augmented
Language Models (RALMs) have shown strong potential in knowledge-intensive tasks, yet they remain vulnerable when retrieved contexts are noisy or irrelevant. Robustness against such contexts requires two distinct capabilities: abstention when contexts are uninformative, and selective extraction when relevant evidence is buried in noise. Yet existing methods face two key limitations: they do not train separately for these two capabilities, and they adapt the model at a coarse layer- or module-level granularity, overlooking that only a small subset of
neurons is strongly activated for a given input. We propose NeuRIT, a
Neuron-guided Robust Instruction-Tuning framework built on a localization-first perspective. NeuRIT mines context-aware
neurons associated with relevant and irrelevant context processing, and uses them as anchors to selectively adapt both the identified
neuron groups and the layers in which they concentrate. NeuRIT then performs two-stage instruction tuning that teaches complementary behaviors: suppress generation when there is nothing to extract, and extract relevant evidence when there is. NeuRIT consistently outperforms strong baselines across diverse QA
benchmarks and generator backbones. Our code is available at https://github.com/HYU-ARK-Lab/NeuRIT.
Read the full abstract ▾
All authors Jae O Lee, Jaemin Kim, Sumyeong Ahn, Seo Yeon Park
Computation and Language
The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level
Jeremy Herbst, Stefan Wermter and Jae Hee Lee
Mixture-of-Experts (MoE) architectures have become the dominant choice for scaling Large
Language Models (LLMs), activating only a subset of parameters per
token. While MoE architectures are primarily adopted for computational efficiency, it remains an open question whether their sparsity makes them inherently easier to interpret than dense feed-forward networks (FFNs). We compare MoE experts and dense FFNs using $k$-sparse probing and find that expert
neurons are consistently less polysemantic, with the gap widening as routing becomes sparser. This suggests that sparsity pressures both individual
neurons and entire experts toward monosemanticity. Leveraging this finding, we zoom out from the
neuron to the expert level as a more effective unit of analysis. We validate this approach by automatically interpreting hundreds of experts. This analysis allows us to resolve the debate on specialization: experts are neither broad domain specialists (e.g., biology) nor simple
token-level processors. Instead, they function as fine-grained task experts, specializing in linguistic operations or semantic tasks (e.g., closing brackets in $\LaTeX{}$). Our findings suggest that MoEs are inherently interpretable at the expert level, providing a clearer path toward large-scale model interpretability. Code is available at: https://github.com/jerryy33/MoE_analysis.
Read the full abstract ▾
Computation and Language
Adam's Law: Textual Frequency Law on Large Language Models
Hongyuan Adam Lu, Z.L., Victor Wei et al. (+5 more)
While textual frequency has been validated as relevant to human cognition in reading speed, its relatedness to Large
Language Models (LLMs) is seldom studied. We propose a novel research direction in terms of textual data frequency, which is an understudied topic, to the best of our knowledge. Our framework is composed of three units. First, this paper proposes Textual Frequency Law (TFL), which indicates that frequent textual data should be preferred for LLMs for both prompting and fine-tuning. Since many LLMs are closed-source in their training data, we propose using online resources to estimate the sentence-level frequency. We then utilize an input paraphraser to paraphrase the input into a more frequent textual expression. Next, we propose Textual Frequency
Distillation (TFD) by querying LLMs to conduct story completion by further extending the sentences in the
datasets, and the resulting corpora are used to adjust the initial estimation. Finally, we propose Curriculum Textual Frequency Training (CTFT) that fine-tunes LLMs in an increasing order of sentence-level frequency. Experiments are conducted on our curated
dataset Textual Frequency Paired
Dataset (TFPD) on math
reasoning, machine translation, commonsense
reasoning and agentic tool calling. Results show the effectiveness of our framework.
Read the full abstract ▾
All authors Hongyuan Adam Lu, Z.L., Victor Wei, Zefan Zhang, Zhao Hong, Qiqi Xiang, Bowen Cao, Wai Lam
Computation and Language
Do Lexical and Contextual Coreference Resolution Systems Degrade Differently under Mention Noise? An Empirical Study on Scientific Software Mentions
Atilla Kaan Alkan, Felix Grezes, Jennifer Lynn Bartlett et al. (+3 more)
We present our participation in the SOMD 2026 shared task on cross-document software mention coreference resolution, where our systems ranked second across all three subtasks. We compare two fine-tuning-free approaches: Fuzzy Matching (FM), a lexical string-similarity method, and Context Aware Representations (CAR), which combines mention-level and document-level
embeddings. Both achieve competitive performance across all subtasks (CoNLL F1 of 0.94-0.96), with CAR consistently outperforming FM by 1 point on the official test set, consistent with the high surface regularity of software names, which reduces the need for complex semantic
reasoning. A controlled noise-injection study reveals complementary failure modes: as boundary noise increases, CAR loses only 0.07 F1 points from clean to fully corrupted input, compared to 0.20 for FM, whereas under mention substitution, FM degrades more gracefully (0.52 vs. 0.63). Our inference-time analysis shows that FM scales superlinearly with corpus size, whereas CAR scales approximately linearly, making CAR the more efficient choice at large scale. These findings suggest that system selection should be informed by both the noise profile of the upstream mention detector and the scale of the target corpus. We release our code to support future work on this underexplored task.
Read the full abstract ▾
All authors Atilla Kaan Alkan, Felix Grezes, Jennifer Lynn Bartlett, Anna Kelbert, Kelly Lockhart, Alberto Accomazzi
Computation and Language
AstroConcepts: A Large-Scale Multi-Label Classification Corpus for Astrophysics
Atilla Kaan Alkan, Felix Grezes, Sergi Blanco-Cuaresma et al. (+5 more)
Scientific multi-label text classification suffers from extreme class imbalance, where specialized terminology exhibits severe power-law distributions that challenge standard classification approaches. Existing scientific corpora lack comprehensive controlled vocabularies, focusing instead on broad categories and limiting systematic study of extreme imbalance. We introduce AstroConcepts, a corpus of English abstracts from 21,702 published astrophysics papers, labeled with 2,367 concepts from the Unified Astronomy Thesaurus. The corpus exhibits severe label imbalance, with 76% of concepts having fewer than 50 training examples. By releasing this resource, we enable systematic study of extreme class imbalance in scientific domains and establish strong baselines across traditional, neural, and vocabulary-constrained LLM methods. Our evaluation reveals three key patterns that provide new insights into scientific text classification. First, vocabulary-constrained LLMs achieve competitive performance relative to domain-adapted models in astrophysics classification, suggesting a potential for parameter-efficient approaches. Second, domain adaptation yields relatively larger improvements for rare, specialized terminology, although absolute performance remains limited across all methods. Third, we propose frequency-stratified evaluation to reveal performance patterns that are hidden by aggregate scores, thereby making robustness assessment central to scientific multi-label evaluation. These results offer actionable insights for scientific NLP and establish
benchmarks for research on extreme imbalance.
Read the full abstract ▾
All authors Atilla Kaan Alkan, Felix Grezes, Sergi Blanco-Cuaresma, Jennifer Lynn Bartlett, Daniel Chivvis, Anna Kelbert, Kelly Lockhart, Alberto Accomazzi
Computation and Language
Brief Is Better: Non-Monotonic Chain-of-Thought Budget Effects in Function-Calling Language Agents
Xuan Qi
How much should a
language agent think before taking action? Chain-of-thought (CoT)
reasoning is widely assumed to improve
agent performance, but the relationship between
reasoning length and accuracy in structured tool-use settings remains poorly understood. We present a systematic study of CoT budget effects on function-calling
agents, sweeping six
token budgets (0–512) across 200 tasks from the Berkeley Function Calling Leaderboard v3 Multiple
benchmark. Our central finding is a striking non-monotonic pattern on Qwen2.5-1.5B-Instruct: brief
reasoning (32
tokens) dramatically improves accuracy by 45% relative over direct answers, from 44.0% to 64.0%, while extended
reasoning (256
tokens) degrades performance well below the no-CoT baseline, to 25.0% (McNemar p < 0.001). A three-way error decomposition reveals the mechanism. At d = 0, 30.5% of tasks fail because the model selects the wrong function from the candidate set; brief CoT reduces this to 1.5%, effectively acting as a function-routing step, while long CoT reverses the gain, yielding 28.0% wrong selections and 18.0% hallucinated functions at d = 256. Oracle analysis shows that 88.6% of solvable tasks require at most 32
reasoning tokens, with an average of 27.6
tokens, and a finer-grained sweep indicates that the true optimum lies at 8–16
tokens. Motivated by this routing effect, we propose Function-Routing CoT (FR-CoT), a structured brief-CoT method that templates the
reasoning phase as "Function: [name] / Key args: [...]," forcing commitment to a valid function name at the start of
reasoning. FR-CoT achieves accuracy statistically equivalent to free-form d = 32 CoT while reducing function
hallucination to 0.0%, providing a structural reliability guarantee without budget tuning.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
MTI: A Behavior-Based Temperament Profiling System for AI Agents
Jihoon Jeong
AI models of equivalent capability can exhibit fundamentally different behavioral patterns, yet no standardized instrument exists to measure these dispositional differences. Existing approaches either borrow human personality dimensions and rely on self-report (which diverges from actual behavior in LLMs) or treat behavioral variation as a defect rather than a trait. We introduce the Model Temperament Index (MTI), a behavior-based profiling system that measures AI
agent temperament across four axes: Reactivity (environmental sensitivity), Compliance (instruction-behavior
alignment), Sociality (relational resource allocation), and Resilience (stress resistance). Grounded in the Four Shell Model from Model Medicine, MTI measures what
agents do, not what they say about themselves, using structured examination protocols with a two-stage design that separates capability from disposition. We profile 10 small
language models (1.7B-9B parameters, 6 organizations, 3 training paradigms) and report five principal findings: (1) the four axes are largely independent among instruction-tuned models (all |r| < 0.42); (2) within-axis facet dissociations are empirically confirmed – Compliance decomposes into fully independent formal and stance facets (r = 0.002), while Resilience decomposes into inversely related cognitive and
adversarial facets; (3) a Compliance-Resilience paradox reveals that opinion-yielding and fact-vulnerability operate through independent channels; (4) RLHF reshapes temperament not only by shifting axis scores but by creating within-axis facet differentiation absent in the unaligned base model; and (5) temperament is independent of model size (1.7B-9B), confirming that MTI measures disposition rather than capability.
Read the full abstract ▾
Computation and Language
GaelEval: Benchmarking LLM Performance for Scottish Gaelic
Peter Devine, William Lamb, Beatrice Alex et al. (+5 more)
Multilingual large
language models (LLMs) often exhibit emergent 'shadow' capabilities in
languages without official support, yet their performance on these
languages remains uneven and under-measured. This is particularly acute for morphosyntactically rich minority
languages such as Scottish Gaelic, where translation
benchmarks fail to capture structural competence. We introduce GaelEval, the first multi-dimensional
benchmark for Gaelic, comprising: (i) an expert-authored morphosyntactic MCQA task; (ii) a culturally grounded translation
benchmark and (iii) a large-scale cultural knowledge Q&A task. Evaluating 19 LLMs against a fluent-speaker human baseline ($n=30$), we find that Gemini 3 Pro Preview achieves $83.3\%$ accuracy on the linguistic task, surpassing the human baseline ($78.1\%$). Proprietary models consistently outperform open-weight systems, and in-
language (Gaelic) prompting yields a small but stable advantage (+$2.4\%$). On the cultural task, leading models exceed $90\%$ accuracy, though most systems perform worse under Gaelic prompting and absolute scores are inflated relative to the manual
benchmark. Overall, GaelEval reveals that frontier models achieve above-human performance on several dimensions of Gaelic grammar, demonstrates the effect of Gaelic prompting and shows a consistent performance gap favouring proprietary over open-weight models.
Read the full abstract ▾
All authors Peter Devine, William Lamb, Beatrice Alex, Ignatius Ezeani, Dawn Knight, Mícheál J. Ó Meachair, Paul Rayson, Martin Wynne
Computation and Language · Cross-listed from cs.AI
TSQueryBench: LLM-as-a-Judge for Time Series Explanations
Preetham Sivalingam, Murari Mandal, Dhruv Kumar et al. (+1 more)
Natural
language explanations of time series data are increasingly produced by foundation models in high stakes domains, making factual correctness critical. Evaluating such explanations differs fundamentally from standard natural
language generation: correctness requires verifying numerical claims against structured data rather than similarity to reference text. While LLM as a Judge has emerged as a scalable paradigm for text evaluation, its applicability to numerically grounded time series explanations remains unstudied. We introduce TSQueryBench, a controlled synthetic
benchmark of 500 time series instances across 10 query types, each paired with correct, partially correct, and incorrect explanations. We evaluate six large
language models across four tasks: explanation generation, relative ranking, independent scoring, and multi anomaly detection. Our central finding is a consistent generation evaluation asymmetry: models that fail to generate numerically correct explanations nonetheless reliably identify or score correct ones. These results show that rubric guided LLM evaluation is substantially more reliable than generation for time series
reasoning, supporting LLM judges as scalable evaluators in numerically grounded settings. Code and data: https://github.com/Prxxthxm/TSQueryBench/
Read the full abstract ▾
All authors Preetham Sivalingam, Murari Mandal, Dhruv Kumar, Saurabh Deshpande
Computation and Language
Reliable Control-Point Selection for Steering Reasoning in Large Language Models
Haomin Zhuang, Hojun Yoo, Xiaonan Luo et al. (+2 more)
Steering vectors offer a training-free mechanism for controlling
reasoning behaviors in large
language models, but constructing effective vectors requires identifying genuine behavioral signals in the model's hidden states. For behaviors that can be toggled via
prompts, this is straightforward. However, many
reasoning behaviors – such as self-reflection – emerge spontaneously and resist
prompt-level control. Current methods detect these behaviors through keyword matching in chain-of-thought traces, implicitly assuming that every detected boundary encodes a genuine behavioral signal. We show that this assumption is overwhelmingly wrong: across 541 keyword-detected boundaries, 93.3% are behaviorally unstable, failing to reproduce the detected behavior under re-generation from the same prefix. We develop a probabilistic model that formalizes intrinsic
reasoning behaviors as
stochastic events with context-dependent trigger probabilities, and show that unstable boundaries dilute the steering signal. Guided by this analysis, we propose stability filtering, which retains only boundaries where the model consistently reproduces the target behavior. Combined with a content-subspace projection that removes residual question-specific noise, our method achieves 0.784 accuracy on MATH-500 (+5.0 over the strongest baseline). The resulting steering vectors transfer across models in the same architecture family without re-extraction, improving Nemotron-Research-
Reasoning-1.5B (+5.0) and DeepScaleR-1.5B-Preview (+6.0). Code is available at https://github.com/zhmzm/stability-steering.
Read the full abstract ▾
All authors Haomin Zhuang, Hojun Yoo, Xiaonan Luo, Kehan Guo, Xiangliang Zhang
Computation and Language
Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representations
Haitong Sun, Stephen McIntosh, Kwanghee Choi et al. (+3 more)
Speech representations from self-supervised
speech models (S3Ms) are known to be sensitive to phonemic contrasts, but their sensitivity to prosodic contrasts has not been directly measured. The ABX discrimination task has been used to measure phonemic contrast in S3M representations via minimal pairs. We introduce prosodic ABX, an extension of this framework to evaluate prosodic contrast with only a handful of examples and no explicit labels. Also, we build and release a
dataset of English and Japanese minimal pairs and use it along with a Mandarin
dataset to evaluate contrast in English stress, Japanese pitch accent, and Mandarin tone. Finally, we show that model and layer rankings are often preserved across several experimental conditions, making it practical for low-resource settings.
Read the full abstract ▾
All authors Haitong Sun, Stephen McIntosh, Kwanghee Choi, Eunjung Yeo, Daisuke Saito, Nobuaki Minematsu
Computation and Language
Optimizing RAG Rerankers with LLM Feedback via Reinforcement Learning
Yuhang Wu, Xiangqing Shen, Fanfan Wang et al. (+4 more)
Rerankers play a pivotal role in refining
retrieval results for
Retrieval-Augmented Generation. However, current reranking models are typically optimized on static human annotated relevance labels in isolation, decoupled from the downstream generation process. This isolation leads to a fundamental misalignment: documents identified as topically relevant by information
retrieval metrics often fail to provide the actual utility required by the LLM for precise answer generation. To bridge this gap, we introduce ReRanking Preference Optimization (RRPO), a reinforcement learning framework that directly aligns reranking with the LLM's generation quality. By formulating reranking as a sequential decision-making process, RRPO optimizes for context utility using LLM feedback, thereby eliminating the need for expensive human annotations. To ensure training stability, we further introduce a reference-anchored deterministic baseline. Extensive experiments on knowledge-intensive
benchmarks demonstrate that RRPO significantly outperforms strong baselines, including the powerful list-wise reranker RankZephyr. Further analysis highlights the versatility of our framework: it generalizes seamlessly to diverse readers (e.g., GPT-4o), integrates orthogonally with query expansion modules like Query2Doc, and remains robust even when trained with noisy supervisors.
Read the full abstract ▾
All authors Yuhang Wu, Xiangqing Shen, Fanfan Wang, Cangqi Zhou, Zhen Wu, Xinyu Dai, Rui Xia
Computation and Language · Cross-listed from cs.LG
Ouroboros: Dynamic Weight Generation for Recursive Transformers via Input-Conditioned LoRA Modulation
Jaber Jaber and Osama Jaber
Recursive
transformers reuse a shared weight block across multiple depth steps, trading parameters for compute. A core limitation: every step applies the same transformation, preventing the model from composing distinct operations across depth. We present Ouroboros, a system that attaches a compact Controller hypernetwork to a recursive
transformer block. The Controller observes the current hidden state, produces a per-step diagonal modulation vector, and applies it to frozen SVD-initialized LoRA bases, making each recurrence step input-dependent. We combine this with gated recurrence (bias-initialized to 88% retention) and per-step LayerNorm for stable deep iteration. On Qwen2.5-3B split into a Prelude/Recurrent/Coda architecture (17 of 36 layers retained), Ouroboros reduces training loss by 43.4% over the unmodified 17-layer baseline, recovering 51.3% of the performance gap caused by layer removal. The full system adds only 9.2M trainable parameters (Controller, gate, and per-step norms) yet outperforms equivalently-sized static per-step LoRA by 1.44 loss points at depth 1 and remains ahead across all tested depths (1, 4, 8, 16) and ranks (8, 32, 64). We also find that gated recurrence is essential: without it, recursive layer application makes the model strictly worse. These gains are measured on the training distribution; on held-out text, the Controller does not yet improve over the baseline, a limitation we attribute to frozen downstream layers and discuss in detail. Code: https://github.com/RightNow-AI/ouroboros
Read the full abstract ▾
Computation and Language
Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding
Tao Jin, Phuong Minh Nguyen and Naoya Inoue
Speculative decoding accelerates large
language model inference by drafting multiple candidate
tokens and verifying them in a single forward pass. Candidates are organized as a tree: deeper trees accept more
tokens per step, but adding depth requires sacrificing breadth (fallback options) under a fixed verification budget. Existing training-free methods draft from a single
token source and shape their trees without distinguishing candidate quality across origins. We observe that two common training-free
token sources – n-gram matches copied from the input context, and statistical predictions from prior forward passes – differ sharply in acceptance rate (~6x median gap, range 2-18x across five models and five
benchmarks). We prove that when such a quality gap exists, the optimal tree is anisotropic (asymmetric): reliable
tokens should form a deep chain while unreliable
tokens spread as wide branches, raising the depth ceiling of balanced trees. We realize this structure in GOOSE, a training-free framework that builds an adaptive spine tree: a deep chain of high-acceptance context-matched
tokens with wide branches of low-acceptance alternatives at each node. The resulting tree provably accepts at least as many
tokens per step as either source alone. On five LLMs (7B-33B) and five
benchmarks, GOOSE achieves 1.9-4.3x lossless speedup, outperforming balanced-tree baselines by 12-33% under the same budget.
Read the full abstract ▾
Computation and Language
BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs
Nicolas Boizard, Théo Deschamps-Berger, Hippolyte Gisserot-Boukhlef et al. (+2 more)
Transforming causal generative
language models into bidirectional encoders offers a powerful alternative to BERT-style architectures. However, current approaches remain limited: they lack consensus on optimal training objectives, suffer from catastrophic forgetting at scale, and fail to flexibly integrate the vast ecosystem of specialized generative models. In this work, through systematic ablations on the Gemma3 and Qwen3 families, we identify the key factors driving successful adaptation, highlighting the critical role of an often-omitted prior masking phase. To scale this process without original pre-training data, we introduce a dual strategy combining linear weight merging with a lightweight multi-domain data mixture that mitigates catastrophic forgetting. Finally, we augment our encoders by merging them with specialized causal models, seamlessly transferring modality- and domain-specific capabilities. This open-source recipe, designed for any causal decoder LLM, yields BidirLM, a family of five encoders that outperform alternatives on text,
vision, and audio representation
benchmarks.
Read the full abstract ▾
All authors Nicolas Boizard, Théo Deschamps-Berger, Hippolyte Gisserot-Boukhlef, Céline Hudelot, Pierre Colombo
Computation and Language
Tracking the emergence of linguistic structure in self-supervised models learning from speech
Marianne de Heer Kloots, Martijn Bentum, Hosein Mohebbi et al. (+3 more)
Self-supervised
speech models learn effective representations of spoken
language, which have been shown to reflect various aspects of linguistic structure. But when does such structure emerge in model training? We study the encoding of a wide range of linguistic structures, across layers and intermediate checkpoints of six Wav2Vec2 and HuBERT models trained on spoken Dutch. We find that different levels of linguistic structure show notably distinct layerwise patterns as well as learning trajectories, which can partially be explained by differences in their degree of abstraction from the acoustic signal and the timescale at which information from the input is integrated. Moreover, we find that the level at which pre-training objectives are defined strongly affects both the layerwise organization and the learning trajectories of linguistic structures, with greater parallelism induced by higher-order prediction tasks (i.e. iteratively refined pseudo-labels).
Read the full abstract ▾
All authors Marianne de Heer Kloots, Martijn Bentum, Hosein Mohebbi, Charlotte Pouw, Gaofei Shen, Willem Zuidema
Computation and Language
Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It?
Alexander Shabalin, Simon Elistratov, Viacheslav Meshchaninov et al. (+2 more)
Diffusion models have become a standard approach for generative modeling in continuous domains, yet their application to discrete data remains challenging. We investigate why Gaussian
diffusion models with the DDPM solver struggle to sample from discrete distributions that are represented as a mixture of delta-distributions in the continuous space. Using a toy Random Hierarchy Model, we identify a critical sampling interval in which the density of noisified data becomes
multimodal. In this regime, DDPM occasionally enters low-density regions between modes producing out-of-distribution inputs for the model and degrading sample quality. We show that existing heuristics, including self-conditioning and a solver we term q-sampling, help alleviate this issue. Furthermore, we demonstrate that combining self-conditioning with switching from DDPM to q-sampling within the critical interval improves generation quality on real data. We validate these findings across conditional and unconditional tasks in multiple domains, including text, programming code, and
proteins.
Read the full abstract ▾
All authors Alexander Shabalin, Simon Elistratov, Viacheslav Meshchaninov, Ildus Sadrtdinov, Dmitry Vetrov
Computation and Language
$k$NNProxy: Efficient Training-Free Proxy Alignment for Black-Box Zero-Shot LLM-Generated Text Detection
Kahim Wong, Kemou Li, Haiwei Wu et al. (+1 more)
LLM-generated text (LGT) detection is essential for reliable forensic analysis and for mitigating LLM misuse. Existing LGT detectors can generally be categorized into two broad classes: learning-based approaches and zero-shot methods. Compared with learning-based detectors, zero-shot methods are particularly promising because they eliminate the need to train task-specific classifiers. However, the reliability of zero-shot methods fundamentally relies on the assumption that an off-the-shelf proxy LLM is well aligned with the often unknown source LLM, a premise that rarely holds in real-world black-box scenarios. To address this discrepancy, existing proxy
alignment methods typically rely on supervised fine-tuning of the proxy or repeated interactions with commercial APIs, thereby increasing deployment costs, exposing detectors to silent API changes, and limiting robustness under domain shift. Motivated by these limitations, we propose the $k$-nearest neighbor proxy ($k$NNProxy), a training-free and query-efficient proxy
alignment framework that repurposes the $k$NN
language model ($k$NN-LM)
retrieval mechanism as a domain adapter for a fixed proxy LLM. Specifically, a lightweight datastore is constructed once from a target-reflective LGT corpus, either via fixed-budget querying or from existing
datasets. During inference, nearest-neighbor evidence induces a
token-level predictive distribution that is interpolated with the proxy output, yielding an aligned prediction without proxy fine-tuning or per-
token API outputs. To improve robustness under domain shift, we extend $k$NNProxy into a mixture of proxies (MoP) that routes each input to a domain-specific datastore for domain-consistent
retrieval. Extensive experiments demonstrate strong detection performance of our method.
Read the full abstract ▾
All authors Kahim Wong, Kemou Li, Haiwei Wu, Jiantao Zhou
Computation and Language
SAFE: An LLM-as-Verifier Framework for Evidence-Grounded Multi-Hop Reasoning
Daeyong Kwon, Soyoung Yoon and Seung-won Hwang
Multi-hop QA
benchmarks often reward Large
Language Models (LLMs) for spurious correctness, where models reach correct answers through invalid intermediate
reasoning. We propose SAFE, an LLM-as-verifier framework for evidence-grounded multi-hop QA. Rather than judging only the final answer after generation, SAFE verifies
reasoning during generation by checking intermediate steps against the provided passages and previous
reasoning trajectory. To make this process checkable, SAFE decomposes
reasoning into atomic, evidence-grounded units represented with Knowledge Graph (KG) triples. At train-time, SAFE verifies
benchmark supervision under KG-grounded constraints and constructs reliable verifier training data. At inference-time, an external verifier checks each generated step, identifies invalid
reasoning, and provides correction feedback before errors propagate. Across three multi-hop QA
benchmarks, SAFE improves accuracy by 8.8 pp on average. These results show that evidence-grounded multi-hop QA benefits from shifting LLM-based evaluation from post-hoc answer judgment to stepwise
reasoning verification.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
RuleForge: Automated Generation and Validation for Web Vulnerability Detection at Scale
Ayush Garg, Sophia Hager, Jacob Montiel et al. (+5 more)
Security teams face a challenge: the volume of newly disclosed Common Vulnerabilities and Exposures (CVEs) far exceeds the capacity to manually develop detection mechanisms. In 2025, the National Vulnerability Database published over 48,000 new vulnerabilities, motivating the need for automation. We present RuleForge, an AWS internal system that automatically generates detection rules–JSON-based patterns that identify malicious HTTP requests exploiting specific vulnerabilities–from structured
Nuclei templates describing CVE details.
Nuclei templates provide standardized, YAML-based vulnerability descriptions that serve as the structured input for our rule generation process. This paper focuses on RuleForge's architecture and operational deployment for CVE-related threat detection, with particular emphasis on our novel LLM-as-a-judge (Large
Language Model as judge) confidence validation system and systematic feedback integration mechanism. This validation approach evaluates candidate rules across two dimensions–sensitivity (avoiding false negatives) and specificity (avoiding false positives)–achieving AUROC of 0.75 and reducing false positives by 67% compared to synthetic-test-only validation in production. Our 5x5 generation strategy (five parallel candidates with up to five refinement attempts each) combined with continuous feedback loops enables systematic quality improvement. We also present extensions enabling rule generation from unstructured data sources and demonstrate a proof-of-concept agentic workflow for multi-event-type detection. Our lessons learned highlight critical considerations for applying LLMs to cybersecurity tasks, including overconfidence mitigation and the importance of domain expertise in both
prompt design and quality review of generated rules through human-in-the-loop validation.
Read the full abstract ▾
All authors Ayush Garg, Sophia Hager, Jacob Montiel, Aditya Tiwari, Michael Gentile, Zach Reavis, David Magnotti, Wayne Fullen
Computation and Language · Cross-listed from cs.IR
Do We Need Bigger Models for Science? Task-Aware Retrieval with Small Language Models
Florian Kelber, Matthias Jobst, Yuni Susanti et al. (+1 more)
Scientific knowledge discovery increasingly relies on large
language models, yet many existing scholarly assistants depend on proprietary systems with tens or hundreds of billions of parameters. Such reliance limits reproducibility and accessibility for the research community. In this work, we ask a simple question: do we need bigger models for scientific applications? Specifically, we investigate to what extent carefully designed
retrieval pipelines can compensate for reduced model scale in scientific applications. We design a lightweight
retrieval-augmented framework that performs task-aware routing to select specialized
retrieval strategies based on the input query. The system further integrates evidence from full-text scientific papers and structured scholarly metadata, and employs compact instruction-tuned
language models to generate responses with citations. We evaluate the framework across several scholarly tasks, focusing on scholarly question answering (QA), including single- and multi-document scenarios, as well as biomedical QA under domain shift and scientific text compression. Our findings demonstrate that
retrieval and model scale are complementary rather than interchangeable. While
retrieval design can partially compensate for smaller models, model capacity remains important for complex
reasoning tasks. This work highlights
retrieval and task-aware design as key factors for building practical and reproducible scholarly assistants.
Read the full abstract ▾
All authors Florian Kelber, Matthias Jobst, Yuni Susanti, Michael Färber
Computation and Language
Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite
Klaudia Thellmann, Bernhard Stadler and Michael Färber
Machine-translated
benchmark datasets reduce costs and offer scale, but noise, loss of structure, and uneven quality weaken confidence. What matters is not merely whether we can translate, but also whether we can measure and verify translation reliability at scale. We study translation quality in the EU20
benchmark suite, which comprises five established
benchmarks translated into 20
languages, via a three-step automated quality assurance approach: (i) a structural corpus audit with targeted fixes; (ii) quality profiling using a neural metric (COMET, reference-free and reference-based) with translation service comparisons (DeepL / ChatGPT / Google); and (iii) an LLM-based span-level translation error landscape. Trends are consistent:
datasets with lower COMET scores exhibit a higher share of accuracy/mistranslation errors at span level (notably HellaSwag; ARC is comparatively clean). Reference-based COMET on MMLU against human-edited samples points in the same direction. We release cleaned/corrected versions of the EU20
datasets, and code for reproducibility. In sum, automated quality assurance offers practical, scalable indicators that help prioritize review – complementing, not replacing, human gold standards.
Read the full abstract ▾
Computation and Language
How to measure the optimality of word or gesture order with respect to the principle of swap distance minimization
Ramon Ferrer-i-Cancho
The structure of all the permutations of a sequence can be represented as a permutohedron, a graph where vertices are permutations and two vertices are linked if a swap of adjacent elements in the permutation of one of the vertices produces the permutation of the other vertex. It has been hypothesized that word orders in
languages minimize the swap distance in the permutohedron: given a source order, word orders that are closer in the permutohedron should be less costly and thus more likely. Here we explain how to measure the degree of optimality of word order variation with respect to swap distance minimization. We illustrate the power of our novel mathematical framework by examining spontaneous gestures produced by speakers from four
languages (two SVO
languages and two VSO
languages, which represent two linguistic families). We show that crosslinguistic gestures are at least $77\%$ optimal. It is unlikely that the multiple times where crosslinguistic gestures hit optimality are due to chance. We establish the theoretical foundations for research on the optimality of word or gesture order with respect to swap distance minimization in communication systems. Finally, we introduce the quadratic assignment problem (QAP) into
language research as an umbrella for multiple optimization problems and, accordingly, postulate a general principle of optimal assignment that unifies various linguistic principles including swap distance minimization.
Read the full abstract ▾
Computation and Language
Reliable News or Propagandist News? A Neurosymbolic Model Using Genre, Topic, and Persuasion Techniques to Improve Robustness in Classification
Géraud Faye, Benjamin Icard, Morgane Casanova et al. (+6 more)
Among news disorders, propagandist news are particularly insidious, because they tend to mix oriented messages with factual reports intended to look like reliable news. To detect propaganda, extant approaches based on
Language Models such as BERT are promising but often overfit their training
datasets, due to biases in data collection. To enhance classification robustness and improve generalization to new sources, we propose a neurosymbolic approach combining non-contextual text
embeddings (fastText) with symbolic conceptual features such as genre, topic, and persuasion techniques. Results show improvements over equivalent text-only methods, and ablation studies as well as explainability analyses confirm the benefits of the added features. Keywords: Information disorder, Fake news, Propaganda, Classification, Topic modeling, Hybrid method, Neurosymbolic model, Ablation, Robustness
Read the full abstract ▾
All authors Géraud Faye, Benjamin Icard, Morgane Casanova, Guillaume Gadek, Guillaume Gravier, Wassila Ouerdane, Céline Hudelot, Sylvain Gatepaille, Paul Égré
Computation and Language
ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues
Bhaskara Hanuma Vedula, Darshan Anghan, Ishita Goyal et al. (+2 more)
Large
Language Models increasingly suppress biased outputs when demographic identity is stated explicitly, yet may still exhibit implicit biases when identity is conveyed indirectly. Existing
benchmarks use name based proxies to detect implicit biases, which carry weak associations with many social demographics and cannot extend to dimensions like age or socioeconomic status. We introduce ImplicitBBQ, a QA
benchmark that evaluates implicit bias through characteristic based cues, demographically associated attributes that signal implicitly, across age, gender, region, religion, caste, and socioeconomic status. Evaluating 11 models, we find that implicit bias in ambiguous contexts is over six times higher than explicit bias in open weight models. Notably, this bias is distributed unevenly across demographics: caste emerges as the most severe while gender is the least affected. Safety prompting and chain-of-thought
reasoning fail to substantially close this gap; even few-shot prompting, which reduces implicit bias by 79%, leaves caste bias at four times the level of any other dimension. These findings indicate that current
alignment and prompting strategies address the surface of bias evaluation while leaving demographically associated stereotypic associations largely unresolved. We publicly release our code and
dataset for model providers and researchers to
benchmark potential mitigation techniques.
Read the full abstract ▾
All authors Bhaskara Hanuma Vedula, Darshan Anghan, Ishita Goyal, Ponnurangam Kumaraguru, Abhijnan Chakraborty
Computation and Language
Is Clinical Text Enough? A Multimodal Study on Mortality Prediction in Heart Failure Patients
Oumaima El Khettari, Virgile Barthet, Guillaume Hocquet et al. (+3 more)
Accurate short-term mortality prediction in heart failure (HF) remains challenging, particularly when relying on structured electronic health record (EHR) data alone. We evaluate
transformer-based models on a French HF cohort, comparing text-only, structured-only,
multimodal, and LLM-based approaches. Our results show that enriching clinical text with entity-level representations improves prediction over CLS
embeddings alone, and that supervised
multimodal fusion of text and structured variables achieves the best overall performance. In contrast, large
language models perform inconsistently across modalities and decoding strategies, with text-only
prompts outperforming structured or
multimodal inputs. These findings highlight that entity-aware
multimodal transformers offer the most reliable solution for short-term HF outcome prediction, while current LLM prompting remains limited for clinical decision support.
Read the full abstract ▾
All authors Oumaima El Khettari, Virgile Barthet, Guillaume Hocquet, Joconde Weller, Emmanuel Morin, Pierre Zweigenbaum
Computation and Language
SURE: Synergistic Uncertainty-aware Reasoning for Multimodal Emotion Recognition in Conversations
Yiqiang Cai, Chengyan Wu, Bolei Ma et al. (+4 more)
Multimodal emotion recognition in conversations (MERC) requires integrating
multimodal signals while being robust to noise and modeling contextual
reasoning. Existing approaches often emphasize fusion but overlook uncertainty in noisy features and fine-grained
reasoning. We propose SURE (Synergistic Uncertainty-aware
REasoning) for MERC, a framework that improves robustness and contextual modeling. SURE consists of three components: an Uncertainty-Aware Mixture-of-Experts module to handle modality-specific noise, an Iterative
Reasoning module for multi-turn
reasoning over context, and a
Transformer Gate module to capture intra- and inter-modal interactions. Experiments on
benchmark MERC
datasets show that SURE consistently outperforms state-of-the-art methods, demonstrating its effectiveness in robust
multimodal reasoning. These results highlight the importance of uncertainty modeling and iterative
reasoning in advancing
emotion recognition in conversational settings.
Read the full abstract ▾
All authors Yiqiang Cai, Chengyan Wu, Bolei Ma, Bo Chen, Yun Xue, Julia Hirschberg, Ziwei Gong
Computation and Language · Cross-listed from cs.CV
Yansong Guo, Chaoyang Zhu, Jiayi Ji et al. (+2 more)
Video Large
Language Models (VideoLLMs) have demonstrated impressive capabilities in
video understanding, yet the massive number of input
video tokens incurs a significant computational burden for deployment. Existing methods mainly prune
video tokens at input level while neglecting the inherent information structure embedded in
videos and large
language models (LLMs). To address this, we propose HieraVid, a hierarchical
pruning framework that progressively and dynamically reduces visual redundancy. Based on two observations that
videos possess the segment-frame structure and LLMs internally propagate multi-modal information unidirectionally, we decompose
pruning into three levels: 1) segment-level, where
video tokens are first temporally segmented and spatially merged; 2) frame-level, where similar frames within the same segment are jointly pruned to preserve diversity; 3) layer-level, redundancy gradually shrinks as LLM layer increases w/o compromising performance. We conduct extensive experiments on four widely used
video understanding
benchmarks to comprehensively evaluate the effectiveness of HieraVid. Remarkably, with only 30% of
tokens retained, HieraVid achieves new state-of-the-art performance, while maintaining over 98% and 99% of the performance of LLaVA-
Video-7B and LLaVA-OneVision-7B, respectively.
Read the full abstract ▾
All authors Yansong Guo, Chaoyang Zhu, Jiayi Ji, Jianghang Lin, Liujuan Cao
Computation and Language
Beyond Detection: Ethical Foundations for Automated Dyslexic Error Attribution
Samuel Rose and Debarati Chakraborty
Dyslexic spelling errors exhibit systematic phonological and orthographic patterns that distinguish them from the errors produced by typically developing writers. While this observation has motivated dyslexic-specific spell-checking and assistive writing tools, prior work has focused predominantly on error correction rather than attribution, and has largely neglected the ethical risks. The risk of harmful labelling, covert screening, algorithmic bias, and institutional misuse that automated classification of learners entails requires the development of robust ethical and legal frameworks for research in this area. This paper addresses both gaps. We formulate dyslexic error attribution as a binary classification task. Given a misspelt word and its correct target form, determine whether the error pattern is characteristic of a dyslexic or non-dyslexic writer. We develop a comprehensive feature set capturing orthographic, phonological, and morphological properties of each error, and propose a twin-input neural model evaluated against traditional machine learning baselines under writer-independent conditions. The neural model achieves 93.01% accuracy and an F1-score of 94.01%, with phonetically plausible errors and vowel confusions emerging as the strongest attribution signals. We situate these technical results within an explicit ethics-first framework, analysing
fairness across subgroups, the interpretability requirements of educational deployment, and the conditions, consent, transparency, human oversight, and recourse, under which a system could be responsibly used. We provide concrete guidelines for ethical deployment and an open discussion of the systems limitations and misuse potential. Our results demonstrate that dyslexic error attribution is feasible at high accuracy while underscoring that feasibility alone is insufficient for deployment in high-stakes educational contexts.
Read the full abstract ▾
Computation and Language
From Guessing to Placeholding: A Cost-Theoretic Framework for Uncertainty-Aware Code Completion
Liang Zhu, Haolin Chen, Lidong Zhao et al. (+1 more)
While Large
Language Models (LLMs) have demonstrated exceptional proficiency in code completion, they typically adhere to a Hard Completion (HC) paradigm, compelling the generation of fully concrete code even amidst insufficient context. Our analysis of 3 million real-world interactions exposes the limitations of this strategy: 61% of the generated suggestions were either edited after acceptance or rejected despite exhibiting over 80% similarity to the user's subsequent code, suggesting that models frequently make erroneous predictions at specific
token positions. Motivated by this observation, we propose Adaptive Placeholder Completion (APC), a collaborative framework that extends HC by strategically outputting explicit placeholders at high-
entropy positions, allowing users to fill directly via IDE
navigation. Theoretically, we formulate code completion as a cost-minimization problem under uncertainty. Premised on the observation that filling placeholders incurs lower cost than correcting errors, we prove the existence of a critical
entropy threshold above which APC achieves strictly lower expected cost than HC. We instantiate this framework by constructing training data from filtered real-world edit logs and design a cost-based reward function for reinforcement learning. Extensive evaluations across 1.5B–14B parameter models demonstrate that APC reduces expected editing costs from 19% to 50% while preserving standard HC performance. Our work provides both a theoretical foundation and a practical training framework for uncertainty-aware code completion, demonstrating that adaptive abstention can be learned end-to-end without sacrificing conventional completion quality.
Read the full abstract ▾
All authors Liang Zhu, Haolin Chen, Lidong Zhao, Xian Wu