Computation and Language
Instruction Complexity Induces Positional Collapse in Adversarial LLM Evaluation
Jon-Paul Cacioli
When instructed to underperform on multiple-choice evaluations, do
language models engage with question content or fall back on positional shortcuts? We map the boundary between these regimes using a six-condition
adversarial instruction-specificity gradient administered to two instruction-tuned LLMs (Llama-3-8B and Llama-3.1-8B) on 2,000 MMLU-Pro items. Distributional screening (response-position
entropy) and an independent content-engagement criterion (difficulty-accuracy correlation) jointly characterise each condition. The gradient reveals three regimes rather than a monotonic transition. Vague
adversarial instructions produce moderate accuracy reduction with preserved content engagement. Standard sandbagging and capability-imitation instructions produce positional
entropy collapse with partial content engagement. A two-step answer-aware avoidance instruction produces extreme positional collapse, with near-total concentration on a single response position (99.9% and 87.4%) and no measurable content sensitivity. This was the only multi-step instruction tested, and it produced the most extreme shortcut. The attractor position matches each model's content-absent null-
prompt default. The effect replicates across both models and four academic domains. Distributional collapse and content engagement can co-occur (50% concordance between screening criteria), indicating that
entropy-based screening and difficulty-based content assessment capture partially independent dimensions of response validity. Results suggest that instruction complexity can determine whether
adversarial compliance uses content-aware or content-blind mechanisms in small instruction-tuned LLMs under greedy decoding.
Read the full abstract ▾
Computation and Language
Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairs
Serpil Karabüklü, Kanishka Misra, Shester Gueuwou et al. (+3 more)
Models of sign
language have historically lagged behind those for spoken
language (text and
speech). Recent work has greatly improved their performance on tasks like sign
language translation and isolated sign recognition. However, it remains unclear to what extent existing models capture various linguistic phenomena of sign
language, and how well they use cues from the multiple articulators used in sign
language (hands, upper body, face). We introduce a new
benchmark dataset for American Sign
Language, ASL Minimal Translation Pairs (ASL-MTP), divided into multiple types of sign
language phenomena and corresponding minimal pairs of translations, for performing such linguistic analyses. As a case study, we use ASL-MTP to analyze a state-of-the-art ASL-to-English translation model. We conduct a targeted analysis of the model by ablating various input cues during training and inference and evaluating on the phenomena in ASL-MTP. Our results show that, while the model performs above chance level on most of the phenomena, it relies strongly on manual cues while often missing crucial non-manual cues.
Read the full abstract ▾
All authors Serpil Karabüklü, Kanishka Misra, Shester Gueuwou, Diane Brentari, Greg Shakhnarovich, Karen Livescu
Computation and Language · Cross-listed from cs.AI
When Roles Fail: Epistemic Constraints on Advocate Role Fidelity in LLM-Based Political Statement Analysis
Juergen Dietrich
Democratic discourse analysis systems increasingly rely on multi-
agent LLM pipelines in which distinct evaluator models are assigned
adversarial roles to generate structured, multi-perspective assessments of political statements. A core assumption is that models will reliably maintain their assigned roles. This paper provides the first systematic empirical test of that assumption using the TRUST pipeline. We develop an epistemic stance classifier that identifies advocate roles from
reasoning text without relying on surface vocabulary, and measure role fidelity across 60 political statements (30 English, 30 German) using four metrics: Role Drift Index (RDI), Expected Drift Distance (EDD), Directional Drift Index (DDI), and
Entropy-based Role Stability (ERS). We identify two failure modes - the Epistemic Floor Effect (fact-check results create an absolute lower bound below which the legitimizing role cannot be maintained) and Role-Prior Conflict (training-time knowledge overrides role instructions for factually unambiguous statements) - as manifestations of a single mechanism: Epistemic Role Override (ERO). Model choice significantly affects role fidelity: Mistral Large outperforms Claude Sonnet by 28pp (67% vs. 39%) and exhibits a qualitatively different failure mode - role abandonment without polarity reversal - compared to Claude's active switch to the opposing stance. Role fidelity is
language-robust. Fact-check provider choice is not universally neutral: Perplexity significantly reduces Claude's role fidelity on German statements (Delta = -15pp, p = 0.007) while leaving Mistral unaffected. These findings have direct implications for multi-
agent LLM validation: a system validated without role fidelity measurement may systematically misrepresent the epistemic diversity it was designed to provide.
Read the full abstract ▾
Computation and Language
Selective Augmentation: Improving Universal Automatic Phonetic Transcription via G2P Bootstrapping
Tobias Bystrich, Julia M. Pritzen, Christoph A. Schmidt et al. (+1 more)
In the field of universal automatic phonetic transcription (APT), clean and diverse training transcriptions are required. However, such high-quality data is limited. We propose the bootstrapping approach Selective Augmentation to improve the available training transcriptions by selectively transferring distinctions between
languages. Based on the model MultIPA, we exemplarily show that we could increase the accuracy of an existing feature (plosive voicing) and add a new feature (plosive aspiration) by augmenting the existing training data using information from a separate helper
language (Hindi). We describe intrinsic challenges of the evaluation and develop objective metrics to determine the success: Voicing accuracy was increased by 17.6% by reducing the number of false positives. Additionally, aspiration recognition was introduced: While the baseline transcribed 0% of German /p, t, k/ as aspirated, our approach transcribed them as aspirated in 61.2% of the cases. Introducing aspiration recognition to APT models allowed for the tenuis class to be successfully reduced by 32.2%, which also reduces the conflations between the test
language's plosives.
Read the full abstract ▾
All authors Tobias Bystrich, Julia M. Pritzen, Christoph A. Schmidt, Claudia Wich-Reif
Computation and Language
Path-Lock Expert: Separating Reasoning Mode in Hybrid Thinking via Architecture-Level Separation
Shouren Wang, Wang Yang, Chuang Ma et al. (+7 more)
Hybrid-thinking
language models expose explicit /think and /no_think modes, but current designs do not separate them cleanly. Even in /no_think mode, models often emit long and self-reflective responses, causing
reasoning leakage. Existing work reduces this issue through better data curation and multi-stage training, yet leakage remains because both modes are still encoded in the same feed-forward parameters. We propose Path-Lock Expert (PLE), an architecture-level solution that replaces the single MLP in each decoder layer with two semantically locked experts, one for /think and one for /no_think, while keeping
attention,
embeddings, normalization, and the
language-model head shared. A deterministic control-
token router selects exactly one expert path for the entire sequence, so inference preserves the dense model's per-
token computation pattern and each expert receives mode-pure updates during supervised fine-tuning. Across math and science
reasoning benchmarks, PLE maintains strong /think performance while producing a substantially stronger mode separation, a /no_think mode with higher accuracy and far less
reasoning leakage. On Qwen3-4B, for example, compared to the SFT-only baseline on AIME24, PLE generates 17x fewer reflective
tokens (6.01 vs. 0.35 per answer) and 2x shorter outputs (8665 vs. 4101
tokens), and improves /no_think accuracy from 35.33% to 44.67%, while maintaining /think-mode performance (61.33% vs. 60.00%). These results suggest that controllable hybrid thinking is fundamentally an architectural problem, and separating mode-specific feed-forward pathways is a simple and effective solution.
Read the full abstract ▾
All authors Shouren Wang, Wang Yang, Chuang Ma, Debargha Ganguly, Vikash Singh, Chaoda Song, Xinpeng Li, Xianxuan Long, Vipin Chaudhary, Xiaotian Han
Computation and Language
Semantic Structure of Feature Space in Large Language Models
Austin C. Kozlowski and Andrei Boutyline
We show that the
geometric relations between semantic features in large
language models' hidden states closely mirror human psychological associations. We construct feature vectors corresponding to 360 words and project them on 32 semantic axes (e.g. beautiful-ugly, soft-hard), and find that these projections correlate highly with human ratings of those words on the respective semantic scales. Second, we find that the cosine similarities between the semantic axes themselves are highly predictive of the correlations between these scales in the survey. Third, we show that substantial variance across the 32 semantic axes lies on a low-dimensional subspace, reproducing patterns typical of human semantic associations. Finally, we demonstrate that steering a word on one semantic axis causes spillover effects on the model's rating of that word on other semantic scales proportionate to the cosine similarity between those semantic axes. These findings suggest that features should be understood not only in isolation but through their
geometric relations and the meaningful subspaces they form.
Read the full abstract ▾
Computation and Language
Cross-Lingual Response Consistency in Large Language Models: An ILR-Informed Evaluation of Claude Across Six Languages
Camelia Baluta
This paper introduces a systematic evaluation framework grounded in the Interagency
Language Roundtable (ILR) Skill Level Descriptions and applies it to Claude (Sonnet 4.6) across six
languages: English, French, Romanian, Spanish, Italian, and German. We administer a battery of 12 semantically equivalent
prompt clusters spanning ILR complexity levels 1 through 3+, collect 216 responses (12
prompts, 6
languages, 3 runs), and analyze outputs through a two-layer methodology combining automated quantitative metrics with expert ILR qualitative assessment. Quantitative analysis reveals that French responses are approximately 30% longer than German responses on identical
prompts, and that creative and affective clusters show the highest cross-lingual surface divergence. Qualitative analysis, conducted by a six-
language professional with 12 years of ILR/OPI assessment experience, identifies five cross-lingual variation patterns: systematic differences in pragmatic disambiguation strategies, aesthetic and literary tradition divergence in creative output,
language-internal technical terminology norms, cultural calibration gaps evidenced by the absence of culture-specific content in favor of culturally neutralized templates, and
language-specific institutional referral behavior in emotional support responses. We argue that ILR-informed expert judgment applied to LLM outputs constitutes a novel and underreported evaluation methodology that complements purely computational
benchmarks, and that cross-lingual output variation in Claude is interpretable, domain-dependent, and consequential for equitable
multilingual AI deployment.
Read the full abstract ▾
Computation and Language
Exploring the Limits of Pruning: Task-Specific Neurons, Model Collapse, and Recovery in Task-Specific Large Language Models
M. K. Khalidi Siam, Md. Tausif-Ul-Islam, Md. Reshad Romim Khan et al. (+5 more)
Neuron pruning is widely used to reduce the computational cost and parameter footprint of large
language models, yet it remains unclear whether
neurons in task-specific models contribute uniformly to task performance. In this work, we provide empirical evidence for the existence and importance of task-specific
neurons through a systematic
pruning study on
language models specialized for mathematical
reasoning and code generation. We introduce an activation-based selectivity metric to identify
neurons with low contribution to the target task and prune them while preserving target-task accuracy, and compare selective
pruning with random
pruning. Selective
pruning consistently outperforms random
pruning, indicating that activation-based selectivity provides a systematic advantage over random
pruning. Reverse
pruning experiments further show that removing a small subset of highly task-specific
neurons (~10%) causes complete performance collapse, suggesting that there exist task specific
neurons and critical task information is concentrated in a small portion of the network. In contrast, selective
pruning of less critical
neurons (~30% - ~35%) reduces accuracy but still preserves significant performance. We also observed consistent reductions in parameters and runtime VRAM usage, along with improved inference throughput as
pruning increases. Experiments on both 1.5B and 7B models reveal a robustness threshold around 15-20%
pruning, beyond which accuracy loss and generation failures increase sharply. Fine-tuning substantially recovers performance across
pruning levels, particularly for aggressively pruned models. These findings provide empirical evidence of
neuron specialization in task-specific
language models and offer insights into
pruning robustness, model redundancy, and post-
pruning recoverability.
Read the full abstract ▾
All authors M. K. Khalidi Siam, Md. Tausif-Ul-Islam, Md. Reshad Romim Khan, Mohammed Ali Hossain, Mushfiqul Amin, Labib Hasan Khan, Niloy Farhan, Farig Sadeque
Computation and Language
Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversations
Mingqian Zheng, Malia Morgan, Liwei Jiang et al. (+2 more)
Current LLM safety
alignment techniques improve model robustness against
adversarial attacks, but overlook whether and how LLMs can recover helpfulness when benign users clarify their intent. We introduce CarryOnBench, the first interactive
benchmark that measures whether LLMs can revise their interpretation of user intent and recover utility, while remaining safe through multi-turn conversations. Starting from 398 seemingly harmful queries with benign underlying intents, we simulate 5,970 conversations by varying user follow-up sequences, evaluating 14 models on both intent-aligned utility and safety. CarryOnBench yields 1,866 different conversation flows of 4–12 turns, totaling 23,880 model responses. We design Ben-Util, a checklist-based metric that evaluates how well each model response fulfills the user's benign information need using atomic items. At turn one, models fulfill only 10.5–37.6% of the user's benign information need. When the same query includes the benign intent upfront, models fulfill 25.1–72.1%, confirming that models withhold information due to intent misinterpretation, not limited knowledge. With benign clarifications in multi-turn conversations, 13 of 14 models approach or exceed this single-turn baseline, yet recovery cost varies across models. We identify two failure modes invisible to single-turn evaluations—unsafe recovery, where a model updates at disproportionate safety cost and redundant recovery, where a model recycles prior responses rather than providing new information. Moreover, conversations converge to similar harmfulness levels regardless of how conservative the model starts. These findings expose a gap that single-turn evaluations miss—whether a model is appropriately cautious or simply unresponsive to clarified user intent.
Read the full abstract ▾
All authors Mingqian Zheng, Malia Morgan, Liwei Jiang, Carolyn Rose, Maarten Sap
Computation and Language · Cross-listed from cs.LG
Detecting Clinical Discrepancies in Health Coaching Agents: A Dual-Stream Memory and Reconciliation Architecture
Samuel L Pugh, Eric Yang, Alexander Muir Sutherland et al. (+1 more)
As Large
Language Model (LLM)
agents transition from single-session tools to persistent systems managing longitudinal healthcare journeys, their
memory architectures face a critical challenge: reconciling two imperfect sources of truth. The patient's evolving self-report is current but prone to recall bias, while the Electronic Health Record (EHR) is medically validated but frequently stale. General-purpose
agent memory systems optimize for coherence by overwriting older facts with the user's latest statement, a pattern that risks safety failures when applied to clinical data. We introduce a Dual-Stream
Memory Architecture that strictly separates the patient narrative from the structured clinical record (FHIR), governed by a dedicated Reconciliation Engine that evaluates every extracted
memory against the patient's FHIR profile and classifies discrepancies by type, severity, and the specific FHIR resources involved. We evaluate this architecture on 26 patients across 675 longitudinal wellness coaching sessions, using a hybrid
dataset that interleaves real provider-patient transcripts with synthetic, FHIR-grounded clinical scenarios. In isolated testing, the engine detects 84.4% of designed clinical discrepancies with 86.7% safety-critical recall. By coupling extraction and reconciliation evaluation on the same data, we directly quantify a 13.6% error cascade, tracing the degradation to clinical details lost during
memory extraction from unstructured conversation rather than to downstream classification errors. These findings establish that validating patient-reported
memories against clinical records is both feasible and necessary for safe deployment of longitudinal health
agents.
Read the full abstract ▾
All authors Samuel L Pugh, Eric Yang, Alexander Muir Sutherland, Alessandra Breschi
Computation and Language
CL-bench Life: Can Language Models Learn from Real-Life Context?
Shihan Dou, Yujiong Shen, Chenhao Huang et al. (+35 more)
Today's AI assistants such as OpenClaw are designed to handle context effectively, making context learning an increasingly important capability for models. As these systems move beyond professional settings into everyday life, the nature of the contexts they must handle also shifts. Real-life contexts are often messy, fragmented, and deeply tied to personal and social experience, such as multi-party conversations, personal archives, and behavioral traces. Yet it remains unclear whether current frontier
language models can reliably learn from such contexts and solve tasks grounded in them. To this end, we introduce CL-bench Life, a fully human-curated
benchmark comprising 405 context-task pairs and 5,348 verification rubrics, covering common real-life scenarios. Solving tasks in CL-bench Life requires models to reason over complex, messy real-life contexts, calling for strong real-life context learning abilities that go far beyond those evaluated in existing
benchmarks. We evaluate ten frontier LMs and find that real-life context learning remains highly challenging: even the best-performing model achieves only 19.3% task solving rate, while the average performance across models is only 13.8%. Models still struggle to reason over contexts such as messy group chat histories and fragmented behavioral records from everyday life. CL-bench Life provides a crucial testbed for advancing real-life context learning, and progress on it can enable more intelligent and reliable AI assistants in everyday life.
Read the full abstract ▾
All authors Shihan Dou, Yujiong Shen, Chenhao Huang, Junjie Ye, Jiayi Chen, Junzhe Wang, Qianyu He, Shichun Liu, Changze Lv, Jiahang Lin, Jiazheng Zhang, Ming Zhang, Shaofan Liu, Tao Ji, Zhangyue Yin, Cheng Zhang, Huaibing Xie, Jianglu Hu, Jingcheng Deng, Lincheng Li, Minda Hu, Shaolei Wang, Syrus Zhao, Weichao Wang, Yan Lei, Yang Liu, Yanling Xiao, Yiting Liu, Zenan Xu, Zhen Guo, Ziliang Zhao, Pluto Zhou, Tao Gui, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang, Di Wang, Shunyu Yao
Computation and Language
Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling
Zhen Zhang, Changyi Yang, Zijie Xia et al. (+11 more)
Tokens are the fundamental units of computation in modern autoregressive models, and generation length directly influences both inference cost and
reasoning performance. Despite its importance, existing approaches model length primarily at the coarse sequence level. We introduce the Length Value Model (LenVM), a
token-level framework that estimates the remaining generation length at every decoding step. By formulating length modeling as a value estimation problem and assigning a constant negative reward to each generated
token, LenVM predicts a bounded, discounted return that is a monotone proxy for the remaining generation horizon. This value formulation provides annotation-free, dense, unbiased, and scalable supervision. Experiments on LLMs and VLMs show that LenVM supports exact control, continuous performance–efficiency steering, length prediction, and interpretation. On LIFEBench-
token, it raises the exact-length score of Qwen2.5-7B-Instruct from $30.9$ to $64.8$ in one pass and reaches $83.6$ when combined with LCG. Near $200$
tokens on GSM8K, LenVM retains about $63\%$ Pass@1 versus $6\%$ under a hard
token budget. Across other settings, LenVM improves matched-length quality over prompting and EOS calibration, while its length-prediction accuracy improves with scale and exceeds specialized predictors. Its
token-level values also identify markers of shifts between longer and shorter trajectories. These results establish generation length as an effective scalable
token-level value signal for control, prediction, and future value-based training.
Read the full abstract ▾
All authors Zhen Zhang, Changyi Yang, Zijie Xia, Zhen Yang, Chengzhi Liu, Zhaotiao Weng, Yepeng Liu, Haobo Chen, Jin Pan, Chenyang Zhao, Yuheng Bu, Alkesh Patel, Zhe Gan, Xin Eric Wang
Computation and Language · Cross-listed from cs.IR
Hypencoder Revisited: Reproducibility and Analysis of Non-Linear Scoring for First-Stage Retrieval
Arne Eichholtz, Yongkang Li, Jutte Vijverberg et al. (+2 more)
The Hypencoder, proposed by Killingback et al., is a
retrieval framework that replaces the fixed inner-product scoring function used in standard bi-encoders with a query-specific neural network (the $q$-net), whose weights are generated by a hypernetwork from the contextualized query
embeddings. This design enables more expressive relevance estimation while preserving independent query and document encoding. In this work, we conduct a reproducibility study of the Hypencoder and extend the original analysis in three directions. Our reproduction confirms that the Hypencoder outperforms a similarly trained bi-encoder baseline on in-domain and out-of-domain
benchmarks, and that the proposed efficient search
algorithm substantially reduces query latency with minimal performance loss. On hard
retrieval tasks, we find partial support: the Hypencoder outperforms the baseline on DL-Hard and FollowIR, but not on TREC TOT, where checkpoint incompatibility and fine-tuning sensitivity complicate full verification. Beyond reproduction, we investigate three extensions: (i)~integrating alternative pre-trained encoders into the Hypencoder framework, where we find that performance gains depend on the encoder and fine-tuning strategy; (ii)~comparing query latency against a Faiss-based bi-encoder pipeline, revealing that standard bi-encoder
retrieval remains faster under both exhaustive and efficient search settings; and (iii)~evaluating
adversarial robustness, where we find that the $q$-net's non-linear scoring does not provide a consistent robustness disadvantage over inner-product scoring. Our code is publicly available at https://github.com/arneeichholtz/Hypencoder-reprod.
Read the full abstract ▾
All authors Arne Eichholtz, Yongkang Li, Jutte Vijverberg, Tobias Groot, Mohammad Aliannejadi
Computation and Language · Cross-listed from cs.LG
Dynamic Adversarial Fine-Tuning Reorganizes Refusal Geometry
Wenhao Lan, Shan Li, Xinhua Lai et al. (+4 more)
Safety-aligned
language models must refuse harmful requests without broad over-refusal, but it remains unclear how dynamic
adversarial fine-tuning changes refusal-control carriers: Kullback–Leibler (KL)-constrained directions or small subspaces that causally modulate refusal without large safe-
prompt distribution shifts. We study a 7B backbone under supervised fine-tuning (SFT) and Robust Refusal Dynamic Defense (R2D2), aligning HarmBench, StrongREJECT, and XSTest evaluations with five-anchor geometry measurements, causal interventions, and sparse adaptive stress tests. R2D2 drives fixed-source HarmBench attack success to zero at early checkpoints; however, these checkpoints also exhibit maximal XSTest refusal and fail a benign-utility audit. Later checkpoints partially recover utility-facing behavior while reopening attack success, with adaptive GCG attack success rate rising to 0.415 at step 250 and 0.613 at step 500. Internally, R2D2 preserves a late-layer admissible refusal-control carrier through step 100 and then relocates the best admissible carrier to an early layer; SFT relocates earlier yet remains less robust. Effective rank stays near 1.24, and SFT shows larger principal-angle drift, arguing against both dimensional expansion and drift magnitude as sufficient explanations. Causal interventions support a low-dimensional but utility-coupled carrier. These results support a geometry-reorganization account of R2D2 along a robustness–utility frontier, without establishing adaptive robustness.
Read the full abstract ▾
All authors Wenhao Lan, Shan Li, Xinhua Lai, Meiqi Wu, Junbin Yang, Haihua Shen, Yijun Yang
Computation and Language
Gongbo Zhang, Wen Wang, Ye Tian et al. (+1 more)
Diffusion large
language models (dLLMs) offer parallel decoding and bidirectional context, but state-of-the-art dLLMs require billions of parameters for competitive performance. While existing
distillation methods for dLLMs reduce inference steps within a single architecture, none address cross-architecture knowledge transfer, in which the teacher and student differ in architecture,
attention mechanism, and tokenizer. We present TIDE, the first framework for cross-architecture dLLM
distillation, comprising three modular components: (1)
TIDAL, which jointly modulates
distillation strength across training progress and
diffusion timestep to account for the teacher's noise-dependent reliability; (2) CompDemo, which enriches the teacher's context via complementary mask splitting to improve predictions under heavy masking; and (3) Reverse CALM, a cross-tokenizer objective that inverts chunk-level likelihood matching, yielding bounded gradients and dual-end noise filtering. Distilling 8B dense and 16B MoE teachers into a 0.6B student via two heterogeneous pipelines outperforms the baseline by an average of 1.53 points across eight
benchmarks, yielding notable gains in code generation, where HumanEval scores reach 48.78 compared to 32.3 for the AR baseline.
Read the full abstract ▾
All authors Gongbo Zhang, Wen Wang, Ye Tian, Li Yuan
Computation and Language
Select to Think: Unlocking SLM Potential with Local Sufficiency
Wenxuan Ye, Yangyang Zhang, Xueli An et al. (+2 more)
Small
language models (SLMs) offer efficient deployment, yet they often lag behind their larger counterparts (LLMs) in
reasoning. Existing remedies either invoke an LLM at points of
reasoning divergence, incurring substantial latency and cost, or rely on standard
distillation, which is limited by the SLM's capacity to accurately mimic the LLM's complex generative distribution. We address this dilemma by identifying local sufficiency: at divergence points, the LLM's preferred
token often resides within the SLM's top-K next-
token predictions, even when failing to emerge as the SLM top-1 choice. We therefore propose Select to Think (S2T), which reframes the LLM's role from open-ended generation to selection among the SLM's proposals, simplifying the supervision signal to discrete candidate rankings. Leveraging this, we introduce S2T-Local, which distills the selection logic into the SLM, empowering it to perform autonomous re-ranking without inference-time LLM dependency. Empirically, a 1.5B SLM's top-8 candidates contain the 32B LLM's choice with a 95% hit rate, and S2T-Local improves the 1.5B SLM's Math Avg. over greedy decoding by 24.1% relative gain, matching the efficacy of 8-path self-consistency with single-trajectory efficiency.
Read the full abstract ▾
All authors Wenxuan Ye, Yangyang Zhang, Xueli An, Georg Carle, Yunpu Ma
Computation and Language · Cross-listed from cs.SE
ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation
Yeheng Chen, Chaoxiang Xie, Yuling Shi et al. (+4 more)
LLMs have achieved strong results on both function-level code synthesis and repository-level code modification, yet a capability that falls between these two extremes – compositional code creation, i.e., building a complete, internally structured class from a specification – remains underserved. Current evaluations are either confined to isolated functions or rely on manually curated class-level tasks that are expensive to scale and increasingly susceptible to data contamination. We introduce ClassEval-Pro, a
benchmark of 300 class-level tasks spanning 11 domains, constructed through an automated three-stage pipeline that combines complexity enhancement, cross-domain class composition, and integration of real-world GitHub code contributed after January 2025. Every task is validated by an LLM Judge Ensemble and must pass test suites with over 90% line coverage. We evaluate five frontier LLMs under five generation strategies. The best model achieves only 45.6% class-level Pass@1, with a 17.7-point gap between the strongest and weakest models, confirming the
benchmark's discriminative power. Strategy choice strongly interacts with model capability: structured approaches such as bottom-up improve weaker models by up to 9.4 percentage points, while compositional generation collapses to as low as 1.3%. Error analysis over 500 manually annotated failures reveals that logic errors (56.2%) and dependency errors (38.0%) dominate, identifying cross-method coordination as the core bottleneck.
Read the full abstract ▾
All authors Yeheng Chen, Chaoxiang Xie, Yuling Shi, Wenhao Zeng, Yongpan Wang, Hongyu Zhang, Xiaodong Gu
Computation and Language
ClawGym: A Scalable Framework for Building Effective Claw Agents
Fei Bai, Huatong Song, Shuang Sun et al. (+11 more)
Claw-style environments support multi-step workflows over local files, tools, and persistent workspace states. However, scalable development around these environments remains constrained by the absence of a systematic framework, especially one for synthesizing verifiable training data and integrating it with
agent training and diagnostic evaluation. To address this challenge, we present ClawGym, a scalable framework that supports the full lifecycle of Claw-style personal
agent development. Concretely, we construct ClawGym-SynData, a diverse
dataset of 13.5K filtered tasks synthesized from persona-driven intents and skill-grounded operations, paired with realistic mock workspaces and hybrid verification mechanisms. We then train a family of capable Claw-style models, termed ClawGym-
Agents, through supervised fine-tuning on black-box rollout trajectories, and further explore reinforcement learning via a lightweight pipeline that parallelizes rollouts across per-task sandboxes. To support reliable evaluation, we further construct ClawGym-Bench, a
benchmark of 200 instances calibrated through automated filtering and human-LLM review. Relevant resources have been released at https://github.com/ClawGym.
Read the full abstract ▾
All authors Fei Bai, Huatong Song, Shuang Sun, Daixuan Cheng, Yike Yang, Chuan Hao, Renyuan Li, Feng Chang, Yuan Wei, Ran Tao, Bryan Dai, Jian Yang, Wayne Xin Zhao, Ji-Rong Wen
Computation and Language
HealthNLP_Retrievers at ArchEHR-QA 2026: Cascaded LLM Pipeline for Grounded Clinical Question Answering
Md Biplob Hosen, Md Alomgeer Hussein, Md Akmol Masud et al. (+3 more)
Patient portals now give individuals direct access to their electronic health records (EHRs), yet access alone does not ensure patients understand or act on the complex clinical information contained in these records. The ArchEHR-QA 2026 shared task addresses this challenge by focusing on grounded question answering over EHRs, and this paper presents the system developed by the HealthNLP_Retrievers team for this task. The proposed approach uses a multi-stage cascaded pipeline powered by the Gemini 2.5 Pro large
language model to interpret patient-authored questions and retrieve relevant evidence from lengthy clinical notes. Our architecture comprises four integrated modules: (1) a few-shot query reformulation unit which summarizes verbose patient queries; (2) a heuristic-based evidence scorer which ranks clinical sentences to prioritize recall; (3) a grounded response generator which synthesizes professional-caliber answers restricted strictly to identified evidence; and (4) a high-precision many-to-many
alignment framework which links generated answers to supporting clinical sentences. This cascaded approach achieved competitive results. Across the individual tracks, the system ranked 1st in question interpretation, 5th in answer generation, 7th in evidence identification, and 9th in answer-evidence
alignment. These results show that integrating large
language models within a structured multi-stage pipeline improves grounding, precision, and the professional quality of patient-oriented health communication. To support reproducibility, our source code is publicly available in our GitHub repository
Read the full abstract ▾
All authors Md Biplob Hosen, Md Alomgeer Hussein, Md Akmol Masud, Omar Faruque, Tera L Reynolds, Lujie Karen Chen
Computation and Language
MoRFI: Monotonic Sparse Autoencoder Feature Identification
Dimitris Dimakopoulos, Shay B. Cohen and Ioannis Konstas
Large
language models (LLMs) acquire most of their factual knowledge during the pre-training stage, through next
token prediction. Subsequent stages of post-training often introduce new facts outwith the parametric knowledge, giving rise to
hallucinations. While it has been demonstrated that supervised fine-tuning (SFT) on new knowledge may exacerbate the problem, the underlying mechanisms are still poorly understood. We conduct a controlled fine-tuning experiment, focusing on closed-book QA, and identify latent directions causally implicated in this degradation. Specifically, we fine-tune Llama 3.1 8B, Gemma 2 9B and Mistral 7B v03 on seven controlled mixtures of a single QA
dataset, controlling for the percentage of new knowledge and number of training epochs. By measuring performance on the test set, we validate that incrementally introducing new knowledge increases
hallucinations, with the effect being more pronounced with prolonged training. We leverage pre-trained sparse autoencoders (SAEs) to analyze residual stream activations across various checkpoints for each model and propose Monotonic Relationship Feature Identification (MoRFI) for capturing causally relevant latents. MoRFI filters SAE features that respond monotonically to controlled fine-tuning data mixtures of a target property. Our findings are consistent with exposure to unknown facts disrupting the model's ability to retrieve stored knowledge along a set of directions in the residual stream. Our pipeline reliably discovers them across distinct models, partially recovering lost knowledge through single-latent interventions.
Read the full abstract ▾
Computation and Language
What Kind of Language is Easy to Language-Model Under Curriculum Learning?
Nadine El-Naggar, Tatsuki Kuribayashi and Ted Briscoe
Many of the thousands of attested
languages share common configurations of features, creating a spectrum from typologically very rare (e.g., object-verb-subject word order) or impossible
languages to very common combinations of features (e.g., subject-object-verb word order). One central question is under what conditions such typological tendencies can be predicted, and specifically whether the learning bias of
language models (LMs) is sufficient to reproduce such patterns. In this study, we add one dimensionality to such analysis – the learning scenario for LMs – to explore its interaction with the inductive bias of LMs. Specifically, as a first study, we examine the effect of curriculum learning (CL), as a developmentally motivated learning scenario, i.e., starting with simpler sentences rather than randomly-ordered input. We expand existing LM-based exploration (El-Naggar et al., 2025a,b) with a simple CL variant and find that CL substantially impacts the apparent inductive bias of LMs.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data
Bao Pham, Mohammed J. Zaki, Luca Ambrogioni et al. (+2 more)
When do
language diffusion models memorize their training data, and how to quantitatively assess their true generative regime? We address these questions by showing that Uniform-based Discrete
Diffusion Models (UDDMs) fundamentally behave as Associative
Memories (AMs) $\textit{with emergent creative capabilities}$. The core idea of an AM is to reliably recover stored data points as $\textit{memories}$ by establishing distinct basins of attraction around them. Historically, models like Hopfield networks use an explicit energy function to guarantee these stable attractors. We broaden this perspective by leveraging the observation that energy is not strictly necessary, as basins of attraction can also be formed via conditional likelihood maximization. By evaluating
token recovery of $\textit{training}$ and $\textit{test}$ examples, we identify in UDDMs a sharp memorization-to-generalization transition governed by the size of the training
dataset: as it increases, basins around training examples shrink and basins around unseen test examples expand, until both later converge to the same level. Crucially, we can detect this transition using only the conditional
entropy of predicted
token sequences: memorization is characterized by vanishing conditional
entropy, while in the generalization regime the conditional
entropy of most
tokens remains finite. Thus, conditional
entropy offers a practical probe for the memorization-to-generalization transition in deployed models.
Read the full abstract ▾
All authors Bao Pham, Mohammed J. Zaki, Luca Ambrogioni, Dmitry Krotov, Matteo Negri
Computation and Language
HalluCiteChecker: A Lightweight Toolkit for Hallucinated Citation Detection and Verification in the Era of AI Scientists
Yusuke Sakai, Hidetaka Kamigaito and Taro Watanabe
We introduce HalluCiteChecker, a toolkit for detecting and verifying hallucinated citations in scientific papers. While AI assistant technologies have transformed the academic writing process, including citation recommendation, they have also led to the emergence of hallucinated citations that do not correspond to any existing work. Such citations not only undermine the credibility of scientific papers but also impose an additional burden on reviewers and authors, who must manually verify their validity during the review process. In this study, we formalize hallucinated citation detection as an NLP task and provide a corresponding toolkit as a practical foundation for addressing this problem. Our package is lightweight and can perform verification in seconds on a standard laptop. It can also be executed entirely offline and runs efficiently using only CPUs. We hope that HalluCiteChecker will help reduce reviewer workload and support organizers by enabling systematic pre-review and publication checks. Our code is released under the Apache 2.0 license on GitHub and is distributed as an installable package via PyPI. A demonstration
video is available on YouTube.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding
Hayate Iso, Tiyasa Mitra, Sudipta Mondal et al. (+15 more)
RL post-training of frontier
language models is increasingly bottlenecked by autoregressive rollout generation, making rollout acceleration a central systems challenge. Many existing efficiency methods improve throughput by changing the rollout or optimization regime, for example, through off-policy execution, replay, or lower-precision generation. We study speculative decoding as a lossless acceleration primitive for RL rollouts that preserves the target model's output distribution. We implement speculative decoding in NeMo-RL with a vLLM backend, supporting both synchronous and asynchronous pipelines and enabling speculation during RL rollouts. This benefit is realizable across speculation mechanisms, such as pretrained MTP heads, small external draft models or even techniques such as Eagle3, which are traditionally applied after RL phase. This yields a deployment path for state-of-the-art speculative decoding inside RL training. In a
reasoning post-training workload at 8B scale under synchronous RL, speculative decoding improves rollout throughput by 1.8x. Using a high-fidelity performance simulator, we project that combining speculative decoding with asynchronous RL yields up to 2.5x end-to-end training speedup at 235B scale.
Read the full abstract ▾
All authors Hayate Iso, Tiyasa Mitra, Sudipta Mondal, Rasoul Shafipour, Venmugil Elango, Terry Kong, Yuki Huang, Seonjin Na, Izzy Putterman, Benjamin Chislett, Maor Ashkenazi, Joseph Guman, Gerald Shen, Tugrul Konuk, Ashwath Aithal, Ritika Borkar, Ran Zilberstein, Bita Rouhani
Computation and Language
Decoupling Knowledge and Task Subspaces for Composable Parametric Retrieval Augmented Generation
Weihang Su, Hanwen Zhang, Qingyao Ai et al. (+1 more)
Parametric
Retrieval-Augmented Generation (PRAG) encodes external documents into lightweight parameter modules that can be retrieved and merged at inference time, offering a promising alternative to in-context
retrieval augmentation. Despite its potential, many PRAG implementations train document adapters with task-supervised objectives, which may cause each adapter to encode both document-specific facts and reusable task-solving behavior. This
entanglement may make adapter composition less reliable: when multiple adapters are merged at inference time, their overlapping task behaviors can accumulate together with document-specific updates, potentially making the merged adapter less stable and less focused on the intended document knowledge. To examine this issue, we explore Orthogonal Subspace Decomposition (OSD), an adapter-training setup that separates reusable task behavior from document-specific knowledge adapters. Concretely, we first train a Task LoRA to capture reusable task behavior, and then train document LoRAs to encode document-specific knowledge in a orthogonal subspace. This setup provides a controlled way to examine how orthogonalizing task and document LoRA updates affects adapter composition in multi-document PRAG. Experiments across multiple knowledge-intensive tasks and model scales suggest that this orthogonalization strategy can improve compositional robustness in parametric RAG, especially when multiple document adapters are merged.
Read the full abstract ▾
All authors Weihang Su, Hanwen Zhang, Qingyao Ai, Yiqun Liu
Computation and Language
Domain-Adapted Small Language Models for Reliable Clinical Triage
Manar Aljohani, Brandon Ho, Kenneth McKinley et al. (+2 more)
Accurate and consistent Emergency Severity Index (ESI) assignment remains a persistent challenge in emergency departments, where highly variable free-text triage documentation contributes to mistriage and workflow inefficiencies. This study evaluates whether open-source small
language models (SLMs) can serve as reliable,
privacy-preserving decision-support tools for clinical triage. We systematically compared multiple SLMs across diverse prompting pipelines and found that clinical vignettes, concise summaries of triage narratives, yielded the most accurate predictions. The SLM, Qwen2.5-7B, demonstrated the strongest balance of accuracy, stability, and computational efficiency. Through large-scale domain adaptation using expert-curated and silver-standard pediatric triage data, fine-tuned Qwen2.5-7B models substantially reduced discordance and clinically significant errors, outperforming all baseline SLMs and advanced proprietary large
language models (LLMs, e.g., GPT-4o). These findings highlight the feasibility of institution-specific SLMs for reliable,
privacy-preserving ESI decision support and underscore the importance of targeted fine-tuning over more complex inference strategies.
Read the full abstract ▾
All authors Manar Aljohani, Brandon Ho, Kenneth McKinley, Dennis Ren, Xuan Wang
Computation and Language
Swap distance minimization shapes the order of subject, object and verb in languages of the world
Jairo Rios-El-Yazidi and Ramon Ferrer-i-Cancho
Languages of the world vary concerning the order of subject (S), object (O) and verb (V). The most frequent dominant orders are SOV and SVO, and researchers have tailored models to this fact. However, there are still
languages whose dominant order does not conform to these expectations or even lack a dominant order. Here we approach word order variation through swap distance, namely the distance between orders in the permutohedron, a graph where the vertices are the six possible orders of S, O and V, and two orders are connected if they differ by a swap of adjacent constituents. Then word order variation becomes a problem of assigning probabilities to word orders so as to minimize the average swap distance. Although previous work suggests that swap distance minimization may influence synchronic and diachronic variation as well as word order acceptability, here we provide large-scale cross-linguistic evidence across families, macroareas and sources that word order variation within
languages is shaped by the principle of swap distance minimization, even when the dominant order is not SOV/SVO and even when a dominant order is lacking.
Read the full abstract ▾
Computation and Language
From Black-Box Confidence to Measurable Trust in Clinical AI: A Framework for Evidence, Supervision, and Staged Autonomy
Serhii Zabolotnii, Viktoriia Holinko and Olha Antonenko
Trust in clinical artificial intelligence (AI) cannot be reduced to model accuracy, fluency of generation, or overall positive user impression. In medicine, trust must be engineered as a measurable system property grounded in evidence, supervision, and operational boundaries of AI autonomy. This article proposes a practical framework for trustworthy clinical AI built around three principles: evidence, supervision, and staged autonomy. Rather than replacing deterministic clinical logic wholesale with end-to-end black-box models, the proposed approach combines a deterministic core, a patient-specific AI assistant for contextual validation, a multi-tier model escalation mechanism, and a human supervision layer for verification, escalation, and risk control. We demonstrate that trust also depends on selective verification of clinically critical findings, bounded clinical context, disciplined
prompt architecture, and careful evaluation on realistic cases. Classifier-driven modular prompting is examined as an incremental path to scaling clinical depth without sacrificing
prompt performance and without waiting for complete rule-based coverage. To operationalize trust, a set of trust metrics is proposed, built on metrological principles – measurement uncertainty, calibration, traceability – enabling quantitative rather than subjective assessment of each architectural layer. In this perspective, trustworthy clinical AI emerges not as a property of an individual model, but as an architectural outcome of a system into which evidence trails, human oversight, tiered escalation, and graduated action rights are embedded from the outset.
Read the full abstract ▾
Computation and Language
Differentially-Private Text Rewriting reshapes Linguistic Style
Stefan Arnold
Differential
Privacy (DP) for text matured from disjointed word-level substitutions to contiguous sentence-level rewriting by leveraging the generative capacity of
language models. While this form of text privatization is best suited for balancing formal
privacy guarantees with grammatical coherence, its impact on the register identity of text remains largely unexplored. By conducting a multidimensional stylistic profiling of differentially-private rewriting, we demonstrate that the cost of
privacy extends far beyond lexical variation. Specifically, we find that rewriting under
privacy constraints induces a systematic functional mutation of the text's communicative signature. This shift is characterized by the severe attrition of interactive markers, contextual references, and complex subordination. By comparing autoregressive paraphrasing against bidirectional substitution across a spectrum of
privacy budgets, we observe that both architectures force convergence toward a non-involved and non-persuasive register. This register-blind sanitization effectively preserves semantic content but structurally homogenizes the nuanced stylistic markers that define human-authored discourse.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
Dongxin Guo, Jikun Wu and Siu Ming Yiu
Large
reasoning models such as DeepSeek-R1 and OpenAI o1 generate extended chains of thought spanning thousands of
tokens, yet their integration with
retrieval-augmented generation (RAG) remains fundamentally misaligned. Current RAG systems optimize for providing context before
reasoning begins, while
reasoning models require evidence injection during multi-step inference chains. We introduce ReaLM-Retrieve, a
reasoning-aware
retrieval framework that addresses this mismatch through three key innovations: (1) a step-level uncertainty detector that identifies knowledge gaps at
reasoning-step granularity rather than
token or sentence level; (2) a
retrieval intervention policy that learns when external evidence maximally benefits ongoing
reasoning; and (3) an efficiency-optimized integration mechanism that reduces per-
retrieval overhead by 3.2x compared to naive integration. Experiments on MuSiQue, HotpotQA, and 2WikiMultiHopQA demonstrate that ReaLM-Retrieve achieves on average 10.1% absolute improvement in answer F1 over standard RAG (range: 9.0-11.8% across the three
benchmarks) while reducing
retrieval calls by 47% compared to fixed-interval approaches like IRCoT (all improvements significant at p<0.01, paired bootstrap). On the challenging MuSiQue
benchmark requiring 2-4 hop
reasoning, our method achieves 71.2% F1 with an average of only 1.8
retrieval calls per question. Analysis shows that ReaLM-Retrieve also improves
retrieval quality itself, achieving 81.3% Recall@5 with consistently higher precision and MRR than fixed-interval baselines on supporting evidence, establishing new state-of-the-art efficiency-accuracy trade-offs for
reasoning-intensive
retrieval tasks.
Read the full abstract ▾