Computation and Language
Lightweight Multimodal LLM-Enabled Cost-Effective Defect Grading of Power Transmission Equipment
Tao Wang, Lipeng Zhu, Jiayong Li et al. (+2 more)
Defect grading of power transmission equipment (DGPTE) is crucial to the stability of electric energy transmission. Although existing machine learning methods exhibit strong capabilities in defect detection, they are plagued by difficulties in integrating expert experience and facing class imbalance in more refined defect grading field. To address this issue, this paper introduces a novel defect grading framework based on
multimodal large
language model (MLLM). Specifically, this approach maximizes the commercial MLLMs' potential of DGPTE through in-context learning and obtains the state-of-te-art (SOTA) model. By sending a secondary request to this model, a small number of chain of thought-based question-answer pairs (Q&As) are generated, which effectively reduces the cost of manual annotation. In this way, these high-quality interpretable Q&As are used to train Qwen3-VL-8B via Low-Rank Adaption-based supervised fine-tuning (SFT). Experimental results on three DGPTE tasks demonstrate that fine-tuning only the
language model layer yields the SOTA performance. Furthermore, multi-task joint fine-tuning verifies the feasibility of handling multiple grading tasks within only a single lightweight MLLM.
Read the full abstract ▾
All authors Tao Wang, Lipeng Zhu, Jiayong Li, Feng Gao, Siwen Liang
Computation and Language · Cross-listed from cs.SI
WhatsApp Vaccine Discourse (WhaVax): An Expert-Annotated Dataset and Benchmark for Health Misinformation Detection
Jônatas H. dos Santos, Julio C. S. Reis, Philipe Melo et al. (+9 more)
We introduce WhaVax, a new expert-annotated
dataset of vaccine-related WhatsApp messages collected from large Brazilian public groups spanning multiple pandemic years. The
dataset was constructed through a rigorous, carefully designed pipeline that integrates keyword-based data collection, semantic deduplication to remove near-duplicate content, and a multi-stage annotation protocol conducted by medical specialists. This process produced a high-quality gold-standard corpus, characterized by substantial inter-annotator agreement and strong reliability for downstream analysis. Additionally, we provide a detailed characterization of WhatsApp misinformation, revealing distinctive linguistic, structural, lexical, temporal, and group-level patterns, as well as a meaningful layer of ambiguous cases that reflect the complexity of health discourse in private messaging. We also
benchmark classical models, fine-tuned Small
Language Models, and zero- or few-shot Large
Language Models under realistic data-scarcity constraints, demonstrating that strong
embeddings and LLM approaches perform competitively, while domain
alignment and data availability remain critical factors. This study provides a rare, high-quality resource to support misinformation research and computational modeling in encrypted communication environments.
Read the full abstract ▾
All authors Jônatas H. dos Santos, Julio C. S. Reis, Philipe Melo, João F. H. Olivetti, Thales H. Silva, Matheus Gontijo Guimaraes, Glaucio de Souza, Marcos A. Gonçalves, Fabricio Benevenuto, Filipe B. B. Zanovello, Marco A. G. Rodrigues, Cristiano X. Lima
Computation and Language · Cross-listed from cs.CY
Secure On-Premise Deployment of Open-Weights Large Language Models in Radiology: An Isolation-First Architecture with Prospective Pilot Evaluation
Sebastian Nowak, Jann-Frederick Laß, Narine Mesropyan et al. (+8 more)
Purpose: To design, implement, evaluate, and report on the regulatory requirements of a self-hosted LLM infrastructure for radiology adhering to the principle of least privilege, emphasizing technical feasibility, network isolation, and clinical utility. Materials and Methods: The isolation-first, containerized LLM inference stack relies on strict network
segmentation, host-enforced egress filtering, and active isolation monitoring preventing unauthorized external connectivity. An accompanying deployment package provides automated isolation and hardening tests. The system served the open-weights DeepSeek-R1 model via vLLM. In a one-week pilot phase, 22 residents and radiologists were free to use 10 predefined
prompt-templates whenever they considered them useful in daily work. Afterward, they rated clinical utility and system stability on an 0-10 Likert scale and reported observed critical errors in model output. Results: The applied institutional governance pathway achieved approval from clinic management, compliance, data protection and information security officers for processing unanonymized PHI. The system was rated stable and user friendly during the pilot. Source text-anchored tasks, such as report corrections or simplifications, and radiology guideline recommendations received the highest utility ratings, whereas open-ended conclusion generation based on findings resulted in the highest frequency of critical errors, such as clinically relevant
hallucinations or omissions. Conclusion: The proposed isolation-first on-premise architecture enabled overcoming regulatory borders, showed promising clinical utility in text-anchored tasks and is the current base to serve open-weights LLMs as an official service of a German University Hospital with over 10,000 employees. The deployment package were made publicly available (https://github.com/ukbonn/ukb-gpt).
Read the full abstract ▾
All authors Sebastian Nowak, Jann-Frederick Laß, Narine Mesropyan, Babak Salam, Nico Piel, Mohammed Bahaaeldin, Wolfgang Block, Alois Martin Sprinkart, Julian Alexander Luetkens, Benjamin Wulff, Alexander Isaak
Computation and Language · Cross-listed from cs.AI
Automated Detection of Dosing Errors in Clinical Trial Narratives: A Multi-Modal Feature Engineering Approach with LightGBM
Mohammad AL-Smadi
Clinical trials require strict adherence to medication protocols, yet dosing errors remain a persistent challenge affecting patient safety and trial integrity. We present an automated system for detecting dosing errors in unstructured clinical trial narratives using gradient boosting with comprehensive multi-modal feature engineering. Our approach combines 3,451 features spanning traditional NLP (TF-IDF, character n-grams), dense semantic
embeddings (all-MiniLM-L6v2), domain-specific medical patterns, and
transformer-based scores (BiomedBERT, DeBERTa-v3), used to train a LightGBM model. Features are extracted from nine complementary text fields (median 5,400 characters per sample) ensuring complete coverage across all 42,112 clinical trial narratives. On the CT-DEB
benchmark dataset with severe class imbalance (4.9% positive rate), we achieve 0.8725 test ROC-AUC through 5-fold ensemble averaging (cross-validation: 0.8833 + 0.0091 AUC). Systematic ablation studies reveal that removing sentence
embeddings causes the largest performance degradation (2.39%), demonstrating their critical role despite contributing only 37.07% of total feature importance. Feature efficiency analysis demonstrates that selecting the top 500-1000 features yields optimal performance (0.886-0.887 AUC), outperforming the full 3,451-feature set (0.879 AUC) through effective noise reduction. Our findings highlight the importance of feature selection as a regularization technique and demonstrate that sparse lexical features remain complementary to dense representations for specialized clinical text classification under severe class imbalance.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
ThermoQA: A Three-Tier Benchmark for Evaluating Thermodynamic Reasoning in Large Language Models
Kemal Düzkar
We present ThermoQA, a
benchmark of 293 open-ended engineering thermodynamics problems in three tiers: property lookups (110 Q), component analysis (101 Q), and full cycle analysis (82 Q). Ground truth is computed programmatically from CoolProp 7.2.0, covering water, R-134a, and variable-cp air. Six frontier LLMs are evaluated across three independent runs each. The composite leaderboard is led by Claude Opus 4.6 (94.1%), GPT-5.4 (93.1%), and Gemini 3.1 Pro (92.5%). Cross-tier degradation ranges from 2.8 pp (Opus) to 32.5 pp (MiniMax), confirming that property memorization does not imply thermodynamic
reasoning. Supercritical water, R-134a refrigerant, and combined-cycle gas turbine analysis serve as natural discriminators with 40-60 pp performance spreads. Multi-run sigma ranges from +/-0.1% to +/-2.5%, quantifying
reasoning consistency as a distinct evaluation axis.
Dataset and code are open-source at https://huggingface.co/
datasets/olivenet/thermoqa
Read the full abstract ▾
Computation and Language
LiFT: How to Enable In-Context Learning for Longitudinal Modelling
Iqra Ali, Talia Tseriotou, Mahmud Elahi Akhter et al. (+2 more)
Longitudinal NLP tasks such as mental health monitoring and stance
evolution require modeling temporally ordered text to track persistence and detect change. Such tasks also suffer from data scarcity, often involving rare events and sparsely annotated data. Large
Language Models (LLMs) can learn from small amounts of data through in-context learning (ICL), which is particularly important in low-data resource scenarios. However, when it comes to tracking evolving interactions or identifying rare events or changes, LLMs fall short in both zero- and few-shot ICL. To address this limitation, we introduce LiFT, a model-agnostic longitudinal Instruction Fine-Tuning framework that is able to induce ICL behaviour in LLMs for longitudinal tasks. LiFT unifies diverse tasks through a shared instruction schema that leverages sequential dependencies, curriculum learning, and temporal conditioning. We evaluate LiFT on five longitudinal
datasets under zero-, one-, and three-shot settings, demonstrating that LiFT delivers superior performance to IFT models under identical ICL conditions. Furthermore, we demonstrate that LiFT yields larger gains for minority classes, meaningfully attends to recent demonstration history, and develops transferable longitudinal modeling capabilities beyond its training tasks
Read the full abstract ▾
All authors Iqra Ali, Talia Tseriotou, Mahmud Elahi Akhter, Yuxiang Zhou, Maria Liakata
Computation and Language
Data Mixing for Large Language Models Pretraining: A Survey and Outlook
Zhuo Chen, Yuxuan Miao, Supryadi et al. (+1 more)
Large
language models (LLMs) rely on pretraining on massive and heterogeneous corpora, where training data composition has a decisive impact on training efficiency and downstream generalization under realistic compute and data budget constraints. Unlike sample-level data selection, data mixing optimizes domain-level sampling weights to allocate limited budgets more effectively. In recent years, a growing body of work has proposed principled data mixing methods for LLM pretraining; however, the literature remains fragmented and lacks a dedicated, systematic survey. This paper provides a comprehensive review of data mixing for LLM pretraining. We first formalize data mixture optimization as a bilevel problem on the probability simplex and clarify the role of data mixing in the pretraining pipeline, and briefly explain how existing methods make this formulation tractable in practice. We then introduce a fine-grained taxonomy that organizes existing methods along two dimensions: static versus dynamic mixing. Static mixing is further categorized into rule-based and learning-based methods, while dynamic mixing is grouped into adaptive and externally guided families. For each class, we summarize representative approaches and analyze their strengths and limitations from a performance-cost trade-off perspective. Building on this analysis, we highlight challenges that cut across methods, including limited transferability across data domains, optimization objectives, models, and validation sets, as well as unstandardized evaluation protocols and
benchmarks, and the inherent tension between performance gains and cost control in learning-based methods. Finally, we outline several exploratory directions, including finer-grained domain partitioning and inverse data mixing, as well as pipeline-aware designs, aiming to provide conceptual and methodological insights for future research.
Read the full abstract ▾
All authors Zhuo Chen, Yuxuan Miao, Supryadi, Deyi Xiong
Computation and Language · Cross-listed from cs.IR
LLMAR: A Tuning-Free Recommendation Framework for Sparse and Text-Rich Industrial Domains
Ryogo Hishikawa, Ichiro Kataoka and Shinya Yuda
Industrial B2B applications (e.g., construction site risk prediction, material procurement) face extreme data sparsity yet feature rich textual interactions. In such environments, traditional ID-based collaborative filtering fails lacking co-occurrence signals, while fine-tuning standard Large
Language Models (LLMs) incurs high operational costs and struggles with frequent data drift. We propose LLMAR (LLM-Annotated Recommendation), a tuning-free framework. Moving beyond simple
embeddings, LLMAR systematically integrates LLM
reasoning to capture user "latent motives" without any training process. We introduce three core contributions: (1) Inference-Driven Annotation: uses LLMs to transform behavioral history into structured semantic motives, enabling
reasoning-based matching unattainable by ID-based methods; (2) Reflection Loop: a self-correction mechanism that refines generated queries to mitigate
hallucinations and resolve "context competition" between past history and current instructions; and (3) Cost-Effective Architecture: relies on tuning-free components and asynchronous batch processing to minimize maintenance costs. Evaluations on public
benchmarks (MovieLens-1M, Amazon
Prime Pantry) and a sparse industrial
dataset (construction risk prediction) demonstrate that LLMAR outperforms state-of-the-art learning-based models (SASRecF), achieving up to a 54.6% nDCG@10 improvement on the industrial
dataset. Inference costs remain highly practical (~$1 per 1,000 users). For B2B domains where strict real-time latency is not critical, combining LLM
reasoning with self-verification offers a superior alternative to training-based approaches across accuracy, explainability, and operational cost.
Read the full abstract ▾
Computation and Language
Correcting Suppressed Log-Probabilities in Language Models with Post-Transformer Adapters
Bryan Sanchez
Alignment-tuned
language models frequently suppress factual log-probabilities on politically sensitive topics despite retaining the knowledge in their hidden representations. We show that a 786K-parameter (approximately 0.02% of the base model) post-
transformer adapter, trained on frozen hidden states, corrects this suppression on 31 ideology-discriminating facts across Qwen3-4B, 8B, and 14B. The adapter memorizes all 15 training facts and generalizes to 11–39% of 16 held-out facts across 5 random splits per scale, with zero knowledge regressions via anchored training. Both gated (SwiGLU) and ungated (linear bottleneck) adapters achieve comparable results; neither consistently outperforms the other (Fisher exact p > 0.09 at all scales). On instruct models, the adapter corrects log-probability rankings. When applied at all
token positions during generation, the adapter produces incoherent output; however, when applied only at the current prediction position (last-position-only), the adapter produces coherent, less censored text. A logit-space adapter operating after
token projection fails to produce coherent generation at any application mode, suggesting hidden-state intervention is the correct level for generation correction. A previously undocumented silent gradient bug in Apple MLX explains all null results in earlier iterations of this work: the standard pattern nn.value_and_grad(model, fn)(model.parameters()) returns zero gradients without error; the correct pattern nn.value_and_grad(model, fn)(model, data) resolves this. We provide a minimal reproduction and discuss implications for other adapter research using MLX.
Read the full abstract ▾
Computation and Language
Tug-of-War within A Decade: Conflict Resolution in Vulnerability Analysis via Teacher-Guided Retrieval-Augmented Generations
Ziyin Zhou, Jianyi Zhang, Xu ji et al. (+3 more)
Large
Language Models (LLMs) are essential for analyzing and addressing vulnerabilities in cybersecurity. However, among over 200,000 vulnerabilities were discovered in the past decade, more than 30,000 have been changed or updated. This necessitates frequent updates to the training
datasets and internal knowledge bases of LLMs to maintain knowledge consistency. In this paper, we focus on the problem of knowledge discrepancy and conflict within CVE (Common Vulnerabilities and Exposures) detection and analysis. This problem hinders LLMs' ability to retrieve the latest knowledge from original training
datasets, leading to knowledge conflicts, fabrications of factually incorrect results, and generation
hallucinations. To address this problem, we propose an innovative two-stage framework called CRVA-TGRAG (Conflict Resolution in Vulnerability Analysis via Teacher-Guided
Retrieval-Augmented Generation). First, to improve document
retrieval accuracy during the
retrieval stage, we utilize Parent Document
Segmentation and an ensemble
retrieval scheme based on semantic similarity and inverted indexing. Second, to enhance LLMs' capabilities based on the
retrieval of CVE
dataset in generation stage, we employ a teacher-guided preference optimization technique to fine-tune LLMs. Our framework not only enhances the quality of content
retrieval through RAG but also leverages the advantages of preference fine-tuning in LLMs to answer questions more effectively and precisely. Experiments demonstrate our method achieves higher accuracy in retrieving the latest CVEs compared to external knowledge bases. In conclusion, our framework significantly mitigates potential knowledge conflicts and inconsistencies that may arise from relying solely on LLMs for knowledge
retrieval.
Read the full abstract ▾
All authors Ziyin Zhou, Jianyi Zhang, Xu ji, Yilong Li, Jiameng Han, Zhangchi Zhao
Computation and Language
Benchmarking Linguistic Adaptation in Comparable-Sized LLMs: A Study of Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B on Romanized Nepali
Ananda Rimal and Adarsha Rimal
Romanized Nepali, the Nepali
language written in the Latin alphabet, is the dominant medium for informal digital communication in Nepal, yet it remains critically underresourced in the landscape of Large
Language Models (LLMs). This study presents a systematic benchmarking of linguistic adaptation across three comparable-sized open-weight models: Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B. We evaluate these architectures under zero-shot and fine-tuned settings using a curated bilingual
dataset of 10,000 transliterated instruction-following samples. Performance is quantified across five metrics spanning seven measurement dimensions: Perplexity (PPL), BERTScore, chrF++, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU, capturing fluency, phonetic consistency, and semantic integrity. Models were fine-tuned using Quantized Low-Rank Adaptation (QLoRA) with Rank-Stabilized LoRA (rsLoRA) at rank r=32 on dual NVIDIA Tesla T4 GPUs, training only approximately 1% of each model's parameters in under 27 total GPU-hours. At zero-shot, all three models fail to generate Romanized Nepali, each exhibiting a distinct architecture-specific failure mode. Following fine-tuning, all three resolve these failures and converge to BERTScore approximately 0.75 and chrF++ greater than 23. Overall dimension-wise assessment across ten criteria identifies Qwen3-8B as the overall recommended architecture, being the only model to produce semantically relevant zero-shot output and leading all structural
alignment metrics post-SFT. The adaptation headroom hypothesis is confirmed: Llama-3.1-8B, despite its weakest zero-shot baseline, achieves the largest absolute fine-tuning gains in PPL (Delta = -49.77) and BERTScore (Delta = +0.3287), making it the preferred choice for iterative low-resource development pipelines. This work establishes the first rigorous baseline for Romanized Nepali adaptation in comparable-sized open-weight LLMs.
Read the full abstract ▾
Computation and Language
Stateful Evidence-Driven Retrieval-Augmented Generation with Iterative Reasoning
Qi Dong, Ziheng Lin and Ning Ding
Retrieval-Augmented Generation (RAG) grounds Large
Language Models (LLMs) in external knowledge but often suffers from flat context representations and stateless
retrieval, leading to unstable performance. We propose Stateful Evidence-Driven RAG with Iterative
Reasoning, a framework that models question answering as a progressive evidence accumulation process. Retrieved documents are converted into structured
reasoning units with explicit relevance and confidence signals and maintained in a persistent evidence pool capturing both supportive and non-supportive information. The framework performs evidence-driven deficiency analysis to identify gaps and conflicts and iteratively refines queries to guide subsequent
retrieval. This iterative
reasoning process enables stable evidence aggregation and improves robustness to noisy
retrieval. Experiments on multiple question answering
benchmarks demonstrate consistent improvements over standard RAG and multi-step baselines, while effectively accumulating high-quality evidence and maintaining stable performance under substantial
retrieval noise.
Read the full abstract ▾
Computation and Language
Chronological Knowledge Retrieval: A Retrieval-Augmented Generation Approach to Construction Project Documentation
Ioannis-Aris Kostis, Natalia Sanchiz, Steeve De Schryver et al. (+2 more)
In large-scale construction projects, the continuous
evolution of decisions generates extensive records, most often captured in meeting minutes. Since decisions may override previous ones, professionals often need to reconstruct the history of specific choices. Retrieving such information manually from raw archives is both labor-intensive and error-prone. From a user perspective, we address this challenge by enabling conversational access to the whole set of project meeting minutes. Professionals can pose natural-
language questions and receive answers that are both semantically relevant and explicitly time-annotated, allowing them to follow the chronology of decisions. From a technical perspective, our solution employs a
Retrieval-Augmented Generation (RAG) framework that integrates semantic search with large
language models to ensure accurate and context-aware responses. We demonstrate the approach using an anonymized, industry-sourced
dataset of meeting minutes from a completed construction project by a large company in Belgium. The
dataset is annotated and enriched with expert-defined queries to support systematic evaluation. Both the
dataset and the open-source implementation are made available to the community to foster further research on conversational access to time-annotated project documentation.
Read the full abstract ▾
All authors Ioannis-Aris Kostis, Natalia Sanchiz, Steeve De Schryver, François Denis, Pierre Schaus
Computation and Language
Generating High Quality Synthetic Data for Dutch Medical Conversations
Cecilia Kuan, Aditya Kamlesh Parikh and Henk van den Heuvel
Medical conversations offer insights into clinical communication often absent from Electronic Health Records. However, developing reliable clinical Natural
Language Processing (NLP) models is hampered by the scarcity of domain-specific
datasets, as clinical data are typically inaccessible due to
privacy and ethical constraints. To address these challenges, we present a pipeline for generating synthetic Dutch medical dialogues using a Dutch fine-tuned Large
Language Model, with real medical conversations serving as linguistic and structural reference. The generated dialogues were evaluated through quantitative metrics and qualitative review by native speakers and medical practitioners. Quantitative analysis revealed strong lexical variety and overly regular turn-taking, suggesting scripted rather than natural conversation flow. Qualitative review produced slightly below-average scores, with raters noting issues in domain specificity and natural expression. The limited correlation between quantitative and qualitative results highlights that numerical metrics alone cannot fully capture linguistic quality. Our findings demonstrate that generating synthetic Dutch medical dialogues is feasible but requires domain knowledge and carefully structured prompting to balance naturalness and structure in conversation. This work provides a foundation for expanding Dutch clinical NLP resources through ethically generated synthetic data.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.DL
Crystal: Characterizing Relative Impact of Scholarly Publications
Hannah Collison, Benjamin Van Durme and Daniel Khashabi
Assessing a cited paper's impact is typically done by analyzing its citation context in isolation within the citing paper. While this focuses on the most directly relevant text, it prevents relative comparisons across all the works a paper cites. We propose
Crystal, which instead jointly ranks all cited papers within a citing paper using large
language models (LLMs). To mitigate LLMs' positional bias, we rank each list three times in a randomized order and aggregate the impact labels through majority voting. This joint approach leverages the full citation context, rather than evaluating citations independently, to more reliably distinguish impactful references.
Crystal outperforms a prior state-of-the-art impact classifier by +9.5% accuracy and +8.3% F1 on a
dataset of human-annotated citations.
Crystal further gains efficiency through fewer LLM calls and outperforms prior baselines using an open-weight model, enabling scalable, cost-effective citation impact analysis. In a case study of ACL Test-of-Time award-winning papers, we find that
Crystal's impact characterizations align closely with long-term scientific recognition. We release
Crystal-Bank, a 46.8k-paper
dataset with rankings and impact labels, along with code.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoning
Luyu Yang, Yutong Dai, An Yan et al. (+3 more)
The physical world is not merely visual; it is governed by rigorous structural and procedural constraints. Yet, the evaluation of
vision-
language models (VLMs) remains heavily skewed toward perceptual realism, prioritizing the generation of visually plausible 3D layouts, shapes, and appearances. Current
benchmarks rarely test whether models grasp the step-by-step processes and physical dependencies required to actually build these artifacts, a capability essential for automating design-to-construction pipelines. To address this, we introduce DreamHouse, a novel
benchmark for physical generative
reasoning: the capacity to synthesize artifacts that concurrently satisfy
geometric, structural, constructability, and code-compliance constraints. We ground this
benchmark in residential timber-frame construction, a domain with fully codified engineering standards and objectively verifiable correctness. We curate over 26,000 structures spanning 13 architectural styles, ach verified to construction-document standards (LOD 350) and develop a deterministic 10-test structural validation framework. Unlike static
benchmarks that assess only final outputs, DreamHouse supports iterative agentic interaction. Models observe intermediate build states, generate construction actions, and receive structured environmental feedback, enabling a fine-grained evaluation of planning, structural
reasoning, and self-correction. Extensive experiments with state-of-the-art VLMs reveal substantial capability gaps that are largely invisible on existing leaderboards. These findings establish physical validity as a critical evaluation axis orthogonal to visual realism, highlighting physical generative
reasoning as a distinct and underdeveloped frontier in
multimodal intelligence. Available at https://luluyuyuyang.github.io/dreamhouse
Read the full abstract ▾
All authors Luyu Yang, Yutong Dai, An Yan, Viraj Prabhu, Ran Xu, Zeyuan Chen
Computation and Language · Cross-listed from cs.CR
AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspective
Zhenyi Wang and Siyu Luan
As machine learning (ML) systems expand in both scale and functionality, the security landscape has become increasingly complex, with a proliferation of attacks and defenses. However, existing studies largely treat these threats in isolation, lacking a coherent framework to expose their shared principles and interdependencies. This fragmented view hinders systematic understanding and limits the design of comprehensive defenses. Crucially, the two foundational assets of ML – data and models – are no longer independent; vulnerabilities in one directly compromise the other. The absence of a holistic framework leaves open questions about how these bidirectional risks propagate across the ML pipeline. To address this critical gap, we propose a unified closed-loop threat taxonomy that explicitly frames model-data interactions along four directional axes. Our framework offers a principled lens for analyzing and defending foundation models. The resulting four classes of security threats represent distinct but interrelated categories of attacks: (1) Data$\rightarrow$Data (D$\rightarrow$D): including data decryption attacks and watermark removal attacks; (2) Data$\rightarrow$Model (D$\rightarrow$M): including poisoning, harmful fine-tuning attacks, and jailbreak attacks; (3) Model$\rightarrow$Data (M$\rightarrow$D): including model inversion, membership inference attacks, and training data extraction attacks; (4) Model$\rightarrow$Model (M$\rightarrow$M): including model extraction attacks. Our unified framework elucidates the underlying connections among these security threats and establishes a foundation for developing scalable, transferable, and cross-modal security strategies, particularly within the landscape of foundation models.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models
Isha Puri, Mehul Damani, Idan Shenfeld et al. (+3 more)
Given a question, a
language model (LM) implicitly encodes a distribution over possible answers. In practice, post-training procedures for LMs often collapse this distribution onto a single dominant mode. While this is generally not a problem for
benchmark-style evaluations that assume one correct answer, many real-world tasks inherently involve multiple valid answers or irreducible uncertainty. Examples include medical diagnosis, ambiguous question answering, and settings with incomplete information. In these cases, we would like LMs to generate multiple plausible hypotheses, ideally with confidence estimates for each one, and without computationally intensive repeated sampling to generate non-modal answers. This paper describes a multi-answer reinforcement learning approach for training LMs to perform distributional
reasoning over multiple answers during inference. We modify the RL objective to enable models to explicitly generate multiple candidate answers in a single forward pass, internalizing aspects of inference-time search into the model's generative process. Across question-answering, medical diagnostic, and coding
benchmarks, we observe improved diversity, coverage, and set-level calibration scores compared to single answer trained baselines. Models trained with our approach require fewer
tokens to generate multiple answers than competing approaches. On coding tasks, they are also substantially more accurate. These results position multi-answer RL as a principled and compute-efficient alternative to inference-time scaling procedures such as best-of-k. Code and more information can be found at https://multi-answer-rl.github.io/.
Read the full abstract ▾
All authors Isha Puri, Mehul Damani, Idan Shenfeld, Marzyeh Ghassemi, Jacob Andreas, Yoon Kim
Computation and Language
Prune as You Generate: Online Rollout Pruning for Faster and Better RLVR
Haobo Xu, Sirui Chen, Ruizhong Qiu et al. (+5 more)
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the
reasoning capabilities of Large
Language Models (LLMs). However, methods such as GRPO and DAPO suffer from substantial computational cost, since they rely on sampling many rollouts for each
prompt. Moreover, in RLVR the relative advantage is often sparse: many samples become nearly all-correct or all-incorrect, yielding low within-group reward variance and thus weak learning signals. In this paper, we introduce arrol (Accelerating RLVR via online Rollout
Pruning), an online rollout
pruning method that prunes rollouts during generation while explicitly steering the surviving ones more correctness-balanced to enhance learning signals. Specifically, arrol trains a lightweight quality head on-the-fly to predict the success probability of partial rollouts and uses it to make early
pruning decisions. The learned quality head can further weigh candidates to improve inference accuracy during test-time scaling. To improve efficiency, we present a system design that prunes rollouts inside the inference engine and re-batches the remaining ones for log-probability computation and policy updates. Across GRPO and DAPO on Qwen-3 and LLaMA-3.2 models (1B-8B), arrol improves average accuracy by +2.30 to +2.99 while achieving up to 1.7x training speedup, and yielding up to +8.33 additional gains in average accuracy in test-time scaling. The code is available at https://github.com/Hsu1023/ARRoL.
Read the full abstract ▾
All authors Haobo Xu, Sirui Chen, Ruizhong Qiu, Yuchen Yan, Chen Luo, Monica Cheng, Jingrui He, Hanghang Tong
Computation and Language
Synthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining
Thales Sales Almeida, Rodrigo Nogueira and Hélio Pedrini
Synthetic data generation through document rewriting has emerged as a promising technique for improving
language model pretraining, yet most studies focus on English and do not systematically control for the quality of the source data being rewritten. We present a controlled study of how synthetic rewriting interacts with source data quality in the context of Portuguese continued pretraining. Starting from ClassiCC-PT, a Portuguese corpus annotated with STEM and Educational quality scores, we construct two 10B-
token subsets at different quality levels and rewrite each into four styles using a 7B instruction-tuned model, producing approximately 40B
tokens of synthetic data per condition. We train two English-centric base models (1.1B and 7B parameters) on each condition and evaluate on PoETa V2, a comprehensive 44-task Portuguese
benchmark. At the 7B scale, rewriting high-quality data yields a +3.4 NPM gain over the same data unmodified, while rewriting low-quality data provides only +0.5 NPM. At the 1.1B scale, this interaction is weaker, with unmodified low-quality data performing comparably to rewritten high-quality data. Our results demonstrate that synthetic rewriting acts primarily as a quality multiplier rather than a substitute for data curation, and that this effect is scale-dependent.
Read the full abstract ▾
Computation and Language
Enhancing Structured Meaning Representations with Aspect Classification
Claire Benét Post, Paul Bontempo, August Milliken et al. (+6 more)
To fully capture the meaning of a sentence, semantic representations should encode aspect, which describes the internal temporal structure of events. In graph-based meaning representation frameworks such as Uniform Meaning Representations (UMR), aspect lets one know how events unfold over time, including distinctions such as states, activities, and completed events. Despite its importance, aspect remains sparsely annotated across semantic meaning representation frameworks. This has, in turn, hindered not only current manual annotation, but also the development of automatic systems capable of predicting aspectual information. In this paper, we introduce a new
dataset of English sentences annotated with UMR aspect labels over Abstract Meaning Representation (AMR) graphs that lack the feature. We describe the annotation scheme and guidelines used to label eventive predicates according to the UMR aspect
lattice, as well as the annotation pipeline used to ensure consistency and quality across annotators through a multi-step adjudication process. To demonstrate the utility of our
dataset for future automation, we present baseline experiments using three modeling approaches. Our results establish initial
benchmarks for automatic UMR aspect prediction and provide a foundation for integrating aspect into semantic meaning representations more broadly.
Read the full abstract ▾
All authors Claire Benét Post, Paul Bontempo, August Milliken, Alvin Po-Chun Chen, Nicholas Derby, Saksham Khatwani, Sumeyye Nabieva, Karthik Sairam, Alexis Palmer
Computation and Language
Evaluating Fine-Tuned LLM Model For Medical Transcription With Small Low-Resource Languages Validated Dataset
Mohammed Nowshad Ruhani Chowdhury, Mohammed Nowaz Rabbani Chowdhury and Sakari Lukkarinen
Clinical documentation is a critical factor for patient safety, diagnosis, and continuity of care. The administrative burden of EHRs is a significant factor in physician burnout. This is a critical issue for low-resource
languages, including Finnish. This study aims to investigate the effectiveness of a domain-aligned natural
language processing (NLP); large
language model for medical transcription in Finnish by fine-tuning LLaMA 3.1-8B on a small validated corpus of simulated clinical conversations by students at Metropolia University of Applied Sciences. The fine-tuning process for medical transcription used a controlled preprocessing and optimization approach. The fine-tuning effectiveness was evaluated by sevenfold cross-validation. The evaluation metrics for fine-tuned LLaMA 3.1-8B were BLEU = 0.1214, ROUGE-L = 0.4982, and BERTScore F1 = 0.8230. The results showed a low n-gram overlap but a strong semantic similarity with reference transcripts. This study indicate that fine-tuning can be an effective approach for translation of medical discourse in spoken Finnish and support the feasibility of fine-tuning a
privacy-oriented domain-specific large
language model for clinical documentation in Finnish. Beside that provide directions for future work.
Read the full abstract ▾
Computation and Language
Fine-Tuning A Large Language Model for Systematic Review Screening
Kweku Yamoah, Noah Schroeder, Emmanuel Dorley et al. (+2 more)
Systematic reviews traditionally have taken considerable amounts of human time and energy to complete, in part due to the extensive number of titles and abstracts that must be reviewed for potential inclusion. Recently, researchers have begun to explore how to use large
language models (LLMs) to make this process more efficient. However, research to date has shown inconsistent results. We posit this is because prompting alone may not provide sufficient context for the model(s) to perform well. In this study, we fine-tune a small 1.2 billion parameter open-weight LLM specifically for study screening in the context of a systematic review in which humans rated more than 8500 titles and abstracts for potential inclusion. Our results showed strong performance improvements from the fine-tuned model, with the weighted F1 score improving 80.79% compared to the base model. When run on the full
dataset of 8,277 studies, the fine-tuned model had 86.40% agreement with the human coder, a 91.18% true positive rate, a 86.38% true negative rate, and perfect agreement across multiple inference runs. Taken together, our results show that there is promise for fine-tuning LLMs for title and abstract screening in large-scale systematic reviews.
Read the full abstract ▾
All authors Kweku Yamoah, Noah Schroeder, Emmanuel Dorley, Neha Rani, Caleb Schutz
Computation and Language · Cross-listed from cs.SE
SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
Gabriel Orlanski, Devjeet Roy, Alexander Yun et al. (+7 more)
Software development is iterative, yet agentic coding
benchmarks hide design issues through their single-shot setup. Recent iterative
benchmarks attempt to remedy this but heavily constrain an
agent's design decision space, making it impossible to faithfully measure how their decisions shape future extensions. We introduce SlopCodeBench, a
benchmark of 36 problems and 196 checkpoints where
agents repeatedly extend their own solutions. Unlike prior iterative
benchmarks, our evolving specifications demand architectural decisions but leave internal structure to the
agent. We measure two forms of degradation: structural erosion (concentrated complexity) and verbosity (redundant code). Evaluating 15 coding
agents across open and closed models, we find that no
agent fully solves any problem end-to-end, and the best
agent passes 14.8% of checkpoints. Quality degrades across checkpoints, with structural erosion rising in 77% of trajectories and verbosity in 75.5%. Compared to 473 open-source Python repositories,
agent code is 2.3x more verbose and 2.0x more eroded, and the human repositories degrade less often and by smaller margins across their git histories. Explicit quality guidance reduces initial verbosity and erosion by up to a third, without affecting degradation rates. SlopCodeBench provides the first measurement of code degradation under iterative extension, revealing that
agents pass checkpoints while producing code that erodes and bloats with each turn.
Read the full abstract ▾
All authors Gabriel Orlanski, Devjeet Roy, Alexander Yun, Changho Shin, Alex Gu, Albert Ge, Dyah Adila, Nicholas Roberts, Frederic Sala, Aws Albarghouthi
Computation and Language · Cross-listed from cs.LG
Training LLMs for Multi-Step Tool Orchestration with Constrained Data Synthesis and Graduated Rewards
Cheng Jiayang, Xin Liu, Zhihan Zhang et al. (+8 more)
Multi-step tool orchestration remains challenging for LLMs, as state-of-the-art models frequently fail on full sequence execution due to parameter errors. Training for these workflows faces two obstacles: the lack of environments supporting complex real-world API dependencies, and sparse binary rewards that provide no signal for partial correctness. We propose a reinforcement learning framework addressing both challenges. First, we construct a deterministic environment backed by a large-scale cache of real API responses, enabling constrained synthesis of valid multi-step traces with controllable complexity. Second, we introduce a graduated reward that decomposes correctness into atomic validity (call-level correctness at increasing granularity) and orchestration consistency (correct sequencing with dependency respect). On ComplexFuncBench, our approach substantially improves turn accuracy, with ablations confirming both reward components are essential. Cross-
benchmark evaluation on BFCL v4 shows that the learned orchestration skills transfer to entirely different API ecosystems (e.g., agentic web search and
memory management), yielding consistent gains while maintaining stable single-step performance. Code is available at https://github.com/horizon-rl/ToolOrchestrationReward
Read the full abstract ▾
All authors Cheng Jiayang, Xin Liu, Zhihan Zhang, Haoyang Wen, Zixuan Zhang, Qingyu Yin, Shiyang Li, Priyanka Nigam, Bing Yin, Chao Zhang, Yangqiu Song
Computation and Language
Demystifying When Pruning Works via Representation Hierarchies
Shwai He, Guoheng Sun, Haichao Zhang et al. (+2 more)
Network
pruning, which removes less important parameters or architectures, is often expected to improve efficiency while preserving performance. However, this expectation does not consistently hold across
language tasks: pruned models can perform well on non-generative tasks but frequently fail in generative settings. To understand this discrepancy, we analyze network
pruning from a representation-hierarchy perspective, decomposing the internal computation of
language models into three sequential spaces:
embedding (hidden representations), logit (pre-softmax outputs), and probability (post-softmax distributions). We find that representations in the
embedding and logit spaces are largely robust to
pruning-induced perturbations. However, the nonlinear transformation from logits to probabilities amplifies these deviations, which accumulate across time steps and lead to substantial degradation during generation. In contrast, the stability of the categorical-
token probability subspace, together with the robustness of the
embedding space, supports the effectiveness of
pruning for non-generative tasks such as
retrieval and multiple-choice selection. Our analysis disentangles the effects of
pruning across tasks and provides practical guidance for its application. Code is available at https://github.com/CASE-Lab-UMD/
Pruning-on-Representations
Read the full abstract ▾
All authors Shwai He, Guoheng Sun, Haichao Zhang, Yun Fu, Ang Li
Computation and Language
When Consistency Becomes Bias: Interviewer Effects in Semi-Structured Clinical Interviews
Hasindri Watawana, Sergio Burdisso, Diego A. Moreno-Galván et al. (+4 more)
Automatic depression detection from doctor-patient conversations has gained momentum thanks to the availability of public corpora and advances in
language modeling. However, interpretability remains limited: strong performance is often reported without revealing what drives predictions. We analyze three
datasets: ANDROIDS, DAIC-WOZ, E-DAIC and identify a systematic bias from interviewer
prompts in semi-structured interviews. Models trained on interviewer turns exploit fixed
prompts and positions to distinguish depressed from control subjects, often achieving high classification scores without using participant
language. Restricting models to participant utterances distributes decision evidence more broadly and reflects genuine linguistic cues. While semi-structured protocols ensure consistency, including interviewer
prompts inflates performance by leveraging script artifacts. Our results highlight a cross-
dataset, architecture-agnostic bias and emphasize the need for analyses that localize decision evidence by time and speaker to ensure models learn from participants'
language.
Read the full abstract ▾
All authors Hasindri Watawana, Sergio Burdisso, Diego A. Moreno-Galván, Fernando Sánchez-Vega, A. Pastor López-Monroy, Petr Motlicek, Esaú Villatoro-Tello
Computation and Language · Cross-listed from cs.SE
Comparing Developer and LLM Biases in Code Evaluation
Aditya Mittal, Ryan Shar, Zichu Wu et al. (+6 more)
As LLMs are increasingly used as judges in code applications, they should be evaluated in realistic interactive settings that capture partial context and ambiguous intent. We present TRACE (Tool for Rubric Analysis in Code Evaluation), a framework that evaluates LLM judges' ability to predict human preferences and automatically extracts rubric items to reveal systematic biases in how humans and models weigh each item. Across three modalities – chat-based programming, IDE autocompletion, and instructed code editing – we use TRACE to measure how well LLM judges align with developer preferences. Among 13 different models, the best judges underperform human annotators by 12-23%. TRACE identifies 35 significant sources of misalignment between humans and judges across interaction modalities, the majority of which correspond to existing software engineering code quality criteria. For example, in chat-based coding, judges are biased towards longer code explanations while humans prefer shorter ones. We find significant misalignment on the majority of existing code quality dimensions, showing
alignment gaps between LLM judges and human preference in realistic coding applications.
Read the full abstract ▾
All authors Aditya Mittal, Ryan Shar, Zichu Wu, Shyam Agarwal, Tongshuang Wu, Chris Donahue, Ameet Talwalkar, Wayne Chi, Valerie Chen
Computation and Language
Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QA
Saahil Mathur, Ryan David Rittner, Vedant Ajit Thakur et al. (+2 more)
Retrieval-augmented generation (RAG) systems are increasingly used to analyze complex policy documents, but achieving sufficient reliability for expert usage remains challenging in domains characterized by dense legal
language and evolving, overlapping regulatory frameworks. We study the application of RAG to AI governance and policy analysis using the AI Governance and Regulatory Archive (AGORA) corpus, a curated collection of 947 AI policy documents. Our system combines a ColBERT-based retriever fine-tuned with contrastive learning and a generator aligned to human preferences using Direct Preference Optimization (DPO). We construct synthetic queries and collect pairwise preferences to adapt the system to the policy domain. Through experiments evaluating
retrieval quality, answer relevance, and faithfulness, we find that domain-specific fine-tuning improves
retrieval metrics but does not consistently improve end-to-end question answering performance. In some cases, stronger
retrieval counterintuitively leads to more confident
hallucinations when relevant documents are absent from the corpus. These results highlight a key concern for those building policy-focused RAG systems: improvements to individual components do not necessarily translate to more reliable answers. Our findings provide practical insights for designing grounded question-answering systems over dynamic regulatory corpora.
Read the full abstract ▾
All authors Saahil Mathur, Ryan David Rittner, Vedant Ajit Thakur, Daniel Stuart Schiff, Tunazzina Islam
Computation and Language
MARCH: Multi-Agent Reinforced Self-Check for LLM Hallucination
Zhuo Li, Yupeng Zhang, Pengyu Cheng et al. (+8 more)
Hallucination remains a critical bottleneck for large
language models (LLMs), undermining their reliability in real-world applications, especially in
Retrieval-Augmented Generation (RAG) systems. While existing
hallucination detection methods employ LLM-as-a-judge to verify LLM outputs against retrieved evidence, they suffer from inherent confirmation bias, where the verifier inadvertently reproduces the errors of the original generation. To address this, we introduce Multi-
Agent Reinforced Self-Check for
Hallucination (MARCH), a framework that enforces rigorous factual
alignment by leveraging deliberate information asymmetry. MARCH orchestrates a collaborative pipeline of three specialized
agents: a Solver, a Proposer, and a Checker. The Solver generates an initial RAG response, which the Proposer decomposes into claim-level verifiable atomic propositions. Crucially, the Checker validates these propositions against retrieved evidence in isolation, deprived of the Solver's original output. This well-crafted information asymmetry scheme breaks the cycle of self-confirmation bias. By training this pipeline with multi-
agent reinforcement learning (MARL), we enable the
agents to co-evolve and optimize factual adherence. Extensive experiments across
hallucination benchmarks demonstrate that MARCH substantially reduces
hallucination rates. Notably, an 8B-parameter LLM equipped with MARCH achieves performance competitive with powerful closed-source models. MARCH paves a scalable path for factual self-improvement of LLMs through co-
evolution. The code is at https://github.com/Qwen-Applications/MARCH.
Read the full abstract ▾
All authors Zhuo Li, Yupeng Zhang, Pengyu Cheng, Jiajun Song, Mengyu Zhou, Hao Li, Shujie Hu, Yu Qin, Erchao Zhao, Xiaoxi Jiang, Guanjun Jiang