Computation and Language · Cross-listed from cs.LG
Pre-AF 13: An Interpretable Atrial Fibrillation Risk Score Mined from Discharge Reports
Olga Shakhmatova, Dmitrii Kriukov, Daniil Larionov et al. (+10 more)
Background. Atrial fibrillation (AF) is the most prevalent cardiac arrhythmia and a major determinant of prognosis. Established AF risk scores rely on factors (older age, hypertension) nearly ubiquitous among patients with cardiovascular disease (CVD), offering limited stratification in this high-risk group. Most target long-term (5-10 year) rather than medium-term prediction. We developed interpretable ML models predicting AF risk over a 24-month and entire follow-up horizon in CVD patients using routinely collected hospital data. Methods. Single-center retrospective study of electronic health records from the National Research Cardiology Center (Russia) for patients aged >=18 with CVD but without pre-existing AF, hospitalized more than once between January 2012 and May 2019. A custom NLP pipeline transformed unstructured discharge reports into 73 structured features, combining a rule-based parser with
transformer-based NER. Using LightAutoML we built a full model (73 features), a simple model (reduced subset), and a linear model for a bedside risk score. Performance was assessed by ROC AUC, compared with CHARGE-AF, C2HEST, MHS, and HAVOC, and interpreted via SHAP. Results. Of 80,576 records from 45,000 patients, 17,562 met inclusion criteria; 1,438 (8.19%) developed AF. The full model reached ROC AUC 0.735 (24-month) and 0.696 (entire follow-up); the simple model was nearly identical (0.725, 0.696). All non-linear models outperformed the four clinical risk scores (ROC AUC 0.53-0.64). The simple model uses 13 features and is named Pre-AF 13. SHAP identified age and left atrial volume as dominant predictors. A linear risk score (Pre-AF 9) stratified observed 24-month AF incidence from ~7% to 36%. Conclusion. Interpretable ML models built from routinely collected EHR data identify high-AF-risk CVD patients, outperforming established clinical risk scores.
Read the full abstract ▾
All authors Olga Shakhmatova, Dmitrii Kriukov, Daniil Larionov, Nikita Khromov, Iaroslav Bespalov, Alexander Zolotarev, Kirill Grishchenkov, Ekaterina Ivanova, Miron Kuznetsov, Ilya Sochenkov, Elizaveta Panchenko, Artem Shelmanov, Dmitry V. Dylov
Computation and Language
Continual LLM Upcycling: A Predictor-Gated Bank-Wise Sparsity Training Recipe for Dense-to-Sparse LLMs
Ruixuan Huang, Jinyuan Shi, Hantao Huang et al. (+5 more)
We study dense-to-sparse continual training as a way to construct channel-sparse large
language models from dense checkpoints. Starting from a Qwen2.5-8B dense backbone, we continue training at 32K context and introduce a predictor-gated sparse SwiGLU FFN in the 32K stage. For each
token and layer, we use a low-rank predictor to produce FFN-channel routing logits. We then apply a bank-wise top-k rule to retain 16 channels in every 64-channel bank, yielding 4x sparsity in the FFN intermediate activation. Unlike post-hoc sparse inference methods, the routing module is placed on the main
language modeling path and optimized during continual training, enabling the dense model to be upcycled into a hardware-oriented sparse model. We report the architecture, training recipe,
benchmark performance, and training lessons. We also identify a layer-local long-context failure mode on RULER-CWE and propose a single-layer repair
algorithm that substantially improves the affected length range.
Read the full abstract ▾
All authors Ruixuan Huang, Jinyuan Shi, Hantao Huang, Yifan Huang, Ziyi Guan, Hao Zeng, Ian En-Hsu Yen, Minghui Yu
Computation and Language
Attention Expansion: Enhancing Keyphrase Extraction from Long Documents with Attention-Augmented Contextualized Embeddings
Roberto Martínez-Cruz, Alvaro J. López-López and José Portela
Pre-trained
language models (PLMs) have achieved strong performance in keyphrase extraction (KPE), largely due to their ability to generate rich contextualized representations. However, long-document KPE remains challenging because salient keyphrase evidence may be scattered across distant document sections that cannot be jointly captured within the limited context window of most PLMs. Although long-context large
language models (LLMs) can process broader textual contexts, their computational cost limits their practicality for efficient and high-throughput KPE. To overcome this limitation, we propose an
attention expansion mechanism that augments PLM
token representations with information from surrounding out-of-context chunks using pre-trained word
embeddings. The proposed mechanism expands the effective contextual scope of PLM-based KPE models without requiring full-document
attention or expensive LLM-based inference. We evaluate our approach across five PLM backbones, including general-purpose, scientific, task-specific, and long-context encoders, using two training regimes and five
benchmark corpora from scientific and news domains. Experimental results demonstrate that
attention expansion consistently enhances KPE performance across all evaluation settings, outperforming state-of-the-art models and yielding notable improvements in F1 score. The improvements extend to domain-specific, task-specialized, and native long-context models, showing that the proposed mechanism provides complementary information rather than merely compensating for limited input length. These results establish
attention expansion as an efficient and effective strategy for long-document KPE.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Models
Leonard Engmann, Christian Medeiros Adriano and Holger Giese
Interpretability methods routinely use population-level summary statistics over observed model behaviour to license claims about the effects of targeted interventions on specific computations; in Pearl's terms, they treat rung-1 associational evidence as if it supported rung-2 interventional conclusions, a move whose validity is rarely tested. We examine one concrete instance: the use of routing statistics in Mixture-of-Experts (MoE)
pruning, where utilization rates, activation norms, and routing weight distributions are treated as predictors of which experts can be removed without functional cost. A
token-level interventional audit across three high-redundancy MoE architectures (OLMoE-1B-7B-0924, Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite) finds no observational metric predicts causal expert importance in any model: across all 60 metric-layer combinations effect sizes stay below Cohen's $d = 0.23$, and no metric is reliably positive under our corrected, dual-test criterion. A per-
token routing weight control, run with identical $n$, rules out insufficient power, recovering a signal whose CI excludes zero at OLMoE's final MoE layer ($d = +0.231$, 95% CI $[+0.09, +0.37]$, $p = 0.0013$). Existing
pruning methods succeed in this regime not by identifying dispensable experts but because early-layer redundancy renders most selection criteria interchangeable. Our results provide an explicit counterexample to the common inferential step from population-level observational summaries to
token-level interventional claims about expert importance, and illustrate how interventional audits can calibrate the evidential standards for interpretability claims.
Read the full abstract ▾
Computation and Language
REAL: A Reasoning-Enhanced Graph Framework for Long-Term Memory Management of LLMs
Keer Lu, Liwei Chen, Guoqing Jiang et al. (+3 more)
Large
Language Models (LLMs) are increasingly expected to interact with users over long time horizons. However, due to their finite context window, LLMs cannot retain all past interactions, making long-term
memory management essential for storing, updating, and retrieving historical information beyond the context limit. Although recent
memory systems attempt to address this issue by storing historical information externally, existing approaches suffer from three key limitations: flat text-based
memory organizations fail to capture explicit relations among
memories, structured
memory systems often destructively overwrite evolving facts, and current
retrieval mechanisms remain query-agnostic and passive when evidence is incomplete. REAL constructs long-term conversational
memory as a temporal and confidence-aware directed property graph, where each atomic fact is represented with entities, relations, valid-time intervals, confidence scores, and exploration intent labels. During
memory construction, REAL adopts a non-destructive temporal update strategy that preserves parallel fact versions and their validity intervals, enabling faithful tracking of fact
evolution. During
retrieval, REAL anchors query-relevant root entities, decouples their exploration intents, and performs semantic evaluator-guided hybrid beam search to extract compact
memory subgraphs. It further incorporates counterfactual inference to repair unreliable
retrieval states and recover missing
memory evidence through implicit logical relations. Comprehensive experiments demonstrate that REAL substantially improves long-term
memory performance over flat-text, graph-based, and existing
memory baselines, achieving an average improvement of 22.72%.
Read the full abstract ▾
All authors Keer Lu, Liwei Chen, Guoqing Jiang, Zhiheng Qin, Yunhuai Liu, Wentao Zhang
Computation and Language · Cross-listed from cs.AI
Infini Memory: Maintainable Topic Documents for Long-Term LLM Agent Memory
Suozhao Ji, Baodong Wu, Zehao Wang et al. (+8 more)
Long-term LLM
agents need persistent
memory that can track changing facts and provide relevant evidence across sessions. Existing
memory systems often store observations as isolated records, summaries, or indexed fragments, which makes evidence aggregation, fact revision, and
memory maintenance difficult. We propose Infini
Memory, a maintainable text-based persistent
memory architecture that treats
agent memory as topic-structured documents. Each topic document serves as a semantic unit for collecting related evidence, preserving metadata, and revising facts over time. New observations are first staged in a buffer and periodically consolidated into coherent textual contexts. At inference time, an agentic
retrieval procedure lets the LLM read
memory through iterative tool calls rather than a single
retrieval step. On MemoryAgentBench, Infini
Memory achieves 64.7% overall score. Ablations show that topic-structured maintenance and iterative evidence inspection improve complementary aspects of long-term
memory use.
Read the full abstract ▾
All authors Suozhao Ji, Baodong Wu, Zehao Wang, Lei Xia, Qingping Li, Ruisong Wang, Wenbo Ding, Zhenhua Zhu, Boxun Li, Guohao Dai, Yu Wang
Computation and Language
Multilingual Word-Level Forced Alignment with Self-Supervised Representations and Learned Dynamic Programming
Roy Weber, Meidan Zehavi, Rotem Rousso et al. (+1 more)
We present a method for accurate
multilingual word-level forced
alignment, consisting of an
alignment encoder and a learned
alignment decoder. The encoder integrates two representations: one from the Massively
Multilingual Speech (MMS) model and another from a self-supervised phoneme boundary detector (UnSupSeg). It learns to fuse them and to estimate word-boundary probabilities over long temporal contexts. The
alignment decoder is a learned dynamic programming that combines encoder outputs with segmental features over the MMS and UnSupSeg representations to infer final word boundaries. Trained iteratively on TIMIT and Buckeye, the proposed approach outperforms Montreal Forced Aligner (MFA) and MMS-based
alignment on both
datasets. On unseen
languages (Dutch, German, and Hebrew), the proposed model achieves performance consistently better than or on par with existing
alignment approaches, indicating its potential to scale to 1100+
languages supported by MMS without further training.
Read the full abstract ▾
All authors Roy Weber, Meidan Zehavi, Rotem Rousso, Joseph Keshet
Computation and Language
Are We Evaluating Knowledge or Phrasing? Mitigating MCQA Sensitivity with ParaEval
João Maria Janeiro, Mathurin Videau, Andrea Caciolai et al. (+3 more)
Multiple-choice (MCQA)
benchmarks are the standard for evaluating pretrained large
language models, but their reliance on log-likelihood scoring makes them unreliable. Specifically, standard scores are highly sensitive to the exact phrasing (surface form) of the answers, conflating a model's familiarity with a specific phrase with its actual capability. We demonstrate this flaw using a controlled testbed of 1B-8B models trained on the same knowledge. Despite having identical knowledge, standard metrics falsely report a performance gap of over 2 points. To solve this, we propose ParaEval, an evaluation framework that queries models using multiple paraphrases per answer option. By scoring each model based on its most favorable phrasing, ParaEval successfully reduces the false performance gap to below 1 point. We confirm that these evaluation artifacts, and the improvements from ParaEval, persist in frontier 70B and 120B open-source models. Ultimately, ParaEval provides a robust and efficient way to evaluate true underlying capability rather than surface-form familiarity.
Read the full abstract ▾
All authors João Maria Janeiro, Mathurin Videau, Andrea Caciolai, Benjamin Piwowarski, Patrick Gallinari, Loic Barrault
Computation and Language
Speaker Group Encoding in Self-supervised Speech Recognition Models
Felix Herron, Solange Rossato Alexandre Allauzen, Benoit Favre et al. (+1 more)
We investigate what self-supervised
speech recognition models (S3Ms) learn about speaker groups (SGs). We examine several states of S3Ms: pretrained, finetuned on speaker identification (SID), finetuned on automatic
speech recognition (ASR), and ASR-finetuned using a
fairness enhancing
algorithm. We find that S3Ms encode information about several speaker group categories (SGCs), including their gender, age, dialect, ethnicity, and whether they are a native speaker. We find that finetuning for SID amplifies certain SGCs, namely those whose variance is more phonetic in nature, though it does not amplify other SGCs, namely those whose variance is more semantic in nature. On the other hand, finetuning for ASR discards phonetically variant speaker group information (SGI) but retains semantically variant SGI. We find that ASR
algorithms designed for
fairness improvement change to what extent SGI is encoded in S3Ms; however, this is primarily true for for phonetically variant SGCs, and less true for semantically variant SGCs. We discuss how SGI is encoded by each layer, and identify subdimensions of
embeddings responsible for encoding different SGCs. Finally, we discuss how our findings could be beneficial in designing fairer ASR
algorithms.
Read the full abstract ▾
All authors Felix Herron, Solange Rossato Alexandre Allauzen, Benoit Favre, François Portet
Computation and Language
Xin Wang, Hui Shen, Boyuan Zheng et al. (+7 more)
The scalability of Large
Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard
attention, motivating the adoption of linear
attention mechanisms with sub-quadratic cost. To improve representation capacity under long contexts, recent approaches organize
memory in a multi-state manner. However, existing multi-state linear
attention methods rely on fixed state merging policies that cannot adapt to dynamically varying
token importance, irreversibly obscuring critical
tokens and causing severe error accumulation over long sequences. To address this limitation, we propose DLA, a dynamic
memory modeling framework for multi-state linear
attention. DLA introduces (i) Information-Aware Dynamic State Merging, which adaptively determines state boundaries based on
token-level information variation, preserving high-resolution representations around semantic transitions while aggressively summarizing stable regions, and (ii) Capacity-Bounded
Memory Modeling, which maintains a fixed-size, chronologically ordered state cache by selectively merging adjacent low-information states to control
memory growth with minimal information loss. We pre-train DLA on two different linear
attention models and evaluate on 16
datasets across three categories. Experimental results demonstrate the superiority of DLA over state-of-the-art.
Read the full abstract ▾
All authors Xin Wang, Hui Shen, Boyuan Zheng, Xueshen Liu, Minkyoung Cho, Zhongwei Wan, Zesen Zhao, Zhuoqing Mao, Shen Yan, Mi Zhang
Computation and Language · Cross-listed from cs.LG
How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMs
Zhichen Dong, Yang Li, Yuhan Sun et al. (+9 more)
Token-level credit assignment remains a key obstacle for reinforcement learning (RL) in large
language models (LLMs), where RL recipes typically treat all
tokens equally, failing to distinguish decisive
reasoning steps from routine formatting or fluent filler. Recent attempts leverage model-internal signals to assign finer-grained credit, but these are often point-wise heuristics that ignore the global structure of information propagation. We propose FlowTracer, an RL framework that traces answer-targeted
reasoning flow on an
attention-induced directed acyclic graph in which nodes correspond to
tokens and edge capacities come from aggregated
attention weights and derives
token credit from this global structure. The edge capacities are reweighted to retain only the influence that can reach the answer region, while enforcing local flow conservation so intermediate
tokens neither lose nor gain effective mass due to path length or irrelevant branches. On this graph, FlowTracer extracts an information-flow backbone connecting the question to the answer and scores
tokens by flow throughput, revealing high-impact hubs and aggregation checkpoints that mediate long-range dependencies. These derived importances are used to shape
token-level rewards, enabling learning signals to focus precisely on the
tokens that route information toward (or away from) correct answers and delivering consistent performance gains across a range of
reasoning tasks.
Read the full abstract ▾
All authors Zhichen Dong, Yang Li, Yuhan Sun, Weixun Wang, Yijia Luo, Zinian Peng, Taiheng Ye, Chao Yang, Wenbo Su, Yu Cheng, Bo Zheng, Junchi Yan
Computation and Language
Small Data, Big Noise: Adversarial Training for Robust Parameter-Efficient Fine-Tuning
Eitan Cohen, Idan Simai and Uri Shaham
Parameter-Efficient Fine-Tuning (PEFT) has become essential for adapting foundation models to downstream NLP tasks. However, current PEFT methods often struggle with robustness to noise and performance degradation on limited training data. We propose SDBN (Small Data Big Noise), a unified framework that brings
adversarial training to PEFT - a combination that remains less studied in the PEFT setting despite its complementary strengths - to enhance model robustness and generalization, outperforming alternative approaches. We also introduce two variants of the method that use discrete uncertainty sets: SDBN-h, which enumerates character-level edits and selects worst-case variants using gradients, and SDBN-p, which uses LLM-generated variants for robust optimization in generative tasks. Experiments across multiple
benchmarks reveal substantial improvements, particularly in low-resource settings and under both word-level and character-level corruptions. This framework addresses the less explored intersection of
adversarial training and parameter-efficient adaptation, without introducing additional parameters or only modest computational overhead, making PEFT deployments more reliable in real-world scenarios where data scarcity and linguistic variability often coexist
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Causal Ensemble Agent: Hierarchical Causal Discovery with LLM-guided Expert Reweighting
Xinyu Li, Yuanyuan Wang, Haoxuan Li et al. (+7 more)
Causal discovery aims to uncover causal structures from observational data, which is crucial for real-world decision-making. However, different causal discovery
algorithms can produce divergent results that conflict with each other, complicating the identification of accurate causal graphs. Traditional approaches rely on numerical values and statistical assumptions, often ignoring rich domain-specific information, such as feature descriptions, which could also help structure learning. While recent works explore using Large
Language Models (LLMs) to infer causal relations via direct queries, such methods can be unreliable due to a lack of
alignment with the actual data. To address these limitations, we propose Causal Ensemble
Agent (CEA), a novel framework that aggregates structural insights from statistical discovery experts across different graph levels via linear opinion pooling, and uses an LLM as a meta-referee to dynamically reweight experts when the aggregated confidence is close to the decision boundary, thereby composing an improved and more complete causal graph. Extensive experiments on both synthetic and real-world
datasets demonstrate that CEA achieves the strongest overall performance across a wide range of causal discovery methods, highlighting the effectiveness of using LLMs for meta-analysis in causal discovery.
Read the full abstract ▾
All authors Xinyu Li, Yuanyuan Wang, Haoxuan Li, Chuan Zhou, Erdun Gao, Bo Han, Tongliang Liu, Kun Zhang, Howard Bondell, Mingming Gong
Computation and Language
ParaBridge: Bridging Paralinguistic Perception and Dialogue Behavior in Speech Language Models
Yuxiang Wang, Qinke Ni, Shengbo Cai et al. (+3 more)
Speech carries more information than just words: a child's voice, a fearful tone, or a noisy background should all lead a sufficiently competent spoken-dialogue assistant to different replies. Current
Speech Language Models (SLMs) can recognize such paralinguistic cues but often ignore them in open-ended dialogue. We observe that a simple paralinguistic instruction scaffold at the inference stage narrows this perception-behavior gap, suggesting that the relevant cues are already latent in the model. Such scaffolds, however, remain brittle under multi-turn context and competing instructions. Therefore, we propose ParaBridge, an on-policy self-
distillation method that turns a brittle inference-time scaffold into stable model behavior. During training, the scaffold serves only as a temporary privileged view; the scaffold-free model rolls out its own response, while the scaffolded view supplies dense, full-vocabulary next-
token targets along its trajectory. This supervision teaches when non-lexical cues should affect the reply without the need for curated dialogues, human labels, or external reward models. On Qwen3-Omni-thinking, ParaBridge raises scaffold-free VoxSafeBench SAR from $14.6\%$ to $40.3\%$ and improves EchoMind average rating from $3.27$ to $3.92$. It also preserves general ability, with MMAU-Pro, VoiceBench, and GPQA all within $0.4$ points of the original model. Beyond the training distribution, ParaBridge generalizes to unseen paralinguistic cues, transfers from safety-oriented training to empathy-oriented dialogue, and works on a different SLM backbone.
Read the full abstract ▾
All authors Yuxiang Wang, Qinke Ni, Shengbo Cai, Wan Lin, Liqiang Zhang, Zhizheng Wu
Computation and Language
Hidden Consensus:Preference-Validity Compression in Human Feedback
Dorcas Chia Ern Chua, Karen Myn Hui Lee, Jia Yue Tan et al. (+9 more)
Standard RLHF pipelines often reduce heterogeneous human judgments into a single scalar reward target. We argue that this reduction can mis-measure
alignment in structurally plural societies, where disagreement may reflect culturally, historically, linguistically, regionally, or normatively grounded interpretations rather than annotation noise. We call this failure Preference-Validity Compression, the collapse of multiple plural-valid response options into a single optimization target. Using Malaysia as a diagnostic setting, we analyze RLHF-style feedback aggregation through preference events linking
prompts, responses, and acceptability judgments across interpretive frames. Across 321 preference events from 20 participants and 107 trio-annotated
prompts, 79% of
prompts contain more than one majority-supported response that single-winner aggregation would discard, and apparent dominance gaps between top responses diminish when all majority-supported options are considered. Participants frequently select multiple acceptable responses, and discarded responses demonstrably reflect coherent local, practical, or cultural frames. These findings show that majority aggregation in this corpus measures argmax acceptability rather than plural
alignment. We treat this as a measurement-validity issue and argue that future
alignment methods should satisfy Validity-Preserving Consistency, remaining stable across plural-valid interpretive frames rather than collapsing them into a single reward target.
Read the full abstract ▾
All authors Dorcas Chia Ern Chua, Karen Myn Hui Lee, Jia Yue Tan, Zhen Xue Gue, Norzalena Abdul Hamid, Azima Binti Azmi, Keat Mei Yeong, Aizat Izyani binti Mujab, Hafsah Noor Azam, Chee Guo Khoo, Han Ying Lim, Chee Seng Chan
Computation and Language
Benchmarking Knowledge Editing using Logical Rules
Tatiana Moteu Ngoli, NDah Jean Kouagou, Hamada M. Zahera et al. (+1 more)
Large
Language Models (LLMs) are increasingly deployed in real-world applications that require access to up-to-date knowledge. However, retraining LLMs is computationally expensive. Therefore, knowledge editing techniques are crucial for maintaining current information and correcting erroneous assertions within pre-trained models. Current
benchmarks for knowledge editing primarily focus on recalling edited facts, often neglecting their logical consequences. To address this limitation, we introduce a new
benchmark designed to evaluate how knowledge editing methods handle the logical consequences of a single fact edit. Our
benchmark extracts relevant logical rules from a knowledge graph for a given edit. Then, it generates multi-hop questions based on these rules to assess the impact on logical consequences. Our findings indicate that while existing knowledge editing approaches can accurately insert direct assertions into LLMs, they frequently fail to inject entailed knowledge. Specifically, experiments with popular methods like ROME and FT reveal a substantial performance gap, up to 24%, between evaluations on directly edited knowledge and on entailed knowledge. This highlights the critical need for semantics-aware evaluation frameworks in knowledge editing.
Read the full abstract ▾
All authors Tatiana Moteu Ngoli, NDah Jean Kouagou, Hamada M. Zahera, Axel-Cyrille Ngonga Ngomo
Computation and Language
Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Models
Jing Xiong, Qi Han, Shansan Gong et al. (+6 more)
Diffusion large
language models (dLLMs) re-encode the entire prefix at every denoising step, causing recomputation that scales quadratically with context length and becomes prohibitive for long-context scenarios. We propose Prefilling-dLLM, a training-free prefill-decode disaggregation framework for dLLMs that partitions the prefix into N chunks, caches their KV representations once, and selects the top-K most relevant chunks with intra-chunk
token sparsity for decoding, showing that sparse prefilling can outperform dense
attention while reducing per-step complexity from quadratic in the full sequence length to quadratic only in the decode length. On LongBench and InfiniteBench, Prefilling-dLLM achieves state-of-the-art quality among dLLM acceleration methods, and an
attention kernel that parallelizes decoding over the non-contiguously cached chunk KV yields 9.1–28.0x speedup at 8K–32K contexts. We further show that beginning-of-sequence
tokens prepended to each chunk act as periodic
attention anchors that eliminate the lost-in-the-middle phenomenon. Code is available at https://github.com/menik1126/Prefilling-dLLM.
Read the full abstract ▾
All authors Jing Xiong, Qi Han, Shansan Gong, Yunta Hsieh, Boyuan Zheng, Chengyue Wu, Chaofan Tao, Chenyang Zhao, Ngai Wong
Computation and Language
LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization
Haoyu Wang, Xingyu Yu, Haiyan Zhao et al. (+2 more)
Quantization-aware training (QAT) is essential for extremely low-bit large
language models (LLMs). Current QAT methods are mainly based on scalar quantization (SQ), which enables efficient optimization but suffers from severe performance degradation at 2-bit precision. On the other hand, vector quantization (VQ) provides substantially higher representational capacity, but its discrete codebook lookup prevents end-to-end training. We propose LC-QAT, a 2-bit weight-only VQ-QAT framework that represents quantized weights via a learned affine mapping over discrete vectors, which yields a high-quality PTQ initialization and enables fully differentiable end-to-end optimization without explicit codebook lookup in the training forward pass. This strong post-training initialization makes LC-QAT highly data-efficient. Experiments across diverse LLMs demonstrate that LC-QAT consistently outperforms state-of-the-art QAT methods while using only 0.1%–10% of the training data. Our results establish LC-QAT as a practical and scalable solution for extreme low-bit model deployment. Codes are publicly available at https://github.com/AI9
Stars/UniSVQ.
Read the full abstract ▾
All authors Haoyu Wang, Xingyu Yu, Haiyan Zhao, Fengxiang Wang, Xu Han
Computation and Language · Cross-listed from cs.LG
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
Guozheng Li, Xiyan Fu and Yiwen Guo
Current reinforcement learning from human feedback (RLHF) methods primarily rely on scalar rewards from a trained reward model (RM). While effective, scalar rewards are often noisy and fail to capture fine-grained preference differences, whereas RM hidden states encode richer semantic and preference information. We introduce the representation-aware advantage estimation, which leverages RM hidden states and models them as auxiliary signals for better advantage estimation. Specifically, we propose the Graph-based Advantage Estimation (GraphAE), treat each sampled group as a graph, where nodes correspond to responses and edges capture their similarity in the RM hidden space. Then advantages are computed via graph propagation, enabling each sample to incorporate contextual information from its neighbors. GraphAE is lightweight and can be seamlessly integrated into existing group-based RL
algorithms. We apply GraphAE to GRPO, GSPO and RLOO, and conduct extensive experiments on different models and
benchmarks. Empirical results show consistent improvements across three
benchmarks, with gains of up to + 6.3 on Arena-Hard-v0.1, + 8.27 on AlpacaEval 2.0, and + 0.22 on MT-Bench. These results demonstrate that leveraging RM representations leads to more sample efficient and robust RLHF.
Read the full abstract ▾
Computation and Language
UniSVQ: 2-bit Unified Scalar-Vector Quantization
Haoyu Wang, Haiyan Zhao, Xingyu Yu et al. (+4 more)
Post-training quantization at the 2-bit level enables low-cost deployment and inference acceleration for large
language models (LLMs). Scalar quantization (SQ) and vector quantization (VQ) are two primary quantization methods, however, the former suffers from significant performance degradation, and the latter incurs computational and storage overhead. We propose UniSVQ, a unified 2-bit quantization framework that bridges scalar and vector quantization by parameterizing codewords as an affine transform of integer
lattices. This structure preserves compatibility with optimized integer kernels while retaining much of VQ's flexibility. We further introduce a data-driven block-wise fine-tuning strategy to directly minimize quantization reconstruction error. Extensive experiments across multiple LLM families and zero-shot
benchmarks demonstrate that UniSVQ consistently outperforms state-of-the-art SQ methods and achieves performance comparable to advanced VQ methods, while providing higher inference throughput. Codes are publicly available at https://github.com/AI9
Stars/UniSVQ.
Read the full abstract ▾
All authors Haoyu Wang, Haiyan Zhao, Xingyu Yu, Zhangyang Yao, Xu Han, Zhiyuan Liu, Maosong Sun
Computation and Language · Cross-listed from cs.LG
Advancing the State-of-the-Art in Empirical Privacy Auditing
Nicole Mitchell, Galen Andrew, Arun Ganesh et al. (+2 more)
Parameter-efficient fine-tuning of large
language models (LLMs) can exhibit problematic memorization of individual training examples. Empirical
privacy auditing (EPA) quantifies this risk by measuring realistic data leakage on membership inference (MI) or reconstruction attacks. A key challenge in EPA is designing “canary” examples that are mixed with the
privacy-sensitive training data. We propose generating synthetic canaries via high-temperature sampling ($T \geq 0.8$) from LLMs, using
prompts tailored to the
privacy-sensitive training data. These canaries act as high-influence outliers, ensuring high identifiability and hence strong audits. Further, since the canaries are themselves non-private, they are inspectable and can be inserted with repetition without jeopardizing the
privacy of the real data. An important use of models fine-tuned on
privacy-sensitive data is the generation of synthetic data. This also comes with
privacy risk. We introduce a powerful synthetic data audit based on fine-tuning an auxiliary model on the synthetic data.
Auditing the auxiliary model for the original canaries then provides a strong estimate of the
privacy leakage through the synthetic data. Finally, leveraging our strong
auditing methodologies, we perform a systematic investigation into the interacting effects of model capacity and canary
entropy on memorization.
Read the full abstract ▾
All authors Nicole Mitchell, Galen Andrew, Arun Ganesh, Brendan McMahan, Peter Kairouz
Computation and Language · Cross-listed from cs.MA
Decoupling Thought from Speech: Knowledge-Grounded Counterfactual Reasoning for Resilient Multi-Agent Argumentation
Jakub Masłowski and Jarosław A. Chudziak
Multi-
agent debate frameworks have been shown to improve large
language model performance in convergent tasks, but they are currently optimized in a way that heavily favors final output accuracy rather than stability of the process. During long-horizon exchanges reactive systems under sustained perturbations often experience logic degradation, argument repetition, and role drift. To structurally prevent the identity loss and maintain the process fidelity, we introduce Knowledge-Grounded Counterfactual
Reasoning (KG-CFR), a dual-stage architecture that enforces a strict separation of concerns between a private,
retrieval-augmented planning buffer, and a public execution layer. We assess this system in Dynamic Resource Allocation under Uncertainty (DRAU), a dedicated 1v1v1 environment, introducing diversity as distinct from standard debate settings. Over 270 completely factorial crisis simulation trajectories with
stochastic environmental shocks, KG-CFR prevents judge-detected critical post-shock degradation (defined as a quality shift, $\Delta \le -0.20$) in more than 95% of perturbed runs, increasing the overall argument quality from 0.694 to 0.822. Our primary contribution is the demonstration of architectural decoupling being an important factor of systemic resilience enhancement under sustained pressure without quality loss. Furthermore, we introduce custom vector metrics for discourse divergence and plan-execution
alignment that provide strong, directionally consistent evidence of operational stability. Our ablation experiments suggest that the proper doctrinal grounding can be an equally important factor for argument quality, as the prospective planning. KG-CFR, according to our initial metric evaluations, reduces semantic looping, by preserving the
agent's consistency with the original plan.
Read the full abstract ▾
Computation and Language
Detecting Speculative Language in Biomedical Texts using Recurrent Neural Tensor Networks
Dhruv Dixit
In this investigation, we delve into the automated detection of speculative
language within biomedical articles by utilizing distributed sentence representations and advanced deep learning techniques. The implications of such identification extend to information
retrieval, multi-document summarization, and the exploration of new knowledge. Our exploration encompasses two distinct approaches for acquiring distributed sentence representations: the Paragraph Vector model and the Recursive Neural Tensor Network. These methodologies are then rigorously compared against three foundational baseline
algorithms: Support Vector Machines, Naive Bayes, and pattern matching. Our findings reveal that the Recursive Neural Tensor Network (RNTN) demonstrates a slight performance edge (F1 = 0.885) over the top-performing baseline, the linear bigram SVM (F1 = 0.881). Meanwhile, the Paragraph Vector model proves less effective (F1 = 0.368), even after extensive training using an expansive, unlabeled
dataset. We engage in a comprehensive discourse on the factors influencing these performance disparities and provide insightful recommendations for future research directions.
Read the full abstract ▾
Computation and Language
Large Language Models as Modal Models in Linguistics
Haruto Suzuki and Saku Sugawara
The rapid advancement of large
language models (LLMs) has intensified debates about their significance for linguistic theory. These debates are commonly divided into three positions: insulationism, which regards LLMs as irrelevant to human
language; eliminativism, which claims that LLMs can replace traditional linguistic theories; and conciliationism, which views them as useful tools for linguistic research. To clarify these positions, this paper applies the framework of modal modeling from the philosophy of science. We argue that LLMs possess genuine epistemic value as minimal models, even without structural correspondence to human cognition. In particular, they can provide how-possibly explanations (HPEs) by testing modal claims about
language acquisition and linguistic competence. We then examine the conditions under which LLMs could qualify as how-actually explanations (HAEs) of human
language, drawing on the mechanistic account of scientific explanation. We argue that current LLMs do not yet satisfy these requirements. On the basis of this analysis, we propose understanding the explanatory power of LLMs as lying on a continuum between HPEs and HAEs. This framework avoids both overstating and understating their explanatory significance and offers a more precise basis for evaluating the role of LLMs in the scientific study of
language.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
ERAlign: Energy-based Representation Alignment of GNNs and LLMs on Text-attributed Graphs
Xianlin Zeng, Fan Xia and Xiangyu Chen
Text-attributed Graphs (TAGs) incorporate textual node attributes with graph structures to describe rich relational semantics. Recent efforts to integrate Graph Neural Networks (GNNs) and Large
Language Models (LLMs) have shown promise for learning on TAGs, yet achieving well-aligned representations remains challenging. Prior studies largely rely on heuristics that perform coarse-grained matching. They lack sufficient constraints and ignore distributional
alignment, leading to representation drift and limited generalization. Building on Energy-based Models (EBMs), we propose an Energy-based Representation
Alignment (ERAlign) framework that projects GNN-encoded graph structure and LLM-derived text
embeddings in a shared latent space to achieve distribution consistency. Concretely, layer-wise
alignment is quantified by a distance metric and optimized via an EBM objective. By decreasing energy values, our framework yields well-aligned representations for downstream tasks. During training, we introduce Energy Discrepancy (ED) to avoid high sampling costs associated with intractable normalization. ED also carries theoretical guarantees of higher training efficiency and reduced energy landscape distortion. Empirical evaluations on eight TAG
datasets demonstrate that ERAlign obtains state-of-the-art performance across varying levels of supervision and cross-task transfer scenarios.
Read the full abstract ▾
Computation and Language
LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake
Haonan Wang, Jiaxiang Liu, Yurong Liu et al. (+11 more)
Recent large
language models (LLMs) have shown rapid progress in reading-based question answering (QA), where evidence is explicitly provided or can be trivially retrieved. In contrast, real-world questions are often not paired with accurate evidence documents. The useful evidence resides in massive data lakes, making search a prerequisite for answering. However, there is a lack of comprehensive
benchmarks that require both searching and
reasoning over large data lakes. To this end, we introduce LakeQA, a comprehensive
benchmark for search-centric question answering over data lakes that jointly emphasizes searching and
reasoning capabilities. LakeQA is built on a heterogeneous collection of approximately 9.5 TB of text resources from Wikipedia and open-source government data, spanning structured and unstructured data. To ensure task quality, each sample is annotated by at least one Ph.D.-level expert. Each task requires long-horizon multi-hop
reasoning with implicit intermediate steps:
agents need to discover the correct documents and then compose evidence across sources to produce the answer. Experimental results on seven frontier LLMs demonstrate that LakeQA is challenging. For instance, GPT-5.2 achieves only an exact-match score of 18.37% on LakeQA. Overall, LakeQA provides a realistic testbed for developing LLM
agents that can both find and analyze data in modern data lakes.
Read the full abstract ▾
All authors Haonan Wang, Jiaxiang Liu, Yurong Liu, Austin Senna Wijaya, Tianle Zhou, Eden Wu, Yijia Chen, Wanting You, Reya Vir, Daniela Pinto, Grace Fan, Yusen Zhang, Juliana Freire, Eugene Wu
Computation and Language · Cross-listed from cs.SI
Leveraging Social Media Data for COVID-19 Studies
Nur Hafieza Ismail, Nur Shazwani Kamarudin and Nurol Husna Che Rose
Nowadays, social media networks have become widely preferred sources of information. Especially during the time of the Coronavirus disease 2019 COVID 19 pandemic, social media has been one of the most used platforms to get the latest news and information related to COVID 19. Social media are popular because they offer free access to their registered users and allow them to do posting, disseminate information, and respond to others postings. With almost 4.6 billion social media users worldwide, it is not surprising the significant amount of information shared through these platforms could affect how people perceive and cope with the pandemic that we are facing right now. With decent use, social media can be a beneficial digital tool to spread reliable news and public awareness for patients, clinicians, and society. Specifically, this chapter describes linguistic, visual, and emotional indicators expressed in user disclosures. Thus, in this chapter, the related studies of social media platforms usage during the COVID 19 pandemic are explored and discussed in detail. This chapter also categorizes social media data used, introduces different deployed machine learning, feature engineering, natural
language processing, and survey methods, and outlines directions for future research.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference
Jaeseong Lee, Seung-won Hwang and Samyam Rajbhandari
Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup. However, its strict 50% sparsity constraint often causes non-negligible accuracy degradation under post-training
pruning. Meanwhile, existing relaxed sparsity formats either require specialized compiler support or introduce runtime overheads that limit end-to-end speedup. We propose Spense, a practical hybrid sparse-dense format that splits each weight
matrix into a 2:4 sparse region and a dense region. This design relaxes the effective sparsity constraint while remaining compatible with existing high-performance sparse and dense GEMM libraries, avoiding both custom compiler support and input activation expansion. Building on this format, we introduce SpenseGPT, a one-shot post-training
pruning method that produces sparse and dense regions. Notably, we show that selecting the right dense regions is important, and we devise two different strategies to choose them. Experiments on Qwen3-32B and Seed-OSS-36B demonstrate that our method achieves up to 1.2x end-to-end decoding speedup on B200 GPUs with FP8 precision, while preserving accuracy. To the best of our knowledge, this is the first one-shot
pruning demonstration of real-world end-to-end LLM decoding speedup from semi-structured sparse tensor cores on recent GPUs such as B200s, while maintaining model quality.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsampling
Guodong Lin, Ziqi Chen, Yuxiang Fu et al. (+2 more)
The rapid progress of large
language models (LLMs) has opened up a new frontier for automatic
speech recognition (ASR), making their effective integration a critical and challenging research direction. To this end, this work proposes a projector-based LLM-ASR framework targeting the key challenges of
multilingual generalization and modality
alignment. Our approach incorporates a Mixture of Experts (MoE) architecture to improve cross-lingual adaptability, and a Continuous Integrate-and-Fire (CIF) mechanism for dynamic downsampling and modality
alignment. Experimental results show that the combination of these components yields substantial performance improvements, surpassing strong baseline models. The proposed method represents a step toward building more accurate, robust, and generalizable LLM-based ASR systems.
Read the full abstract ▾
All authors Guodong Lin, Ziqi Chen, Yuxiang Fu, Ke Li, Wei-Qiang Zhang
Computation and Language · Cross-listed from cs.LG
Parallel Causal Associative Fields: Gated Sparse Memory for Long-Context Language Modeling
Muhammad Ahmed
Transformers achieve strong
language modeling performance by providing direct
token-to-
token communication paths, but causal self-
attention scales quadratically with context length. Recurrent and state-space models reduce this cost, yet compress history into sequentially updated fixed-size states. This paper studies a third primitive: a parallel content-addressed
memory over causal successor records. The proposed Parallel Causal Associative Field (PCAF) writes local records from a context window into hash buckets, retrieves a bounded candidate set for the current query, forms a sparse cache distribution over successor
tokens, and mixes that cache with a parametric local
language model through a learned gate. The resulting model maintains sparse long-context access while avoiding a single fixed recurrent state bottleneck. We evaluate PCAF under full autoregressive pretraining on WikiText-103 and PG-19 using a distributed Google Cloud TPU v4-32 pod. At 303M parameters and context length T = 2048, PCAF-semantic reaches 36.31 perplexity on WikiText-103 and 52.45 perplexity on PG-19, compared with 47.49 and 53.84 for a matched dense
Transformer. PCAF-semantic simultaneously processes 0.61-0.62M
tokens/s across the TPU pod, versus 0.43M
tokens/s for dense and local
attention baselines. Supporting 41M-parameter multi-seed sweeps and single-GPU component ablations show that the associative cache,
retrieval capacity, and learned gate materially affect the speed-quality trade-off.
Read the full abstract ▾