Computation and Language
Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models
Amogh Sheth, Andrew Lin, Biruk Assefa et al. (+2 more)
Large
language models (LLMs) excel at multi-step
reasoning but incur substantial inference cost. We introduce Causal Attribution
Pruning (CAP), a training-free method that identifies critical
attention heads by measuring their causal impact on
reasoning tasks and uses these head-level scores to guide fine-grained weight
pruning. For each
attention head, CAP estimates the expected performance degradation when the head is masked during forward passes on a small calibration set of
reasoning problems. These causal scores are then converted into weight-level importance values for the corresponding projection
matrices. Unlike magnitude-only or activation-based criteria, CAP's interventional measurement directly captures each head's functional contribution, yielding relative accuracy gains of up to 61% over Wanda on ARC-Challenge at 20% sparsity. We evaluate CAP on GSM8K, StrategyQA, and ARC-Challenge using Llama-3-8B-Instruct and Mistral-7B-Instruct at 10%, 20%, and 50% sparsity. At moderate sparsity (10-20%), CAP improves over Wanda in most model-
benchmark configurations. with especially large gains on ARC-Challenge for Llama-3. Our results suggest that
attention-head-level causal attribution can better preserve
reasoning performance on downstream
benchmarks than correlational
pruning criteria at equivalent sparsity, while remaining limited by coarse MLP attribution at 50% sparsity.
Read the full abstract ▾
All authors Amogh Sheth, Andrew Lin, Biruk Assefa, Yi Wen Huang, Yuhao Ge
Computation and Language · Cross-listed from cs.LG
Staged Factorial Screening for Budget-Constrained Micro-Pretraining
Felipe Chavarro Polania
Budget-constrained micro-pretraining often requires triaging many candidate recipes on a shared accelerator before larger search budgets are spent. We study whether a staged fractional-factorial workflow can recover stable early effect structure in this setting. On a fixed autoresearch-derived single-GPU training loop, we run 613 experiments across pilot and follow-up screens at 2, 5, and 10 minutes; full 16-condition seeded reruns at 5 and 10 minutes; targeted seeded anchor checks; same-host greedy and matched-cost random baselines; a 60-minute bridge package; and bounded Windows A100 and Linux L40S anchor continuations through 24 hours. Main penalties from total batch, depth, and width are largest at short budgets and relax as budget increases. Within the predeclared seeded full-screen families, D, A, B, and C retain non-zero estimates at 5 and 10 minutes after within-budget Benjamini-Hochberg correction, while E does not. Random search can reach strong incumbents in this 32-condition space, but repeatedly in the same low-penalty region and without factor attribution. The 60-minute bridge anchor has the lowest mean, although that package does not separate workflow refinement from the larger bridge model's capacity advantage. In bounded 12-hour and 24-hour three-anchor continuations on both hosts, the bridge has the lowest sample mean while the non-bridge ordering stays host-sensitive. We therefore present a bounded methods result: use short designed screens to identify high-penalty directions, confirm promising anchors under repeated runs, and refine locally inside the reduced space. The evidence supports a bridge-centered recommendation through 24 hours on two hosts, not hardware-invariant ranking or general hyperparameter-optimization superiority.
Read the full abstract ▾
Computation and Language
Thoughts-as-Planning: Latent World Models for Chain-of-Thoughts Optimization via Reinforcement Planning
Dong Liu, Yanxuan Yu and Ying Nian Wu
The success of large
language models (LLMs) across diverse NLP tasks has elevated the importance of
reasoning chain optimization as a critical step in aligning model behavior with task objectives. Existing
reasoning chain tuning methods often rely on black-box heuristics or gradient-free search, which lack interpretability, generalization, and sample efficiency. In this work, we introduce Thoughts-as-Planning, a novel framework that formalizes
reasoning chain optimization as a sequential decision-making process over a latent semantic space. We model the LLM as a partially observable environment and learn a latent world model that simulates the effect of
reasoning chain edits on downstream outputs. A proximity-preserving
embedding space is constructed to encode
reasoning chain-response dynamics, enabling planning via gradient descent or reinforcement learning. Our method supports multi-scale abstraction, allowing
reasoning chain edits at
token, segment, and instruction levels to be integrated into a unified planner. Through extensive experiments on
language understanding and generation tasks, we demonstrate that Thoughts-as-Planning outperforms state-of-the-art
reasoning chain tuning baselines in efficiency, robustness, and generalization, while offering interpretability through its structured planning trajectory. Our code is available at https://github.com/FastLM/Thoughts-as-Planning.
Read the full abstract ▾
Computation and Language
Analyzing LLM Reasoning to Uncover Mental Health Stigma
Sreehari Sankar, Aliakbar Nafar, Mona Barman et al. (+8 more)
While large
language models (LLMs) are increasingly being explored for mental health applications, recent studies reveal that they can exhibit stigma toward individuals with psychological conditions. Existing evaluations of this stigma primarily rely on multiple-choice questions (MCQs), which fail to capture the biases embedded within the models' underlying logic. In this paper, we analyze the intermediate
reasoning steps of LLMs to uncover hidden stigmatizing
language and the internal rationales driving it. We leverage clinical expertise to categorize common patterns of stigmatizing
language directed at individuals with psychological conditions and use this framework to identify and tag problematic statements in LLM
reasoning. Furthermore, we rate the severity of these statements, distinguishing between overt prejudice and more subtle, less immediately harmful biases. To broaden the
reasoning domain and capture a wider array of patterns, we also extend an existing mental health stigma
benchmark by incorporating additional psychological conditions. Our findings demonstrate that evaluating model
reasoning not only exposes substantially more stigma than traditional MCQ-based methods but also helps identify the flaws in the LLMs' logic and their understanding of mental health conditions.
Read the full abstract ▾
All authors Sreehari Sankar, Aliakbar Nafar, Mona Barman, Hannah K. Heitz, Ashwin Kumar, Pouria Tohidi, Dailun Li, Danish Hussain, Russell DuBois, Hamed Hasheminia, Farshad Majzoubi
Computation and Language · Cross-listed from cs.AI
Leverage Laws: A Per-Task Framework for Human-Agent Collaboration
Stan Loosmore
We propose a per-task leverage ratio for human-
agent collaboration: human work displaced by an
agent, divided by the human time required to specify the task, resolve mid-run interrupts, and review the result. The denominator decomposes into three channels through which a conserved per-task information requirement must flow, each with its own time-cost scalar. We show that information density itself is directional and bounded by separate ceilings on human-to-
agent and
agent-to-human flow, and that the asymptotic behavior of leverage decomposes into two scaling axes (capability and
memory) with a non-zero floor on the planning term set by irreducible task novelty bounded by human throughput. We extend this per-task analysis to a windowed leverage measure that accommodates recurring tasks, spawned subtasks, and amortized system-design investment. The per-task ceiling does not bind the windowed measure, though both remain bounded: $L_{\text{task}}$ by per-task novelty, $L_{\text{window}}$ by the stock of accumulated planning investment that pays out within the window. The framework operationalizes aspects of earlier qualitative work on supervisory control (Sheridan, 1992), common ground (Clark & Brennan, 1991), and mixed-initiative interaction (Horvitz, 1999) within a single normative ratio, and produces a list of testable empirical questions that we leave as open problems.
Read the full abstract ▾
Computation and Language
Dual-Track CoT: Budget-Aware Stepwise Guidance for Small LMs
Sagnik Chatterjee, Atharva Patil and Sricharan Ramesh
Large
Language Models (LLMs) solve many
reasoning tasks via chain-of-thought (CoT) prompting, but smaller models (about 7 to 8B parameters) still struggle with multi-step
reasoning under tight compute and
token budgets. Existing test time
reasoning methods such as self consistency (sampling multiple rationales and voting), Tree-of-Thoughts (search over intermediate thoughts), and critique revise loops improve performance, but often at high
token cost and without fine-grained step-level control. This project1 aims to address that gap: can Small
Language Models (SLMs) reason reliably using the same or fewer
tokens? This question is both scientific and practical. Scientifically, it probes whether process supervision and simple test-time controls (such as
token budgets and rejection of redundant steps) can substitute for model scale or large sampling counts. Practically, many deployments (on-device, low-latency, or cost-constrained settings) cannot afford huge models or dozens of sampled rationales per query. A method that improves SLM
reasoning at fixed cost would therefore be directly useful.
Read the full abstract ▾
Computation and Language
Faithful Autoformalization via Roundtrip Verification and Repair
Daneshvar Amrollahi, Jerry Lopez and Clark Barrett
When an LLM formalizes natural
language, how do we know the output is faithful? We propose a roundtrip verification approach which does not require ground-truth annotations: formalize a statement, translate the result back to natural
language, re-formalize, and use a formal tool to check logical equivalence. When the two formalizations agree, this provides evidence of a faithful formalization. When they disagree, a stage-level diagnosis localizes the error to a specific translation step, and a scoped repair operator attempts to correct that step. We evaluate the framework on two statutory domains (the Texas Transportation Code and the Texas Parks and Wildlife Code) using two LLMs (Claude Opus~4.6 and GPT-5.2) with three repair baselines. Diagnosis-guided scoped repair is the most effective method, with effectiveness contingent on the reliability of the diagnosis function. Across both domains and both models, under our full repair system, rules that fail the equivalence check show 1.4x-2.5x more natural
language inference (NLI) drift than rules that pass it.
Read the full abstract ▾
Computation and Language
Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models
Dan Shi, Zhuowen Han, Simon Ostermann et al. (+3 more)
Reinforcement learning (RL)-based post-training often improves the
reasoning performance of large
language models (LLMs) beyond the training domain, while supervised fine-tuning (SFT) frequently leads to general capabilities forgetting. However, the mechanisms underlying this contrast remain unclear. To bridge this gap, we present a feature-level mechanistic analysis methodology to probe RL generalization using a controlled experimental setup, where RL- and SFT-tuned models are trained from the same base model on identical data. Leveraging our interpretability framework, we align internal activations across models within a shared feature space and analyze how features evolve during post-training. We find that SFT rapidly introduces many highly specialized features that stabilize early in training, whereas RL induces more restrained and continually evolving feature changes that largely preserve base models' representations. Focusing on samples where RL succeeds but the base model fails, we identify a compact, task-agnostic set of features that directly mediate generalization across diverse tasks. Feature-level interventions confirm their causal role: disabling these features significantly degrades RL models' generalization performance, while amplifying them improves base models' performance. The code is available at https://github.com/danshi777/RL-generalization.
Read the full abstract ▾
All authors Dan Shi, Zhuowen Han, Simon Ostermann, Renren Jin, Josef van Genabith, Deyi Xiong
Computation and Language
Dont́ Stop Early: Scalable Enterprise Deep Research with Controlled Information Flow and Evidence-Aware Termination
Prafulla Kumar Choubey, Kung-Hsiang Huang, Pranav Narayanan Venkit et al. (+5 more)
Enterprise deep research often fails to produce decision-ready reports due to uneven information coverage, context explosion, and premature stopping. We propose a scalable Enterprise Deep Research (EDR) architecture to address these failures. Our system (i) decomposes requests into coverage-driven objectives via outline generation with reflection, (ii) localizes context with dependency-guided execution and explicit information sharing, and (iii) enforces evidence-based completion criteria so
agents iteratively collect information until sufficiency conditions are met. We evaluate on an internal sales enablement task and the public DeepResearch Bench
benchmark, where our proposed system design achieves the strongest overall performance compared with competitive deep-research baselines. The results show that dependency-controlled context and explicit evidence sufficiency criteria reduce premature stopping and improve the consistency and depth of enterprise research outputs.
Read the full abstract ▾
All authors Prafulla Kumar Choubey, Kung-Hsiang Huang, Pranav Narayanan Venkit, Jiaxin Zhang, Vaibhav Vats, Yu Li, Xiangyu Peng, Chien-Sheng Wu
Computation and Language
A Survey on LLM-based Conversational User Simulation
Bo Ni, Leyao Wang, Yu Wang et al. (+27 more)
User simulation has long played a vital role in computer science due to its potential to support a wide range of applications.
Language, as the primary medium of human communication, forms the foundation of social interaction and behavior. Consequently, simulating conversational behavior has become a key area of study. Recent advancements in large
language models (LLMs) have significantly catalyzed progress in this domain by enabling high-fidelity generation of synthetic user conversation. In this paper, we survey recent advancements in LLM-based conversational user simulation. We introduce a novel taxonomy covering user granularity and simulation objectives. Additionally, we systematically analyze core techniques and evaluation methodologies. We aim to keep the research community informed of the latest advancements in conversational user simulation and to further facilitate future research by identifying open challenges and organizing existing work under a unified framework.
Read the full abstract ▾
All authors Bo Ni, Leyao Wang, Yu Wang, Branislav Kveton, Franck Dernoncourt, Yu Xia, Hongjie Chen, Reuben Leura, Samyadeep Basu, Subhojyoti Mukherjee, Puneet Mathur, Nesreen Ahmed, Junda Wu, Li Li, Huixin Zhang, Ruiyi Zhang, Tong Yu, Sungchul Kim, Jiuxiang Gu, Zhengzhong Tu, Alexa Siu, Zichao Wang, David Seunghyun Yoon, Nedim Lipka, Namyong Park, Zihao Lin, Trung Bui, Yue Zhao, Tyler Derr, Ryan A. Rossi
Computation and Language
Dynamic Decision Learning: Test-Time Evolution for Abnormality Grounding in Rare Diseases
Jun Li, Mingxuan Liu, Jiazhen Pan et al. (+4 more)
Clinical abnormality grounding for rare diseases is often hindered by data scarcity, making supervised fine-tuning impractical and single-pass inference highly unstable. We propose Dynamic Decision Learning (DDL), a framework that enables frozen large
vision-
language models (LVLMs) to refine their decisions across both
language and visual spaces by optimizing instructions and consolidating predictions under visual perturbations. This process improves localization quality and produces a consensus-based reliability score that quantifies model confidence. Results on
brain imaging
benchmarks, including a rare-disease
dataset with 281 pathology types across models ranging from 3B to 72B parameters, show that DDL improves mAP@75 by up to 105% on rare-disease cases and outperforms adaptation baselines and supervised fine-tuning. Furthermore, DDL demonstrates stronger calibration between reliability scores and localization accuracy under severe distribution shifts and increasing task difficulty. Code is available at: https://lijunrio.github.io/DDL/
Read the full abstract ▾
All authors Jun Li, Mingxuan Liu, Jiazhen Pan, Che Liu, Wenjia Bai, Cosmin I. Bercea, Julia A. Schnabel
Computation and Language · Cross-listed from cs.LG
PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference
Ishan Patel and Ishan Joshi
We present PolyKV, a system in which multiple concurrent inference
agents share a single, asymmetrically compressed KV cache pool. Rather than allocating a separate KV cache per
agent – the standard paradigm – PolyKV writes a compressed cache once and injects it into N independent
agent contexts via HuggingFace DynamicCache objects. Compression is asymmetric: Keys are quantized at int8 (q8_0) to preserve softmax stability, while Values are compressed using TurboQuant MSE – a Fast Walsh-Hadamard Transform (FWHT) rotation followed by 3-bit Lloyd-Max quantization with centroids tuned to N(0,1). We evaluate across two model scales (SmolLM2-1.7B-Instruct and Llama-3-8B-Instruct), three context lengths (600-7,194
tokens), and up to 15 concurrent
agents. PolyKV achieves a stable 2.91x compression ratio across all configurations. On Llama-3-8B with 15
agents sharing a 4K-
token context, PolyKV reduces KV cache
memory from 19.8 GB to 0.45 GB – a 97.7% reduction – while maintaining only +0.57% perplexity degradation and a mean BERTScore F1 of 0.928. PPL delta does not grow with
agent count and improves as context length increases, inverting to -0.26% at 1,851 coherent
tokens. To our knowledge, no prior work combines a single shared, lossy-compressed KV pool with multi-reader concurrent
agent access.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks
Lawrence Keunho Jang, Jing Yu Koh, Daniel Fried et al. (+1 more)
Existing web
agent benchmarks have largely converged on short, single-site tasks that frontier models are approaching saturation on. However, real world web use consists of long-horizon, multi-site workflows. Common web
navigation tasks, such as comparing products across different domains, planning trips across multiple services, or summarizing information from multiple search queries, require sustained context and cross-site
reasoning over potentially hours of browsing. To capture and evaluate such behaviors, we introduce Odysseys: a
benchmark of 200 long-horizon web tasks derived from real world browsing sessions evaluated on the live Internet. We find that binary pass/fail evaluation is inadequate for long-horizon settings and introduce a rubric-based evaluation, annotating each Odysseys task with an average of 6.1 graded rubrics. We demonstrate that this yields higher agreement with humans and provides a more fine-grained signal than commonly used trajectory-level LLM-as-a-judge evaluation metrics. We tested several leading frontier models and find that the strongest models achieve a success rate of 44.5%, which leaves substantial room for future improvements. Beyond task success, we argue that efficiency is a first-class concern for long-horizon
agents. We introduce a Trajectory Efficiency metric (rubric score per step) and find that even frontier
agents achieve only 1.15%, marking an evident need for
agents that can succeed efficiently and not simply eventually. Odysseys isolates the critical evaluation of long-horizon proficiency in open-web environments, providing a realistic
benchmark to measure progress towards computer-use
agents that can potentially productively operate for hours. We release our tasks, evaluation scripts, and other results at https://odysseys-website.pages.dev
Read the full abstract ▾
All authors Lawrence Keunho Jang, Jing Yu Koh, Daniel Fried, Ruslan Salakhutdinov
Computation and Language
Xinming Tu, Tianze Wang, Yingzhou Lu et al. (+3 more)
As
benchmarks grow in complexity, many apparent
agent failures are not failures of the
agent at all—they are failures of the
benchmark itself: broken specifications, implicit assumptions, and rigid evaluation scripts that penalize valid alternative approaches. We propose employing frontier LLMs as systematic auditors of evaluation infrastructure, and realize this
vision through BenchGuard, the first framework explicitly designed for joint cross-artifact
auditing of execution-based
agent benchmarks. BenchGuard cross-verifies all
benchmark artifacts via structured LLM protocols, optionally incorporating
agent solutions or execution traces as additional diagnostic evidence. Deployed on two prominent scientific
benchmarks, BenchGuard identified 12 author-confirmed issues in ScienceAgentBench—including fatal errors rendering tasks unsolvable—and exactly matched 83.3% of expert-identified issues on the BIXBench Verified-50 subset, catching defects that prior human review missed entirely. A full audit of 50 complex bioinformatics tasks costs under USD 15, making automated
benchmark auditing a practical and valuable complement to human review. A preliminary native-format audit of ProgramBench further demonstrates cross-format applicability. These findings point toward AI-assisted
benchmark development, where frontier models serve not only as subjects of evaluation but as active participants in validating the evaluation infrastructure itself.
Read the full abstract ▾
All authors Xinming Tu, Tianze Wang, Yingzhou Lu, Kexin Huang, Yuanhao Qu, Sara Mostafavi
Computation and Language
Independent-Component-Based Encoding Models of Brain Activity During Story Comprehension
Kamya Hari, Taha Binhuraib, Jin Li et al. (+2 more)
Encoding models provide a powerful framework for linking continuous stimulus features to neural activity; however, traditional voxelwise approaches are limited by measurement noise, inter-subject variability, and redundancy arising from spatially correlated voxels encoding overlapping neural signals. Here, we propose an independent component (IC)-based encoding framework that dissociates stimulus-driven and noise-driven signals in fMRI data. We decompose continuous fMRI data from naturalistic story listening into ICs using one subset of the data, and train encoding models on independent data to predict IC time series from large
language model representations of linguistic input. Across subjects, a subset of ICs exhibited consistently high predictivity. These ICs were spatially and temporally consistent across subjects and included cognitive networks known to respond during story listening (auditory and
language). Auditory component time series were strongly correlated with acoustic stimulus features, highlighting the interpretability of identified component time series. Components identified as noise or motion-related artifacts by ICA-AROMA showed uniformly poor predictive performance, confirming that highly predicted components reflect genuine stimulus-related neural signals rather than confounds. Overall, IC-based encoding models enable analyses at the level of functional networks, accommodating the variability in network locations across individuals and providing interpretable results that are easy to compare across subjects. Code provided at: https://github.com/kamyahari/IC-Encoding-Models.git
Read the full abstract ▾
All authors Kamya Hari, Taha Binhuraib, Jin Li, Cory Shain, Anna A. Ivanova
Computation and Language
ADE: Adaptive Dictionary Embeddings – Scaling Multi-Anchor Representations to Large Language Models
Orhan Demirci, Sezer Aptourachman and Ayd{\i}n Kaya
Word
embeddings are fundamental to natural
language processing, yet traditional approaches represent each word with a single vector, creating representational bottlenecks for polysemous words and limiting semantic expressiveness. While multi-anchor representations have shown promise by representing words as combinations of multiple vectors, they have been limited to small-scale models due to computational inefficiency and lack of integration with modern
transformer architectures. We introduce Adaptive Dictionary
Embeddings (ADE), a framework that successfully scales multi-anchor word representations to large
language models. ADE makes three key contributions: (1) Vocabulary Projection (VP), which transforms the costly two-stage anchor lookup into a single efficient
matrix operation; (2) Grouped Positional Encoding (GPE), a novel positional encoding scheme where anchors of the same word share positional information, preserving semantic coherence while enabling anchor-level variation; and (3) context-aware anchor reweighting, which leverages self-
attention to dynamically compose anchor contributions based on sequence context. We integrate these components into the Segment-Aware
Transformer (SAT), which provides context-aware reweighting of anchor contributions at inference time. We evaluate ADE on AG News and DBpedia-14 text classification
benchmarks. With 98.7% fewer trainable parameters than DeBERTa-v3-base, ADE surpasses DeBERTa on DBpedia-14 (98.06% vs. 97.80%) and approaches it on AG News (90.64% vs. 94.50%), while compressing the
embedding layer over 40x – demonstrating that multi-anchor representations are a practical and parameter-efficient alternative to single-vector
embeddings in modern
transformer architectures.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning
Minkyu Kim, Vincent-Daniel Yun, Youngrae Kim et al. (+3 more)
Depth
pruning improves the inference efficiency of large
language models by removing
Transformer blocks. Prior work typically treats layer redundancy as an inherent structural property of pretrained networks, emphasizing importance criteria and search
algorithms to identify removable layers. In this study, we empirically investigate depth
pruning from a functional perspective. Evaluating representative LLM families across diverse calibration configurations and multiple search
algorithms, we show that different configurations produce different
pruning patterns. Furthermore, under a fixed calibration configuration, complex search
algorithms yield marginal performance improvements over simple one-shot methods, converging to similar pruned subsets. Overall, our results suggest that the calibration configuration plays a substantially larger role than the choice of search
algorithm in shaping
pruning patterns and calibration perplexity, while contributing comparably to variance in downstream
reasoning accuracy. This indicates that future
pruning efforts may benefit from prioritizing the calibration configuration over search complexity.
Read the full abstract ▾
All authors Minkyu Kim, Vincent-Daniel Yun, Youngrae Kim, Suin Cho, Woosang Lim, Sunwoo Lee
Computation and Language
Yunsu Kim, Kaden Uhlig and Joern Wuebker
Agent benchmarks remain largely English-centric, while their
multilingual versions are often built with machine translation (MT) and limited post-editing. We argue that, for agentic tasks, this minimal workflow can easily break
benchmark validity through query-answer misalignment or culturally off-target context. We propose a refined workflow for adapting English
benchmarks into multiple
languages with explicit functional
alignment, cultural
alignment, and difficulty calibration using both automated checks and human review. Using this workflow, we introduce
GAIA-v2-LILT, a re-audited
multilingual extension of
GAIA covering five non-English
languages. In experiments, our workflow improves
agent success rates by up to 32.7% over minimally translated versions, bringing the closest audited setting to within 3.1% of English performance while substantial gaps remain in many other cases. This indicates that a substantial share of the
multilingual performance gap is
benchmark-induced measurement error, motivating task-level
alignment when adapting English
benchmarks across
languages. The data is available as part of the MAPS package at https://huggingface.co/
datasets/Fujitsu-FRE/MAPS/viewer/
GAIA-v2-LILT. We also release the code used in our experiments at https://github.com/lilt/
gaia-v2-lilt.
Read the full abstract ▾
Computation and Language
Large Language Models Explore by Latent Distilling
Yuanhao Zeng, Ao Lu, Lufei Li et al. (+3 more)
Generating diverse responses is crucial for test-time scaling of large
language models (LLMs), yet standard
stochastic sampling mostly yields surface-level lexical variation, limiting semantic exploration. In this paper, we propose Exploratory Sampling (ESamp), a decoding approach that explicitly encourages semantic diversity during generation. ESamp is motivated by the well-known observation that neural networks tend to make lower-error predictions on inputs similar to those encountered before, and incur higher prediction error on novel ones. Building on this property, we train a lightweight Distiller at test time to predict deep-layer hidden representations of the LLM from its shallow-layer representations to model the LLM's depth-wise representation transitions. During decoding, the Distiller continuously adapts to the mappings induced by the current generation context. ESamp uses the prediction error as a novelty signal to reweight candidate
token extensions conditioned on the current prefix, thereby biasing decoding toward less-explored semantic patterns. ESamp is implemented with an asynchronous training–inference pipeline, with less than 5% worst case overhead (1.2% in the optimized release). Empirical results show that ESamp significantly boosts the Pass@k efficiency of
reasoning models, showing superior or comparable performance to strong
stochastic and heuristic baselines. Notably, ESamp achieves robust generalization across mathematics, science, and code generation
benchmarks and breaks the trade-off between diversity and coherence in creative writing. Our code has released at: https://github.com/LinesHogan/tLLM.
Read the full abstract ▾
All authors Yuanhao Zeng, Ao Lu, Lufei Li, Zheng Zhang, Yexin Li, Kan Ren
Computation and Language · Cross-listed from cs.RO
Libra-VLA: Achieving Learning Equilibrium via Asynchronous Coarse-to-Fine Dual-System
Yifei Wei, Linqing Zhong, Yi Liu et al. (+4 more)
Vision-
Language-Action (VLA) models are a promising paradigm for generalist robotic manipulation by grounding high-level semantic instructions into executable physical actions. However, prevailing approaches typically adopt a monolithic generation paradigm, directly mapping visual-linguistic features to high-frequency motor commands in a flat, non-hierarchical fashion. This strategy overlooks the inherent hierarchy of robotic manipulation, where complex actions can be naturally modeled in a Hybrid Action Space, decomposing into discrete macro-directional reaching and continuous micro-pose
alignment, severely widening the semantic-actuation gap and imposing a heavy representational burden on grounding high-level semantics to continuous actions. To address this, we introduce Libra-VLA, a novel Coarse-to-Fine Dual-System VLA architecture. We explicitly decouple the learning complexity into a coarse-to-fine hierarchy to strike a training
equilibrium, while simultaneously leveraging this structural modularity to implement an asynchronous execution strategy. The Semantic Planner predicts discrete action
tokens capturing macro-directional intent, while the Action Refiner conditions on coarse intent to generate high-frequency continuous actions for precise
alignment. Crucially, our empirical analysis reveals that performance follows an inverted-U curve relative to action decomposition granularity, peaking exactly when the learning difficulty is balanced between the two sub-systems. With the asynchronous design, our approach offers a scalable, robust, and responsive solution for open-world manipulation.
Read the full abstract ▾
All authors Yifei Wei, Linqing Zhong, Yi Liu, Yuxiang Lu, Xindong He, Maoqing Yao, Guanghui Ren
Computation and Language · Cross-listed from cs.LG
Intrinsic Mutual Information as a Modulator for Preference Optimization
Peng Liao, Peijia Zheng, Lingbo Li et al. (+2 more)
Offline preference optimization methods, such as Direct Preference Optimization (DPO), offer significant advantages in aligning Large
Language Models (LLMs) with human values. However, achieving optimal performance with these methods typically involves additional hyperparameter tuning, resulting in substantial time overhead. Although prior work has proposed a range of improvements, these methods remain limited in effectiveness and have not fully eliminated reliance on hyperparameter tuning. In this work, we propose RMiPO, a lightweight and efficient framework for offline preference optimization. RMiPO leverages intrinsic Response-level Mutual information for Preference Optimization with hyperparameter modulation, dynamically decoupling preference contributions at negligible additional computational cost. Extensive experimental results demonstrate that RMiPO achieves consistently superior performance over existing methods while reducing training overhead by more than 15%. Our code is available at https://github.com/liavonpenn/rmipo.
Read the full abstract ▾
All authors Peng Liao, Peijia Zheng, Lingbo Li, Shangsong Liang, Lin Chen
Computation and Language
Sentiment and Emotion Classification of Indonesian E-Commerce Reviews via Multi-Task BiLSTM and AutoML Benchmarking
Hermawan Manurung, Ibrahim Al-Kahfi, Ahmad Rizqi et al. (+1 more)
Indonesian marketplace reviews mix standard vocabulary with slang, regional loanwords, numeric shorthands, and emoji, making lexicon-based sentiment tools unreliable in practice. This paper describes a two-track classification pipeline applied to the PRDECT-ID
dataset, which contains 5,400 product reviews from 29 Indonesian e-commerce categories, each labeled for binary sentiment (Positive/Negative) and five-class
emotion (Happy, Sad, Fear, Love, Anger). The first track applies TF-IDF vectorization with a PyCaret AutoML sweep across standard classifiers. The second track is a PyTorch Bidirectional Long Short-Term
Memory (BiLSTM) network with a shared encoder and two task-specific output heads. A preprocessing module applies 14 sequential cleaning steps, including a 140-entry slang dictionary assembled from marketplace corpora. Four configurations are benchmarked: BiLSTM Baseline, BiLSTM Improved, BiLSTM Large, and TextCNN. Training uses class-weighted cross-
entropy loss, ReduceLROnPlateau scheduling, and early stopping. Both tracks are deployed as Gradio applications on Hugging Face Spaces. Source code is publicly available at https://github.com/ikii-sd/pba2026-crazyrichteam.
Read the full abstract ▾
All authors Hermawan Manurung, Ibrahim Al-Kahfi, Ahmad Rizqi, Martin Clinton Tosima Manullang
Computation and Language
Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
Parsa Ashrafi Fashi, Utkarsh Saxena, Mehdi Rezagholizadeh et al. (+7 more)
Hybrid sequence models that combine efficient
Transformer components with linear sequence modeling blocks are a promising alternative to pure
Transformers, but most are still pretrained from scratch and therefore fail to reuse existing
Transformer checkpoints. We study upcycling as a practical path to convert pretrained
Transformer LLMs into hybrid architectures while preserving short-context quality and improving long-context capability. We call our solution HyLo (HYbrid LOng-context): a long-context upcycling recipe that combines architectural adaptation with efficient
Transformer blocks, Multi-Head Latent
Attention (MLA), and linear blocks (Mamba2 or Gated DeltaNet), together with staged long-context training and teacher-guided
distillation for stable optimization. HyLo extends usable context length by up to $32\times$ through efficient post-training and reduces KV-cache
memory by more than $90\%$, enabling up to 2M-
token prefill and decoding in our vLLM inference stack, while comparable Llama baselines run out of
memory beyond 64K context. Across 1B- and 3B-scale settings (Llama- and Qwen-based variants), HyLo delivers consistently strong short- and long-context performance and significantly outperforms state-of-the-art upcycled hybrid baselines on long-context evaluations such as RULER. Notably, at similar scale, HyLo-Qwen-1.7B trained on only 10B
tokens significantly outperforms JetNemotron (trained on 400B
tokens) on GSM8K, Lm-Harness common sense
reasoning and RULER-64K.
Read the full abstract ▾
All authors Parsa Ashrafi Fashi, Utkarsh Saxena, Mehdi Rezagholizadeh, Aref Jafari, Akash Haridas, Mingyu Yang, Vansh Bhatia, Guihong Li, Vikram Appia, Emad Barsoum
Computation and Language · Cross-listed from cs.AI
Case-Specific Rubrics for Clinical AI Evaluation: Methodology, Validation, and LLM-Clinician Agreement Across 823 Encounters
Aaryan Shah, Andrew Hines, Alexia Downs et al. (+6 more)
Objective. Clinical AI documentation systems require evaluation methodologies that are clinically valid, economically viable, and sensitive to iterative changes. Methods requiring expert review per scoring instance are too slow and expensive for safe, iterative deployment. We present a case-specific, clinician-authored rubric methodology for clinical AI evaluation and examine whether LLM-generated rubrics can approximate clinician agreement. Materials and Methods. Twenty clinicians authored 1,646 rubrics for 823 clinical cases (736 real-world, 87 synthetic) across primary care, psychiatry, oncology, and behavioral health. Each rubric was validated by confirming that an LLM-based scoring
agent consistently scored clinician-preferred outputs higher than rejected ones. Seven versions of an EHR-embedded AI
agent for clinicians were evaluated across all cases. Results. Clinician-authored rubrics discriminated effectively between high- and low-quality outputs (median score gap: 82.9%) with high scoring stability (median range: 0.00%). Median scores improved from 84% to 95%. In later experiments, clinician-LLM ranking agreement (tau: 0.42-0.46) matched or exceeded clinician-clinician agreement (tau: 0.38-0.43), attributable to both ceiling compression and LLM rubric improvement. Discussion. This convergence supports incorporating LLM rubrics alongside clinician-authored ones. At roughly 1,000 times lower cost, LLM rubrics enable substantially greater evaluation coverage, while continued clinical authorship grounds evaluation in expert judgment. Ceiling compression poses a methodological challenge for future inter-rater agreement studies. Conclusion. Case-specific rubrics offer a path for clinical AI evaluation that preserves expert judgment while enabling automation at three orders lower cost. Clinician-authored rubrics establish the baseline against which LLM rubrics are validated.
Read the full abstract ▾
All authors Aaryan Shah, Andrew Hines, Alexia Downs, Denis Bajet, Paulius Mui, Fabiano Araujo, Laura Offutt, Aida Rutledge, Elizabeth Jimenez
Computation and Language
Green Shielding: A User-Centric Approach Towards Trustworthy AI
Aaron J. Li, Nicolas Sanchez, Hao Huang et al. (+8 more)
Large
language models (LLMs) are increasingly deployed, yet their outputs can be highly sensitive to routine, non-
adversarial variation in how users phrase queries, a gap not well addressed by existing red-teaming efforts. We propose Green Shielding, a user-centric agenda for building evidence-backed deployment guidance by characterizing how benign input variation shifts model behavior. We operationalize this agenda through the CUE criteria:
benchmarks with authentic Context, reference standards and metrics that capture true Utility, and perturbations that reflect realistic variations in the Elicitation of model behavior. Guided by the PCS framework and developed with practicing physicians, we instantiate Green Shielding in medical diagnosis through HealthCareMagic-Diagnosis (HCM-Dx), a
benchmark of patient-authored queries, together with structured reference diagnosis sets and clinically grounded metrics for evaluating differential diagnosis lists. We also study perturbation regimes that capture routine input variation and show that
prompt-level factors shift model behavior along clinically meaningful dimensions. Across multiple frontier LLMs, these shifts trace out Pareto-like tradeoffs. In particular, neutralization, which removes common user-level factors while preserving clinical content, increases plausibility and yields more concise, clinician-like differentials, but reduces coverage of highly likely and safety-critical conditions. Together, these results show that interaction choices can systematically shift task-relevant properties of model outputs and support user-facing guidance for safer deployment in high-stakes domains. Although instantiated here in medical diagnosis, the agenda extends naturally to other decision-support settings and agentic AI systems.
Read the full abstract ▾
All authors Aaron J. Li, Nicolas Sanchez, Hao Huang, Ruijiang Dong, Jaskaran Bains, Katrin Jaradeh, Zhen Xiang, Bo Li, Feng Liu, Aaron Kornblith, Bin Yu
Computation and Language
The Chameleon's Limit: Investigating Persona Collapse and Homogenization in Large Language Models
Yunze Xiao, Vivienne J. Zhang, Chenghao Yang et al. (+2 more)
Large
language models (LLMs) have powered multi-
agent simulations such as survey respondents or application users. However, LLMs lack the population-level diversity required by these simulations—responses from LLMs with detailed human profiles cluster into classes, unlike human responses that uniformly fill the entire space. In this paper, we formalize this as Persona Collapse and quantify its population-level structure with three diagnostic metrics: Coverage (how much of the persona space is occupied), Uniformity (how evenly
agents spread across it), and Complexity (how rich the resulting behaviors are). Across eight LLMs on personality (BFI-44), moral
reasoning, and self-introduction, persona collapse is universal but task-dependent: CoSER-Llama-8B collapses the most in personality while being the most diverse in moral
reasoning. Moreover, sampling more personas cannot mitigate the collapse. We further show a fidelity–diversity tradeoff: the models with the highest per-persona fidelity produce the most polarized populations. Finally, persona-targeted fine-tuning further narrows the population instead of diversifying it.
Read the full abstract ▾
All authors Yunze Xiao, Vivienne J. Zhang, Chenghao Yang, Weihao Xuan, Jen-tse Huang
Computation and Language
Contextual Linear Activation Steering of Language Models
Brandon Hsu, Daniel Beaglehole, Adityanarayanan Radhakrishnan et al. (+1 more)
Linear activation steering is a powerful approach for controlling and specializing large
language models by adding linear representations of concepts to their activations at inference. While effective, existing methods often steer with a fixed strength, independent of how much steering any given input requires. In this work, we introduce Contextual Linear Activation Steering (CLAS), which decomposes activation steering into a fixed steering direction and a context-dependent steering coefficient. This decomposition preserves the interpretability of the steering direction while enabling the steering strength to vary with context. It also enables CLAS to perform selective steering: steering multiple behaviors across different portions of a single generation, which cannot be achieved using a fixed steering strength. In addition, across a wide range of steering
benchmarks and
language models, CLAS consistently outperforms standard linear activation steering and matches or exceeds ReFT and LoRA in settings with limited labeled data. Notably, CLAS achieves this using directions that also serve as reliable concept monitors, a property that directions learned via fine-tuning lack. We therefore propose CLAS as an interpretable and accurate method for controlling and specializing large
language models.
Read the full abstract ▾
All authors Brandon Hsu, Daniel Beaglehole, Adityanarayanan Radhakrishnan, Mikhail Belkin
Computation and Language
Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination
Lirong Gao, Zeqing Wang, Yuyan Cai et al. (+6 more)
While Large
Language Models (LLMs) have increasingly assisted in historical tasks such as text processing, their capacity for professional-level historical
reasoning remains underexplored. Existing
benchmarks primarily assess basic knowledge breadth or lexical understanding, failing to capture the higher-order skills, such as evidentiary
reasoning,that are central to historical research. To fill this gap, we introduce ProHist-Bench, a novel
benchmark anchored in the Chinese Imperial Examination (Keju) system, a comprehensive microcosm of East Asian political, social, and intellectual history spanning over 1,300 years. Developed through deep interdisciplinary collaboration, ProHist-Bench features 400 challenging, expert-curated questions across eight dynasties, accompanied by 10,891 fine-grained evaluation rubrics. Through a rigorous evaluation of 18 LLMs, we reveal a significant proficiency gap: even state-of-the-art LLMs struggle with complex historical research questions. We hope ProHist-Bench will facilitate the development of domain-specific
reasoning LLMs, advance computational historical research, and further uncover the untapped potential of LLMs. We release ProHist-Bench at https://github.com/inclusionAI/ABench/tree/main/ProHist-Bench.
Read the full abstract ▾
All authors Lirong Gao, Zeqing Wang, Yuyan Cai, Jiayi Deng, Yanmei Gu, Yiming Zhang, Jia Zhou, Yanfei Zhang, Junbo Zhao
Computation and Language
Benchmarking Source-Sensitive Reasoning in Turkish: Humans and LLMs under Evidential Trust Manipulation
Sercan Karakaş and Yusuf Şimşek
This paper investigates whether source trustworthiness shapes Turkish evidential morphology and whether large
language models (LLMs) track this sensitivity. We study the past-domain contrast between -DI and -mIs in controlled cloze contexts where the information source is overtly external, while only its perceived reliability is manipulated (High-Trust vs. Low-Trust). In a human production experiment, native speakers of Turkish show a robust trust effect: High-Trust contexts yield relatively more -DI, whereas Low-Trust contexts yield relatively more -mIs, with the pattern remaining stable across sensitivity analyses. We then evaluate 10 LLMs in three prompting paradigms (open gap-fill, explicit past-tense gap-fill, and forced-choice A/B selection). LLM behavior is highly model- and
prompt-dependent: some models show weak or local trust-consistent shifts, but effects are generally unstable, often reversed, and frequently overshadowed by output-compliance problems and strong base-rate suffix preferences. The results provide new evidence for a trust-/commitment-based account of Turkish evidentiality and reveal a clear human-LLM gap in source-sensitive evidential
reasoning.
Read the full abstract ▾
Computation and Language
DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference
Zahra Dehghanighobadi and Asja Fischer
Long-context
reasoning is a critical capability of large
language models (LLMs), enabling applications such as long-document understanding, summarization, and code generation. However, efficient autoregressive inference relies on the key-value (KV) cache, whose
memory footprint grows linearly with sequence length, leading to a major
memory bottleneck. To mitigate this overhead, KV cache
pruning methods discard cached
tokens with low
attention scores during inference. Most existing methods apply a uniform
pruning ratio across layers, implicitly assuming that all layers contribute equally to overall model performance. We show that this assumption is suboptimal, as layers differ significantly in their sensitivity to
pruning. We propose DepthKV, a layer-dependent
pruning framework that allocates a fixed global KV budget across layers based on their sensitivity, rather than using a uniform allocation. Across multiple models and tasks, DepthKV consistently outperforms uniform
pruning at the same global
pruning ratio, demonstrating more effective utilization of the KV cache budget through layer-dependent allocation.
Read the full abstract ▾