Computation and Language · Cross-listed from cs.LG
Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation
Maxime Griot, Paul Steven Scotti and Tanishq Mathew Abraham
Reasoning models produce long chain-of-thought traces that are costly to distill and encourage verbose student outputs. We study post-hoc compression of such traces before knowledge
distillation. Two teachers, Qwen3.5-397B-A17B and gpt-oss-120B, generate about 283k correct traces each; two instruction-tuned models then compress them to 8.6-21.0% of their original character length. Across a 48-run main grid plus seven Qwen-teacher truncation ablations, compressed traces reduce training
tokens to 12-30% of raw, speed up training by 2.0-7.6x, and shorten inference outputs by 3-19x with smaller reductions under the shorter gpt-oss teacher. However, raw traces retain the highest downstream accuracy at every scale and for both teachers. A length-matched raw-trace truncation ablation shows that compression is not merely benefiting from a smaller
token budget: model-compressed traces usually beat or match naive truncation, especially for smaller students, while maintaining shorter inference outputs. Overall,
reasoning-trace compression offers an accuracy-efficiency trade-off rather than a free improvement: students retain up to 96% of raw-trace accuracy while gaining up to 18x higher per-
token efficiency, and at the 0.8B scale under LoRA compressed traces narrow the raw-vs-compressed gap but do not exceed raw.
Read the full abstract ▾
Computation and Language
Beyond Alignment: Value Diversity as a Collective Property in Multicultural Agent Systems
Shaoyang Xu, Jingshen Zhang, Long P. Hoang et al. (+2 more)
Multicultural multi-
agent systems are increasingly deployed in globally diverse settings, where different
agents are grounded in different cultural backgrounds. Existing cultural evaluation focuses on value
alignment: how closely a single
agent matches a target culture. Yet
alignment is a per-
agent property and cannot reveal whether a system, taken as a whole, preserves the cultural plurality it is meant to represent. We propose value diversity as a system-level evaluation axis for multicultural
agent systems, defined through the dissimilarity between culturally conditioned
agents' responses on a shared value survey. Using the World Values Survey, we evaluate 19 cultures and 18 backbone models across a wide range of system configurations. We find that diversity is largely uncorrelated with
alignment, indicating that the two capture complementary system properties, and that current multicultural
agent systems fall substantially below human societies in value diversity. Mixed-backbone systems narrow this gap but do not close it, and the gap persists across culture compositions and
agent scales. Social interaction further erodes diversity by driving
agents toward consensus, and a participatory budgeting case study shows that this homogenization narrows the breadth of collective decision-making. Together, our results establish value diversity as a distinct evaluation axis for multicultural multi-
agent systems and reveal a persistent homogenization tendency in current LLM-based societies. Our code and data are publicly available at https://github.com/iNLP-Lab/MultiAgent-Diversity.
Read the full abstract ▾
All authors Shaoyang Xu, Jingshen Zhang, Long P. Hoang, Jinyuan Li, Wenxuan Zhang
Computation and Language · Cross-listed from cs.AI
Framing, Judging, Steering: An Assessable Competency Model for Teach-ing Students to Reason With Generative AI
Alexander Apartsin and Yehudit Aperstein
Generative AI makes answers easy and understanding hard, and uncritical use invites cognitive offloading. Schools still measure unaided performance, yet the real task is to produce good work with AI: framing an ill-defined task, judging the output, and steering the model toward a better result. This ability is rarely assessed in its own right; where measured, it collapses into one "prompting" score that cannot diagnose why AI use succeeds or fails. We propose CoRe-3 (Co-
Reasoning), a competency model factoring productive AI use into three assessable skills we abbreviate FJS: Framing (specifying an ill-defined task before invoking AI), Judging (evaluating output for errors and unstated assumptions), and Steering (iteratively redirecting the model). Its distinguishing claim is the separation of pre-generation Framing from post-generation Steering, with Judging as the gate between. We ground the skills in theory, state five testable propositions, and instantiate them in CoReasoningLab, an open platform that presents flawed AI output and scores them independently. Over simulated learners (generated and graded by different models), the skills dissociate: each tracks its own manipulated competence while staying flat in the others, and grades become correlated when one competence is shared across all three (convergent and discriminant validity), across grader backends from two providers. Human-rater agreement and outcomes are next; we release the instrument, data, and protocol.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
The Self-Correction Illusion: Role Relabeling Gates Explicit Error Flagging in Large Language Models
Kuan-Yen Chen, Fang-Yi Su, Shih-Yen Lin et al. (+2 more)
Recent works show that LLM
agents struggle to correct errors in their own
reasoning traces, despite their ability to correct errors from external sources. We ask whether this reflects a capability deficit or an artifact of the role labeling. To test this, we design a training-free intervention, source-conditioned role relabeling, that keeps the erroneous claim byte-identical and varies only its message role. The claim is presented inside the
agent's "<thought>", a user message, a tool response, or a system "<
memory>" block. We test 12 model-domain combinations spanning closed-weight APIs and open-weight models from 70B-class down to smaller families. Relabeling "<thought>" to an external role increases the explicit-correction rate by 23 to 93 percentage points, significant in 10 of 12 experimental settings. This suggests that these models' failure to detect a self-generated error is largely an artifact of how the claim is role-labeled in the chat template, rather than a pure cognitive deficit. The most effective role label is domain-dependent: "<
memory>" dominates in most math experiments, while a user message dominates in logical deduction. Recognizing role-label handling as a key experimental variable in instruction tuning presents a more direct path to closing the self-correction gap.d
Read the full abstract ▾
All authors Kuan-Yen Chen, Fang-Yi Su, Shih-Yen Lin, Bao Li, Jung-Hsien Chiang
Computation and Language
Measuring the sensitivity of LLM-based structured extraction to prompt, model, and schema choices in clinical discharge summaries
Martin Murin
Large
language models are increasingly used for structured extraction from clinical free-text notes, but the sensitivity of their output to upstream configuration choices is less understood than their accuracy on fixed
benchmarks. This work measures that sensitivity without human-annotated ground truth, by holding the extraction task fixed and varying one choice at a time. The fixed schema comprises 17 clinical documentation flags on a three-way yes/no/not_documented value set and a 47-tag vocabulary for the primary admission reason. Three
prompt variants expressing this schema were each run at two model sizes on MIMIC-IV v3.1 discharge summaries. Cross-
prompt agreement was measured by Cohen's kappa on ICD-stratified subsets. A paired same-note comparison isolated the effect of model choice, and a post-hoc collapse of the three-way flags to binary tested the schema's contribution to disagreement. On the three-way flags, the two models reach the same pooled cross-
prompt agreement (median kappa 0.69 and 0.68); the larger model raises agreement on some fields and lowers it on others, a redistribution rather than the absence of an effect. Collapsing the schema to binary dissolves most of the cross-
prompt disagreement, locating it on the absence-versus-silence distinction rather than on whether the finding is present. On the multi-class admission categorization, changing the model reassigns the dominant tag on close to half of all notes while changing the
prompt phrasing reassigns it on roughly one in eight, and the larger model places far less mass on residual catch-all categories (44% to 26%). These patterns indicate a schema-imposed source of disagreement concentrated on the absence-versus-silence axis and a dominance of model over
prompt phrasing on multi-class categorization, identified by a reusable methodology for
auditing extraction reproducibility on a population-scale deployment.
Read the full abstract ▾
Computation and Language
Large Language Models are Perplexed by some Political Parties
Paul Lerner and François Yvon
Large
Language Models (LLMs) are increasingly used, including in political applications, but their political
fairness has been little studied. We assess it using perplexity, posing that a fair model should give equal probability to all political groups. However, we find, across ten LLMs and three
datasets covering 37
languages, that LLMs are more perplexed by the texts of far right and nationalist parties than of social-democratic parties. We find this to be consistent with previous work on translation
fairness, to the point that perplexity correlates with downstream translation metrics. Our method is applicable to both base LLMs as well as their instruction-tuned counterpart, and we find that both are highly correlated, suggesting that the political
fairness of LLMs stems from their pretraining, and is hardly affected by instruction-tuning.
Read the full abstract ▾
Computation and Language
Epistemic Injustice in Language Models: An Audit of Pretraining Filters and Guardrails
Marco Antonio Stranisci, A Pranav, Rossana Damiano et al. (+2 more)
Modern
language models rely on pretraining filters to remove undesirable content from training corpora and inference-time guardrails to suppress undesirable outputs during deployment. In this paper, we examine how these filtering and moderation decisions produce forms of epistemic erasure and reveal tensions both across automated systems and between these systems and human judgment. We audit four pretraining filters and three inference-time guardrails on Common Crawl sentences containing gender and regional-origin mentions, together with a manually annotated subset of 500 sentences. Our analysis shows that filtering and guardrail decisions are strongly associated with blocklist-based lexical cues, while frequently failing to flag content containing private information or explicit hate
speech. At the same time, marginalized groups, particularly transgender people, women, and Central Americans, are significantly over-flagged across systems. Human annotators, by contrast, would retain 88.5% of filter-flagged and 91.3% of guardrail-flagged content, often recognizing representational harms arising from tensions of content removal that current systems fail to capture. Taken together, our findings document a form of epistemic erasure in which mentions of marginalized groups are disproportionately removed before pretraining and additionally suppressed again at inference time.
Read the full abstract ▾
All authors Marco Antonio Stranisci, A Pranav, Rossana Damiano, Christian Hardmeier, Anne Lauscher
Computation and Language
To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection
Erfan Loweimi, Mengjie Qian, Kate Knill et al. (+7 more)
When retrieving a person from a
video archive by voice and face, should the system be
multimodal or not? In real-world broadcast archives, unlike curated
benchmarks, a target may be heard but unseen, seen but unheard, or both. Fusing scores from an absent modality injects noise, degrading precision below the best unimodal system. We propose a query-adaptive framework that detects active modalities via cross-modal score consistency: when both modalities are active, files retrieved by one also score highly on the other; this agreement breaks down when a modality is absent. Classifiers driven by these cross-modal features achieve 89% detection accuracy. On the BBC Rewind corpus (with over 12,000 broadcast
videos) the adaptive system attains 94.2% P@1, outperforming speaker-only (82.9%), face-only (93.4%), and fixed fusion (90.0%), recovering 64% of the gap to an oracle with ground-truth modality labels (96.6%).
Read the full abstract ▾
All authors Erfan Loweimi, Mengjie Qian, Kate Knill, Guanfeng Wu, Chi-Ho Chan, Abbas Haider, Muhammad Awan, Josef Kittler, Hui Wang, Mark Gales
Computation and Language
Better Literary Translation: A Multi-Aspect Data Generation and LLM Training Approach
Zhihao Lin, Ziqi Zhu, Hao Huang et al. (+2 more)
Literary translation poses unique challenges due to the scarcity of high-quality annotated data and the need to balance expression fluency with literary effect. We present a multi-aspect iterative refinement framework that generates high-quality translation references and preference data through specialized LLM translators, each targeting a distinct quality dimension. We leverage the generated data for supervised fine-tuning and reinforcement learning. Experiments show that our generated references outperform the original ground truth for SFT by 8.65 CEA100 points. For reinforcement learning, we find that DPO leads to performance degradation in this setting, while leveraging an explicit reward model for GRPO yields an additional 1.51 point improvement. We attribute this to the stability of two-stage training and GRPO's online exploration capability. Our resulting models, LitMT-8B and LitMT-14B, achieve 67.25 and 69.07 CEA100 respectively on the MetaphorTrans English-to-Chinese literary translation
benchmark, competitive with Claude Sonnet 4.5 at 68.43, and demonstrate strong generalization to out-of-domain literary work (i.e., O. Henry).
Read the full abstract ▾
All authors Zhihao Lin, Ziqi Zhu, Hao Huang, Guanghui Wang, Peiyang He
Computation and Language · Cross-listed from cs.AI
Evolving Agents in the Dark: Retrospective Harness Optimization via Self-Preference
Wenbo Pan, Shujie Liu, Chin-Yew Lin et al. (+5 more)
AI
agents rely on a harness of skills, tools, and workflows to solve complex problems. Continually improving this harness is essential for adapting to new tasks. However, existing optimization methods typically require ground-truth validation sets, yet such labeled data is difficult to acquire in practical deployment settings. To address this problem, we introduce Retrospective Harness Optimization (RHO), a self-supervised method that optimizes the
agent harness using only past trajectories. Specifically, RHO selects a diverse coreset of challenging tasks from past trajectories and re-solves them in parallel. The
agent analyzes these rollouts using self-validation and self-consistency, then generates candidate harness updates and selects the most effective one by its own pairwise self-preference. We evaluate RHO across three diverse domains, spanning software engineering, technical work, and knowledge work. Notably, a single optimization round improves the pass rate on SWE-Bench Pro from 59% to 78% without any external grading. Furthermore, our analysis demonstrates that RHO effectively targets prior failure modes. As a result, the optimized harness alters the
agent's behavior patterns and sustains higher accuracy during long-horizon sessions.
Read the full abstract ▾
All authors Wenbo Pan, Shujie Liu, Chin-Yew Lin, Jingying Zeng, Xianfeng Tang, Xiangyang Zhou, Yan Lu, Xiaohua Jia
Computation and Language · Cross-listed from cs.SE
Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement
Xin Wang, Liangtai Sun, Yaoming Zhu et al. (+8 more)
Existing code-generation
benchmarks score a single mapping from a complete
prompt to a one-shot output. However, real web development is different. Users seldom write a full spec at the start; many requirements only become clear once they look at an intermediate result and react to it. We present Asuka-Bench, a
benchmark that pairs underspecified user intent with multi-round refinement, grounded in browser-rendered behavior. Each task is resolved through a closed loop: a Code
Agent generates a web project, a UI
Agent executes test cases on the deployed site, and a User LLM turns evaluation outcomes into natural-
language feedback for the next round. The
benchmark comprises 50 web tasks with 784 evaluation criteria and 2402 expected outcomes. We
benchmark 8 LLMs across 2
agent frameworks. The results separate models clearly: weighted Task Pass Rate varies by 38 percentage points and models also differ substantially in their ability to repair from feedback. Asuka-Bench is also far from saturated: even the strongest model completes only 52% of projects after three rounds.
Read the full abstract ▾
All authors Xin Wang, Liangtai Sun, Yaoming Zhu, Shuang Zhou, Jiaxing Liu, Fengjiao Chen, Lin Qiu, Xuezhi Cao, Xunliang Cai, Licheng Zhang, Zhendong Mao
Computation and Language · Cross-listed from cs.CV
MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering
Qing Yang, Pengcheng Huang, Xinze Li et al. (+6 more)
Long-
video question answering remains challenging for
Vision-
Language Models (VLMs), as answer-relevant evidence is often sparse, transient, and temporally dispersed across lengthy
video contexts. Existing frame-centric approaches improve efficiency through uniform sampling, query-aware frame selection, visual-
token compression, and adaptive resolution strategies. However, they still rely on isolated and fragmented frames as the fundamental evidence units, limiting VLMs' ability to effectively capture coherent event-level semantics. To address this limitation, we propose MemoryCard, a
video-
memory-based augmentation framework that organizes long
videos into self-contained
Memory Cards. Specifically, MemoryCard first performs a self-reading process over
videos and aligned utterances to segment the
video into semantically coherent units, each corresponding to a distinct topic or event. For each unit, it generates an event-level
video gist and selects representative visual moments, which are then rendered into unified
Memory Cards for
retrieval and question answering. Experimental results demonstrate that MemoryCard consistently improves long-
video QA performance under comparable visual-
token budgets, achieving up to a 21.8% relative improvement in accuracy. All code is available at https://github.com/NEUIR/MemoryCard.
Read the full abstract ▾
All authors Qing Yang, Pengcheng Huang, Xinze Li, Zhenghao Liu, Yukun Yan, Yu Gu, Ge Yu, Gang Li, Maosong Sun
Computation and Language
ACE-SQL: Adaptive Co-Optimization via Empirical Credit Assignment for Text-to-SQL
Xiaobing Chen, Ai Jian, Eryu Guo et al. (+1 more)
Text-to-SQL maps natural
language questions to executable SQL queries. Modern databases often contain large and complex schemas, making schema linking a critical step for accurate SQL generation. Existing methods either rely on full-schema generation, which leaves schema linking implicit within a large search space, or use a separate retriever trained with static gold-column supervision, whose targets may be suboptimal for the current generator policy. To address this issue, we propose Adaptive Co-optimization via Empirical Credit Assignment for Text-to-SQL (ACE-SQL), a reinforcement learning (RL) framework that jointly optimizes schema
retrieval and SQL generation under execution feedback. ACE-SQL constructs an online column-set pool from generator rollouts and derives adaptive on-policy
retrieval targets from the column set most frequently associated with execution-correct rollouts. This induces bidirectional adaptation, where the retriever adapts toward column sets that the generator can execute correctly, while the generator adapts to the retriever's evolving schema selections under execution feedback. With approximately 3k synthetic Text-to-SQL question-database pairs for RL training, ACE-SQL achieves 65.3% greedy execution accuracy on BIRD Dev while using 0.93k output
tokens per query. The repository is available at https://github.com/xbchen1/ACE-SQL.
Read the full abstract ▾
All authors Xiaobing Chen, Ai Jian, Eryu Guo, Zhiqi Pang
Computation and Language
Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)
Christopher J. Wedge, Joshua Stutter, Danny Dixon et al. (+1 more)
Large
language models (LLMs) have fundamentally transformed the landscape of Natural
Language Processing (NLP), although they remain susceptible to errors.
Retrieval-augmented generation (RAG) systems have emerged as a common deployment scenario seeking to both avoid the well known risk of the LLM “hallucinating” information, and to enable
reasoning and question answering over proprietary information that the LLM did not have access to during training without resorting to expensive model fine-tuning. In this work, we explore the idea of using a lightweight graph structure with a relatively simple graph schema, to support the RAG subsystem via a dedicated toolset. We design an agentic system with a variety of vector search and graph query tools operating over a structured
dataset based on a curated subset of English Wikipedia articles, and evaluate its performance on questions from MoNaCo, a challenging Wikipedia based
benchmark of complex question answering (QA) tasks. Our results show that the introduction of graph-based tools can significantly increase the precision and recall of factual correctness, can halve the number of hallucinated answers, and achieves the highest fine-grained truthfulness score among the three evaluated scenarios. All this with a modest increase in
token usage.
Read the full abstract ▾
All authors Christopher J. Wedge, Joshua Stutter, Danny Dixon, Jacek Cała
Computation and Language
Representing Research Attention as Contextually Structured Flows
Jessica Rodrigues, Angelo Salatino, Gard Jenset et al. (+1 more)
Research metrics use
attention as evidence of societal impact. Yet
attention serves as evidence only once interpreted, and its meaning depends on its contextual structure, not on volume alone. Altmetrics represents signals in isolation, keeping a count of the
attention an output received, or a sequence of when. We address this with
attention flows, representations that situate an output's
attention in the contexts through which it is distributed. To evaluate the flow, we build a
benchmark of analogy queries, each testing whether the relationship between two outputs, applied to a third, yields a fourth. The count and sequence baselines fail to recover these relationships, whereas flows learned as dynamic contextualised representations recover them. The recovered structure also survives partial observation and rests on its contexts instead of volume. These findings support
attention represented as contextually structured for research evaluation.
Read the full abstract ▾
All authors Jessica Rodrigues, Angelo Salatino, Gard Jenset, Scott Hale
Computation and Language
EMBER: Efficient Memory via Budgeted Evidence Retention for Long-Horizon Agents
Yilong Li, Suman Banerjee and Tong Che
Long-horizon
agents can archive large histories, but future answers still incur
retrieval, rereading, and context costs. When retained
memory misses answer-relevant evidence, the system must return to larger portions of the raw history. We study budgeted evidence survival: before the query is known, which source evidence should be retained so that it remains recoverable and usable under a fixed retained source-evidence
token budget? We instantiate this setting as Budgeted Pre-Query Retention, where
memory is written during ingestion and later read without access to the full raw stream. We introduce EMBER, a learned retention policy that constructs a compact, source-backed evidence state. EMBER stores evidence capsules: verbatim source excerpts paired with
retrieval keys and update metadata, preserving both grounding and read-time access. Post-query outcome feedback trains the writer to preserve evidence across the ingestion-
retrieval-answer chain. On LongMemEval-RR, our LongMemEval-derived retained-evidence protocol, EMBER-14B reaches 0.3017 F1 at the 8192-
token retained-evidence comparison point, compared with 0.1765 for the strongest non-EMBER budgeted baseline. Across retained source-evidence budgets, EMBER improves F1, Retain-Recall, and Read-Recall, indicating that long-horizon
memory depends on retaining evidence within the budget rather than rereading larger histories.
Read the full abstract ▾
Computation and Language
Staying with the Uncertainty: Uncertainty-Scaffolding Strategies for Artificial Moral Advisors in LLM-to-LLM Simulated Conversations
Salvatore Greco, Hainiu Xu, Jacopo Domenicucci et al. (+2 more)
LLMs are increasingly deployed as Artificial Moral Advisors (AMA) in a variety of contexts: what kind of conversational patterns should they display? In this paper, we study how AMA can help their interlocutors "stay with the uncertainty". We propose three modes of uncertainty (Perspective-Multiplying, Tension-Preserving, Process-Reflecting) and compare them against three control conditions (Baseline, Persuasive, Sycophantic). A user-
agent LLM engages in a dialogue on an ethical dilemma with an AMA following a specific uncertainty strategy, and completes pre- and post-conversation questionnaires. We further examine the effect of two persona
prompt formats (Declarative and Narrative). We found that (1) no single model dominates as a simulated user
agent, with open models aligning with human ambiguity through between-persona divergence and closed models through within-persona hedging; (2) declarative personas better capture initial stance diversity while narrative personas show more realistic belief revision; (3) all six AMA strategies produce distinguishable conversational patterns; and (4) uncertainty strategies differ not in how much stance revision they produce, but in the quality of engagement they sustain.
Read the full abstract ▾
All authors Salvatore Greco, Hainiu Xu, Jacopo Domenicucci, Yulan He, Sylvie Delacroix
Computation and Language · Cross-listed from cs.SD
GLASS: GRPO-Trained LoRA for Acoustic Style Steering in Zero-Shot Text-to-Speech
Jaehoon Kang, Yejin Lee and Kyuhong Shim
We propose GLASS, a framework for composable acoustic style control in zero-shot autoregressive text-to-
speech (TTS) that learns controls from post-generation rewards rather than style labels. In zero-shot TTS, a speaker
prompt often entangles speaker identity with prosodic attributes such as speaking rate and pitch, making it difficult to change style without changing the
prompt itself. GLASS instead treats each acoustic attribute as a reward-defined control direction. For each control axis, GLASS freezes the TTS backbone and trains one lightweight LoRA adapter with Group Relative Policy Optimization (GRPO), using
speech-
token length and mean F0 as style rewards and WER as an intelligibility anchor. Because each control is represented as a LoRA weight update, independently trained adapters can be swapped, interpolated, and composed through linear LoRA arithmetic without retraining the backbone. Experiments on speaking rate and pitch control show targeted style shifts while preserving naturalness, speaker similarity, and intelligibility, and demonstrate smooth interpolation and multi-axis composition across independently trained adapters.
Read the full abstract ▾
Computation and Language
Huiyuan Zheng, Houtao Zhang, Boyang Wang et al. (+2 more)
Current evaluations for
Multimodal Large
Language Models (MLLMs) overwhelmingly focus on utility-driven objectives, leaving model behavior under logic-neutral scenarios largely underexplored. Stochasticity is essential in scenarios where multiple actions are equally valid, such as recommending travel itineraries or daily schedules where multiple options have similar utility. In such settings, deterministic policies may lead to repetitive behaviors and reduced coverage of valid alternatives. To bridge this gap, we propose RandomBench, a
benchmark designed to evaluate whether MLLMs can maintain distributionally neutral behavior when selecting among equivalent options. We further introduce three metrics, including RI, BCI, BII, to quantify
entropy and distributional bias. Experiments reveal a pervasive phenomenon termed
Stochastic Collapse, where MLLMs fail to maintain uniform randomness under explicit random instructions, with top-1 probabilities reaching 97% from the ideal one quarter baseline and RI dropping to 0.068 in Claude Sonnet 4.6. Extensive ablation studies further demonstrate that these deviations persist across
languages and representation formats, highlighting the robustness of distributional collapse in logic-neutral decision settings.
Read the full abstract ▾
All authors Huiyuan Zheng, Houtao Zhang, Boyang Wang, Qingyi Si, Hongcheng Guo
Computation and Language
YouZhi: Towards High-Concurrency Financial LLMs via Adaptive GQA-to-MLA Transition
PSBC LLM Team, Huawei LLM Team: Ruihan Long, Junjie Wu et al. (+55 more)
Large
language models (LLMs) drive significant financial innovations, yet their high-concurrency deployment is severely bottlenecked by KV cache
memory overhead, which inflates infrastructure costs and throttles scalability. To address this, we propose YouZhi-LLM, a highly efficient financial LLM empowered by a comprehensive structural transition and training pipeline natively built on the Huawei Ascend ecosystem. At its algorithmic core, YouZhi-LLM features a layer-adaptive GQA-to-MLA transition framework that dynamically assigns per-layer FreqFold sizes, maximizing KV-cache compression while minimizing perplexity degradation. To recover representation capacity and inject domain expertise, the Ascend-based training pipeline seamlessly integrates generalized knowledge
distillation with financial-specific supervised fine-tuning. Evaluations demonstrate the superiority of this systematic approach, with the adaptive transition reducing perplexity degradation by up to 35% over uniform baselines. Crucially, when evaluated on Ascend NPUs via vLLM-Ascend, the massive KV-cache reduction translates directly into deployment efficiency. Compared to their respective base models, YouZhi-7B yields a 12.3% improvement in average financial
benchmark score alongside a 2.69$\times$ increase in maximum concurrency; similarly, YouZhi-14B achieves a 7.0% accuracy gain and a 2.43$\times$ concurrency boost, establishing a new paradigm for cost-effective, high-throughput financial inference.
Read the full abstract ▾
All authors PSBC LLM Team, Huawei LLM Team: Ruihan Long, Junjie Wu, Tianan Zhang, Duo Zhang, Yaozong Wu, Jinbin Fu, Chang Liu, Zhentao Tang, Wenshuang Yang, Xin Wang, Zhihao Song, Ning Huang, Wenjing Xu, Shuai Zong, Shupei Sun, Sen Wang, Jing Hu, Bin Wang, Xinyu Wang, Junkui Ju, Zequn Ding, Jie Ran, Man Luo, Shixiong Kai, Linkai Hou, Kaichao Liang, Hu Zhao, Yang Zhao, Shucheng Lin, Wei Yu, Chenghan Jiang, Jingjing Ding, Jiahui Zhang, Tian Jin, Yuhang Zhang, Dong Guo, Wei Sun, Jun Xie, Jianwei Li, Lei Cao, Pei Li, Jiabin Li, Jia Yuan, Rui Yuan, Jing Zhu, Mingxuan Yuan, Zhangcheng Lv, Xin Jiang, Xiuhong Fei, Xiaozhe Ren, Yulong Li, Zhipeng Zhang, Hang Wang, Zhaohui Xu, Rui Zhao, Yibo He, Xinzhuang Niu
Computation and Language
Analysis of the Neglect-Zero Effect in Large Language Models
Jin Tanaka, Daiki Matsuoka, Ryoma Kumon et al. (+1 more)
We investigate the extent to which the
language processing of LLMs resembles human cognitive processes, focusing on a human cognitive bias called the $\textit{neglect-zero effect}$. This effect refers to the human tendency to ignore $\textit{zero-models}$, which are configurations that render a proposition vacuously true by virtue of an empty set. We focus on two types of inferences driven by the neglect-zero effect, and examine how LLMs process these inferences by comparing their behavior with that in an inference that does not involve the neglect-zero effect. For this purpose, we employ a paradigm based on $\textit{structural priming}$, where recent exposure to a preceding sentence (the $\textit{prime}$) facilitates the processing of a subsequent sentence (the $\textit{target}$) due to their structural similarity. We prepare
primes to force LLMs to consider the zero-model, and analyze whether they also consider it in the target. The results suggest that the neglect-zero effect may not occur in the LLMs analyzed in this study. Our code is available at https://github.com/ynklab/neglect_zero
Read the full abstract ▾
All authors Jin Tanaka, Daiki Matsuoka, Ryoma Kumon, Hitomi Yanaka
Computation and Language
TARPO: Token-Wise Latent-Explicit Reasoning via Action-Routing Policy Optimization
Liting Zhang, Shiwan Zhao, Xuyang Zhao et al. (+3 more)
Latent
reasoning has emerged as a promising alternative to discrete Chain-of-Thought (CoT) in large
language models (LLMs), enabling more expressive
reasoning by operating over continuous representations. However, the inherently deterministic nature of continuous representations limits policy exploration in reinforcement learning (RL). To address this, we propose TARPO (
Token-Wise Latent-Explicit
Reasoning via Action-Routing Policy Optimization), a pure RL framework that adaptively switches between discrete
token generation and continuous latent
reasoning at each step. TARPO introduces a lightweight action head router that observes the current hidden state and samples a routing decision from a binary mode-selection space, preserving the stochasticity of discrete
token sampling from the vocabulary. The LLM backbone and router are jointly optimized end-to-end with a shared group-relative advantage signal. Extensive experiments across Qwen2.5 (from 1.5B to 7B) and Llama-3.1-8B backbones demonstrate that TARPO consistently outperforms existing explicit and latent
reasoning RL baselines across diverse
benchmarks. Further analysis shows that TARPO learns adaptive
token-wise switching behaviors while maintaining stable training dynamics. Our code is available at https://github.com/NKU-LITI/TARPO-master.
Read the full abstract ▾
All authors Liting Zhang, Shiwan Zhao, Xuyang Zhao, Zichen Xu, Jianye Wang, Qicheng Li
Computation and Language
ReverseEOL: Improving Training-free Text Embeddings via Text Reversal in Decoder-only LLMs
Ailiang Lin, Zhuoyun Li, Yusong Wang et al. (+3 more)
Recent advances in Large
Language Models (LLMs) have opened new avenues for generating training-free text
embeddings. However, the causal
attention in decoder-only LLMs prevents earlier
tokens from attending to future context, leading to biased contextualized representations. In this work, we propose Reverse prompting with Explicit One-word Limitation (ReverseEOL), a simple yet effective method for enhancing the representational capability of frozen LLMs. ReverseEOL augments the standard forward
embedding with an additional reversed
embedding derived from the reversed input text. Since reversing the input exposes each
token to context inaccessible in the original order, the resulting reversed
embedding effectively provides complementary information to the original one. As a result, combining the forward and reversed
embeddings yields a richer final representation. Comprehensive experiments on STS and MTEB
benchmarks demonstrate that ReverseEOL significantly improves the performance of existing training-free baselines across a broad range of LLMs with diverse architectures and scales. Extensive ablations and analyses further confirm the necessity of our reversal mechanism.
Read the full abstract ▾
All authors Ailiang Lin, Zhuoyun Li, Yusong Wang, Keyu Mao, Kotaro Funakoshi, Manabu Okumura
Computation and Language
Forgive or forget: Understanding the context of hate in audio retrieval systems
Arghya Pal, Sailaja Rajanala, Raphael C.-W. Phan et al. (+1 more)
Handling toxic
retrieval in text-to-audio systems is challenging due to contextual dependencies. Existing strategies (e.g., rephrasing, summarization) risk altering intent or omitting details. We propose a post hoc causal debiasing framework with a sentiment-controlled mediator to preserve semantic relevance while suppressing harmful
speech. Our approach is model-agnostic and integrates seamlessly with existing
retrieval pipelines. We introduce two variants: Forgive, which re-ranks and filters toxic audio via logit adjustment, and Forget, which generates counterfactual toxic
prompts to mitigate harmful
retrievals. Experiments show consistent toxicity reduction with minimal loss in
retrieval accuracy, improving both safety and reliability.
Read the full abstract ▾
All authors Arghya Pal, Sailaja Rajanala, Raphael C.-W. Phan, Shekhar Nayak
Computation and Language
Towards Truly Multilingual ASR: Generalizing Code-Switching ASR to Unseen Language Pairs
Gio Paik, Hyunseo Shin and Soungmin Lee
Automatic
Speech Recognition (ASR) has become a key technology for human–AI interaction. However, code-switching ASR (CS-ASR) remains particularly challenging due to the severe scarcity of
multilingual CS
speech resources across diverse
language pairs. Existing approaches primarily improve CS-ASR performance through synthetic CS
speech generation or pair-specific fine-tuning on limited bilingual
datasets. Nevertheless, these approaches face an inherent scalability limitation, as support for CS must be developed separately for
language pairs whose number grows combinatorially with the number of supported
languages. In this work, we investigate whether CS capabilities learned from a limited set of seen
language pairs can generalize to unseen
language pairs through model merging and domain generalization methods. Our experiments show that merged bilingual CS-ASR models modestly generalize to unseen
language pairs, suggesting limited transfer of bilingual CS capabilities across
language pairs.
Read the full abstract ▾
Computation and Language
Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads
Ruoxi Sun, Quantong Qiu, Juntao Li et al. (+3 more)
While
Multimodal Large
Language Models (MLLMs) demonstrate remarkable proficiency on complex
vision-
language tasks, the mechanisms by which they extract query-relevant visual features from complex, noisy contexts remain opaque. In this paper, we present an in-depth interpretability study that uncovers a profound structural property within MLLMs: functional sparsity in cross-modal
retrieval. Leveraging a
token-level metric termed
Retrieval Attention Mass (RAM), we identify and characterize a highly specialized subset of
attention heads, referred to as Context-aware
Retrieval (CoRe) heads. Across diverse visual domains and model scales, we observe a clear functional division: CoRe heads act as dedicated information extractors, while most other heads distribute
attention over broader contextual regions. Causal interventions further demonstrate the necessity of these specialized heads. Ablating only the top 5% of CoRe heads causes significant degradation in
multimodal reasoning performance, whereas ablating lower-ranked heads has minimal effect. Moreover, acceleration experiments validate the utility of CoRe heads, showing that leveraging this localized sparsity significantly accelerates inference while maintaining robust task performance. Our findings reveal a structural principle of functional sparsity within MLLMs, refining the current understanding of mechanistic interpretability and laying a theoretical foundation that can inspire future architecture design and model optimization.
Read the full abstract ▾
All authors Ruoxi Sun, Quantong Qiu, Juntao Li, Zecheng Tang, Yihang Lou, Min Zhang
Computation and Language
ProSPy: A Profiling-Driven SQL-Python Agentic Framework for Enterprise Text-to-SQL
Zhaorui Yang, Huawei Zheng, Sen Yang et al. (+14 more)
Large
language models have substantially advanced Text-to-SQL systems, yet applying them to enterprise-scale databases remains challenging. Real-world databases often contain large and heterogeneous schemas, incomplete metadata, dialect-specific SQL syntax, and complex analytical questions that are difficult to solve with a single SQL query. To address these challenges, we propose ProSPy, a Profiling-driven SQL–Python agentic framework for enterprise-scale Text-to-SQL. ProSPy structures the
reasoning process into four stages: it first extracts fine-grained data evidence through automatic profiling, progressively prunes large schemas into task-relevant contexts, fetches intermediate views through a dialect-agnostic SQL interface, and finally performs flexible downstream analysis with Python. This design combines the efficiency of SQL over large databases with the flexibility of Python-based analysis, while reducing reliance on unreliable metadata and improving robustness across SQL dialects. Experiments on Spider 2.0-Lite and Spider 2.0-Snow show that ProSPy consistently outperforms strong baselines with both open-source and proprietary models, achieving execution accuracies of 60.15% and 60.51% with Claude-4.5-Opus, without majority voting. Further analysis shows that ProSPy is robust to SQL dialect variations and achieves a favorable trade-off between schema recall and precision.
Read the full abstract ▾
All authors Zhaorui Yang, Huawei Zheng, Sen Yang, Yuhui Zhang, Haoxuan Li, Zhizhen Yu, Xuan Yi, Chen Hou, Defeng Xie, Chao Hu, Minfeng Zhu, Dazhen Deng, Haozhe Feng, Danqing Huang, Yingcai Wu, Peng Chen, Wei Chen
Computation and Language · Cross-listed from cs.AI
Statistical Priors for Implicit Preferences: Decoupling Skill Selection as a Local Harness in Personal Agents
Zeyu Gan, Huayi Tang and Yong Liu
As Large
Language Model (LLM) capabilities advance, locally deployed personal
agents relying on API-based remote models and external skills have emerged as a novel paradigm. With the rapid expansion of available skills, enabling personal
agents to learn and adapt to implicit user preferences becomes a critical challenge. However, local deployment constraints preclude complex centralized selection
algorithms, creating an urgent need for a lightweight local preference harness. This paper explores the implementation of such a harness through a novel architecture that strictly decouples statistical preference learning from semantic intent parsing. Specifically, we leverage localized statistical results to influence and modulate the selection decisions of the remote LLM. Extensive evaluations demonstrate that our decoupled approach achieves the lowest cumulative regret and highest test accuracy, significantly outperforming traditional
memory-augmented
agents.
Read the full abstract ▾
Computation and Language
Can LLMs Be Constrained to the Past? Improving Knowledge Cutoff through Recall-Based Prompting
Michiro Asai, Ailiang Lin, Yu Kishimoto et al. (+4 more)
Prompted knowledge cutoff instructs a large
language model (LLM) to act as if information beyond a specified cutoff date were unavailable. However, prior work mainly relies on direct-answer generation, which struggles when post-cutoff knowledge is not explicitly queried but is only causally related to the question. To address this limitation, we propose two recall-based prompting strategies: Self-Recall (SR), which asks the model to restate its cutoff constraint, and Question-Recall (QR), which requires the model to recall question-relevant information valid under the cutoff. Across three existing
benchmarks, our methods outperform both direct-answer prompting and conventional step-by-step
reasoning baselines, with particularly strong improvements on counterfactual questions. To investigate robustness across different cutoff settings, we further construct the Multi-cutoff Historical Event
Benchmark (MHEB), which evaluates the same question under multiple cutoff years. Results show that knowledge cutoff performance varies with cutoff distance, while combining SR and QR consistently yields the best performance.
Read the full abstract ▾
All authors Michiro Asai, Ailiang Lin, Yu Kishimoto, Takao Obi, Satoshi Kosugi, Kotaro Funakoshi, Manabu Okumura
Computation and Language · Cross-listed from cs.LG
CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction
Mohammad Anas Jawad and Cornelia Caragea
Existing calibration methods for Large
Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's behavioral robustness to irrelevant or misleading information. In this paper, we argue that a model's true confidence should reflect its stability under cognitive pressure. We introduce CaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model's susceptibility to distraction. CaliDist quantifies how an LLM's predictions and uncertainty change when its input
prompt is perturbed with semantic distractors. This stability (or lack thereof) signal is then used to adaptively scale the model's initial confidence score. Our extensive experiments on seven Natural
Language Understanding classification
benchmarks using six distinct LLMs show that CaliDist consistently achieves lower Expected Calibration Error (ECE) and Brier Score compared with strong baselines. Remarkably, our method reduces the ECE from 23% to 7% on average–a relative improvement of 70%–demonstrating that behavioral stability is a powerful signal for calibration. We make our code and
datasets available at github.com/anas-jawad/CaliDist.
Read the full abstract ▾