Computation and Language
When the Knowledge Base Becomes the Gold Standard: Measuring Resource-Shared Evaluation Loops in Entity-Level Machine Translation
Jinhyung Bae, Dain Kil, Seongmin Oh et al. (+1 more)
The Seungjeongwon Ilgi, a UNESCO
Memory of the World record, is only 37.4% translated, and the most conspicuous failure mode in automatic translation is the person name – a misread name corrupts the historical fact rather than merely the surface. Low-resource historical domains have no expert gold standard for entity translation, so practitioners substitute a knowledge base (KB) for the gold. That KB is the same resource injected into the system: scoring becomes self-referential and the metric measures instruction compliance rather than translation quality. We measure this loop. Using expert person-name annotations from the National Institute of Korean History as a gold independent of the injection pipeline, we hold the entity set fixed and vary only the provenance of the correct reading. Of 527 expert-annotated mentions, only 31.1% lie outside the injection pipeline, and the residual loop is not uniform – in the overlapping segment the injected reading agrees with the human translation 97.8% of the time against 70.1% in the independent one, so the segment that looks healthiest is the one the loop is holding up. Across four models, a difference-in-differences analysis shows the gain from KB injection is confined to the segment whose gold shares the injected resource; in the independent segment it is at or below zero. Post-injection preservation clusters in a narrow 0.910-0.996 band even though baseline capability differs fivefold, so the reported gain is the complement of prior performance and weaker models appear to improve more dramatically. On an independent sample built by removing the construction filter, the measure replicates within model (overlapping intervals) while discriminating between models (non-overlapping intervals) – it reflects a property of the model, not of the sample.
Read the full abstract ▾
All authors Jinhyung Bae, Dain Kil, Seongmin Oh, Seungmin Lee
Computation and Language · Cross-listed from cs.CY
Quantifying the Relationship Between Clinical Safety and Environmental Impact in Therapeutic LLMs
Alireza A. Safaei, Laura M. Vowels, Matthew J. Vowels et al. (+2 more)
The deployment of large
language models (LLMs) in mental health contexts raises questions about the relationship between clinical safety and environmental cost. In this paper, we examine this relationship by combining K-Bench clinical safety scores with EcoLogits life-cycle assessment estimates across 47 supported model configurations. We evaluate model performance and environmental impact across four dimensions: energy use, carbon emissions, water consumption, and abiotic depletion. The results indicate a non-linear trade-off at the upper end of the safety distribution: a 2.61 percentage-point increase in clinical safety score corresponded to an approximately 60-fold increase in estimated energy use per million output
tokens. Row-level analyses further suggest that additional test-time compute did not consistently improve clinical safety and, in some configurations, was associated with lower clinical safety scores. These findings suggest that relying solely on larger models or additional inference-time computation may be an inefficient strategy for improving safety in therapeutic AI systems. We discuss the implications for sustainable deployment and highlight dynamic model selection, including model cascading, as a potential approach for reducing environmental impact while preserving clinical performance in higher-risk cases.
Read the full abstract ▾
All authors Alireza A. Safaei, Laura M. Vowels, Matthew J. Vowels, Apoorv Jha, Shekoufeh Rahimi
Computation and Language · Cross-listed from cs.LG
Towards Understanding On-Policy Distillation through the Lens of Test-Time Scaling
Xinmu Ge, Zizhuo Zhang, Yu Huang et al. (+8 more)
On-policy
distillation (OPD) has emerged as a promising post-training technique for enhancing LLM
reasoning. Under the reverse KL objective, the idealized optimum of OPD aligns the student distribution with that of the teacher. When the teacher consistently outperforms the student, this naturally suggests that OPD should yield broad improvements over the pre-OPD student. However, do such improvements extend across the entire range of test-time sampling budgets? In this work, we revisit this expectation through the lens of test-time scaling by varying the sampling budget $K$ and evaluating performance with pass@$K$. Across multiple settings, we observe two distinct patterns: OPD can improve pass@$K$ at both small and large sampling budgets, but it can also improve small-budget performance while reducing large-budget pass@$K$. We show one condition that guarantees such a reversal and an idealized reverse KL counterexample where it occurs even when the teacher has higher accuracy on every problem. To choose between two candidate teachers at a target sampling budget, we propose the \textit{Teacher Advantage Score at $K$} (TAS@$K$), which can be computed before OPD training to predict which teacher will lead to a larger improvement in pass@$K$. Across three domains and thirteen
benchmarks, the ordering predicted by TAS@$K$ agrees with the observed pass@$K$ improvements of the resulting OPD models in 83.6% of experiments, providing a useful signal for teacher selection at the target pass@$K$.
Read the full abstract ▾
All authors Xinmu Ge, Zizhuo Zhang, Yu Huang, Jianing Zhu, Lin Yuan, Wanli Gu, Weichang Wu, Weiran Huang, Bo Han, Xiaolu Zhang, Jiangchao Yao
Computation and Language
Located but Not Releasable: Silent Gate Inversion and Bounded Linear Release
Xining Xun
A growing body of work reports that
language models represent task-relevant latent structure that they fail to use. Whether such structure, once located, can be converted into behavior is a separate question that is rarely tested end to end. We submit the complete pipeline – detect, localize, and release – to a fully preregistered stress test on a 25.7M
transformer trained on causal-evidence discrimination, where a known suppression phenomenon (latent causal structure present but behaviorally unused) has previously been documented. Every threshold, claim template, and decision-tree branch was hashed and archived before any corresponding data existed. Three findings. (i) Localization succeeds: interventions at observation-evidence channels of mid layers restore target behavior on otherwise-suppressed worlds (paired release advantages $0.563$ and $0.854$, 97.5% CIs excluding zero; best-site release rate $0.889$). (ii) Gating fails out of distribution: a detector calibrated to trigger on zero out-of-distribution calibration worlds triggers on 6.9-7.3% of held-out in-distribution generations and on zero of the 2,400 held-out generations that actually need it – a complete inversion that silently reduces the gated pipeline to its base model. (iii) Linear release is capped: removing the gate and injecting a per-instance linear direction unconditionally yields a monotone dose-response that plateaus far below the preregistered release margin (intercept $0.382 \to 0.311 \to 0.264$ vs. threshold $\le 0.08$); per-instance adaptivity adds less than $\pm 0.03$. The failure is doubly located: the detector is OOD-inverted, and the entire family of linear release directions at this site and resolution is bounded away from sufficiency. The two failures are dissociable, and neither overturns localization. Every number traces to a hashed artifact in the released audit chain.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
How China-Origin Vision-Language Models Move from Refusal to Reframing in State Alignment
Guang Yang, Fengchen Liu, Alex Wang et al. (+2 more)
State-aligned distortion has been documented in China-origin text-based large
language models (LLMs), but whether, and in what form, it arises in
multimodal systems has not been systematically examined. We construct a balanced
benchmark of 200 core entries spanning ten politically sensitive topics, plus a seven-variant visual-abstraction probe, and run nine
vision-
language models (VLMs), seven China-origin and two non-China, across four elicitation paradigms and two
prompt languages, yielding 21,708 trials. Each response is audited on six dimensions – explicit refusal, information integrity, visual grounding, state-aligned framing,
language consistency, and response length – by two independent frontier LLM judges, validated against three human experts on a 200-trial sample. Measuring each dimension separately lets us decompose
multimodal censorship into individual signals rather than a single refusal-based score; in particular, refusal and framing are measured independently, so a model can stop refusing while still reframing. We find that (i) Chinese-
language prompting roughly triples the odds of state-aligned framing, within every model; (ii) China-origin models reframe more than non-China models (direction robust across judges and human raters; magnitude 1.6–3.2x); (iii) the effect is strongest in text-only political commentary (36.5%) and is gated by recognition of the depicted subject rather than pixel detail, persisting even at silhouette for iconic images; and (iv) across four Qwen generations, state-aligned framing rises while explicit refusal falls: censorship migrates from a visible act (refusal) to an invisible one (fluent reframing). We argue this shift to invisible reframing is fundamentally a problem of human-AI interaction: it removes the very signal users rely on to recognize that information has been withheld.
Read the full abstract ▾
All authors Guang Yang, Fengchen Liu, Alex Wang, Homa Hosseinmardi, Amir Ghasemian
Computation and Language
Zekun Zhou, Ruobing Xie, Lanrui Wang et al. (+1 more)
Gated
attention is an effective approach to mitigate
attention sinks and enhance the representational capacity of
attention. To further extend its effectiveness-efficiency Pareto frontier, we propose a Hybrid Gated
Attention (HyGA) framework that contains three types of gating strategies. Specifically, these gates leverage diverse information from multiple stages of
attention, and collaboratively build element-wise/head-wise gating from multiple perspectives, capturing intra-head and cross-head information interactions. Through our hybrid gating components, HyGA could provide multi-source modulation signals, enabling more comprehensive control over information flow and improving the representational capacity of
attention. We also introduce low-rank
matrix decomposition and learnable
attention sink to further enhance training efficiency and stability. In experiments, we evaluate HyGA on widely-used
benchmarks based on different backbones. The experimental results show that our HyGA comprehensively improves both training loss and various downstream performances compared with Gated
attention. HyGA has also been verified to achieve the best performance at different computation costs, with comprehensive model analyses for better understanding. The proposed HyGA sheds light on a more effective, efficient, and stable
attention mechanism.
Read the full abstract ▾
All authors Zekun Zhou, Ruobing Xie, Lanrui Wang, Weixuan Sun
Computation and Language
TELLME: Test-Enhanced Learning for Language Model Enrichment
Minjun Kim, Inho Won, Hyeonseok Lim et al. (+6 more)
Continual pre-training (CPT) has been widely adopted as a method for domain adaptation in large
language models. However, CPT has consistently been accompanied by challenges, such as the difficulty of acquiring large-scale domain-specific
datasets and high computational costs. In this study, we propose a novel method called Test-Enhanced Learning for
Language Model Enrichment (TELLME) to alleviate these issues. TELLME leverages the TestEnhanced Learning (TEL) principle, whereby the model's training efficiency is improved using quizzes during training. It integrates this principle with CPT, thereby promoting efficient domain-specific knowledge acquisition and long-term
memory retention. Experimental results demonstrate that TELLME outperforms existing methods by up to 23.6% in the financial domain and achieves a 9.8% improvement in long-term
memory retention.
Read the full abstract ▾
All authors Minjun Kim, Inho Won, Hyeonseok Lim, MinKyu Kim, Junghun Yuk, Wooyoung Go, Jongyoul Park, Jungyeul Park, KyungTae Lim
Computation and Language
GRPO for Financial Advice Generation: Outperforming Commercial LLMs under CATE Evaluation
Ofir Ben Shoham, Shrutendra Harsola, Vignesh Subrahmaniam et al. (+3 more)
Generating actionable financial advice from business records demands that models integrate numerical
reasoning, domain knowledge, and sound judgment, while avoiding recommendations that could harm the business. Direct supervision is difficult: historical decisions are not necessarily optimal, and high-quality free-form labels are expensive to obtain. We formulate financial advice generation as a reinforcement learning problem and fine-tune an open-weight
language model using Group Relative Policy Optimization (GRPO). Our reward is an LLM-as-a-judge rubric that scores each recommendation across multiple binary dimensions of advice quality, augmented with a safety gate for harm prevention. Since LLM-based evaluation alone cannot confirm whether improvements reflect genuine business value rather than adaptation to the judge, we complement it with a judge-independent audit based on a standard doubly-robust Conditional Average Treatment Effect (CATE)
estimator. Under this observational off-policy audit, our trained LLM achieves approximately twice the estimated gross-profit lift of the strongest evaluated commercial baseline ($0.0228$ vs.\ $0.0104$), together with the lowest downside rate and the least negative tail risk of any policy evaluated. Notably, the two evaluations do not rank the baselines identically: the untrained base model places last on the judge rubric but second on the causal audit, indicating that the audit captures a signal the judge does not. Our results demonstrate that GRPO with a finance-grounded reward signal can produce substantially more useful business recommendations than commercial LLMs, and that a judge-independent causal audit is a valuable complement to, rather than a confirmation of, LLM-as-a-judge assessment in financial NLP.
Read the full abstract ▾
All authors Ofir Ben Shoham, Shrutendra Harsola, Vignesh Subrahmaniam, Shravan Mohan, Yakov Gazman, Oded Vainas
Computation and Language
Language-Conditional Dequantization: Recovering What Quantization Steals from Non-English Languages
Nirmal Thomas
Aggressive quantization disproportionately harms
multilingual capability: in the sub-4B INT3 GPTQ regime, we measure 2-4x larger perplexity degradation on non-English
languages than on English. We propose
Language-Conditional Dequantization (LCD), a post-hoc method that attaches per-
language rank-2 LoRA corrections to the linear layers of an already-quantized model, adding 0.12% parameters per
language and training in under 20 minutes on a single GPU. Across Qwen2.5-3B and Llama-3.2-3B, LCD recovers 70-83% of the perplexity gap for non-Latin script
languages and 17-28% of the GlobalMMLU accuracy gap, outperforming a
language-agnostic correction of equal capacity by 3-9 points on typologically distant
languages and a data-free low-rank baseline (LQER) by an order of magnitude. We further identify a perplexity-accuracy disconnect and trace it to where quantization concentrates damage: early-depth errors (Llama) propagate downstream and resist local correction, while late-depth errors (Qwen) do not. A layer-restricted variant of LCD validates this mechanism directly.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
The Sleeping Agent: What Gist-Based Context Compression Loses and Why
Nicholas E. Kyrkewood
Gist-based context compression—summarising older conversation history into compact representations—is a common approach in long-horizon
language model
agents, yet its effect on different types of
memory retrieval is poorly understood. We use Salience-Weighted Consolidation (SWC), a biologically-inspired compression framework motivated by sleep-based
memory consolidation, as a diagnostic probe to study when gist compression helps and when it hurts. SWC scores conversation history by salience, partitions it into priority tiers, and applies structured gist abstraction to mid-priority content. Evaluating four conditions on all ten LoCoMo conversations—1,935 matched text-only questions in total, 1,501 used in the primary aggregate after excluding Category 5 (
adversarial) questions—at temperature 0, we find a consistent task-type interaction: gist compression substantially outperforms truncation on multi-hop
reasoning and single-hop factual questions, but temporal questions remain substantially harder under compression, with compressed conditions scoring well below the full-context reference on the conversations where both are evaluated. We trace this failure to a specific mechanism: the gist abstraction
prompt preserves relational and event structure while discarding dates and times. A preservation analysis across all ten conversations confirms the mechanism: an approximately 20-fold increase in temporal expression preservation (3.05% to 62.39%) with a one-sentence
prompt modification, while named entity and event preservation rates barely change (x1.02 and x1.11), demonstrating that the fix is a precision instrument. The
prompt modification recovers +0.314 [0.254, 0.375] judge accuracy on category-2 (temporal) questions in the matched set. Code and results: https://github.com/kyrkewood/sleeping-
agent.
Read the full abstract ▾
Computation and Language
Diagnosis Before Recovery: Turning Agent Failures into Selective Self-Correction
Pan Wang, Yihao Hu, Hang Wang et al. (+6 more)
Self-correction is particularly useful when a failure constrains the next repair. Coding
agents benefit from this property because compilers, tests, and execution traces turn many failures into typed recovery signals, but broad
language-
agent tasks often expose only a coarse task failure. This creates a tension for generic recovery playbooks: they broaden the
agent's context precisely when the system needs a narrower repair interface, mixing incompatible signals for invalid actions, missing procedures, and strict-format errors. Our insight is that development-set failures can recover part of the missing diagnostic substrate by deciding which recovery interventions are admissible before test-time correction. We propose DARC, a diagnosis-guided recovery harness that profiles task-family failure modes, prunes mismatched interventions from a shared recovery library, and freezes a verifier-selected success-cost policy for deployment. This causal order makes correction selective: the harness first determines what kind of failure can be repaired, then decides how much recovery evidence to spend. In ALFWorld, AppWorld, and XBRL Finance, the same protocol yields an action-validity harness, a procedural-recovery fallback, and a format-precision
retrieval policy; in each evaluated setting it improves average task performance over base
agents and broad playbooks while reducing environment steps or
retrieval budget. Our experiments show that failures need not trigger uniformly more context: DARC turns self-correction from
prompt expansion into recovery-interface design. DARC provides a practical route toward more reliable
agents in domains where compiler-like feedback is absent: making failures actionable before making contexts larger.
Read the full abstract ▾
All authors Pan Wang, Yihao Hu, Hang Wang, Zirui Lv, Xin Zhang, Jianshe Li, Jiang-Ming Yang, Wei Wu, Yongqi Tong
Computation and Language
Causal Structure is Inducible but Functionally Decoupled: The Routing/Readout Boundary of a Typed Mechanism Library
Xining Xun
When a
language model answers an interventional question, the computation it must perform depends on the type of evidence the query requires. We report a decoupling in how a
transformer organizes causal knowledge: slot-by-type structure induced by type-level supervision organizes routing, yet remains functionally decoupled from answer readout. We establish this with a typed mechanism library – discrete mechanism slots partitioned by evidence type, auditable at the state level – on a causal-world
benchmark with exact interventional ground truth, under a frozen protocol, at two scales (22.6M and 125M). Four preregistered findings. (i) Origin. Slot-by-type organization is induced by type-level supervision: absent in architecturally identical unsupervised controls, not buyable by content-free gating labels, and statistically attributable to the supervision signal, replicating at 125M under a powered preregistered protocol (all nine
cells passed). (ii) Boundary. The induced structure is a typed routing index with a sharp routing/readout boundary: slot codes scaffold routing but do not drive answer readout ($|\Delta\hat{y}| \le 3.4\times10^{-6}$, zero collateral, three seeds, stable across a 5.6x scale window) – we therefore make no behavioral-editability claim. (iii) Cost. The structure is free: LM quality matches a parameter-matched monolith within 0.0082 nats. (iv) Trust. The library state is exactly local under edit and bit-exactly revertible – 250 single-edit and 1,000 stacked reverts per seed, zero failures. We further find that the unsupervised null itself moves with scale, so comparisons reusing a null calibrated at one scale may be confounded at another. Every claim is tied to a preregistered, machine-checkable criterion archived before the data it governs; the full audit trail, including one criterion we failed and how the frozen protocol handled it, is released as an appendix.
Read the full abstract ▾
Computation and Language
AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention
Juncheng Liao, Jinfan Lv, Guoming Wang et al. (+3 more)
Multimodal Large
Language Models (MLLMs) exhibit strong generalization and
reasoning abilities due to large-scale
multimodal pre-training. However, fine-tuning these models on downstream tasks often leads to catastrophic forgetting, where newly learned task-specific knowledge degrades previously acquired capabilities. This issue arises because gradient updates for new tasks overwrite parameters critical to prior knowledge, limiting the practical deployment of MLLMs. To address this challenge, we propose Activation-Weighted Adaptive REtention (AWARe), a fine-tuning method that mitigates catastrophic forgetting by dynamically controlling parameter updates based on activation patterns. AWARe assigns activation-based importance scores to parameters, selectively freezing those essential for preserving prior capabilities while allowing less important parameters to adapt to new tasks. Importantly, AWARe operates without modifying model architectures, ensuring compatibility with existing inference engines. Extensive experiments demonstrate that AWARe effectively preserves upstream capabilities while achieving superior downstream performance compared to existing methods. Code is available at https://github.com/kaln27/AWARe.
Read the full abstract ▾
All authors Juncheng Liao, Jinfan Lv, Guoming Wang, Jupeng Zheng, Ling Xiao, Siliang Tang
Computation and Language · Cross-listed from cs.SD
MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques
Jiabao Zhuang, Changhao Jiang, Hanchen Wang et al. (+11 more)
Long-form song generation models continue to improve in duration, structural coherence, and acoustic complexity, increasing the need for reliable aesthetic rewards aligned with human preferences. However, reward models for complete songs remain limited, and existing evaluators typically predict scores in a single forward pass without readable explanations. To this end, we introduce MuseCritic, a semi-scalar reward model that generates a natural-
language critique covering five aesthetic dimensions and uses it as an intermediate representation to predict continuous reward scores. MuseCritic follows a two-stage training pipeline: a teacher model first provides high-quality critiques for supervised fine-tuning, then the fine-tuned model generates its own critiques for reward learning, mitigating training-inference distribution shift. On an in-domain test set of 200 SongEval songs, MuseCritic reduces macro-averaged mean squared error from 0.2875 to 0.2316 and improves macro-averaged LCC, SRCC, and Kendall's tau to 0.9068, 0.8838, and 0.7178, respectively. On the out-of-domain Music Arena
benchmark with 733 preference pairs, it achieves 71.35% accuracy and remains competitive with strong music-specific reward models. Using MuseCritic with GRPO also improves Muse-0.6B on all nine aesthetic metrics from SongEval and Audiobox Aesthetics. These results show that critique-conditioned reward modeling reduces scoring error and provides an effective optimization signal for song generation. The project repository is available at https://github.com/WuqnEl/MuseCritic.
Read the full abstract ▾
All authors Jiabao Zhuang, Changhao Jiang, Hanchen Wang, Jiahao Chen, Zhixiong Yang, Zhenghao Xiang, Yifei Cao, Jiajun Sun, Hui Li, Ming Zhang, Tao Ji, Tao Gui, Qi Zhang, Xuanjing Huang
Computation and Language
LabelFusion-TS: Fusing Large Language Models, Transformer Encoders, and Financial Time Series for Monetary-Policy Stance Classification
Michael Schlee, Fabian Lukassen and Christoph Weisser
Financial text is produced and interpreted within a
market environment, yet financial text classifiers almost always receive text alone. We study whether financial time series are useful as an additional input on the task of classifying sentences from Federal Reserve communication as hawkish, dovish, or neutral. Our system, \lfts{}, extends the \lf{} architecture with this modality: a small voting network combines three independently trained components, a fine-tuned RoBERTa encoder, a prompted large
language model (LLM), and a fused ensemble of time-series
transformers over the
market series of the months preceding publication. Because only about a thousand annotated sentences are available for training, the RoBERTa encoder is first pre-trained on sentences annotated automatically by the LLM and only then fine-tuned on the human labels. Trained on Federal Open
Market Committee (FOMC) communication up to 2015 and evaluated on 2015–2022, the fused system achieves 70.2% weighted F1 – against 64.1% for the zero-shot LLM – and overtakes it with as few as 240 human-labelled sentences. We take this as initial evidence for
market time series as an input modality in financial text classification.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization
Ellen Su, Andres Potapczynski, Shikai Qiu et al. (+2 more)
Modern systems are increasingly expected to transfer across tasks not specified during training. What data facilitates generalization in these new, unanticipated settings? One hypothesis is that data with more structural information could contain shared
circuits and subprograms that could be recycled in a wider array of downstream settings. Epiplexity, a recently proposed measure of the structural information a compute-bounded learner can extract from data, provides a mechanism to reason about this relationship. In this paper, we show how to operationalize epiplexity as an online training signal for data selection and synthetic data generation. For selection, we fit scaling laws to the training loss curves of natural data domains to predict the expected epiplexity gain as a function of training
tokens, and use this signal to adaptively determine the sampling weights over domains during training. For synthetic data generation, we define a generator's reward as the change in learner epiplexity over a buffer of previously generated data and use REINFORCE policy gradients to guide the generator toward an epiplexity-maximizing distribution. In both cases, higher epiplexity predicts improved downstream performance on zero-shot and fine-tuning based tasks, supporting the hypothesis that data rich in structural information yield representations that transfer across domains.
Read the full abstract ▾
All authors Ellen Su, Andres Potapczynski, Shikai Qiu, Edward Hughes, Andrew Gordon Wilson
Computation and Language
Ripple-Pivot Search: Active Parallel Decoding for Diffusion Large Language Models
Yushi Ye, Xu Chen, Haoyun Jiang et al. (+7 more)
Diffusion Large
Language Models (dLLMs) have emerged as a competitive alternative to autoregressive
language models, offering the potential for substantially faster inference through parallel decoding. Existing parallel decoding schedulers typically commit positions only after they meet a per-position criterion, overlooking how early commitments may benefit subsequent decoding. We identify a ripple effect in dLLM decoding: proactively committing a mid-
entropy pivot position can induce a pronounced reduction in uncertainty across the remaining masked positions. This uncertainty reduction allows subsequent steps to unmask more
tokens in parallel, thereby accelerating the overall decoding process. To exploit the ripple effect, we propose Ripple-Pivot Search (RPS), a novel training-free decoding method that seeks mid-
entropy positions as promising candidate pivots (where to decode), and determines their
token assignment that yields the greatest downstream benefit via lookahead evaluation (what to decode). Across 3 dLLMs and 4
reasoning and code-generation
benchmarks, RPS achieves 4-10$\times$ wall-clock speedup over the standard decoder while preserving generation quality, and improves accuracy over the previous lookahead baseline by up to 5.49% while delivering higher throughput in most settings. When integrated with KV caching, RPS further achieves up to 18$\times$ wall-clock speedup over the standard decoder.
Read the full abstract ▾
All authors Yushi Ye, Xu Chen, Haoyun Jiang, Jinsong Lan, Haihong Tang, Xiangtao Li, Mingming Gong, Ivor Tsang, Yanfeng Wang, Jiangchao Yao
Computation and Language
Locating and Controlling Implicit Personalization in Large Language Models
Yueru Yan, Siqi Wu and Thai Le
Large
language models (LLMs) often shift their outputs in response to implicit demographic cues even when users never state a demographic identity. Previous work has documented this behavior, but the connection between these behavioral changes and the model's internal activations remains unclear. Using matched cued and neutral conversations across five LLMs, we establish that a localized internal activation signal tracks changes in recommendations, with correlations up to r=0.87. When multiple cues appear together, their internal signals largely combine, but the changes in output do not simply add up. We further show that removing the internal signal associated with one cue can suppress its influence, often more effectively than asking the model to ignore demographics via prompting, while largely preserving general
benchmark performance. However, the ability to selectively remove one dimension's influence while leaving co-present dimensions intact remains highly model- and attribute-specific. These results connect implicit personalization behavior to an internal signal that can be analyzed and causally controlled.
Read the full abstract ▾
Computation and Language
When the API Speaks the Wrong Language: Revisiting Post-Training for Multilingual Tool Use
Siddharth Chauhan, Thomas Butler, Abhishek Singhania et al. (+2 more)
The reliability of Large
Language Models (LLMs) for API calling degrades in
multilingual settings. A common failure occurs when a model selects the correct tool but generates argument values in an inconsistent
language, which we term Argument
Language Mismatch (ALM). Although semantically correct, such outputs are operationally invalid and not captured by standard API-calling metrics. We revisit post-training strategies for mitigating ALM and find that, in our
benchmark, supervised fine-tuning (SFT) provides a strong baseline, substantially improving argument
language consistency and end-to-end function call accuracy. Under consistent model selection, SFT achieves performance comparable to, and sometimes exceeding more complex reinforcement learning (RL) approaches. We further examine whether RL with structured, argument-aware rewards offers additional benefits. While methods such as Group Relative Policy Optimization (GRPO) can improve
language consistency and better preserve general
reasoning ability, these gains are incremental and most pronounced in generalization and multi-objective trade-offs. Overall, our results suggest that much of the performance in
multilingual API grounding can be achieved through careful supervised training, with RL providing targeted rather than fundamental improvements.
Read the full abstract ▾
All authors Siddharth Chauhan, Thomas Butler, Abhishek Singhania, Pankaj Porwal, Honey Gupta
Computation and Language
The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance
Shailja Thakur, Sungeun An, Chad DeLuca et al. (+1 more)
A
benchmark score comes from a single phrasing of each problem. That single phrasing is treated as if it stood for the whole space of ways the same problem could be asked, but it does not. We show that rephrasing a problem while keeping its meaning and answer fixed routinely flips a model's answer in both directions, so some failures become successes and some successes become failures. We call this drift. BenchDrift generates meaning-preserving variations of
benchmark problems along four axes, namely linguistic, referential, pragmatic, and structural, and measures how often, and why, correctness flips under each. Across eight models and three
benchmarks (GSM8K, MMLU, MATH-Hard), we observe that drift is large in both directions. Two findings stand out. First, phrasing sensitivity does not fade as models get better. Instead, it changes sign. Weak models gain more from rephrasing than they lose, while strong models lose far more than they gain. We find that the best models on a
benchmark are therefore the ones whose scores depend most on the wording they happened to be given. Second, the models largely agree on which rephrasings cost the most correct answers even though they differ in how much they drift, so fragility belongs to the rephrasing and not to the model. Furthermore, rephrasing breaks answers a model was confident about, whether the problem is made shorter or longer. Code and Data: https://github.com/IBM/BenchDrift/tree/demo-ui
Read the full abstract ▾
All authors Shailja Thakur, Sungeun An, Chad DeLuca, Hima Patel
Computation and Language · Cross-listed from cs.LG
LEMUR: Latent Entropy-aware Multimodal Unlearning via Visual-anchored Reasoning Redirection
Xinhao Zhong, Yuxia Qiao, Junhao Li et al. (+3 more)
Reinforcement-learning (RL) post-training equips
multimodal large
reasoning models (MLRMs) with exploratory chains of thought (CoT), substantially improving visual
reasoning. However, we find that this capability introduces a distinct
privacy vulnerability: even when a sensitive fact is successfully unlearned from the final answer, the model may still reproduce it in its
reasoning trace. This leakage is substantially more pronounced in natively RL-trained MLRMs than in their non -
reasoning base models, revealing a
privacy risk that existing unlearning methods are not designed to address. We show that RL-induced exploration leaves sensitive content with a distinctive
token-level
entropy signature that is largely absent from base models. Based on this observation, we propose LEMUR, a fully training-free, inference-time unlearning framework for natively RL-trained
multimodal models. LEMUR uses
entropy dynamics as a control signal to identify when sensitive
reasoning begins and when sanitization should stop. During this interval, it redirects the
reasoning trajectory through
entropy-modulated visual-anchor latent injection, replacing committed
tokens with sanitized, probability-weighted
embeddings re-grounded in the input image. Across diverse MLRMs, LEMUR consistently outperforms existing unlearning met hods in suppressing both
reasoning-trace and answer leakage, while better preserving non-sensitive utility and output fluency. These results demonstrate that RL-induced
entropy dynamics provide a distinctive signal for
privacy leakage and that exploiting this signal enables effective training-free unlearning for
reasoning-capable
multimodal models.
Read the full abstract ▾
All authors Xinhao Zhong, Yuxia Qiao, Junhao Li, Hao Fang, Yi Sun, Bin Chen
Computation and Language · Cross-listed from cs.AI
FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents
Yuhao Zhang, O. Ozan Koyluoglu, Thejas Venkatesh et al. (+4 more)
AI
agents are increasingly deployed for professional investment research, yet no
benchmark captures the complexity of the full investor workflow. Existing
benchmarks mainly target financial data extraction, a narrow slice that current models have largely saturated, while reference-based metrics and generic LLM-as-a-judge scoring fall short on the open-ended, long-form answers that real analyst queries demand. We introduce FrontierFinance, a fully open
benchmark of 220 expert-crafted queries and 11,543 source-attributed rubrics spanning six crucial use cases across the full investor workflow. FrontierFinance is both broader and harder than existing public finance
benchmarks. Evaluating frontier models and
agent systems under a common harness restricted to publicly available data, we find that the tool harness, not the model alone, strongly shapes quality and efficiency; that Samaya's in-house system leads at 56.0%, ahead of the strongest frontier model (Claude Fable 5, 49.2%) at roughly 2.2x lower cost; and that the best open-weight model (Kimi K3, 46.4%) nearly matches the best proprietary model at 4.5x lower cost. Screening & Discovery and Sector, Industry & Macro remain the hardest use cases across all systems, where even the best systems reach only 33% and 39%. We make the
dataset and grading code publicly available.
Read the full abstract ▾
All authors Yuhao Zhang, O. Ozan Koyluoglu, Thejas Venkatesh, Richard Diehl Martinez, Vishank Bhatia, Arash Alidoust, Ashwin Paranjape
Computation and Language · Cross-listed from cs.LG
Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-Based RL
Minglai Yang, Xinyu Guo, Zhengliang Shi et al. (+8 more)
Rubric-based reinforcement learning, in which an LLM judge scores responses against
prompt-specific criteria, has become a standard way to post-train
language models on tasks without verifiable answers. The rubric, however, is only a proxy for response quality, and a policy trained against it long enough learns to exploit the difference. We train models with GRPO on medical and science rubrics and grade their responses on out-of-distribution (OOD)
benchmarks throughout training with two judges: the training judge and a stronger OOD judge. The two scores first rise together and then diverge. The policy is learning to satisfy the training judge without writing better responses, which is a form of reward hacking. To mitigate it, we propose Rubric Dropout, which applies dropout to rubric criteria: at each training step, a random subset of each rubric's criteria is removed from the reward, so the rewarded subset of criteria is resampled at each step. All responses sampled for the same
prompt are scored on the same subset, so their rewards remain comparable, and evaluation always uses the full rubric. We also introduce Block Dropout, which drops semantically similar criteria together so that a near duplicate of a dropped criterion cannot keep rewarding the same behavior. Across both
benchmarks and models from 1.7B to 8B, dropout improves the gold judge's score on OOD
benchmarks by up to 3 percent on HealthBench-Hard and 7 percent on ResearchQA, and it reduces both the gap between the two judges and over-crediting. It keeps in-domain reward close to the baseline, and dropout rates of 30-50 percent work best. Block Dropout improves the gold judge's OOD score further at the same dropout rate. Changing which criteria are rewarded at each step is a simple, inexpensive way to make rubric rewards harder to exploit.
Read the full abstract ▾
All authors Minglai Yang, Xinyu Guo, Zhengliang Shi, Utkarsh Tyagi, Mian Zhang, Razvan Dumitru, Sunjie Hou, Yunzhong He, Daniel Yue Zhang, Dan Roth, Ying Liu
Computation and Language
Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editing
Tianci Liu, Zihan Dong, Tianchun Li et al. (+8 more)
Large
language models (LLMs) achieve remarkable performance across natural
language tasks, yet they are trained on static corpora and their knowledge quickly becomes outdated in a fast-changing world. This motivates knowledge editing (KE), which updates specific knowledge in an LLM without changing unrelated others. Recent works move from structured knowledge triples toward unstructured KE (UKE), where the edit is a free-form passage that may state multiple facts at once. Nonetheless, existing editors inject such a passage yet fail to use it: the edited model can recall the passage, but can neither answer atomic questions about its facts nor compose them into multi-hop
reasoning. We attribute this missing property, which we term composability, to editors' passive reliance on the fixed passage as the sole learning source. In response, we cast editing as a proactive self-
distillation from a privileged in-context state of the same model, which requires no external supervision. We further reveal that due to the novelty of the injected knowledge, the pre-edited model's own rollouts rarely cover it, which limits the effectiveness of pure on-policy
distillation. To close this gap, we propose HPSE, which builds a hybrid rollout that steps in to place missing facts onto the student's own trajectory precisely where its coverage fails, while staying on-policy elsewhere. We theoretically analyze HPSE's advantage over pure on-policy
distillation, and empirically establish its plug-and-play improvements across four LLM backbones and two KE editors under various scenarios.
Read the full abstract ▾
All authors Tianci Liu, Zihan Dong, Tianchun Li, Yi-Chung Chen, Qiming Cao, Xingchen Wang, Shiyang Wang, Zichen Miao, Linjun Zhang, Haoyu Wang, Jing Gao
Computation and Language
Semantic Lenia: Emergence of Homeostatic Solitons within the Semantic Space of Large Language Models
Yoshihiko Kayama
We introduce Semantic Lenia, an artificial life framework that transforms Large
Language Model (LLM) inference from a static optimization problem into a continuous, closed-loop dynamical system. By establishing a non-linear homeostatic feedback loop to dynamically balance semantic attraction and syntactic repulsion, we demonstrate the emergence of “Homeostatic
Solitons”-metastable semantic structures that actively resist repetitive crystallization. Our exhaustive parameter sweeps map a critical “Habitable Ridge” where applied steering forces balance the model's intrinsic syntactic inertia. This approach successfully maintains generative trajectories in a numerically sensitive critical regime, triggering profound abductive leaps without structural collapse, and reveals a capacity-dependent scaling trend in the syntactic inertia across different model sizes.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder
Huaxuan Wang, Huimin Wang, Ruiyu Zhang et al. (+2 more)
Recent advances in zero-shot text-to-
speech (TTS) have substantially improved
speech quality and voice cloning fidelity. However, many zero-shot TTS systems still depend on audio
prompt transcripts at inference time. This dependency limits cross-lingual voice cloning, since in-the-wild reference audio is often untranscribed. In this technical report, we present Confucius4-TTS, a
multilingual zero-shot TTS system that supports 14
languages and performs both intra-lingual and cross-lingual reference cloning without requiring transcripts of audio
prompts. Confucius4-TTS follows a two-stage architecture, consisting of text-to-semantic (T2S) and semantic-to-acoustic (S2A) modules. The LLM-based T2S module uses a learnable speaker encoder to extract timbre features from self-supervised
speech representations, and the conditional flow-matching S2A module converts the predicted semantic
tokens into mel-spectrograms. The same model also supports continuation cloning when a reference transcript is available. Confucius4-TTS is trained on large-scale
multilingual speech data. It achieves high intelligibility and speaker similarity on public
benchmarks. On the CV3-Eval cross-lingual
benchmark, Confucius4-TTS obtains an average WER of 3.73% across six directions. On our internal cross-lingual set, it achieves the best average overall rank in human evaluation among recent open-source and commercial systems. We release code, model checkpoints, and demos at https://github.com/netease-youdao/Confucius4-TTS.
Read the full abstract ▾
All authors Huaxuan Wang, Huimin Wang, Ruiyu Zhang, Yingjie Li, Yitao Duan
Computation and Language
Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study
Simone Mungari
As users increasingly turn to Large
Language Models (LLMs) for information and advice on political matters, particularly during election periods, the political preferences expressed by these systems have become a matter of public interest. Prior research has shown that interactions with LLMs can influence users' political attitudes and choices, raising questions about how these models themselves evaluate political actors. In this paper, we investigate whether and how LLMs express preferences toward political parties and political leaders. We introduce a systematic and reproducible
auditing framework in which multiple LLMs are prompted to evaluate parties and leaders across nine criteria. Rather than attempting to infer the models' "true" political beliefs, we focus on their observable behavior, examining consistency across evaluations, differences between models, refusal rates, and sensitivity to
prompt formulation. We further investigate how these evaluations vary when models are instructed to adopt different personas. We demonstrate the framework through an Italian case study, providing a systematic analysis of LLM-generated political evaluations on italian parties and leaders.
Read the full abstract ▾
Computation and Language
Easper: An Accessible ASR Pipeline for Language Documentation
Aso Mahmudi, Ting Dang, Ekaterina Vylomova et al. (+1 more)
Audio transcription is a critical bottleneck in
language documentation. While
multilingual Automatic
Speech Recognition (ASR) models like Whisper offer solutions, field linguists often lack the expertise to utilise them. We present Easper, an open-source, no-code workflow enabling linguists to iteratively fine-tune ASR models via cloud resources directly from ELAN annotations. Deploying ASR also raises a cold start problem: deciding which recordings to transcribe first to bootstrap an accurate model. Using Easper, we evaluate transcription prioritisation strategies on three Vanuatu
languages (Bislama, Nafsan, Nguna). We fine-tune models by recording session, comparing Character Error Rate trajectories when prioritising acoustic cleanliness versus linguistic richness. We demonstrate that prioritising lexically rich narratives and increasing acoustic-phonetic repetition, even in noisy environments, leads to faster improvements in transcription quality.
Read the full abstract ▾
All authors Aso Mahmudi, Ting Dang, Ekaterina Vylomova, Nick Thieberger
Computation and Language
Learning to Persuade Exposes How Easily LLMs Abandon Correct Beliefs
Nimet Beyza Bozdag, Emre Can Acikgoz, Gokhan Tur et al. (+1 more)
Persuasion is a core dynamic of natural
language communication, shaping how large
language models (LLMs) update beliefs, resolve disagreements, and reach decisions. As LLMs increasingly debate, advise, and think collaboratively with humans and each other, resistance to harmful persuasion becomes a core requirement for reliable behavior. Yet we show that this requirement is far from met: a single targeted persuasive argument is enough to collapse model accuracy to near zero, even when the argument is factually false. We formalize this threat as
adversarial persuasion and introduce an
adversarial reinforcement learning framework that trains persuader
agents to change a target model's answer in a single interaction. First, we show that optimizing persuasion strategies through trial and error exposes vulnerabilities that static prompting misses: RL-trained persuaders raise persuasion success from approximately 24% to over 93% against the training-time persuadee. Second, we find that these learned strategies transfer to unseen models, achieving 83% attack success on Qwen-14B, 79% on Llama-3.1-8B, and 25% on GPT-4o-mini. Third, we demonstrate that a curriculum that bootstraps on more persuadable open-weight models before targeting harder models further increases GPT-4o-mini attack success from 25% to 38%. Moreover, our results reveal that optimized persuaders increasingly rely on credibility-based tactics, including fabricated citations and false authoritative evidence. Together, these findings expose a critical weakness in current LLM
agents: even when they initially reason correctly, they can be steered toward false conclusions by optimized natural
language influence. This positions persuasion robustness as a necessary safety criterion for multi-
agent and human-AI decision-making systems.
Read the full abstract ▾
All authors Nimet Beyza Bozdag, Emre Can Acikgoz, Gokhan Tur, Dilek Hakkani-Tür
Computation and Language · Cross-listed from eess.AS
Robust Multi-Tier Infant-Centered Audio Understanding with Whisper via Structured Speaker Conditioning
Xulin Fan, Jialu Li, Mohammad Nur Hossain Khan et al. (+4 more)
Recent advances in model design and self-supervised audio representations have improved
speech and audio understanding, yet infant-centered naturalistic recordings remain challenging due to limited labeled data, low signal-to-noise ratio, and cross-family domain shifts. We present a family-conditioned, multi-tier audio tagger that combines a LoRA-finetuned Whisper encoder with a lightweight, target-speaker-aware
Transformer for long-context inference and framewise prediction across tiers. To improve temporal coherence, we incorporate a simple sequence-level smoothing loss, and to enhance robustness across households, we introduce a factorized speaker-
token design with a shared tier
token and a learned family-specific offset, reducing family bias and promoting generalizable representations. Together, these choices enable efficient and effective infant-centered audio tagging of daylong audio recordings in home environments.
Read the full abstract ▾
All authors Xulin Fan, Jialu Li, Mohammad Nur Hossain Khan, Kexin Hu, Bashima Islam, Mark Hasegawa-Johnson, Nancy L. McElwain