Computation and Language
Anuvani-LID: Robust and Calibrated Spoken Language Identification for Indian Languages
Stephen Fernandes and Narayan Choudhary
Every multilingual
speech pipeline begins with a
language decision, an error there propagates silently through recognition, translation and routing. In India it spans 22 scheduled
languages across four
language families, heard over short, noisy, narrowband and code-mixed audio. Foundational
speech models increasingly present
language identification (LID) as an inherent latent, generalized capability. Autoregressive decoders in
speech models emit a special
language ID
token, and modern LLMs with audio capabilities can name the
language when prompted. We test this assumption on identical clips and find that whisper and similar
speech models has
tokens for only 15 of the 23
languages, on one second inputs Gemini 3.5 Flash reaches 0.52 macro-F1 and GPT-audio 0.14. We present Anuvani LID, a dedicated 115M parameter LID conformer model trained on 54,643 hours of Indian
speech spanning 22 Indian
languages and Indian accented English. Along with designed and controlled studies on 102 condition
benchmark with speaker and corpus held out tests. Across 37 conditions, it reaches 0.914 mean macro-F1, 5.2 points above Gemini 3.5 Flash and ahead of 14 other systems. It is 24 points ahead of Gemini at one second, scores 0.897 on all 23
languages where no other system exceeds 0.713, and is calibrated to an ECE of 0.008. Reliable Indian LID still requires a robust, dedicated and calibrated model.
Read the full abstract ▾
Computation and Language
Repairing Speech Access After Text Updates with Frozen Affine Fields
Ming-Hao Hsu, Yuxiang Wang, Yingda Shen et al. (+1 more)
Speech-capable
language models inherit most of their knowledge from text-rich
language backbones, and continued model maintenance is likewise dominated by text updates. Knowledge acquired through a text update can nevertheless remain difficult to access when the same query is spoken. We formulate this mismatch as post-update
speech access repair, where the knowledge-bearing checkpoint stays frozen and a detachable operator repairs
speech-only access in one model pass. We introduce Affine Fields for Interface Repair across Modalities (AFFIRM), which first learns and freezes a base correction for
speech hidden states, then adds query-specific corrections with zero mean over the operator-training queries. Paired
speech and text states provide the coordinates that condition these corrections on each query. On balanced synthetic-
speech PopQA, paired comparisons do not establish a clear advantage for AFFIRM over global SALSA steering or low-rank ReFT interventions under the same frozen single-state constraint. On recorded SLUE-SQA-5 questions with text passages, a full-scale PopQA-fitted AFFIRM operator transfers without refitting and improves ROUGE-L by 0.0941 over its paired No Repair baseline. Low-rank ReFT interventions also transfer positively, whereas the matched SALSA variant falls 0.0707 below its paired baseline. These results show that text acquisition and
speech access can diverge after an update and should be verified and repaired separately.
Read the full abstract ▾
All authors Ming-Hao Hsu, Yuxiang Wang, Yingda Shen, Zhizheng Wu
Computation and Language
Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation
Joel Dag, Nikit Srivastava, Hamada M. Zahera et al. (+1 more)
Multilingual models share capacity across
languages, so adding
languages can hurt performance on existing ones. Mixture-of-experts models expand this capacity with little additional per-
token computation. Given this extra capacity, does adding experts reduce what each additional
language costs? To answer this question, we propose a repetition-aware joint scaling law relating number of active parameters, training
tokens per
language, number of experts, and
language coverage. Fitted on 26,300 per-
language loss observations from 64 pretrained configurations, our law reduces prediction error on held-out configurations by 28% relative to the strongest evaluated baseline. At matched number of active parameters and training
tokens per
language, 32 experts reduce the loss penalty of expanding from 12 to 200
languages by 83% compared with a dense model. Our law further shows that broader
language coverage favors more experts across a wider range of compute budgets, guiding capacity allocation as multilingual models scale.
Read the full abstract ▾
All authors Joel Dag, Nikit Srivastava, Hamada M. Zahera, Axel-Cyrille Ngonga Ngomo
Computation and Language
Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation
Zahraa Al Sahili, Ioannis Patras and Matthew Purver
Large
language models (LLMs) are increasingly deployed across
languages, yet their safety and
reasoning capabilities remain predominantly evaluated in English. Translating existing
benchmarks offers a scalable approach to multilingual evaluation, but preserving task-specific meaning across
languages presents significant challenges, particularly for harmful requests, mathematical
reasoning, and conversational interactions. We introduce a
reasoning-guided translation framework for constructing multilingual evaluations designed to preserve the structure and intended meaning of diverse
benchmark tasks. Our approach separates translation strategy selection from large-scale generation, combining
reasoning-guided configuration selection with task-aware validation, targeted repair, and independent quality screening. Using this framework, we develop a source-aligned evaluation collection spanning seven safety, conversational, and mathematical
reasoning resources across 11
languages. The resulting collection achieves 99.94% coverage of intended item–
language pairs in the two subsets requiring repair. Evaluations of open-source LLMs reveal distinct
language-dependent vulnerabilities, including inconsistent refusal behaviour and substantial
reasoning degradation in lower-resource
languages. These findings highlight the importance of task-sensitive
benchmark construction and systematic quality assessment for understanding multilingual model behaviour beyond English.
Read the full abstract ▾
Computation and Language
What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds
Caixin Kang, Ke Wang, Yuchuan Wu et al. (+5 more)
Effective social interaction requires more than fluent responses:
agents must reach plans that fit the participants' needs and circumstances. Existing evaluations of social
agents mainly rely on LLM judges that rate the conversation, without checking the resulting plan. We introduce HICO-Bench, a
benchmark of 180 everyday coordination scenarios, such as resource allocation, scheduling, and authorization, in which two
agents with private information converse and then independently record the plan they agreed on. A program checks Feasibility (the two records match and satisfy both participants' private conditions), and an LLM judge checks Integrity (the conversation supports the recorded plan); passing both is a Verified Success. Each scenario has two versions that share the same public context but differ in one private condition, so the correct plan differs between them. Across thirteen LLMs, 36% of conversations rated highly for goal completion fail Verified Success. To learn from outcomes that arrive only at the end of a conversation, we propose Paired-Credit Policy Optimization (PaCPO). After each turn, PaCPO asks the
agent for a tentative plan, checks it in both versions of the scenario, and rewards the turns after which the tentative plans fit their respective versions. PaCPO raises Qwen3-8B's Verified Success from 20.4% to 28.3%.
Read the full abstract ▾
All authors Caixin Kang, Ke Wang, Yuchuan Wu, Xiaoqiang Wang, Ling Li, Bingxue Qiu, Yongbin Li, Yoichi Sato
Computation and Language
When Occupation-derived Skills Don't Help: On the Limits of Skill-Augmented Career Path Prediction
Ege Süalp, Elena Senger, Toine Bogers et al. (+1 more)
Skills are widely assumed to play a central role in career transitions, motivating their use in career path prediction (CPP) systems. In this work, we critically examine whether explicit skill representations provide complementary signal beyond text-based job representations. We evaluate multiple strategies for incorporating skills into CPP models, including score fusion and neural architectures, under two conditions: predicted skills and ground-truth ESCO annotations. Surprisingly, across two
benchmark datasets, none of these settings yield substantial improvements over strong text-only baselines. Even when using ground-truth skills, gains remain marginal and inconsistent. Through detailed analysis, we identify key factors underlying this result: high redundancy between textual and skill representations, limited discriminative power of ESCO's taxonomy, and representational collapse in skill
embeddings. Furthermore, we find that skill prediction difficulty correlates with overall sample difficulty, limiting the potential for complementary signals. Our findings challenge the common assumption that explicit skill modeling inherently benefits career prediction and suggest that dense text representations already capture most skill-relevant information.
Read the full abstract ▾
All authors Ege Süalp, Elena Senger, Toine Bogers, Barbara Plank
Computation and Language
Cross-Model In-Context Neurofeedback Reveals (Limited) Activation Control in LLMs
Davide Baldelli, Robyn Kozierok, Mirko Bronzi et al. (+1 more)
Probes on activation vectors are widely used to detect misbehavior in
language models, but their success relies on the assumption that models cannot control what their activations encode. Humans can learn, through neurofeedback, to control the firing of even single
neurons, and recent studies investigated whether this ability extends to
language models, reaching conflicting conclusions. No experimental design so far disentangles competence at the task from genuine access to one's own activations. We fill this gap with in-context neurofeedback sessions, in which a "subject" model is asked to maximize (or minimize) a score derived from its activations, shown back to it at every turn. In our first experiment, the sentences are generated either by the subject or by a second "writer" model, while the score always derives from the activations of the subject. We test ten open-weight models from five families, with sizes from 9B to 32B. Some models move their own scores considerably, but no more than a writer of comparable competence moves them. Is this because the task requires no privileged access, or because the subject controls its activations while processing the sentences, whoever wrote them? To answer this, we prefill a fixed sentence as the subject's reply at every turn and ask it to increase or decrease the score. Three out of ten models show small but consistent control effects. We further ask whether models can control their activations to the point of fooling an activation monitor. We make models repeat sentences a monitor would flag, such as a deceptive answer for a deception monitor, and ask them to raise or lower the monitor's score. No model learns to lower it, but one model can raise it. In-context control of a model's own activations might not yet be a threat to activation monitors, but it exists in a few models and needs to be tracked as models evolve.
Read the full abstract ▾
All authors Davide Baldelli, Robyn Kozierok, Mirko Bronzi, Damiano Fornasiere
Computation and Language
TreeSpark: Learning to Draft the Trees You Verify for Speculative Decoding
Jihao Zhao, Qianyu Jiang, Shuaishuai Zu et al. (+6 more)
Parallel drafters such as DFlash and DSpark reduce proposal latency, but early mismatches limit how much of each block can be accepted. Retaining alternatives introduces parent states and candidate choices that reference-conditioned training does not directly supervise. We introduce TreeSpark, a unified training and inference framework extending the semi-autoregressive DSpark architecture. It constructs parent-conditioned candidate trees from one backbone pass and replays their expansion,
pruning, and budgeted packing during training. Survival negative log-likelihood weights retained edges by reference-derived path mass, while a capped coverage hinge promotes target-important alternatives near local selection boundaries. Both objectives reuse reference-chain target distributions without additional target forward passes. We evaluate greedy decoding with Qwen3-4B and Qwen3-8B on nine general
benchmarks and six writing domains. Across targets and suites, TreeSpark improves mean acceptance length over DSpark by 23.4% to 31.6% and mean single-request throughput by 15.1% to 25.0%. Batched Qwen3-4B throughput improves by 12.6% to 17.2%. At fixed
attention capacity and inference policy, the proposed supervision further improves mean acceptance length by approximately 6.4% and mean system throughput by 4.2% to 5.0% on Qwen3-4B.
Read the full abstract ▾
All authors Jihao Zhao, Qianyu Jiang, Shuaishuai Zu, Pengfei Li, Guanghuan Fang, Ben Feng, Biao Qin, Zhibin Wang, Baochuan Yang
Computation and Language · Cross-listed from cs.LG
RM-OPD: On-Policy Distillation for Generative Reward Models
Qinglin Su, Licheng Pan, Zhen Fang et al. (+5 more)
Reinforcement Learning on Mixed-domain preference data (Mix-RL) has become the mainstream approach for training general-purpose Generative Reward Models (GRMs). However, Mix-RL suffers from two critical bottlenecks: the reward sparsity induced by outcome rewards, and the cross-domain interference arising from jointly optimizing heterogeneous capabilities, which together give rise to imbalanced capability acquisition and pervasive reward hacking. Inspired by the success of On-Policy
Distillation (OPD) in multi-domain capability integration, we introduce RM-OPD (Reward Modeling via On-Policy
Distillation), a two-stage framework that separates specialist acquisition from capability integration. First, we cultivate domain-specialized teacher models via single-domain GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; then we consolidate these teachers into one student model through (i) routed on-policy
distillation, which provides dense signed
token-level supervision on student-generated critiques; (ii) domain contribution reweighting, which reweights
token guidance toward target domain shares, preventing specific domains from dominating optimization; and (iii) optional criteria context
distillation, which conditions teachers on domain-specific evaluation criteria to sharpen
token-level supervision. Across multiple RM
benchmarks and two base models, RM-OPD consistently outperforms vanilla Mix-RL and other strong baselines, achieving overall superior performance and exhibiting an emergent "teacher-surpassing" effect. These results establish RM-OPD as a promising paradigm for building generalist GRMs.
Read the full abstract ▾
All authors Qinglin Su, Licheng Pan, Zhen Fang, Jiakang Wang, Qingnan Ren, Yunsheng Lu, Yan Gao, Zhixuan Chu
Computation and Language · Cross-listed from cs.SD
MultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue
Puneet Mathur and Dinesh Manocha
Full-duplex
speech systems must decide when to enter, hold, or relinquish the conversational floor, even as multiple speakers overlap or backchannel without taking it. Existing evaluations study dyadic turn-taking or end-to-end
agent behavior, but lack a unified, controlled evaluation of genuine floor transitions against confusable non-events in multi-party settings. We introduce MultiPartyTurnBench (MPTB), a
benchmark of 240 synthetically generated, channel-separated conversations spanning two to five speakers and six interaction types. MPTB evaluates two complementary tasks: End-of-Turn (EOT) detection distinguishes genuine turn completion from temporary holds, while Interruption (INT) detection distinguishes successful floor takeovers from non-floor-taking
speech. Exact event-conditioned annotations enable causal evaluation using false-positive-constrained recall and signed latency. Across acoustic, semantic, and turn-projection baselines, we find a stark asymmetry: learned systems achieve 82.5-92.6% EOT recall at 4.3-11.8% false-positive rates, while INT detection suffers from missed takeovers or excessive false activations. This asymmetry extends to five full-duplex
agents which achieve 86.1-95.8% balanced accuracy on turn entry, yet perform near chance (46.9-54.7%) on selective interruption handling. Gemini-3.8-Live yields to 97.7% of backchannels, whereas MiniCPM-o-4.5 yields to only 1.4% of genuine interruptions. These findings establish selective floor control as a distinct, unresolved challenge for reliable multi-party voice interaction.
Read the full abstract ▾
Computation and Language
Gap Detection: Mitigating Homogenization through Demand and Defensibility
Xucheng Yu, Haibo Jin, Xiaoqi Wang et al. (+2 more)
Language models often produce semantically similar answers to open-ended questions, even when many plausible alternatives exist. Generating useful alternatives requires identifying both what the query calls for and what existing responses already cover. We introduce Gap Detection, a framework that searches for relevant but unoccupied directions in a shared text
embedding space. Its objective, GapScore, combines query relevance (Demand) with dissimilarity to the nearest existing item (Defensibility). The framework optimizes this product on the unit sphere, maps candidate directions to concept words with a
language model, and uses task-specific filtering and constrained prompting to turn those concepts into text. On five curated topics and 90 INFINITY-CHAT queries, the reported mean relative reductions in output Collision are 12.8% and 10.8%, respectively. We also instantiate the framework for generative engine optimization over four product
markets and for iterative synthetic-data augmentation. In the latter setting, with 100 training examples per method, Gap Detection reduces Collision from 0.324 to 0.298 relative to undirected accumulation and increases the Vendi Score from 27.16 to 29.20; the accuracy difference is one example on a 40-example test set. Component ablations expose the trade-off between semantic separation and relevance, while also showing that better direction scores do not uniformly yield better downstream metrics. These results support explicit gap search as a practical mechanism for diversifying content, with task validity checked separately from
geometric novelty.
Read the full abstract ▾
All authors Xucheng Yu, Haibo Jin, Xiaoqi Wang, Runze Kong, Haohan Wang
Computation and Language · Cross-listed from cs.AI
LLM-Based Policy Compliance Verification of Loan Applications
Florentina Voboril
Banks rely on extensive internal policy frameworks to ensure that corporate lending decisions comply with regulatory requirements and internal risk standards. Reviewing loan applications against these policies is a largely manual and knowledge-intensive process, which can lead to overlooked policy violations or exceptions. This report presents a prototype of an LLM-based policy checker that automatically verifies loan applications against internal lending policies. The proposed approach follows a two-step architecture. First, compliance-relevant rules are automatically extracted from policy documents and converted into structured, machine-readable representations. Second, loan applications are evaluated against the extracted rules to identify policy exceptions, non-compliant findings, missing information, and cases requiring manual review. Evaluation results demonstrate that the rule extraction process is highly stable across multiple runs and consistently captures rules identified as critical by a domain expert. When evaluated on 11 real loan applications with known policy exceptions, the system successfully detected the majority of these exceptions and identified additional compliance issues that had previously been overlooked during manual reviews. While the approach produces false positives and cannot fully replace human judgement, the results indicate that LLM-based policy compliance verification can provide valuable support in the loan application verification process by highlighting potential compliance risks.
Read the full abstract ▾
Computation and Language
Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility
Salima Lamsiyah, Adelaide Danilov, Oleksandr Marchenko Breneur et al. (+1 more)
Sparse autoencoders (SAEs) have emerged as a promising approach to mechanistic interpretability by decomposing
language-model activations into sparse, potentially interpretable features. However, differences in design assumptions, evaluation objectives, and experimental settings complicate conclusions about their effectiveness. We present a critical survey of 138 studies on SAEs in natural
language processing. Our contributions are fourfold: (1) we propose a task-grounded taxonomy connecting dictionary design, feature operations, and NLP functions; (2) we examine how sparsity allocation, geometry, data coverage, and inference affect feature recoverability; (3) we assess evidence for semantic interpretability, mechanistic faithfulness, and behavioral utility; and (4) we synthesize findings across multilingual processing,
retrieval,
reasoning, classification, and behavioral regulation, highlighting the roles of objectives, baselines, supervision, and intervention pipelines. Through comparative analyses of published results, we reconcile apparent disagreements, identify unresolved questions, and propose directions for more rigorous evaluation. An anonymous research register https://anonymous.4open.science/r/sae-research-register-C331/register.html accompanies the survey.
Read the full abstract ▾
All authors Salima Lamsiyah, Adelaide Danilov, Oleksandr Marchenko Breneur, Aria Nourbakhsh
Computation and Language
Why Generate a Click? Plug-in Decision Heads for Web Agents
Chengguang Gan, Yunhao Liang, Hanjun Wei et al. (+3 more)
Web
agents built on
vision language models decide every action by generating text such as click(12), although most of that text encodes a choice among a few action types and a few dozen marked elements. We read these choices directly from the hidden states that the generator computes while it reads the goal and the screenshot. Plug-in decision heads on a frozen generator predict the action type, pool the visual
tokens inside each numbered box to pick the target, and point at a listed text option or at a WRITE slot that leaves the writing of new text to the generator. A confidence cascade executes the heads' action when they are sure and otherwise lets the generator decode from the KV cache of the same forward pass. With Qwen3-VL 4B and 8B generators trained on 213k web steps, a strict threshold keeps the cascade between 0.3 points below and 1.4 points above the generator on seven offline test sets while the heads decide 12 to 64% of the steps. A looser threshold raises step success on unseen Mind2Web websites by 3.1 and 1.8 points, costs up to 2 points on GUIAct and MiniWoB++, and lifts WebArena task success from 12.5 to 13.8% (4B) and from 13.6 to 14.6% (8B) with less than half of the generated
tokens. Training the heads jointly with the generator lowers the generator's own step success by up to 4.9 points; heads on a frozen generator avoid this cost.
Read the full abstract ▾
All authors Chengguang Gan, Yunhao Liang, Hanjun Wei, Qinghao Zhang, Zhixi Cai, Shiwen Ni
Computation and Language
Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training
JiWoo Hwang, Lukas Edman and Alexander Fraser
We investigate two aspects of multilingual pre-training with limited data, namely how
language exposure is distributed across English, Dutch, and Chinese, and how that exposure is ordered during training. In particular, we compare balanced multilingual exposure to
language-dominant exposure, where one
language is exposed with more training data than the others. We also compare mixed training, where multilingual training data is used throughout training, with curriculum training, where monolingual exposure of the dominant
language is followed by balanced multilingual exposure. Our results show that multilingual BabyLM performance varies with both the amount and order of
language exposure. Mixed training outperforms curriculum learning across all 15 matched configurations, indicating that simultaneous multilingual exposure is more effective than staged exposure in our setting. Several
language-dominant mixed training configurations achieve higher scores than the balanced configuration, suggesting that a balanced distribution is not necessarily the highest-performing configuration in our setting. Interestingly, the Dutch-primary configurations achieve the best overall performance.
Read the full abstract ▾
Computation and Language
AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining
Yunhao Wang, Feng Zhang, Tinghao Yu et al. (+2 more)
Synthetic data is increasingly used in
language-model pretraining. Existing knowledge-corpus synthesis pipelines, including WRAP and Nemotron-CC, usually take a source passage or document segment as input and generate rewritten or QA-style samples from that text. This can improve surface form and
prompt diversity, but it does not explicitly require fine-grained source facts – especially secondary entities, list/table entries, and precise temporal attributes – to appear in the synthesized corpus. We propose AFCS (Atomic-Fact-Conditioned Synthesis), a pipeline that first extracts source-supported atomic facts and then uses each retained fact to drive generation. AFCS uses structure-aware extraction over the document tree, source-grounded verification, local context expansion, and multi-format generation. In continued-pretraining experiments with OLMo-3-7B and a 32B base on the same FineWiki source corpus, AFCS outperforms WRAP and Nemotron-CC on SimpleQA, SimpleQA Verified, PopQA, and HotpotQA. Granularity ablations show that fact-level synthesis improves over chunk-level, paragraph-level, and entity-level synthesis; and a 100B-
token mid-training integration improves factual
benchmarks and raises the 12-task general
benchmark average by +1.09.
Read the full abstract ▾
All authors Yunhao Wang, Feng Zhang, Tinghao Yu, Xing Wu, Jiang Zhou
Computation and Language · Cross-listed from cs.AI
Designing Textual Interfaces To Enhance Llm Reasoning On Audio-Visual Question Answering
Yi Wang, Tianzi Wang, Tong Lei et al. (+2 more)
Automatic
prompt optimization (APO) has shown strong effectiveness for text-based LLMs and LLM systems, but its application to audio-visual understanding systems requires suitable textual interfaces through which
multimodal reasoning can be exposed to text-based optimizers. We investigate how the design of textual interfaces affects
prompt optimization for
multimodal reasoning, on the audio-visual question answering task. We compare three interface designs, including a direct inference system consisting of a single omni-modal large
language model (OmniLLM), a captioner-reasoner system, and an agentic architecture in which a primary
agent orchestrates subagent OmniLLMs. Experiments with Qwen3-Omni and Gemini show that direct
prompt optimization provides model-dependent gains; the captioner-reasoner system and the agentic system expose more interpretable optimization interfaces but are harder to optimize, where error analysis of cases affected by optimization is conducted. Our study highlights textual interface design as an important factor in making
prompt optimization effective for omni-modal
reasoning systems. Our implementation details are specified in GitHub repo.
Read the full abstract ▾
All authors Yi Wang, Tianzi Wang, Tong Lei, Dahan Wang, Peter Bell
Computation and Language
Which Knowledge Should LLMs Trust? Uncovering a Generalizable Steering Direction Across Knowledge Conflicts
Yibin Lou, Yi Hu, Jianhui Chen et al. (+3 more)
Large
Language Models (LLMs) frequently encounter conflicts between internal parametric knowledge and external contextual knowledge, yet the mechanisms governing this selection remain unclear. We show that the choice between context and parametric knowledge is linearly represented and can be controlled along a single direction in the model's residual stream. We first demonstrate that this direction generalizes across factual-conflict settings, including context formats, knowledge categories, and
datasets, while consistently shifting behavior between parametric and contextual answers. We then transfer a direction learned from simple factual conflicts to three more complex task families: RAG,
memory, and rule following, where models must acquire information from retrieved documents, interaction histories, or explicit rules. Across four model families and multiple transfer settings, steering consistently improves the use of
prompt-provided information over the unsteered model. These results suggest that the direction captures a transferable preference for external knowledge rather than a mechanism specific to factual conflicts.
Read the full abstract ▾
All authors Yibin Lou, Yi Hu, Jianhui Chen, Chenpeng Wang, Muhan Zhang, Liangming Pan
Computation and Language
What Is a Clarification? Estimating Model-Dependent Input Ambiguity
Jingqin Tang, Tim Tomov, Sebastian Schmidt et al. (+2 more)
Ambiguous inputs can admit multiple interpretations, each potentially yielding a different answer. Existing work largely treats this ambiguity as a property of the input, yet for a particular
language model, only some alternatives may be known and answerable. We therefore distinguish world ambiguity, capturing all alternatives admitted by an input, from model ambiguity, capturing those accessible to the responding model. To characterize these alternatives, we formalize natural-
language clarifications through three desiderata: uniqueness, faithfulness, and answer equivalence. We show that these desiderata induce a one-to-one correspondence between valid answers and equivalence classes of clarifications, implying that any interpretation can be represented either by its answer or by the clarifications that uniquely express it. We leverage this correspondence to introduce \circleclar{}, a black-box method that estimates the set of answers available to a model by checking consistency along an answer–clarification–answer cycle. Across open-domain QA and executable Text-to-SQL, \circleclar{} achieves strong recovery of these answer sets. In QA, the recovered support yields the strongest overall performance in model-dependent ambiguity detection and enables effective selection among answering, clarifying, and abstaining. Together, these results demonstrate the practical value of a model-dependent view of ambiguity and its characterization through an answer-clarification correspondence.
Read the full abstract ▾
All authors Jingqin Tang, Tim Tomov, Sebastian Schmidt, Dominik Fuchsgruber, Stephan Günnemann
Computation and Language
FACES: Fine-grained Gated Semantic Alignment for Multimodal Emotional Support Conversation
Hang Ren, Peidong Wang, Yichen Gao et al. (+4 more)
In recent years, with advances in
multimodal interaction technologies and the growing demand for intelligent emotional support,
Multimodal Emotional Support Conversation (MESC) has emerged as an important task. However, existing systems still suffer from shallow responses and inappropriate support strategies due to weak semantic feedback, cross-dimension reward compensation, and coarse
reasoning supervision. To address these challenges, we propose FACES, a fine-grained gated semantic
alignment framework for MESC. Specifically, FACES integrates multi-source rewards to construct multidimensional semantic and functional
alignment signals, employs a non-compensatory gated fusion mechanism to reduce cross-dimension reward offset, and leverages structured Chain-of-Thought (CoT) anchors to perform
token-level credit assignment. Under a controlled response-length protocol, FACES improves the average MLLM-based evaluation score from 3.61 to 4.04 and achieves an 80.22% pairwise win rate over SOTA. Human evaluation under the same protocol yields a 75.3% preference rate. Results on three external
datasets further support transferability. The code and
dataset are available at https://github.com/OCT-ARR/arr.
Read the full abstract ▾
All authors Hang Ren, Peidong Wang, Yichen Gao, Wen Zhang, Jiayang Yu, Xiaocui Yang, Shi Feng
Computation and Language · Cross-listed from cs.SE
LintEval: Benchmarking LLMs on Linter-Guided File Refactoring
Tongwei Deng, Jinyu Zhu, Yue Cao et al. (+3 more)
Linters are widely used in code maintenance and real-time IDE editing to surface lint alerts related to performance, style, error-prone, and other patterns in source files. However, existing code
benchmarks do not explicitly evaluate linter-guided file refactoring, where models must reduce lint alerts in source files with correct initial functionality while satisfying functionality preservation. We introduce LintEval, a multilingual
benchmark for linter-guided file refactoring across C++, Python, and Java, comprising 13,061 programs with correct initial functionality and 171,662 lint alerts extracted using
language-specific linters. LintEval jointly evaluates functionality preservation, lint alert reduction, new alert avoidance, and edit minimality. Experiments reveal a persistent trade-off: closed-source commercial models reach up to 90.4% Reduction@P but achieve only 67.1%-68.3% Pass@1, while several open-source small models show low or negative Reduction@P. A controlled fine-tuning study further shows that supervision format matters: refactoring-oriented SFT raises Qwen2.5-Coder-7B Reduction@P from 24.7% to 76.0%. LintEval enables controlled evaluation of models that improve source files under functionality preservation.
Read the full abstract ▾
All authors Tongwei Deng, Jinyu Zhu, Yue Cao, Chenxu Wang, Yanzhen Zou, Bing Xie
Computation and Language
A Strong Harness Narrows the Model Gap: A Comparative Evaluation for Legislative Document-to-XML Conversion
Palvika Bansal, Rittika Jindal and Ruichong Wang
Legal-content reproduction requires matching the source exactly in section hierarchy, verbatim text, citations, and amendment spans; a dropped citation or mislabeled amendment carries into the products lawyers rely on. We report an industrial pipeline converting U.S. state legislation into strict, schema-constrained XML, and hypothesize that its accuracy comes from harness engineering, the deterministic tooling built around a model, more than the model itself. Our controlled ablation crosses three harnesses (zero-shot, a single tool-using
agent, and a decomposed multi-
agent pipeline) with three Claude models on 153 gold bills from two jurisdictions. The harness narrows the model gap: the weakest model in the full pipeline beats the strongest model alone (0.948 versus 0.806, paired bootstrap p less than 0.001). Because the two agentic harnesses share identical tools, their gap comes from orchestration: decomposition plus the persisted state and per-
agent context it brings. Across four scored dimensions, the model spread at zero-shot (0.68 to 0.81) narrows to 0.95 at the full pipeline; the residual sits almost entirely in citation normalization, where the canonical identifier is retrieved from an external authority rather than generated from the weights, so a deterministic lookup tool supplies the exactness that a larger model does not.
Read the full abstract ▾
Computation and Language
Gray-Box Hallucination Detection in LLMs with Token Identities
Shaked Goldshtein, Yitzchak Grunbaum, Fabrizio Frasca et al. (+2 more)
Large
language models often produce fluent but factually incorrect output. Gray-box detectors identify such
hallucinations from the
token-probability statistics emitted during generation, avoiding the need for white-box access to hidden states that frontier models do not provide. These detectors, however, typically rely only on probability-derived representations and ignore the identities of the candidate
tokens. They can therefore observe how uncertain the model is, but not what it is uncertain about. Consequently, they cannot distinguish between a model that is torn between "big" and "small" and one that is torn between "big" and "huge" when the two pairs receive the same probability distribution. We introduce GURI (Gray-box Uncertainty Representations with Identity), a lightweight gray-box
hallucination detector that combines candidate-
token probabilities with per-
token vector identifiers. This exposes which alternatives the model considered while preserving the accessibility and transferability of the gray-box setting. Across three open-weight models and three question-answering
benchmarks, GURI increases mean test ROC-AUC from 72.09% to 79.09% over a probability-only baseline. On the closed-source GPT-4.1 model, mean ROC-AUC increases from 67.39% to 71.84%. Among the three open-weight models, GURI outperforms transferred probability-only baseline in 17 of 18 directed cross-model transfers without fine-tuning. For the three open-weight models, cross-
dataset transfer between HotpotQA and TriviaQA with the model held fixed improves mean ROC-AUC over transferred LOS-Net by 3.76 points.
Read the full abstract ▾
All authors Shaked Goldshtein, Yitzchak Grunbaum, Fabrizio Frasca, Guy Bar-Shalom, Haggai Maron
Computation and Language
SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges
Sebastian Nagl
Law students are examined through open-text answers: case solutions, essays and memos that apply the law to a set of facts. Such answers are expensive to grade, human graders disagree widely on them, and large
language models (LLMs) now offer to grade them at scale, both in teaching and in the
benchmarks that test legal AI. This paper systematizes how open-text legal answers can be evaluated. We derive eight desiderata from educational measurement and from the law that governs exams, among them validity, reliability, reasons, credit for defensible alternative solutions, robustness to gaming and human oversight. We then compare eight evaluation paradigms against them, from unstructured human grading to LLM judges with instance-specific rubrics and pairwise comparison. Three findings stand out. A rubric written for the task is the strongest lever for agreement between an LLM judge and human graders, but the reported figures answer different questions and rarely come with the human agreement they must be read against. The central validation gap lies where examination law is most demanding: defensible alternative solutions, follow-on errors, and the examiner's correction of model judgments. And rubric-centred grading is the most auditable family, because it makes the standard, its application and the final decision checkable. From this we derive the trustworthy rubric judge: an instance rubric that credits arguments and alternative paths,evidence checked in code, probes, repeated runs, calibration against independent human grades,logging, and an examiner who reads every script. We state what each safeguard rests on, what remains to be tested, and the uses the evidence supports today.
Read the full abstract ▾
Computation and Language
Choosing the Number of Attention Heads from Intrinsic Dimension: A Small-Scale Study
Arjun Boya, Charan Sai Bandaru, Masthanaiah Pillimetla et al. (+1 more)
The number of
attention heads in a
transformer is usually set by convention or grid search. We test a simple heuristic for setting it in advance: estimate the intrinsic dimension of
token representations without labels, and use roughly that many heads. We estimate intrinsic dimension with a nested-dropout VAE trained on DistilBERT
embeddings of WikiText-2, choosing the KL weight at the knee of the VAE's rate-distortion curve rather than by hand. This gives $k^*=9.2$ (95% CI [8.6, 9.8]). In a small 4-layer
transformer trained from scratch, 10 heads and 32 heads are statistically equivalent on SST-2 and on MRPC (paired TOST over 10 seeds, $\pm1$ point margin), with 69% fewer
attention parameters for the 10-head model. Training 10 heads from the start also slightly outperforms training 32 heads and
pruning to 10 (0.6 points, $p=0.008$). The evidence is narrow: two classification tasks, one small architecture, no pretrained models, and no formal link between intrinsic dimension and head count. We present the result as a practical heuristic worth testing at larger scale, not as a general principle.
Read the full abstract ▾
All authors Arjun Boya, Charan Sai Bandaru, Masthanaiah Pillimetla, Kavana Chidanandappa
Computation and Language
AtomSearch: Evidence-Grounded Tree Search for Agent Skill Optimization
Jia Tong, Senyou Deng, Min Peng et al. (+1 more)
Agent skills adapt frozen models through editable external instructions. Revising several rules at once makes changes difficult to inspect, while judging each local edit only by its immediate score can discard useful intermediate states. We introduce AtomSearch, which keeps revision actions local while using subsequent skill states to guide search. Contrastive atomic editing links each primitive rule edit to failed and successful executions, a behavioral hypothesis, and an intended scope. Skill-State Tree Search (STS) evaluates short edit compositions and selectively retains them as executable states that can be revisited and extended. Outcome
memory supplies later proposals with observed repair and regression statistics. We evaluate AtomSearch across four tasks and five target models. With deepseek-v4-flash, it exceeds SkillOpt on three of four tasks, including a 9.68-percentage-point gain on LiveMathematicianBench, and SkillOpt's final skills contain 2.3-19.7 times as many characters. In 67% of completed searches, the best validation state passes through an ancestor that was not highest-scoring at its depth. These results show how local rule edits can support the exploration of useful revision paths and yield compact skills, with additional offline evaluations used to explore alternative states.
Read the full abstract ▾
All authors Jia Tong, Senyou Deng, Min Peng, Jianfeng Pan
Computation and Language
Does Machine Translation Retain Media Bias?
Silvia V. Gargova, Tim Menzner and Jochen L. Leidner
Within the most recent two decades, machine translation (MT) has been advanced from a research area to a mature, deployed technology for prose text and most
language pairs. However,
language has different dimensions of meaning, and the alluded progress has primarily been demonstrated for propositional meaning. In this small exploratory study, we investigate whether one type of affective meaning, namely different types of linguistically expressed media bias, are retained under automatic translation for two
language pairs, English and German as the source
languages, and Bulgarian as the target
language. We evaluate four state-of-the-art translation methods to answer the question whether and to what degree a loss of non-neutral
language (specifically, 27 types of media bias) can occur as a result of applying MT. Our empirical findings are consistent with the claim that MT does not fully retain existing news bias of its input source text, but also may add extra bias of its own. To the best of our knowledge, this is the first experimental study of MT media bias retention.
Read the full abstract ▾
Computation and Language
To Reason but Not to Stop: Understanding and Recovering Pruned Large Reasoning Models
Minh-Anh Nguyen, Nguyen T. Ha Lan, Xiyan Fu et al. (+2 more)
Weight
pruning can degrade the accuracy of large
reasoning models (LRMs), yet it remains unclear whether it damages
reasoning itself. We find that much of this degradation stems from a failure to stop rather than to reason. As sparsity increases, pruned LRMs increasingly fail to terminate within the
token budget, often after the correct answer has appeared in their
reasoning, instead falling into repetitive loops, even though they still recognize where the dense model would stop. We further find that the hidden-state difference between terminating and looping continuations is consistent across pruned checkpoints, suggesting a shared direction associated with committing to an answer. Building on this, we propose Termination-Guided Activation Steering (TGAS), a training-free method that adds this direction to the residual stream at inference time. Across six LRMs, three
pruning pipelines, and math and code
benchmarks, TGAS improves termination and overall accuracy while reducing repetitive
reasoning and generation length, showing that this failure is actionable.
Read the full abstract ▾
All authors Minh-Anh Nguyen, Nguyen T. Ha Lan, Xiyan Fu, Dung D. Le, Wei Lu
Computation and Language
MemDuet: Learning Dual-Process Memory for Long-Context Reasoning
Jiangcheng Song, Haodong Jing, Yulin Mu et al. (+7 more)
Reasoning over infinitely long documents is becoming a reality with recurrent
memory agents. However, increasing context length exacerbates the trade-off between
reasoning performance and inference efficiency. Inspired by Kahneman's account of fast and slow thinking, we propose MemDuet, a dual-process
memory framework that learns to coordinate a small
language model for fast, parallel evidence selection from raw context chunks (System 1) with a large
language model for
memory-update verification and
memory updates in document order (System 2). We train MemDuet in three stages: supervised fine-tuning of System 1 for evidence selection, Group Relative Policy Optimization (GRPO) of System 2 for verification and
memory rewriting, and joint optimization of both policies to coordinate selection and updating for long-context
reasoning. Experiments on diverse long-context
reasoning benchmarks (RULER-HQA, LongBench v2, and 2WikiMultiHopQA) demonstrate a favorable accuracy-efficiency trade-off as context length grows. Averaged across evaluation settings on the three
datasets, MemDuet achieves an end-to-end inference speedup of 9.74x and an output-
token reduction of 73.91% relative to RL-MemAgent.
Read the full abstract ▾
All authors Jiangcheng Song, Haodong Jing, Yulin Mu, Lesong Tao, Jixin Wang, Jin Cui, Tianjing Ren, Runxuan Xie, Yongqiang Ma, Nanning Zheng
Computation and Language · Cross-listed from cs.LG
MolHop: Scaffold-Hopping Molecular Design with Agentic Reinforcement Learning
Yanyi Su, Chenyan Wang, Yulei Gong et al. (+3 more)
Molecular design is often formulated as an optimization problem over existing molecular spaces. However, molecular optimization and the discovery of structurally distinct replacement molecules represent two fundamentally different search problems. LLM-based molecular design
agents typically perform the former, conducting local refinement within the neighborhood of existing structures to improve target properties through structural modifications. In contrast, the latter requires exploration of a broader chemical space to discover molecules with entirely different scaffolds while preserving the functional characteristics of reference molecules. We propose MolHop, an agentic framework for cross-scaffold molecular discovery that enables models to navigate and edit within a pre-organized chemical space. MolHop organizes approximately 1.6 million molecular scaffolds from commercial molecules into a coarse-to-fine hierarchical action space. The
agent selects new exploration regions through scaffold navigation actions and adjusts candidate properties through molecular editing. The framework maintains historical trajectory
memory to support long-horizon search processes. On drug molecule and commercial compound tasks, connecting Grok-4.6 and DeepSeek-Flash-V4.1 to the MolHop framework substantially improves their cross-scaffold joint success rates. We develop MolHop-9B through agentic reinforcement learning, and MolHop-9B outperforms generative baselines including GPT-5.6-sol, Claude-5-Opus, and Gemini-3.1-Pro, while generating candidate molecules with larger scaffold differences from the reference molecules. These results demonstrate that effective molecular discovery beyond the traditional optimization paradigm requires structured chemical and action spaces, enabling
agents to explore new structural regions rather than merely refine existing molecules.
Read the full abstract ▾
All authors Yanyi Su, Chenyan Wang, Yulei Gong, Yeyun Chen, Xiaohui Yang, Jun Cheng